refactor!: rebuild the "refresh" system into expanded "sync" system

docs/ignore.md

@@ -8,12 +8,6 @@ icon: material/file-document-remove
 
 # :material-file-document-remove: Ignoring Files & Directories
 
-<!-- prettier-ignore -->
-!!! warning "Legacy File Extension Ignoring"
-    TagStudio versions prior to v9.5.4 use a different, more limited method to exclude or include file extensions from your library and subsequent searches. Opening a pre-exiting library in v9.5.4 or later will non-destructively convert this to the newer, more extensive `.ts_ignore` format.
-
-    If you're still running an older version of TagStudio in the meantime, you can access the legacy system by going to "Edit -> Manage File Extensions" in the menubar.
-
 TagStudio offers the ability to ignore specific files and directories via a `.ts_ignore` file located inside your [library's](libraries.md) `.TagStudio` folder. This file is designed to use very similar [glob](<https://en.wikipedia.org/wiki/Glob_(programming)>)-style pattern matching as the [`.gitignore`](https://git-scm.com/docs/gitignore) file used by Git™[^1]. It can be edited within TagStudio or opened to edit with an external program by going to the "Edit -> Ignore Files" option in the menubar.
 
 This file is only referenced when scanning directories for new files to add to your library, and does not apply to files that have already been added to your library.
@@ -131,14 +125,15 @@ The forward slash "`/`" is used as the directory separator. Separators may occur
 A `!` prefix before a pattern negates the pattern, allowing any files matched matched by previous patterns to be un-matched.
 
 - Any matching file excluded by a previous pattern will become included again.
-- **It is not possible to re-include a file if a parent directory of that file is excluded.**
+- **An excluded parent directory can not have files within it reincluded!** (e.g. `Photos/`)
+    - Alternatively, if one or more files within a directory is excluded (e.g. `!Photos/*.jpg`) then files within can be reincluded from there (e.g. `Photos/a.jpg`).
 
 <!-- prettier-ignore-start -->
 === "Example negation"
     ```toml
     # All .jpg files will be ignored, except any located in the 'Photos' folder.
     *.jpg
-    Photos/!*.jpg
+    !Photos/*.jpg
     ```
 === "Escape a ! Symbol"
     ```toml
@@ -147,6 +142,10 @@ A `!` prefix before a pattern negates the pattern, allowing any files matched ma
     ```
 <!-- prettier-ignore-end -->
 
+<!-- prettier-ignore -->
+!!! bug "Directory Exclusion Negation"
+    TagStudio attempts to match the behavior of a `.gitignore` file 1:1, however if you don't have `ripgrep` installed on your system and TagStudio falls back to its internal pattern matcher, excluded directories can be overwritten by further negations, unlike `.gitignore` behavior. Be wary that this is **not officially supported**, and this behavior may be removed at any time.
+
 ---
 
 ### Wildcards

docs/libraries.md

@@ -33,13 +33,68 @@ To create or open a [library](libraries.md), go to **File -> Open/Create Library
 !!! info "Legacy Library Migration"
     If you open a library created with TagStudio **v9.4.2 or earlier** in **[v9.5.0](changelog.md#950-march-3rd-2025) or later**, you'll be walked through a migration process that converts the old `ts_library.json` save file to the new `ts_library.sqlite` format. The original JSON file is preserved and can be easily deleted from the **View -> Library Information** panel once you're satisfied with the migration.
 
-## :material-database-refresh: Refreshing Directories
+## :material-database-sync: Library Syncing
+
+A TagStudio library gets synced with the files found in your chosen content folders and certain metadata attributes (e.g. stats) found on those files. This is a **non-destructive, read-only** process and none of your content files are moved, modified, or deleted. Syncing is indicated by a temporary progress bar, and you can continue to use TagStudio normally while syncing occurs.
+
+Syncing automatically occurs when you open a library by default, and you can manually sync a library at any time by going to **File -> Sync Library** in the menubar to by pressing <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS). If you do not wish for your library to be synced when opened, you can disable this behavior in the settings.
+
+<figure markdown="span">
+  ![Settings -> Sync Library on Open](assets/settings_refresh_library_on_open.png)
+  <figcaption>
+  Settings -> Sync Library on Open
+  </figcaption>
+</figure>
+
+### :material-link-variant: Automatic Relinking
+
+Unlinked entries are file entries in your TagStudio library that have become "unlinked" from their original file on disk, likely as a result of the original file being renamed, moved, or deleted. TagStudio attempts to automatically relink any of these entries as a part of the syncing process, but there are some scenarios where automatic relinking is not possible or too ambiguous and requires a manual review. Below is a complete table of every scenario in which file entries can become unlinked, and whether or not TagStudio can auto-relink them:
+
+|     Case | File Moved? | File Renamed? | File Modified? | Unlinked Entries | Matched Files |             Auto-Relink             |
+| -------: | :---------: | :-----------: | :------------: | :--------------: | :-----------: | :---------------------------------: |
+|  **\#0** |    _No_     |     _No_      |    **Yes**     |        0         |       —       | :material-minus-circle:{.lg .gray}  |
+|  **\#1** |      —      |       —       |       —        |        1         |       0       |  :material-close-circle:{.lg .red}  |
+|  **\#2** |   **Yes**   |    **Yes**    |    **Yes**     |        1         |       0       |  :material-close-circle:{.lg .red}  |
+|  **\#3** |   **Yes**   |    **Yes**    |      _No_      |        1         |       1       | :material-check-circle:{.lg .green} |
+|  **\#4** |   **Yes**   |    **Yes**    |      _No_      |        1         |      2+       |  :material-close-circle:{.lg .red}  |
+|  **\#5** |   **Yes**   |    **Yes**    |      _No_      |        2+        |      Any      |  :material-close-circle:{.lg .red}  |
+|  **\#6** |   **Yes**   |     _No_      |    **Yes**     |        1         |       1       | :material-check-circle:{.lg .green} |
+|  **\#7** |   **Yes**   |     _No_      |      _No_      |        1         |       1       | :material-check-circle:{.lg .green} |
+|  **\#8** |   **Yes**   |     _No_      |      _No_      |        1         |      2+       |  :material-close-circle:{.lg .red}  |
+|  **\#9** |   **Yes**   |     _No_      |      _No_      |        2+        |      Any      |  :material-close-circle:{.lg .red}  |
+| **\#10** |    _No_     |    **Yes**    |    **Yes**     |        1         |       0       |  :material-close-circle:{.lg .red}  |
+| **\#11** |    _No_     |    **Yes**    |      _No_      |        1         |       1       | :material-check-circle:{.lg .green} |
+| **\#12** |    _No_     |    **Yes**    |      _No_      |        1         |      2+       |  :material-close-circle:{.lg .red}  |
+| **\#13** |    _No_     |    **Yes**    |      _No_      |        2+        |      Any      |  :material-close-circle:{.lg .red}  |
+
+#### Explanations
+
+- **Case \#0**: _Modifying file content alone does not create unlinked entries._
+- **Case \#1**: If the original file was deleted, no matches will be found. TagStudio leaves the decision to delete entries up to the user.
+- **Case \#2**: If the original file bears no similarities to the unlinked entry anymore, it is indistinguishable from a deleted or new file.
+- **Case \#3**: The file has been moved and renamed with a high degree of confidence.
+- **Case \#4**: If more than one file is matched with the same stats, the case is too ambiguous.
+- **Case \#5**: If two or more entries share the same stats, it's not clear which entry a matched file belongs to.
+- **Case \#6**: The file has been moved and modified, but since no other file shares its filename, it is assumed to be the same file with a decent degree of confidence.
+- **Case \#7**: The original file has been moved with a high degree of confidence.
+- **Case \#8**: If multiple copies of the same moved file are matched in different locations, the case is ambiguous.
+- **Case \#9**: _Similar to **\#5**._ If two or more entries share the same filename and stats, it's not clear which entry a matched file belongs to.
+- **Case \#10**: _Same as **\#2**._
+- **Case \#11**: The file has been renamed with a high degree of confidence.
+- **Case \#12**: _Same as **\#4**._
+- **Case \#13**: _Same as **\#5**._
+
+Every numbered case above assumes the entry has saved file metadata attributes to help match against (added in **v9.7**), in which case the **file modification date** combined with the **file size** is used as a soft file signature to aid in scenarios such as renames or moves. If no file metadata is stored with the file entry, or if this soft file signature doesn't lead to a confident match, TagStudio falls back to matching by filename alone: a single file found with that name is automatically relinked, while zero or multiple filename matches leave the entry unlinked for manual review.
 
-TagStudio automatically scans for new or updated files when opening a library by default. This behavior can be toggled in the settings if your library is very large and/or located on a slow drive.
+<!-- prettier-ignore -->
+!!! warning
+    There's currently no way to manually specify which remaining unlinked entries should be linked with which files, only to delete the entries from the library. Manual relinking is a high priority feature for future releases.
 
-![Settings -> Automatically Load New Files](assets/settings_refresh_library_on_open.png)
+<!-- prettier-ignore -->
+!!! warning "Switching from a Case-Sensitive to Case-Insensitive Filesystem (i.e. Windows)"
+    If you switch from using TagStudio on a computer with a case-sensitive filesystem to one with a *case-insensitive* one, TagStudio will treat any entries added up to this point with the same filepath + name under case-insensitivity as duplicate entries and merge them. The automatic relinking process will also take case-insensitivity into account when relinking entries.
 
-To manually refresh your library at any time, use **File -> Refresh Directories** from the menu or by using <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS).
+    Currently, TagStudio only uses this case-insensitivity mode when running on Windows. Future versions will be more precise about this distinction, with the aim of determining the case sensitivity on a per-drive basis.
 
 ## :material-database-cog: Library Information Panel
 

docs/roadmap.md

@@ -96,7 +96,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
     - [x] Delete Old Backups **[[v9.5.4](changelog.md#954-september-1st-2025)]**
     - [x] Delete Legacy JSON File **[[v9.5.4](changelog.md#954-september-1st-2025)]**
 - [x] Translations
-- [ ] Search Bar Rework :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
+- [ ] Search Bar Rework :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
     - [ ] Improved Tag Autocomplete :material-chevron-triple-up:{ .priority-high title="High Priority" }
     - [ ] Tags appear as widgets in search bar _(similar to new tag search/create bar)_ :material-chevron-triple-up:{ .priority-high title="High Priority" }
 - [x] Unified Media Player
@@ -136,7 +136,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
     - [x] Theme
     - [x] Thumbnail Generation **[[v9.5.4](changelog.md#954-september-1st-2025)]**
 - [x] Configurable Page Size
-- [ ] Library Settings :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
+- [ ] Library Settings :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
     - [ ] Stored in `.TagStudio` folder :material-chevron-triple-up:{ .priority-high title="High Priority" }
 - [ ] Toggle File Extension Label :material-chevron-double-up:{ .priority-med title="Medium Priority" }
 - [ ] Toggle Duration Label :material-chevron-double-up:{ .priority-med title="Medium Priority" }
@@ -149,13 +149,13 @@ A detailed specification written for the TagStudio tag and/or library format. In
 
 - [x] Per-Library Tags
 - [ ] Global Tags :material-chevron-double-up:{ .priority-med title="Medium Priority" } **[v9.8.x]**
-- [ ] Multiple Root Directories :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
-    - [ ] Ability to store TagStudio data folder separate from library content folder(s) :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
-- [ ] Automatic Entry Relinking :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.8.x]**
-    - [ ] Detect Renames :material-chevron-triple-up:{ .priority-high title="High Priority" }
-    - [ ] Detect Moves :material-chevron-triple-up:{ .priority-high title="High Priority" }
-    - [ ] Detect Deletions :material-chevron-triple-up:{ .priority-high title="High Priority" }
-    - [ ] Performant :material-chevron-triple-up:{ .priority-high title="High Priority" }
+- [ ] Multiple Root Directories :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
+    - [ ] Ability to store TagStudio data folder separate from library content folder(s) :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
+- [x] Automatic Entry Relinking :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.0]**
+    - [x] Detect Renames :material-chevron-triple-up:{ .priority-high title="High Priority" }
+    - [x] Detect Moves :material-chevron-triple-up:{ .priority-high title="High Priority" }
+    - [ ] Some way to safely handle unlinked entries presumed to be from deleted files automatically (deleted after X days?) :material-chevron-double-up:{ .priority-med title="Medium Priority" }
+    - [x] Performant :material-chevron-triple-up:{ .priority-high title="High Priority" }
 - [ ] Background File Scanning :material-chevron-triple-up:{ .priority-high title="High Priority" }
 - [x] Thumbnail Caching **[[v9.5.0](changelog.md#950-march-3rd-2025)]**
     - [ ] Audio Waveform Caching :material-chevron-double-up:{ .priority-med title="Medium Priority" } **[v9.7.x]**
@@ -166,7 +166,7 @@ A detailed specification written for the TagStudio tag and/or library format. In
 File or file-like [entries](entries.md) stored in the library.
 
 - [x] File Entries **[v1.0.0]**
-- [ ] URL Entries / Bookmarks :material-chevron-up:{ .priority-low title="Low Priority" } **[v9.6.x]**
+- [ ] URL Entries / Bookmarks :material-chevron-up:{ .priority-low title="Low Priority" } **[v9.8.x]**
 - [x] Fields
     - [x] Text Lines
     - [x] Text Boxes
@@ -267,7 +267,7 @@ Discrete library objects representing [attributes](<https://en.wikipedia.org/wik
 Sharable TagStudio library data in the form of data packs (tags, colors, etc.) or other formats.
 Packs are intended as an easy way to import and export specific data between libraries and users, while export-only formats are intended to be imported by other programs.
 
-- [ ] Color Packs :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.6.x]**
+- [ ] Color Packs :material-chevron-triple-up:{ .priority-high title="High Priority" } **[v9.7.x]**
     - [ ] Importable
     - [ ] Exportable
     - [x] UUIDs + Namespaces :material-chevron-triple-up:{ .priority-high title="High Priority" }

docs/stylesheets/extra.css

@@ -80,6 +80,7 @@ img {
 th,
 td {
     padding: 0.5em 1em 0.5em 1em !important;
+    min-width: 0 !important;
 }
 
 hr {

docs/usage.md

@@ -12,9 +12,13 @@ icon: material/mouse
 
 To create or open a [library](libraries.md), go to **File -> Open/Create Library** in the menu bar or use <kbd>Ctrl</kbd>+<kbd>O</kbd> (<kbd>⌘ Command </kbd>+<kbd>O</kbd> on macOS) and chose a folder with file contents you'd like to use as a TagStudio library. If a `.TagStudio` folder doesn't already exist inside the directory, TagStudio will create one and automatically scan the folder for files to include. Otherwise, the pre-existing library is opened.
 
-### :material-database-refresh: Refreshing Directories
+### :material-database-sync: Library Syncing
 
-TagStudio automatically scans for new or updated files when opening a library by default. Manually refresh by going to **File -> Refresh Directories** in the menu or by using <kbd>Ctrl</kbd>+<kbd>R</kbd> (<kbd>⌘ Command </kbd>+<kbd>R</kbd> on macOS).
+A TagStudio library gets synced with the files found in your content folders. This is a **non-destructive, read-only** process and none of your files are moved, modified, or deleted. Syncing is indicated by a temporary progress bar, and you can continue to use TagStudio normally while syncing occurs.
+
+### :material-link-variant: Automatic Relinking
+
+Files that become moved, renamed, or modified will try to be automatically relinked during syncing. If a file cannot be found or a match cannot be safely made, it will stay unlinked until the entries are manually deleted under "Fix Unlinked Entries". _Manual relinking is a high priority feature for future versions._
 
 <!-- prettier-ignore -->
 !!! abstract "TagStudio Libraries"
@@ -115,20 +119,6 @@ Creating and adding fields to entries is extremely similar to how [tagging](#tag
 
 ---
 
-## Relinking Moved Files
-
-Inevitably some of the files inside your library will be renamed, moved, or deleted. If a file has been renamed or moved, TagStudio will display the thumbnail as a red broken chain link. To relink moved files or delete these entries, select the "Manage Unlinked Entries" option under the Tools menu. Click the "Refresh" button to scan your library for unlinked entries. Once complete, you can attempt to "Search & Relink" any unlinked file entries to their respective files, or "Delete Unlinked Entries" in the event the original files have been deleted and you no longer wish to keep their entries inside your library.
-
-<!-- prettier-ignore -->
-!!! warning
-    There is currently no method to relink entries to files that have been renamed - only moved or deleted. This is a high priority for future releases.
-
-<!-- prettier-ignore -->
-!!! warning
-    If multiple matches for a moved file are found (matches are currently defined as files with a matching filename as the original), TagStudio will currently ignore the match groups. Adding a GUI for manual selection, as well as smarter automated relinking, are high priorities for future versions.
-
----
-
 ## Launch Arguments
 
 There are a handful of launch arguments you can pass to TagStudio via the command line or a desktop shortcut.

pyproject.toml

@@ -9,7 +9,7 @@ build-backend = "hatchling.build"
 [project]
 name = "TagStudio"
 description = "A User-Focused Photo & File Management System."
-version = "9.6.4"
+version = "9.7.0"
 license = "GPL-3.0-only"
 readme = "README.md"
 requires-python = ">=3.14,<3.15"

src/tagstudio/core/library/alchemy/constants.py

@@ -14,7 +14,7 @@
 
 DB_VERSION_CURRENT_KEY: str = "CURRENT"
 DB_VERSION_INITIAL_KEY: str = "INITIAL"
-DB_VERSION: int = 400
+DB_VERSION: int = 500
 
 TAG_CHILDREN_QUERY = text("""
 WITH RECURSIVE ChildTags AS (

src/tagstudio/core/library/alchemy/db.py

@@ -9,6 +9,8 @@
 from sqlalchemy import Dialect, String, TypeDecorator
 from sqlalchemy.orm import DeclarativeBase
 
+from tagstudio.core.utils.normalization import norm_path
+
 logger = structlog.getLogger(__name__)
 
 
@@ -19,7 +21,7 @@ class PathType(TypeDecorator):
     @override
     def process_bind_param(self, value: Path | None, dialect: Dialect):
         if value is not None:
-            return Path(value).as_posix()
+            return norm_path(Path(value), case_sensitive=True).as_posix()
         return None
 
     @override

src/tagstudio/core/library/alchemy/enums.py

@@ -72,7 +72,10 @@ class ItemType(enum.Enum):
 
 class SortingModeEnum(enum.Enum):
     DATE_ADDED = "file.date_added"
+    DATE_CREATED = "file.date_created"
+    DATE_MODIFIED = "file.date_modified"
     FILE_NAME = "generic.filename"
+    FILE_SIZE = "file.size"
     PATH = "file.path"
     RANDOM = "sorting.mode.random"