Repository navigation
perf: skip re-reading files whose size and mtime are unchanged - #53
Merged
Merged
Conversation
msallin
force-pushed
the
perf/file-stamp
branch
from
October 11, 2026 11:12
c3e677c to
b07b7eb
Compare
Since every compilation resets the file slots, each file a document touches was read and hashed in full on every compile, packages and plugins included. A slot now keeps the path, length and modification time of the file it was read from, and a reset slot whose stamp still matches a fresh stat is served from cache without a read. The stat is taken once per file and compilation and shared by the source and bytes views of the file, and it doubles as the stamp when the file is read. A stamp is only trusted once the file is at least two seconds older than the read that took it, so a rewrite within one tick of a coarse file system clock is still picked up (git's racily-clean check).
msallin
force-pushed
the
perf/file-stamp
branch
from
October 11, 2026 11:28
b07b7eb to
72b74b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #39 (which fixed #28).
Problem
Since #39,
SystemWorld::resetsends every file slot back to disk before each compilation, and the next access doesfs::metadata+fs::read+hash128on every file the document touches, packages and plugins included. A label template importing@preview/tiaoma:0.3.0re-reads its 879 KBzint_typst_plugin.wasmon every compile, about 0.5 ms on top of a 3 ms compile on Windows.Change
SlotCellkeeps a stamp of the file it was read from: path, length and modification time.nowreset are untouched. Package files are not special-cased: the stamp makes them cheap, and vendored local packages keep picking up redeploys.Racy-timestamp trade-off
A stamp is only trusted if the file's mtime is at least 2 s older than the read that took it (git's "racily clean" check), which covers FAT and SMB granularity. An untrusted stamp always falls through to read + fingerprint and is judged again after that read, so a freshly written file settles after its first read past the window. A future mtime is never trusted.
What remains undetected is a rewrite that keeps both length and mtime of a file older than 2 s, e.g. a timestamp-preserving copy over a file of the same size. README and release notes say so, next to the existing advice to replace templates atomically.
Two limits on network shares worth knowing: the guard compares the file's mtime with the local clock, so a server clock running behind narrows the window; and a stat may be answered from the client's attribute cache (NFS
acregmax, SMB file info cache), so a redeploy on such a share can take that cache's lifetime to be seen.Benchmark
Windows 11, release build, warm page cache, files dated an hour back, median of five batches of 200 compiles of one warm compiler, mean of two runs. Per compilation:
read()Per-file cost drops from about 85 µs to about 25 µs, and the 1 MB file no longer scales with its size. The rest is the stat itself:
fs::metadataon Windows opens a handle, which the on-access scanner sees.GetFileAttributesExWwould avoid that, but needs FFI or awindows-sysdependency, so I left it out. The "no re-check" column isSlotCell::resetmade a no-op, for reference only. Absolute numbers differ from #39 because the template content is different; the method is the same.Tests
New
world.rsunit tests:File::set_modified)IsDirectoryand a file put in its place is picked upSystemWorldsees a rewrite of a settled file, i.e. the shared stat does not outlive the compilationis_settledboundaries: future, 0, window minus 1 ms, exactly the window, an hourThe existing
SlotCelltests now go through the same helper. Each of these mutations fails at least one test: disabling the stamp check, disabling the racy guard, keeping a stamp after a failed read, re-stating for the second view, and not clearing the shared stat on reset.tests/file_reload.rsand the managed tests pass either way: their rewrites change the length or move the NTFS mtime, so they cover the behaviour but don't depend on the guard.cargo test --releasepasses 44 native tests anddotnet testpasses 77 managed tests. The only warnings are the three existing CS0618 warnings inTests.cs, which test the deprecatedBufferson purpose.