Add allMatches to Traverse function to improve performance

[jtmaxwell3]

I discovered when parsing wemulujaʼjawype in the Aweti project that a single call to Traverse in DeterministicFsaTraversalMethod could take 4 seconds. It was processing more than 200,000 traversals, even though the input only had 47 characters and the fsa only had 4 states. This was because the DeterministicFsaTraversalInstance data structure included the registers, which encoded the match to that point. Processing 200,000 traversals was especially annoying since the caller only wanted one match.

The registers only record the match, they don't filter the traversal in any way. So it should be possible to traverse in two passes, where the first pass finds traversals that are acceptable without recording the registers and the second pass extracts the registers for each of the successful traversals. But it turns out that the code path that was causing the performance issue only takes the first match. So I optimized for the case when the caller only wanted one match. This can be done in one pass.

To fix the performance issue, I added an allMatches parameter to Traverse, and changed the code to ignore traversals that arrived at a previously processed <State, AnnotationIndex> tuple. Since the only difference between the new traversal and the previously processed traversal is in the registers, we can just ignore the new traversal. This means that in the worst case it only takes O(N * |States|) to find a traversal. I tried returning immediately when I found the first match, but that produces the shortest match and many of the MatcherTests failed. Letting it run to the end finds matches of all lengths. Otherwise, the choice of match doesn't seem to matter.

With this change in place, parsing wemulujaʼjawype went from 83 seconds to 3 seconds and parsing 185 Aweti words went from 4 minutes to 1 minute.

---

This change is 

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.37838% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.41%. Comparing base (b9e7db4) to head (039a97d).

Files with missing lines	Patch %	Lines
src/SIL.Machine/FiniteState/TraversalMethodBase.cs	84.66%	19 Missing and 6 partials ⚠️
.../FiniteState/NondeterministicFsaTraversalMethod.cs	42.50%	23 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #511      +/-   ##
==========================================
+ Coverage   74.35%   74.41%   +0.05%     
==========================================
  Files         456      456              
  Lines       38283    38492     +209     
  Branches     5245     5293      +48     
==========================================
+ Hits        28467    28643     +176     
- Misses       8666     8695      +29     
- Partials     1150     1154       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

The optimization is real and worth having, but the (State, AnnotationIndex) key drops traversal state that filters later arcs, so Match() returns different answers than master — including outright lost matches.

The mechanism you found reproduces, and is worse than you reported. It isn't the state count: TraversalMethodBase.Advance (src/SIL.Machine/FiniteState/TraversalMethodBase.cs:275-292) forks an instance for every Optional annotation at the next offset, so instances grow exponentially in the number of optional annotations and independently of |States|. On a synthetic 2-state FSA with 20 optional annotations I measured 2,097,150 instances popped (~2^(k+1)); at k≥24 it blows past a 5,000,000 budget. Your fix bounds it at ≤ N·|States| — measured ratio 0.5–1.0×, turning never-returns cases into instant. That part holds up completely.

The problem. A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against both master and this branch, comparing success, range, every group range and every variable binding — found:

cases=20000  divergences=112  allMatchesFirstDiff=0

AllMatches().First() is identical in all 20,000 cases, which pins the divergence to exactly the allMatches=false path this PR changes. Two classes:

1. Variable bindings (108 cases). TraversalInstance.VariableBindings is not just a record of the match — CheckInputMatch → Input.Matches unifies against it, so which branch survives decides which later arcs can match. Matcher.Compile (src/SIL.Machine/Matching/Matcher.cs:89) only determinizes when a pattern has no variables, so every variable-bearing pattern — i.e. every alpha-variable phonological rule — runs NondeterministicFsaTraversalMethod, where the new key is blindest. Minimal case:

pattern: high=$v0+      (anchored both ends, 5 annotations)
master : success=True;  range=[0,5); v0=-
this PR: success=False; range=<null>; v0=<unbound>

2. Capture registers (4 cases), on the deterministic path. All four are variable-free, verified to compile with IsDeterministic=True, and all have alternation plus capture groups. They don't merely change captures — they shorten the match range:

pattern: (g0(back=back+)|g1(high=high+back=back+))
master : range=[0,4); g1=[0,4)
this PR: range=[0,1); g0=[0,1)

So "deterministic method only" would not be a sufficient guard; group-bearing patterns diverge there too.

b348f48f does not affect this: re-running the same fuzz against da352c7a produced a byte-identical candidate output file (same SHA-256), same 112 divergences, same gate hit counts. That commit fixes instance-release bookkeeping after the skip decision, not the key itself. Both committed suites stay green on this branch (826/0/3 and 98/0/0), so the existing tests cannot see this.

A narrowing that keeps the win. I prototyped and tested one:

// DeterministicFsaTraversalMethod
bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1;

with NondeterministicFsaTraversalMethod reverted to master (keeping the allMatches parameter only to satisfy ITraversalMethod). Results: fuzz divergences 112 → 0; the group-free microbenchmark sweep is cell-for-cell identical to this PR across all 18 cells, so the speedup is fully preserved where patterns have no capture groups; grouped patterns fall back to master's behaviour exactly; suites unchanged. (GroupNames.Count() should be hoisted out of Traverse — it is O(1) via the ICollection fast path but is evaluated once per call.)

Where this could go further. The rewrite-rule environment matchers look like the best target. RewriteRuleSpec (src/SIL.Machine.Morphology.HermitCrab/PhonologicalRules/RewriteRuleSpec.cs:83-107) reads only leftEnvMatch.Success and leftEnvMatch.VariableBindings from those matches — never the range, never a group. So for environment matching the registers are pure overhead, exactly as your description says, and the sound dedup key there is (State, AnnotationIndex, VariableBindings), with bindings droppable entirely when IgnoreVariables is set. That would extend the bound to the variable-bearing patterns this narrowing has to give up, without any capture-reconstruction pass.

One caveat on the measurements. I could not reproduce the 83s → 3s end to end. On the Aweti grammar I have (aweti-hc.xml re-exported 2026-09-16 from Jun-26 data), that word does not complete on either build — both grow to a 40 GB heap cap in ~5 minutes. Instrumenting it showed zero DeterministicFsaTraversalMethod calls out of 7.3M, i.e. the run dies during analysis and never reaches synthesis where that method is used, so we are clearly not parsing the same grammar you were. Across five grammars (Sena, Amharic, Indonesian, Mbugwe, Aweti; sampled words, parity clean) I measured 0.90×–1.01×. If you can share the exact grammar export you measured, I can re-run the comparison on it.

Happy to open a PR with the narrowed gate, the differential fuzz harness, and a regression test for the lost-match case if that is useful. Separately, the fuzz also found 19 cases where Match() throws NullReferenceException on master (e.g. back=$v0*, anchored to start) — pre-existing and unrelated to this PR; I will file that on its own.

🤖 Analysis performed with Claude Code

[johnml1135]

Rebuilt on the stack you named — master + filter-final-templates-in-analysis + change-add-to-priority-union — and the Aweti word still does not complete here. But the attempt turned up something you'll want to know about #491 independently.

All five of our grammars are partial, so #491's prune is inert by default on every one of them. Morpher.IsPartial is true for each, so AlwaysEnforceFinalTemplates is the only way the prune engages:

grammar	partial morphemes
Aweti	3 (2 stems, 1 affix)
Sena	27
Mbugwe	1
Amharic	1
Indonesian	4

That is presumably not true of whatever you measured on, and it would explain why the stack did nothing for us.

Forcing the guard on does change the behaviour, just not enough. With AlwaysEnforceFinalTemplates = true, the failure mode flips from memory-bound to time-bound: without it the word hits a 40 GB heap cap in ~250 s; with it both arms plateau around 16.7 GB and are still running at 480 s. Real, reproducible, and in the direction you would want — but the word does not finish either way, so there is still at least one more factor.

Measurements, all on Aweti source-list index 182, maxDegreeOfParallelism: 1, 40 GB cap:

arm	guard	result
master + #491 + #494	off	killed, 248 s, 40.17 GB
+ #511 (da352c7)	off	killed, 248 s, 40.16 GB
+ narrowed #511	off	killed, 238 s, 40.08 GB
master + #491 + #494	on	killed at 480 s, 16.64 GB
+ #511	on	killed at 480 s, 16.73 GB

None produced a single parse, so there is nothing to compare for either speed or parity, and I can't yet tell you whether the narrowed gate would cost you your win.

Three things would let me reproduce it rather than keep guessing:

1. The exact aweti-hc.xml you measured on. Mine was re-exported 2026-09-16 from the Jun-26 .fwdata; if yours came out of FLEx earlier or with different export settings it may be a materially different grammar. This is my main suspicion.
2. Whether AlwaysEnforceFinalTemplates was set, given the table above.
3. How you invoked the parse — through FLEx's parser, or Morpher.ParseWord directly, and at what parallelism.

Worth noting for its own sake: when I instrumented the word on plain master, DeterministicFsaTraversalMethod was never invoked at all — 7.3M traversal calls, all nondeterministic. The run dies in analysis and never reaches synthesis, which is where the deterministic method gets used. So on my export I am not even reaching the code path your PR optimizes, which is consistent with everything above.

🤖 Analysis performed with Claude Code

[claude]

Minor: F1. A valid match can be lost when Acceptable reads the registers. ExpandInstances keeps one instance per lattice node, so the registers of every other path are gone before the final arc runs CheckAccepting/Acceptable. SynthesisRewriteRuleSpec.CheckTarget reads match.Range and AnalysisCompoundingSubruleRuleSpec reads captures: if the kept path is rejected, there is no result, even when a dropped path would have passed. Master tries every path. Unverified: from reading the code, not run.

[claude]

Minor: F2. The dedup never fires for patterns with variables. VariableBindings does not override Equals/GetHashCode, and each key holds a fresh Clone(), so this compares references and never matches. Every alpha-variable rule pays for the lattice, the extra copies, and a second CheckAccepting/Acceptable pass, and gets no speedup. NondeterministicTraversal_DedupOnVariableBindingLosesMatch passes for this reason. Unverified: from reading the code, not run.

[claude]

Minor: F3. Match() results change for determinized patterns, and the description does not say so. When either result is lazy the shorter one now sorts first, and this applies on the AllMatches path too. DeterministicTraversal_DedupOnRegistersShortensMatch asserts [0,1), where the fuzz comment above reports master returning [0,4). The comparator is also still intransitive: with a lazy a of length 2, b of length 3 and c of length 1, a<b, b<c and c<a. Is this change intended? Unverified: not run.

[claude]

FYI: F4. The final arc is recorded once per yielded instance, not once. resultCount is read before the Advance loop and never updated, so when 2+ annotations start at the next offset, the same (origInst, arc) is added again. ExtractResults then produces duplicate FstResults and repeats the Acceptable calls. The same happens at NondeterministicFsaTraversalMethod.cs:132. Updating resultCount after recording fixes both.

[claude]

FYI: F5. In a DFA, which path wins at a merge is not well defined. Priorities is null there, so InstanceCompare always returns 0, and the surviving captures depend on incoming-arc order and how Sort handles ties. Master falls back to the Order tiebreak instead. This can change GroupCaptures for determinized patterns with groups. Unverified.

[claude]

FYI: F6. Stack depth during extraction grows with match length. ExpandInstances and ExpandArcInstances recurse once per lattice edge, where master used an explicit stack. A long enough input to the public Matcher could throw StackOverflowException, which cannot be caught. Unverified; HermitCrab words are short.

[claude]

FYI: F7. The comments here break the code-comments standard that --agent-strict enforces. This header is provenance and points to TraversalDedupDifferentialFuzzTests, which is not in the tree. Line 36 still says the key is (State, AnnotationIndex) alone. The second test's name says the dedup "shortens" the match, yet the test asserts that shorter match as correct.

[claude]

FYI: F8. The allMatches parameter is dead here. ExtractResults, ExpandArcInstances and ExpandInstances run only on the !allMatches path, so the parameter is always false and this check always passes. Keeping one element does not need a full sort: a single MinBy scan, as Fst.cs already uses, does the same.

[claude]

1. Verdict: request changes. It is not yet shown that Match() returns the same answers for patterns with captures, Acceptable callbacks, or variables.
2. Most important: F1 at src/SIL.Machine/FiniteState/TraversalMethodBase.cs:642. Keeping one instance per lattice node drops the registers that Acceptable reads (for example SynthesisRewriteRuleSpec.CheckTarget), so a valid match can be lost.
3. Counts: Critical 0, Important 3 (F1-F3), Low 5 (F4-F8).
4. What I ran: gh reads of the PR, its threads and comments; a read of the diff and the surrounding FiniteState/Matching/HermitCrab source; /code-review high 511. ./local_check.sh, ./local_check.sh --agent-strict and dotnet were not permitted in this review environment, so nothing was built or tested here. CSharpier and the strict comment-hygiene scan were not checked.
5. Not verified: every finding comes from reading the code and none was reproduced. I also did not verify the Aweti speedup. The earlier fuzz comment covers parity testing; it was run at da352c7a, not at this head.

API and compatibility: ITraversalMethod and TraversalMethodBase are internal, so no public signature changed. The public Matcher.Match() behavior does change for determinized patterns (F3). No target framework or package changes. machine.py parity: none verified.

Findings: F1 new, unverified; F2 new, unverified; F3 new, unverified; F4 new; F5 new, unverified; F6 new, unverified; F7 new; F8 new.

Reviewed at 039a97d