Add a conformance suite for HermitCrab, with its adequacy argument - #480
johnml1135 wants to merge 2 commits into
Conversation
d826869 to
8f9dcea
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #480 +/- ##
==========================================
- Coverage 74.34% 74.08% -0.27%
==========================================
Files 456 555 +99
Lines 38261 55976 +17715
Branches 5242 7606 +2364
==========================================
+ Hits 28445 41468 +13023
- Misses 8666 12658 +3992
- Partials 1150 1850 +700 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d0b5448 to
9dfb2f6
Compare
de95458 to
4823a05
Compare
Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4823a05 to
3128301
Compare
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct. |
|
Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip). Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed. PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence. |
Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
93f470c to
2933fcc
Compare
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2933fcc to
f412c25
Compare
Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Also folds in the follow-up PR commits: fuzz/break-attempt evidence; admitting PCRE.NET's source generator via a vetted content-hash allowlist; final-template and template-sharing controls; oracle-verified fixtures for per-piece environments, simultaneous mode and iterative epenthesis; reduplication x phonology cases; and a regenerated manifest, ledgers and evidence cards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f412c25 to
d8ff628
Compare
Quick summary
Parser maintainers can compare full parse signatures across HermitCrab and compatible engines.
The suite covers 44 fixtures and 536 cases with explicit expectations and auditable coverage evidence.
It leaves HermitCrab's analysis and synthesis algorithms unchanged.
Where to look
conformance/PROTOCOL.md-- engine adapter contract.conformance/generated/hc-conformance-manifest.v1.json-- fixture paths and hashes.src/SIL.Machine.Morphology.HermitCrab.Conformance/SemanticCoverage/AnalyzerMetadataInspector.cs-- hash-pinned generator admission remains probe-gated.src/SIL.Machine.Morphology.HermitCrab.Conformance/MaterializedRunner.cs-- crash fixtures require an ordinary fixture to complete.Deliberately not included
Validation
dotnet buildanddotnet build -c Release-- succeeded, 0 warnings.dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --no-build-- 593 passed, 1 skipped (symlink test needs elevation), 0 failed.dotnet run --no-build -c Release --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance-- 43 passed, 0 failed.... -- --semantic-coverage --repository-root .-- 0 new gaps, 0 stale lines.python conformance/parity-check.py-- all checks passed.dotnet csharpier check .-- 868 files, clean.pwsh -NoProfile -File scripts/comment-hygiene.ps1 -BaseRef origin/master-- clean.Decisions, and why
IIncrementalGenerator. The compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.expect_crashfixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).*.fwdatais marked binary in.gitattributes; its pinned hash depended oncore.autocrlf, which is why CI failed a test that passed locally.🤖 Generated with Claude Code
This change is