Skip to content

Add a conformance suite for HermitCrab, with its adequacy argument - #480

Open
johnml1135 wants to merge 2 commits into
masterfrom
integrate-conformance-framework
Open

johnml1135 wants to merge 2 commits into
masterfrom
integrate-conformance-framework

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

Quick summary

Parser maintainers can compare full parse signatures across HermitCrab and compatible engines.
The suite covers 44 fixtures and 536 cases with explicit expectations and auditable coverage evidence.
It leaves HermitCrab's analysis and synthesis algorithms unchanged.

Where to look

  • conformance/PROTOCOL.md -- engine adapter contract.
  • conformance/generated/hc-conformance-manifest.v1.json -- fixture paths and hashes.
  • src/SIL.Machine.Morphology.HermitCrab.Conformance/SemanticCoverage/AnalyzerMetadataInspector.cs -- hash-pinned generator admission remains probe-gated.
  • src/SIL.Machine.Morphology.HermitCrab.Conformance/MaterializedRunner.cs -- crash fixtures require an ordinary fixture to complete.

Deliberately not included

  • Counterfactual and coverage-evidence sweeps remain deferred; known oracle gaps are tracked.

Validation

  • dotnet build and dotnet build -c Release -- succeeded, 0 warnings.
  • dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --no-build -- 593 passed, 1 skipped (symlink test needs elevation), 0 failed.
  • dotnet run --no-build -c Release --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance -- 43 passed, 0 failed.
  • ... -- --semantic-coverage --repository-root . -- 0 new gaps, 0 stale lines.
  • python conformance/parity-check.py -- all checks passed.
  • dotnet csharpier check . -- 868 files, clean.
  • pwsh -NoProfile -File scripts/comment-hygiene.ps1 -BaseRef origin/master -- clean.

Decisions, and why
  • PCRE.NET (now on master) ships an IIncrementalGenerator. The compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
  • An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
  • *.fwdata is marked binary in .gitattributes; its pinned hash depended on core.autocrlf, which is why CI failed a test that passed locally.
  • Hand-maintained fixture totals were removed from the README and docs; the manifest is the count.
  • Four fixture grammars keep trailing whitespace: their hashes are pinned in the manifest.

🤖 Generated with Claude Code


This change is Reviewable

@johnml1135
johnml1135 requested a review from ddaspit August 19, 2026 19:27
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from d826869 to 8f9dcea Compare August 19, 2026 19:46
@codecov-commenter

codecov-commenter commented Aug 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.08%. Comparing base (c7146cd) to head (18cf242).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #480      +/-   ##
==========================================
- Coverage   74.34%   74.08%   -0.27%     
==========================================
  Files         456      555      +99     
  Lines       38261    55976   +17715     
  Branches     5242     7606    +2364     
==========================================
+ Hits        28445    41468   +13023     
- Misses       8666    12658    +3992     
- Partials     1150     1850     +700     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from d0b5448 to 9dfb2f6 Compare August 20, 2026 09:10
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from de95458 to 4823a05 Compare September 4, 2026 00:15
johnml1135 added a commit that referenced this pull request Sep 15, 2026
Introduces an engine-agnostic conformance suite for HermitCrab-formalism
morphological parsers: a `conformance/` tree of language grammars and
edge-case micro-grammars, each an HC-XML grammar.xml paired with a
words.yaml ground truth (oracle-provenance, gloss, and per-word parse
signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project
and hc-conformance CLI tool run those fixtures against an engine adapter,
producing coverage and rules CSVs, an interface inventory, dataflow-
obligation ledgers, evidence cards, a FieldWorks-producibility table, and
a JSON manifest for external consumers. PROTOCOL.md is the adapter
contract a second engine (e.g. the in-progress Rust port) implements to
plug into the same suite.

Includes the FieldWorks-producibility classification campaign: for every
HermitCrab construct the suite covers, a verdict on whether FieldWorks'
own HC-XML export path can actually produce a grammar exercising it,
distinguishing genuine data-model limits from mere loader gaps, with five
previously-unresolved verdicts now classified and corrected write-ups for
two more cases initially misread as exporter silence.

Also lands one new edge-case fixture,
edge-cases/chained-output-feature-override-loss, isolating an analysis-
side OutputHeadFeatures retraction bug: a well-formed four-morph parse of
`zudiua` that the founding C# oracle currently loses. The fixture is
deliberately red against hc.dll today (expected, not a broken test) and
documents the mechanism, provenance, and the HC-Rust divergence that
motivated it.

The suite is deliberately stopped at this point rather than finished:
scope, known gaps, and next steps are documented in the suite's own
tracking docs rather than left implicit.

This commit squashes the full conformance-suite branch stack
(integrate-conformance-framework through conformance/fieldworks-
witnesses, 49 commits) into one commit on top of current master, for
PR #480.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from 4823a05 to 3128301 Compare September 15, 2026 12:29
johnml1135 added a commit that referenced this pull request Sep 15, 2026
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix)
against the conformance suite: 36/36 fixtures pass (including
pathological cases), and a generated-word fuzz sweep (every root x
every 0..3-subset of affix rules, up to 6 nonhead roots where the
grammar compounds) produced ~1,300 words across 35 grammars, parsed by
both master and the PR branch with 0 divergences attributable to
PR #494. The one observed diff (edge-cases/truncate-morphotactic,
words `a`/`ga`) is pre-existing oracle nondeterminism in analysis
multiplicity ordering, reproducible on master alone, unrelated to the
syntactic FS change.

This was previously recorded only at tag archive/pr494-conformance;
folding it into the conformance suite's own PR (#480) so the evidence
travels with the suite it was run against. Fixed two references to
branches deleted since the doc was written (perf/pr494-conformance,
conformance/fieldworks-witnesses) to point at their preserved
archive/* tags instead; no other content changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@johnml1135

Copy link
Copy Markdown
Collaborator Author

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

@johnml1135

Copy link
Copy Markdown
Collaborator Author

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

johnml1135 added a commit that referenced this pull request Sep 26, 2026
Introduces an engine-agnostic conformance suite for HermitCrab-formalism
morphological parsers: a `conformance/` tree of language grammars and
edge-case micro-grammars, each an HC-XML grammar.xml paired with a
words.yaml ground truth (oracle-provenance, gloss, and per-word parse
signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project
and hc-conformance CLI tool run those fixtures against an engine adapter,
producing coverage and rules CSVs, an interface inventory, dataflow-
obligation ledgers, evidence cards, a FieldWorks-producibility table, and
a JSON manifest for external consumers. PROTOCOL.md is the adapter
contract a second engine (e.g. the in-progress Rust port) implements to
plug into the same suite.

Includes the FieldWorks-producibility classification campaign: for every
HermitCrab construct the suite covers, a verdict on whether FieldWorks'
own HC-XML export path can actually produce a grammar exercising it,
distinguishing genuine data-model limits from mere loader gaps, with five
previously-unresolved verdicts now classified and corrected write-ups for
two more cases initially misread as exporter silence.

Also lands one new edge-case fixture,
edge-cases/chained-output-feature-override-loss, isolating an analysis-
side OutputHeadFeatures retraction bug: a well-formed four-morph parse of
`zudiua` that the founding C# oracle currently loses. The fixture is
deliberately red against hc.dll today (expected, not a broken test) and
documents the mechanism, provenance, and the HC-Rust divergence that
motivated it.

The suite is deliberately stopped at this point rather than finished:
scope, known gaps, and next steps are documented in the suite's own
tracking docs rather than left implicit.

This commit squashes the full conformance-suite branch stack
(integrate-conformance-framework through conformance/fieldworks-
witnesses, 49 commits) into one commit on top of current master, for
PR #480.

Folds in three review fixes: AdapterEngine's contract in MaterializedRunner
now credits an expect_crash fixture only once the same engine also
completes an ordinary fixture from the same run, so a misconfigured
adapter (bad path, rejected arguments) can no longer register as
reproducing the pinned crash; GuesserSignatureTests' batch-path test is
renamed for the overload it actually calls; and .gitattributes freezes
project.fwdata as binary so its pinned SHA-256 is reproducible across
platforms and CI runners, not just this one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from 93f470c to 2933fcc Compare September 26, 2026 02:55
johnml1135 added a commit that referenced this pull request Sep 26, 2026
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix)
against the conformance suite: 36/36 fixtures pass (including
pathological cases), and a generated-word fuzz sweep (every root x
every 0..3-subset of affix rules, up to 6 nonhead roots where the
grammar compounds) produced ~1,300 words across 35 grammars, parsed by
both master and the PR branch with 0 divergences attributable to
PR #494. The one observed diff (edge-cases/truncate-morphotactic,
words `a`/`ga`) is pre-existing oracle nondeterminism in analysis
multiplicity ordering, reproducible on master alone, unrelated to the
syntactic FS change.

This was previously recorded only at tag archive/pr494-conformance;
folding it into the conformance suite's own PR (#480) so the evidence
travels with the suite it was run against. Fixed two references to
branches deleted since the doc was written (perf/pr494-conformance,
conformance/fieldworks-witnesses) to point at their preserved
archive/* tags instead; no other content changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Sep 26, 2026
Introduces an engine-agnostic conformance suite for HermitCrab-formalism
morphological parsers: a `conformance/` tree of language grammars and
edge-case micro-grammars, each an HC-XML grammar.xml paired with a
words.yaml ground truth (oracle-provenance, gloss, and per-word parse
signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project
and hc-conformance CLI tool run those fixtures against an engine adapter,
producing coverage and rules CSVs, an interface inventory, dataflow-
obligation ledgers, evidence cards, a FieldWorks-producibility table, and
a JSON manifest for external consumers. PROTOCOL.md is the adapter
contract a second engine (e.g. the in-progress Rust port) implements to
plug into the same suite.

Includes the FieldWorks-producibility classification campaign: for every
HermitCrab construct the suite covers, a verdict on whether FieldWorks'
own HC-XML export path can actually produce a grammar exercising it,
distinguishing genuine data-model limits from mere loader gaps, with five
previously-unresolved verdicts now classified and corrected write-ups for
two more cases initially misread as exporter silence.

Also lands one new edge-case fixture,
edge-cases/chained-output-feature-override-loss, isolating an analysis-
side OutputHeadFeatures retraction bug: a well-formed four-morph parse of
`zudiua` that the founding C# oracle currently loses. The fixture is
deliberately red against hc.dll today (expected, not a broken test) and
documents the mechanism, provenance, and the HC-Rust divergence that
motivated it.

The suite is deliberately stopped at this point rather than finished:
scope, known gaps, and next steps are documented in the suite's own
tracking docs rather than left implicit.

This commit squashes the full conformance-suite branch stack
(integrate-conformance-framework through conformance/fieldworks-
witnesses, 49 commits) into one commit on top of current master, for
PR #480.

Folds in three review fixes: AdapterEngine's contract in MaterializedRunner
now credits an expect_crash fixture only once the same engine also
completes an ordinary fixture from the same run, so a misconfigured
adapter (bad path, rejected arguments) can no longer register as
reproducing the pinned crash; GuesserSignatureTests' batch-path test is
renamed for the overload it actually calls; and .gitattributes freezes
project.fwdata as binary so its pinned SHA-256 is reproducible across
platforms and CI runners, not just this one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Sep 26, 2026
Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix)
against the conformance suite: 36/36 fixtures pass (including
pathological cases), and a generated-word fuzz sweep (every root x
every 0..3-subset of affix rules, up to 6 nonhead roots where the
grammar compounds) produced ~1,300 words across 35 grammars, parsed by
both master and the PR branch with 0 divergences attributable to
PR #494. The one observed diff (edge-cases/truncate-morphotactic,
words `a`/`ga`) is pre-existing oracle nondeterminism in analysis
multiplicity ordering, reproducible on master alone, unrelated to the
syntactic FS change.

This was previously recorded only at tag archive/pr494-conformance;
folding it into the conformance suite's own PR (#480) so the evidence
travels with the suite it was run against. Fixed two references to
branches deleted since the doc was written (perf/pr494-conformance,
conformance/fieldworks-witnesses) to point at their preserved
archive/* tags instead; no other content changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from 2933fcc to f412c25 Compare September 26, 2026 04:15
Introduces an engine-agnostic conformance suite for HermitCrab-formalism
morphological parsers: a `conformance/` tree of language grammars and
edge-case micro-grammars, each an HC-XML grammar.xml paired with a
words.yaml ground truth (oracle-provenance, gloss, and per-word parse
signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project
and hc-conformance CLI tool run those fixtures against an engine adapter,
producing coverage and rules CSVs, an interface inventory, dataflow-
obligation ledgers, evidence cards, a FieldWorks-producibility table, and
a JSON manifest for external consumers. PROTOCOL.md is the adapter
contract a second engine (e.g. the in-progress Rust port) implements to
plug into the same suite.

Includes the FieldWorks-producibility classification campaign: for every
HermitCrab construct the suite covers, a verdict on whether FieldWorks'
own HC-XML export path can actually produce a grammar exercising it,
distinguishing genuine data-model limits from mere loader gaps, with five
previously-unresolved verdicts now classified and corrected write-ups for
two more cases initially misread as exporter silence.

Also lands one new edge-case fixture,
edge-cases/chained-output-feature-override-loss, isolating an analysis-
side OutputHeadFeatures retraction bug: a well-formed four-morph parse of
`zudiua` that the founding C# oracle currently loses. The fixture is
deliberately red against hc.dll today (expected, not a broken test) and
documents the mechanism, provenance, and the HC-Rust divergence that
motivated it.

The suite is deliberately stopped at this point rather than finished:
scope, known gaps, and next steps are documented in the suite's own
tracking docs rather than left implicit.

This commit squashes the full conformance-suite branch stack
(integrate-conformance-framework through conformance/fieldworks-
witnesses, 49 commits) into one commit on top of current master, for
PR #480.

Folds in three review fixes: AdapterEngine's contract in MaterializedRunner
now credits an expect_crash fixture only once the same engine also
completes an ordinary fixture from the same run, so a misconfigured
adapter (bad path, rejected arguments) can no longer register as
reproducing the pinned crash; GuesserSignatureTests' batch-path test is
renamed for the overload it actually calls; and .gitattributes freezes
project.fwdata as binary so its pinned SHA-256 is reproducible across
platforms and CI runners, not just this one.

Also folds in the follow-up PR commits: fuzz/break-attempt evidence;
admitting PCRE.NET's source generator via a vetted content-hash allowlist;
final-template and template-sharing controls; oracle-verified fixtures for
per-piece environments, simultaneous mode and iterative epenthesis;
reduplication x phonology cases; and a regenerated manifest, ledgers and
evidence cards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the integrate-conformance-framework branch from f412c25 to d8ff628 Compare September 30, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants