feat: grader file format with test-style assertions

[queso]

Closes #32.

What this adds

Grader files. A scenario can reference a named grader in a TS/JS file:

"grader": { "file": "./grade.eval.ts", "name": "product-named" }

The file's default export is grade(artifactPath, { name: ({ result }) => ... }). result has assert(condition, message, context?), json(), text, path, shouldHave, shouldNotHave, and shouldMatch. Every check runs, so one run reports every failure, with the context data or an artifact excerpt next to each. An exception in the grader counts as one more failure. A grader that makes no checks fails. The docs and the new example (examples/07-grader-file) lead with semantic result.assert checks, per design constraint 1.

The command contract stays the base layer. A file grader compiles to a command grader: bun <pkg>/promptdiff grade --file <abs> --name <name> runs in the sandbox with the same cwd, env, and timeout handling. promptdiff grade is also a public subcommand, so you can run a grader by hand inside a kept sandbox. The artifact path resolves against the grader's cwd, the same place a command grader looks. Grader files import grade from @theaiteam/promptdiff (or bare promptdiff). A Bun plugin resolves both names to the running copy, so no install is needed next to the file. package.json now has an exports entry for consumers.

Missing artifacts are a separate outcome.

• File graders exit 77 when the artifact does not exist. Command graders can exit 77 too; the code is exported to them as $PROMPTDIFF_NO_ARTIFACT_EXIT_CODE. A grader with an explicit expectExitCode: 77 keeps its old meaning.
• Pass rates and Fisher's p use graded runs only. Summaries read 4/4 pass (100%), 1 no-artifact and list each such run as run 3 no artifact: ....
• compare prints a NOTE with both arms' no-artifact counts. An arm with zero graded runs fails its target or regression assertion.
• Reports and receipts include noArtifact only when it is non-zero, so existing output does not change.

Load-time validation. A missing grader file, a bad default export, or an unknown grader name fails when the scenario loads, before any paid run. The check runs promptdiff grade --list in a child process.

compare points at measure. On a baseline-only scenario the error now reads compare requires proposed skill paths; to run one instruction set without a proposed arm, use \promptdiff measure``.

Cache. The baseline cache key includes a hash of the grader file's content.

Open question for review

In compare, a proposed arm that produces the artifact less often than baseline only gets a NOTE. For example, proposed at 1/1 pass plus 4 no-artifact, against baseline at 3/5, passes a target assertion. That follows the rule that a missing artifact is never a failure. For a regression assertion, though, a skill change that stops the agent writing the artifact is arguably the regression you want to catch. One option is to fail regression when proposed's no-artifact count clearly exceeds baseline's. I left that out rather than guess at the threshold.

Known limits

• The cache hashes the grader file but not modules it imports (documented).
• Loading a scenario now executes grader file code in a child process (documented in the security note).
• The promptdiff grade output in example 07 was captured for real. Its measure cost is marked "not measured" because no paid run was made.

Testing

• bun run check: 155 tests pass, typecheck clean. The new tests in test/grade-file.test.ts cover collecting every failure, no-artifact classification for file and command graders, summary lines, config validation errors, and the compare hint.
• Ran promptdiff grade by hand from a directory outside the repo with a bare import { grade } from "promptdiff". Checked --list, a missing artifact (exit 77), two failing checks reported together (exit 1), all checks passing (exit 0), and an unknown name (exit 2).

🤖 Generated with Claude Code

[github-actions]

Nitpick review — comment

The grader-file feature is well-structured and largely well-tested, but the exit-77 reclassification applies to all command graders, not just file graders, so a broken grader script exiting 77 would now be excluded from pass rates instead of counted as a failure — a silent semantic change for existing suites. A few lower-severity items remain: prototype-inherited lookups in grader-file, a potentially degenerate samplingP when an arm has zero graded runs, a test depending on a prebuilt binary, and new timeout/no-artifact semantics that ship without pinning tests. No blocking issues; recommend addressing the exit-77 scoping before merge.

6 inline comment(s).

[github-actions]

Nitpick review — comment

The grader-file feature is well-structured and mostly well-tested, but the exit-77 reclassification in gradeCommand changes semantics for any pre-existing command grader that happens to exit 77 without declaring it: such graders silently move from 'visible failure' to 'noArtifact' (excluded from pass-rate denominators), altering compare statistics with no config change. Two smaller items remain: an unguarded JSON.parse that degrades a load-time diagnostic into an opaque crash, and a CLI help test that depends on a prebuilt binary at the repo root. Worth addressing before merge, none blocking.

1 inline comment(s).

2 previously-acknowledged finding(s) not re-posted (resolved review threads).