diff --git a/CHANGELOG.md b/CHANGELOG.md index df3d67a9..a88736ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- learn(skills): **`unhappy-path-testing.md` — requirement-cited unhappy-path e2e.** Every case cites a requirement (BRD rule, entity doc comment, `not null` error, `Visible:` expression, grant) before it runs; results are PASS / GAP / UNSPECIFIED, and UNSPECIFIED is re-scored against seven house rules (whitespace-as-required, silent truncation, no empty-state text, …) as GAP-HOUSE so a silent requirement cannot hide a user-visible defect. Measured on a warehouse-management demo project, 2026-09-24/25: 45 cases over 5 modules, five overnight runs 16 → 25 of 45 PASS (run 2's 17 GAPs were 10 instrument faults from one deep link), fix loop to 27/45 PASS, 5 GAP, 13 UNSPECIFIED, every GAP an open product Decision. Routed `test,review` at stages 5–6 (`verify`, ondemand) and pointed at from `testing-shape.md`'s owner table. — MendixMau - perf(routing): **CLAUDE.local.md routing carries triggers only.** The `baseline` view (the copy loaded into every session of every project) now cuts each row's "when" cell at its first " — " and replaces the three-paragraph preamble with two lines pointing at `ROUTING.md` for the full text; `readme-baseline` (README, runbook) is unchanged. Fresh scaffold: CLAUDE.local.md 17.5 KB → 11.7 KB (−33%). Field run 2026-10-07 (existing app, one main session per run, 2 runs per side, an entity + 2 microflows + 2 pages): 10/10 steps applied each side, new mx errors 5 vs 5 (the same CE2421 page-binding slip in both), cost $1.39 vs $1.38; the slim lanes opened the MDL pre-flight skill as often or more. Also drops two stray merge-artifact lines from this section. — MendixMau - perf(ui-loop): **fewer mxbuilds, no fixed sleeps, legible screenshots.** Field run 2026-10-07 (a requirements-driven build): `exec.sh` was 22% of tool time at two mxbuilds per exec, journeys and page waits slept a fixed time, and screenshots were 11.5% of re-read tokens. `exec.sh` now skips the pre-flight baseline mxbuild when the model is byte-identical to its last *clean* pass: the verification stamp gains an `errors:` field (`model-stamp.sh check --clean` accepts only `errors: 0`, so a delta-gate pass over pre-existing errors never counts; `SKIP_BASELINE=0` is unchanged). New `tests/e2e/settle.js` waits for the page's own state (no visible Mendix progress/loading indicator, DOM size stable) instead of a fixed time; `journey-runner.js` uses it after each action (an explicit `settleMs` still wins) and `page-audit.js` per page. Probed in a real browser: on a page that loads in 1.8 s the old 1200 ms sleep and networkidle+600 both captured it half-loaded, settle waited for it; on a fast page settle took 775 ms vs 1200. `SETTLE_MODE=fixed` restores the old waits for an A/B run. `page-audit.js` also saves a viewport shot `page-audit-.top.png` beside the full-page one, and `ui-loop.md` says to look at viewport shots: a tall full-page shot reaches the model at ~1024 px high, text unreadable. `shrink-image-read.sh` now works off macOS (ImageMagick or Pillow, python when jq is absent; it was a silent no-op without `sips`). — MendixMau - perf(ui-loop): **a stale app reloads instead of restarting, journeys stop sleeping, worst pages are looked at first.** Field run 2026-10-07, existing app. `test-stack-up.sh` records which model the running app was built from (`.claude/loop/served-model`); when the model has changed since, it hot-reloads with `mxcli docker reload` (adding `--skip-check` when the exec.sh gate already passed this model) and restarts only if that fails: 93 s instead of 179 s. `ui-loop.md` and the script header carry the measured table and point at `mxcli run --local --watch` as the fast loop (~18 s per page change). `journey-runner.js` waits on page state after a combobox action instead of fixed pauses, and a combobox step now opens the widget it names: it used to open the page's first combobox, so on a form with three every step but the first failed. `page-audit.js` clicks the first copy of a nav item a user could actually click, not just the first match: with the sidebar collapsed, every in-nav page faulted while the same item sat in the top menu bar. It also writes `page-audit-look-order.txt`, all pages worst first (`module-review.md` §4a: order, not selection, every page is still looked at). `design-audit.js` drops its fixed 1.2 s wait. Same 7-step journey on the same app: 56 s with `SETTLE_MODE=fixed`, 24 s with settle, 11/11 pass both ways, 3 runs each. — MendixMau diff --git a/README.md b/README.md index 120e7d90..e724582c 100644 --- a/README.md +++ b/README.md @@ -649,6 +649,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Writing DB assertion tests that cross-check UI state against the database | `skills/learned-db-assertions.md` | | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | `skills/fixture-seeding.md` | | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | +| Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | diff --git a/ROUTING.md b/ROUTING.md index 10bd579b..a3f53cdd 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -211,6 +211,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Writing DB assertion tests that cross-check UI state against the database | `skills/learned-db-assertions.md` | test | 6 | ondemand | | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | `skills/fixture-seeding.md` | test,review | 5,6 | ondemand | | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | test,review | 5,6 | ondemand | +| Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | test,review | 5,6 | ondemand | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | test,review | 5,6 | ondemand | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | review | 6 | ondemand | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | review | 6 | ondemand | diff --git a/agents/review-agent.md b/agents/review-agent.md index deff9cbd..b000fbb6 100644 --- a/agents/review-agent.md +++ b/agents/review-agent.md @@ -69,6 +69,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `project-bin/graph-sweep.sh` | Running the wiring rung alone — a module imported but never reached, an element built but wired to nothing, a boundary crossed; mxbuild and e2e are blind to all three | | `skills/fixture-seeding.md` | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | | `skills/journey-proof.md` | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | +| `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | | `skills/learned-skill-ux-audit.md` | UX audit and screenshot-loop discipline | | `skills/learned-skill-scope-delta.md` | Tracking scope delta between the BRD and the built state | diff --git a/agents/test-agent.md b/agents/test-agent.md index ec92c7ba..78cef7c7 100644 --- a/agents/test-agent.md +++ b/agents/test-agent.md @@ -50,6 +50,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/learned-db-assertions.md` | Writing DB assertion tests that cross-check UI state against the database | | `skills/fixture-seeding.md` | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | | `skills/journey-proof.md` | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | +| `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | | `skills/company-brain.md` | Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ | | `skills/close-the-loop.md` | Cutover and retrospective — promoting proven patterns back into the toolkit | diff --git a/bin/gate-check.sh b/bin/gate-check.sh index c1772943..c71fb739 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -1763,8 +1763,8 @@ stage_protocol_paths() { 2) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-brd.md skills/checkpoints/checkpoint-architecture.md skills/image-transcription.md skills/small-project-tier.md skills/kb-generation.md skills/brd-generation.md skills/brd-validation.md" ;; 3) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/learned-mdl-cannot-express.md" ;; 4) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-build.md skills/agent-roles.md skills/small-project-tier.md skills/module-brief.md skills/module-folder-convention.md skills/brd-to-build-plan.md skills/coverage-ledger.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/learned-constants-and-secrets.md" ;; - 5|build-ready) echo "skills/interview-protocol.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/microflow-loop-antipatterns.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; - 6) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 5|build-ready) echo "skills/interview-protocol.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/microflow-loop-antipatterns.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 6) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; 7) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/close-the-loop.md skills/share-demo-package.md skills/handoff-to-studio-pro.md skills/platform-link.md skills/teamserver-alignment.md skills/deploy-to-sandbox.md" ;; *) echo "" ;; # diff --git a/bin/lib/skill-routing.tsv b/bin/lib/skill-routing.tsv index cd670c84..8a2d1d4d 100644 --- a/bin/lib/skill-routing.tsv +++ b/bin/lib/skill-routing.tsv @@ -160,6 +160,7 @@ e2e-harness-base skills/e2e-harness-base.md Standing up or extending the Playwri learned-db-assertions skills/learned-db-assertions.md Writing DB assertion tests that cross-check UI state against the database test 6 ondemand verify fixture-seeding skills/fixture-seeding.md Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness test,review 5,6 ondemand verify journey-proof skills/journey-proof.md Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red test,review 5,6 ondemand verify +unhappy-path-testing skills/unhappy-path-testing.md Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test test,review 5,6 ondemand verify monkey-test skills/monkey-test.md Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works test,review 5,6 ondemand verify learned-skill-ux-audit skills/learned-skill-ux-audit.md UX audit and screenshot-loop discipline review 6 ondemand verify learned-skill-scope-delta skills/learned-skill-scope-delta.md Tracking scope delta between the BRD and the built state review 6 ondemand verify diff --git a/skills/testing-shape.md b/skills/testing-shape.md index c12550c3..471d54d9 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -16,6 +16,7 @@ and the guard for each. Job 2 is why this skill exists; if you shorten it, do no | Non-pipeline app, à-la-carte assurance | `existing-app-assurance.md` (shared toolkit) | | Why a green UI report can still be lying | **§4a below** — merged in, no longer a separate file | | The deep five-rung version of "prove it", with per-rung non-vacuity controls | `journey-proof.md` (shared toolkit) | +| Unhappy paths after the journeys are green — requirement-cited cases, house-rules fallback for UNSPECIFIED, GAP counts with a denominator | `unhappy-path-testing.md` (shared toolkit) | | Requirement traceability the suite asserts against | `coverage-ledger.md` (shared toolkit) | | Why `$?` and tool output are not evidence | `tool-output-is-not-ground-truth.md` (shared toolkit) | | Does the whole journey hang together, not just each piece — a different axis, not a fifth rung | `process-coherence-pass.md` (shared toolkit) | @@ -270,7 +271,7 @@ human looking at the screenshot, not by the suite. bland **"Sign in failed."** on the login page — indistinguishable from a wrong password, and `system$user.failedlogins` stays at **0** because authentication was never reached. Every downstream step then reports as a feature failure. Measured 2026-08-31 on a Mendix 11.13 app: a - 16-step narrated tour scored 0, with each step labelled `(as (not signed in))`, and the only honest + narrated tour of 16 steps scored 0, with each step labelled `(as (not signed in))`, and the only honest evidence was one line in the runtime log — `ERROR - Connector: ... Maximum number of sessions exceeded! (You are currently using a trial license)`. Sixteen "defects" that were one leaked session pool. diff --git a/skills/unhappy-path-testing.md b/skills/unhappy-path-testing.md new file mode 100644 index 00000000..38bdbf96 --- /dev/null +++ b/skills/unhappy-path-testing.md @@ -0,0 +1,126 @@ +# Skill: unhappy-path-testing — every case cites a requirement, or a house rule, before it runs + +**Applies to:** any built module with a form, a role-gated action, or a REST backend; any mxcli project. +**Purpose:** e2e beyond the happy journeys — missing fields, invalid values, wrong state, wrong role, +a down or slow backend, stale data, double clicks — with every case tied to a cite so its result can +be *judged*, not just recorded. Also for the moment an agent must decide whether odd behaviour is a defect. +**Source:** a warehouse-management demo project, 2026-09-24/25 — 45 cases, 5 modules, 5 overnight runs, +then one fix loop. Promoted 2026-10-08. +**Companion skills:** `journey-proof.md` (runs first), `monkey-test.md` (runs after), `learned-db-assertions.md`, +`e2e-harness-base.md`, `testing-shape.md`. + +--- + +## Why it is not fuzzing + +`monkey-test.md` measured random input at **0 defects against 9 for scripted journeys**. Unhappy +paths find defects only when each case asks a *specific* question that a requirement answers. +Without a requirement, a result cannot be judged. The case then records behaviour and the +defect hides in plain sight. + +## The matrix comes first, the code second + +Write `docs/handoffs/unhappy-path-matrix.md` before any spec. One row per case: + +| ID | Requirement cite | Scenario | Expected per requirement | Assertion (UI **and** DB) | Status | + +- **Cite** is a quote with its source, e.g. a BRD business rule, the entity's own doc comment, a + `not null error '…'` in the domain model, a page's `Visible:` expression, or a grant line read + with `DESCRIBE`. Read the **live model**, not `mdlsource/`: a committed script is not an + executed one (`tool-output-is-not-ground-truth.md`). +- **Status** is filled by the run, never by hand: + - **PASS**: the app does what the cite says. + - **GAP**: the app does what the cite forbids, or does not do what it demands. A real finding. + - **UNSPECIFIED**: the cite is silent; the behaviour is recorded, not judged. +- **Reject a source that is about something else.** Check every BRD you cite against the entity + it is meant to cover. In the source project a BRD titled "equipment management" turned out to + describe a different domain's module that happened to share the word; it was read, rejected in + writing at the top of the matrix, and the module's cases were grounded in the live entity's own + doc-comment rules instead. Citing it would have produced confident nonsense. + +## The categories to cover, per module + +For each entity or screen, write at least one case per applicable row: + +1. **Required fields**: blank, and a single space (does `not null` stop whitespace?). +2. **Length and format**: one past the column width, a bad format against a stated pattern. +3. **Duplicates**: a second record with the same business key. +4. **Cancel**: fill the form, cancel, check that no row exists. +5. **Double submit**: two clicks under 300 ms; count the rows. +6. **Wrong state**: act on a record whose state forbids it (released, closed, retired). +7. **Wrong role**: log in as a read-only role, then assert the button is absent **and** that the + microflow refuses (hiding is not access control). +8. **Backend errors** (REST modules): down (connection refused), 4xx with problem details, + 5xx, empty body, slow. Assert the user sees a sentence, not a stack trace or a blank list. +9. **Concurrency**: two sessions, one stale write; the app should say so, not overwrite. +10. **Empty results**: a search that matches nothing; is there an empty-state message? +11. **Security posture**: grants read from the model (anonymous CRUD, over-broad module roles). + Report as findings; do not drive them through Playwright. + +## The house-rules fallback: UNSPECIFIED must not be a hiding place + +In the source project, 13 of 45 cases came back UNSPECIFIED. Reviewing those rows afterwards, at +least 6 were defects a user would report: whitespace passing "required", silent truncation at the +20- and 200-character column widths, no empty-state text. Nothing flagged them, because no +requirement spoke. This is `degrade-to-judgement.md` applied to a test matrix: a missing +requirement changes what you assess *against*, never *whether* you assess. + +So every UNSPECIFIED result is scored a second time against these **house rules**, and a failure +is reported as **GAP-HOUSE**, a separate count from GAP: + +| House rule | Fails when | +|---|---| +| H1 Required means non-blank | A value of only spaces passes a required field | +| H2 Never truncate silently | Stored value is shorter than typed, and no message said so | +| H3 Empty is explained | A list that matches nothing shows a blank area, with no text | +| H4 Errors are sentences | The user sees a code, a stack trace, JSON, or nothing | +| H5 One click, one record | A double submit creates two rows | +| H6 Cancel leaves nothing | A row exists after Cancel | +| H7 Hidden is also refused | A role that cannot see a button can still run its microflow | + +The house rules are not requirements. Put GAP-HOUSE rows in front of the product owner as +Decisions ("fix, or accept as designed?"); do not fix them unasked. + +## Assertions that cannot pass vacuously + +- **UI and DB, both.** "Error text visible" alone passes when the form never saved anyway. Count + rows in the DB before and after, with the same filter the case is about + (`learned-db-assertions.md` for the mechanics). +- **Guard the precondition.** Assert that the record under test exists and is in the state the + case needs, before acting. A case on a released route passes trivially if the seed has none. +- **Log in as a real non-admin role, and assert the username on screen.** A harness that falls + back to an admin account on a failed login makes every role case pass. +- **Assert the nav landed** (page title or a page-unique element) before reading anything — + see `e2e-locators-that-lie.md` for the locator side of this. +- **Never count visible rows** as a total; they stop at the page size. +- For REST modules, **stop or break the backend on purpose** in the down cases, and check that it + really was down (a health probe fails) before asserting the message. + +## Running and reporting + +- One spec per module (`tests/e2e/unhappy/unhappy-.js`), one runner that writes + `results.json` with an ID, status, the cite, the observed behaviour, and a screenshot path per + case. +- The report headline is counts **with the denominator**: `27/45 PASS, 5 GAP, 6 GAP-HOUSE, + 7 UNSPECIFIED, 0 ERROR`. An ERROR (the test itself broke) is never folded into PASS or GAP. + A missing `results.json` is reported as missing, never as `0/45` (`measured-claims.md`). +- Every GAP gets a line: the fix, or the Decision it maps to. In the source project all 5 + remaining GAPs mapped one-to-one to open product Decisions; that mapping is the useful output. +- Look at the screenshots of every GAP and GAP-HOUSE before reporting. A GAP whose screenshot + shows the previous page is a test defect, not an app defect. +- Expect the early runs to be dominated by **instrument faults, not app faults**: in the source + project run 2 reported 17 GAPs, 10 of which were one module's spec using a deep link the + runtime answered with HTTP 404. Triage every GAP cluster that lands on a single module as a + harness suspect first. + +## Where this came from + +A warehouse-management demo project, 2026-09-24 and 25: matrix `docs/handoffs/unhappy-path-matrix.md`, +specs under `tests/e2e/unhappy/`, run log in the overnight handoff. Five runs went +16 → 19 → 22 → 21 → 25 of 45 PASS as harness faults and app fixes landed; the next day's fix loop +took it to **27/45 PASS, 5 GAP, 13 UNSPECIFIED**, with every GAP an open Decision. The house-rules +fallback is the lesson from reviewing the 13 UNSPECIFIED rows afterwards; it had not yet been run +as a second scoring pass in that project, so GAP-HOUSE counts above are the shape, not a measurement. + +Related: `e2e-locators-that-lie.md`, `measured-claims.md`, `tool-output-is-not-ground-truth.md`, +`degrade-to-judgement.md`, `testing-shape.md` §4a (why a green UI report can still be lying).