Skip to content

skills: promote unhappy-path-testing (requirement-cited unhappy-path e2e) - #250

Closed
MendixMau wants to merge 1 commit into
masterfrom
feat/unhappy-path-testing
Closed

skills: promote unhappy-path-testing (requirement-cited unhappy-path e2e)#250
MendixMau wants to merge 1 commit into
masterfrom
feat/unhappy-path-testing

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Direct lane (skills/)

What changed and why (one paragraph):

Promotes skills/unhappy-path-testing.md from a personal toolkit draft into the shared toolkit. It is the requirement-cited form of unhappy-path e2e: the matrix (docs/handoffs/unhappy-path-matrix.md) is written before any spec, one row per case, and every row quotes its source (BRD rule, entity doc comment, not null error text, Visible: expression, grant read with DESCRIBE, from the live model). The run fills Status as PASS / GAP / UNSPECIFIED; UNSPECIFIED is re-scored against seven house rules (H1 required means non-blank, H2 never truncate silently, H3 empty is explained, H4 errors are sentences, H5 one click one record, H6 cancel leaves nothing, H7 hidden is also refused) and reported separately as GAP-HOUSE, so a silent requirement cannot hide a user-visible defect. Eleven categories per module, vacuity guards (UI and DB both, precondition asserted, real non-admin login with username on screen, backend provably down), and a headline that always carries the denominator. It sits between journey-proof.md (before) and monkey-test.md (after).

Field evidence — a warehouse-management demo project, 2026-09-24/25:

  • 45 cases over 5 modules (11 + 10 + 10 + 9 + 5), one spec per module plus a runner writing results.json.
  • Five overnight runs: 16 → 19 → 22 → 21 → 25 of 45 PASS. Run 2's 17 GAPs included 10 from one module whose spec used a deep link the runtime answered with HTTP 404 (instrument fault, not app fault) — the skill now says to triage a single-module GAP cluster as a harness suspect first.
  • Fix loop the next day: 27/45 PASS, 5 GAP, 13 UNSPECIFIED; all 5 GAPs mapped one-to-one to open product Decisions.
  • Reviewing the 13 UNSPECIFIED rows afterwards found at least 6 user-visible defects (whitespace passing required, silent truncation at 20/200 chars, no empty-state text). That review is where the house-rules fallback comes from; the GAP-HOUSE count itself has not yet been run as a second scoring pass, and the skill says so.
  • One BRD was read and rejected because it described a different domain's module with the same name; the skill keeps that as a rule ("reject a source that is about something else") without the project detail.

No project code, data or names are copied; the skill describes the process with the run as the measured source.

Wired:

  • bin/lib/skill-routing.tsv: new row unhappy-path-testing — test,review, stages 5,6, ondemand, group verify; placed after journey-proof, before monkey-test. Re-rendered with bin/render-routing.sh (README.md, ROUTING.md, agents/test-agent.md, agents/review-agent.md).
  • skills/testing-shape.md "Read the owner" table: one pointer row after journey-proof.md.
  • CHANGELOG.md: learn(skills) line under ## Unreleased.
  • skills/testing-shape.md line 273 (pre-existing on master): the pre-commit check-docs-numbering.sh read "16-step narrated tour" as a list count once the file was in the diff; reworded to "narrated tour of 16 steps", same measurement, no content change. query-the-model.md:52 and source-triage.md:267 have the same pre-existing false positive and are left alone (not in this diff).
  • Cross references point only at skills that exist here (journey-proof, monkey-test, testing-shape, e2e-harness-base, learned-db-assertions, measured-claims, tool-output-is-not-ground-truth, degrade-to-judgement) plus e2e-locators-that-lie.md, which is being promoted in a parallel PR.

Checks run (all pass): bin/check-pr-discipline.sh, bin/check-skill-routing.sh, bin/render-routing.sh --check, bin/check-no-client-data.sh, bin/check-portability.sh.

  • No client data anywhere in the diff
  • Size cap: 8 files, well under 400 lines excluding CHANGELOG.md and rendered surfaces
  • Test tier reached: T0 — mechanical gates above; no instrument changed, so no fixture applies
  • For a new skill: routing row added (bin/lib/skill-routing.tsv + bin/render-routing.sh)
  • CHANGELOG.md line appended in this PR, crediting the source project
  • New bug entries: none

🤖 Generated with Claude Code

@MendixMau @claude
skills: promote unhappy-path-testing (requirement-cited unhappy-path …
c3cd9af 
…e2e)

Every unhappy-path case cites a requirement before it runs; results are
PASS / GAP / UNSPECIFIED, and UNSPECIFIED is re-scored against seven house
rules as GAP-HOUSE so a silent requirement cannot hide a user-visible
defect. Measured on a warehouse-management demo project, 2026-09-24/25:
45 cases, 5 modules, 25 -> 27 of 45 PASS, 5 GAP, 13 UNSPECIFIED.

Wired: routing row (test,review; stages 5,6; ondemand; verify) and the
rendered surfaces, a pointer row in testing-shape.md's owner table, and
the CHANGELOG line. testing-shape.md:273 reworded (16-step -> 16 steps)
so the pre-commit numbering check no longer reads a measurement as a
list count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MendixMau MendixMau mentioned this pull request Oct 8, 2026
Merged
5 tasks done
MendixMau added a commit that referenced this pull request Oct 8, 2026
@MendixMau @claude
Integration 2026-10-08: 13 PRs (#258)
9dd9be4 
Squash of integration/2026-10-08, which merges:

- #225 perf(ui-loop): test-stack-up.sh recognises a live mxcli run --local
- #231 fix(sync): sync-project.sh no longer exits 1 silently before its bin/ refresh
- #242 fix(init): local by default, no Dev Container
- #243 docs(lint): process/lint-backlog.md, the lint rollout plan
- #247 new(lint): the lint gate keeps a ledger
- #249 learn(test): skills/e2e-locators-that-lie.md
- #250 learn(skills): unhappy-path-testing.md
- #251 fix(testing-shape): mxcli test --local shares the deploy directory (#230)
- #252 fix(obligations): LOOK staleness reads the newest VALID AT stamp (#155)
- #253 docs(lint): quality-source review, Batch 4 backlog rows and skill additions
- #254 fix(lint): CONV020 works out of the box, page navigation counts as feedback (#229, #245)
- #255 fix(e2e): journey-runner rung 2 runs for a negative-only span claim (#150)
- #256 fix(gates): Stage 6 Surface, source-sufficiency title, ${JOURNEY_DIR} manifest token (#233, #234, #235)

Verified on the branch: tests/run-tests.sh 33/0, tests/test-lint-delivery.sh ALL GREEN, tests/wave2 73/75 (two environmental), render-routing --check, check-scripts, check-portability, leak guard, check-no-private-citations, check-pr-discipline all clean; scaffold smoke run without crashes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau

MendixMau commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Merged into master via integration PR #258 (squash 9dd9be4), on the maintainer's approval. Closing this PR; its branch is part of that squash.

Generated by Claude Code

@MendixMau MendixMau closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MendixMau