From 5971577b3b462f8058766e723a499066d8ea8347 Mon Sep 17 00:00:00 2001 From: MendixMau Date: Thu, 8 Oct 2026 22:07:51 +0800 Subject: [PATCH] skills: promote e2e-locators-that-lie (four Playwright harness defects on Mendix pages) Four measured defects (2026-09-17, user-group administration module) where the Playwright harness reported a defect in the app: widget name unique per page not per DOM, forced click on a tab swallowed without throwing, grid re-render after the response not the click, fixture literal written by the reset script. Routed test/review at Stages 5-6 (diagnose), pointer from e2e-harness-base.md, CHANGELOG line in this commit. The shipped runner does not yet apply rules 1-3; the skill states that and the runner is unchanged. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 1 + README.md | 1 + ROUTING.md | 1 + agents/review-agent.md | 1 + agents/test-agent.md | 1 + bin/gate-check.sh | 4 +- bin/lib/skill-routing.tsv | 1 + skills/e2e-harness-base.md | 1 + skills/e2e-locators-that-lie.md | 121 ++++++++++++++++++++++++++++++++ 9 files changed, 130 insertions(+), 2 deletions(-) create mode 100644 skills/e2e-locators-that-lie.md diff --git a/CHANGELOG.md b/CHANGELOG.md index df3d67a9..33ca0da6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- learn(test): **`skills/e2e-locators-that-lie.md`** — four Playwright harness defects that report a defect in the app: a widget name is unique per page not per DOM (hidden tab pane first, so `.first()` drives or asserts the wrong node), `click({force:true})` on a Mendix tab is swallowed and never throws (so force-then-plain fallback is dead), a grid re-renders on the response not the click (single-read text assertions race), and a fixture literal the reset script writes measures only the reset. Measured 2026-09-17 on a user-group administration module; routed for test/review at Stages 5–6 (diagnose), pointer from `e2e-harness-base.md`. The shipped `project-tests/e2e/journey-runner.js` does not yet apply rules 1–3; the skill says so. — MendixMau - perf(routing): **CLAUDE.local.md routing carries triggers only.** The `baseline` view (the copy loaded into every session of every project) now cuts each row's "when" cell at its first " — " and replaces the three-paragraph preamble with two lines pointing at `ROUTING.md` for the full text; `readme-baseline` (README, runbook) is unchanged. Fresh scaffold: CLAUDE.local.md 17.5 KB → 11.7 KB (−33%). Field run 2026-10-07 (existing app, one main session per run, 2 runs per side, an entity + 2 microflows + 2 pages): 10/10 steps applied each side, new mx errors 5 vs 5 (the same CE2421 page-binding slip in both), cost $1.39 vs $1.38; the slim lanes opened the MDL pre-flight skill as often or more. Also drops two stray merge-artifact lines from this section. — MendixMau - perf(ui-loop): **fewer mxbuilds, no fixed sleeps, legible screenshots.** Field run 2026-10-07 (a requirements-driven build): `exec.sh` was 22% of tool time at two mxbuilds per exec, journeys and page waits slept a fixed time, and screenshots were 11.5% of re-read tokens. `exec.sh` now skips the pre-flight baseline mxbuild when the model is byte-identical to its last *clean* pass: the verification stamp gains an `errors:` field (`model-stamp.sh check --clean` accepts only `errors: 0`, so a delta-gate pass over pre-existing errors never counts; `SKIP_BASELINE=0` is unchanged). New `tests/e2e/settle.js` waits for the page's own state (no visible Mendix progress/loading indicator, DOM size stable) instead of a fixed time; `journey-runner.js` uses it after each action (an explicit `settleMs` still wins) and `page-audit.js` per page. Probed in a real browser: on a page that loads in 1.8 s the old 1200 ms sleep and networkidle+600 both captured it half-loaded, settle waited for it; on a fast page settle took 775 ms vs 1200. `SETTLE_MODE=fixed` restores the old waits for an A/B run. `page-audit.js` also saves a viewport shot `page-audit-.top.png` beside the full-page one, and `ui-loop.md` says to look at viewport shots: a tall full-page shot reaches the model at ~1024 px high, text unreadable. `shrink-image-read.sh` now works off macOS (ImageMagick or Pillow, python when jq is absent; it was a silent no-op without `sips`). — MendixMau - perf(ui-loop): **a stale app reloads instead of restarting, journeys stop sleeping, worst pages are looked at first.** Field run 2026-10-07, existing app. `test-stack-up.sh` records which model the running app was built from (`.claude/loop/served-model`); when the model has changed since, it hot-reloads with `mxcli docker reload` (adding `--skip-check` when the exec.sh gate already passed this model) and restarts only if that fails: 93 s instead of 179 s. `ui-loop.md` and the script header carry the measured table and point at `mxcli run --local --watch` as the fast loop (~18 s per page change). `journey-runner.js` waits on page state after a combobox action instead of fixed pauses, and a combobox step now opens the widget it names: it used to open the page's first combobox, so on a form with three every step but the first failed. `page-audit.js` clicks the first copy of a nav item a user could actually click, not just the first match: with the sidebar collapsed, every in-nav page faulted while the same item sat in the top menu bar. It also writes `page-audit-look-order.txt`, all pages worst first (`module-review.md` §4a: order, not selection, every page is still looked at). `design-audit.js` drops its fixed 1.2 s wait. Same 7-step journey on the same app: 56 s with `SETTLE_MODE=fixed`, 24 s with settle, 11/11 pass both ways, 3 runs each. — MendixMau diff --git a/README.md b/README.md index 120e7d90..75a62ce4 100644 --- a/README.md +++ b/README.md @@ -674,6 +674,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Task | Skill to load | |---|---| | Reading a whole class of tool defects (a retest, a new mxcli release, an audit) — for one CE code or symptom use bin/bug-lookup.sh instead; the ledger is 32k words | `bug-logs/mxcli-bugs.md` | +| An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures | `skills/e2e-locators-that-lie.md` | | Any refused, denied or blocked command — BEFORE rewriting a permission rule and before telling the user a tool is blocked. A rule matches the START of the command line, so an allowlisted tool prefixed with cd matches nothing | `skills/agent-permission-friction.md` | | Studio Pro will not load the project, or the .mpr looks gutted — recover before relaunching SP, never git checkout | `skills/mpr-corruption-and-sp-load-errors.md` | | Preparing an mxcli/Studio Pro bug for submission — scope pinning, read-back-vs-write-path verification, gate-sensitivity negative controls, severity scoping, before it's called filable | `skills/bug-submission-checklist.md` | diff --git a/ROUTING.md b/ROUTING.md index 10bd579b..a595e38f 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -237,6 +237,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Always relevant for | Load this | Agent(s) | Stage(s) | Tier | |---|---|---|---|---| | Reading a whole class of tool defects (a retest, a new mxcli release, an audit) — for one CE code or symptom use bin/bug-lookup.sh instead; the ledger is 32k words | `bug-logs/mxcli-bugs.md` | mdl,gate | 5,6 | ondemand | +| An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures | `skills/e2e-locators-that-lie.md` | test,review | 5,6 | ondemand | | Any time an exit code, a tool's output or a subagent's report is about to become a stated finding — verify before you conclude | `skills/tool-output-is-not-ground-truth.md` | all | - | baseline | | Any refused, denied or blocked command — BEFORE rewriting a permission rule and before telling the user a tool is blocked. A rule matches the START of the command line, so an allowlisted tool prefixed with cd matches nothing | `skills/agent-permission-friction.md` | all | - | ondemand | | Studio Pro will not load the project, or the .mpr looks gutted — recover before relaunching SP, never git checkout | `skills/mpr-corruption-and-sp-load-errors.md` | mdl,gate | - | ondemand | diff --git a/agents/review-agent.md b/agents/review-agent.md index deff9cbd..bd681c04 100644 --- a/agents/review-agent.md +++ b/agents/review-agent.md @@ -70,6 +70,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/fixture-seeding.md` | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | | `skills/journey-proof.md` | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | +| `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures | | `skills/learned-skill-ux-audit.md` | UX audit and screenshot-loop discipline | | `skills/learned-skill-scope-delta.md` | Tracking scope delta between the BRD and the built state | | `skills/company-brain.md` | Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ | diff --git a/agents/test-agent.md b/agents/test-agent.md index ec92c7ba..70da8dab 100644 --- a/agents/test-agent.md +++ b/agents/test-agent.md @@ -51,6 +51,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/fixture-seeding.md` | Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness | | `skills/journey-proof.md` | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | +| `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures | | `skills/company-brain.md` | Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ | | `skills/close-the-loop.md` | Cutover and retrospective — promoting proven patterns back into the toolkit | | `skills/report-schema.md` | Writing or reading docs/report.json — the append-only contract every instrument writes to and every renderer reads; open BEFORE building a new instrument or a second renderer | diff --git a/bin/gate-check.sh b/bin/gate-check.sh index c1772943..8899bc22 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -1763,8 +1763,8 @@ stage_protocol_paths() { 2) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-brd.md skills/checkpoints/checkpoint-architecture.md skills/image-transcription.md skills/small-project-tier.md skills/kb-generation.md skills/brd-generation.md skills/brd-validation.md" ;; 3) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/learned-mdl-cannot-express.md" ;; 4) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-build.md skills/agent-roles.md skills/small-project-tier.md skills/module-brief.md skills/module-folder-convention.md skills/brd-to-build-plan.md skills/coverage-ledger.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/learned-constants-and-secrets.md" ;; - 5|build-ready) echo "skills/interview-protocol.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/microflow-loop-antipatterns.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; - 6) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 5|build-ready) echo "skills/interview-protocol.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/microflow-loop-antipatterns.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 6) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; 7) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/close-the-loop.md skills/share-demo-package.md skills/handoff-to-studio-pro.md skills/platform-link.md skills/teamserver-alignment.md skills/deploy-to-sandbox.md" ;; *) echo "" ;; # diff --git a/bin/lib/skill-routing.tsv b/bin/lib/skill-routing.tsv index cd670c84..581b5693 100644 --- a/bin/lib/skill-routing.tsv +++ b/bin/lib/skill-routing.tsv @@ -161,6 +161,7 @@ learned-db-assertions skills/learned-db-assertions.md Writing DB assertion tests fixture-seeding skills/fixture-seeding.md Establishing the data and identities a journey run needs — BEFORE it runs. Derive and measure with project-bin/fixture-manifest.sh first; interview only the residue, and never seed from inside the harness test,review 5,6 ondemand verify journey-proof skills/journey-proof.md Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red test,review 5,6 ondemand verify monkey-test skills/monkey-test.md Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works test,review 5,6 ondemand verify +e2e-locators-that-lie skills/e2e-locators-that-lie.md An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures test,review 5,6 ondemand diagnose learned-skill-ux-audit skills/learned-skill-ux-audit.md UX audit and screenshot-loop discipline review 6 ondemand verify learned-skill-scope-delta skills/learned-skill-scope-delta.md Tracking scope delta between the BRD and the built state review 6 ondemand verify company-brain skills/company-brain.md Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ all - ondemand spine diff --git a/skills/e2e-harness-base.md b/skills/e2e-harness-base.md index cbbb3d69..30f39d2b 100644 --- a/skills/e2e-harness-base.md +++ b/skills/e2e-harness-base.md @@ -385,6 +385,7 @@ Write a gap report in `tests/results/YYYY-MM-DD-gap-report.md`: ## Mendix-specific quirks to know - **`.mx-name-*` selectors are stable** — always use these, never rely on position or text +- **Stable is not unique in the DOM** — a tabbed page renders hidden panes at zero height, so `.mx-name-x` can match two nodes with the hidden one first. Use `:visible`, plain click before `force`, and poll text — `e2e-locators-that-lie.md` has the four measured defects - **Widget names come from MDL** — the name you gave a widget in `CREATE PAGE` is the class - **Post-login modal** — Mendix shows a "Welcome" or consent modal after login; always dismiss it - **3.5s wait after login** — less and the modal check may race; more is safe diff --git a/skills/e2e-locators-that-lie.md b/skills/e2e-locators-that-lie.md new file mode 100644 index 00000000..17e60868 --- /dev/null +++ b/skills/e2e-locators-that-lie.md @@ -0,0 +1,121 @@ +# Playwright locators that lie about a Mendix page — four harness defects that accuse the app + +**Applies to:** any mxcli project with a Playwright e2e suite against Mendix pages — hand-written specs first (see "Runner status" below). +**Purpose:** stop an e2e rung from reporting a confident, specific defect **in the app** that is really a defect in the instrument. Three of the four below were false FAILs; one was a false PASS, which is worse. +**Companion skills:** `e2e-harness-base.md` (widget discovery, `.mx-name-*` selectors), `journey-proof.md` (rung 1 landing guard and the UI text rung), `fixture-seeding.md` (where fixture values come from), `tool-output-is-not-ground-truth.md` (verify before you conclude), `unhappy-path-testing.md` (absence assertions on denied affordances). + +Measured 2026-09-17 on a user-group administration module (a tabbed detail page with a members +grid and a per-row role dropdown). Net effect before the four were found: a module with no UI +defect looked broken, and an interaction that did nothing looked green. + +--- + +## 1. A widget name is unique per PAGE, not per rendered DOM + +Studio Pro guarantees `dataGrid21` is unique in the page model. It guarantees nothing about how many +times that page renders it. A tabbed page renders each pane's widgets, and the hidden panes are in +the DOM at zero height. + +``` +.mx-name-dataGrid21 → 2 nodes: [0] hidden, height 0 + [1] visible, height 686 ← the one the user sees +``` + +`.first().isVisible()` therefore answered **false** about a grid that was on screen, and the landing +guard reported "never appeared". Worse, `.first().click()` and `.first()` on a `