fix(objectql)!: a per-aggregation filter and having refuse a non-boolean $exists / $null with INVALID_FILTER / 400, in the drivers' words (#20981)

[objectstack-fleet]

Fixes #20981
Clause-②: no (narrowing)

The engine's in-process evaluator for a per-aggregation filter and a having now refuses a non-boolean $exists / $null with INVALID_FILTER / 400, in the words every driver's where refuses it in. It is judged once, before any driver is asked for a row, and per row as the floor, beside the existing $empty gate. true / false answer exactly as before.

What was wrong

Measured on origin/main 7a606a9a3 through engine.aggregate on driver-memory AND driver-sql (better-sqlite3). The object has a text field name: row a holds "won" (amount 10), b holds null (amount 1), and c has no value (amount 100). The two drivers answered identically, because the engine evaluates both clauses itself:

clause · comparand	$exists	$null
filter · "yes" / 1 / "false"	sum 10 (the valued row)	sum 111 (every row)
filter · 0 / null	sum 101 (the no-value rows)	sum 111 (every row)
having · "yes" / 1 / "false"	the won group	both groups
having · 0 / null	the null group	both groups
filter / having · true	10 / won	101 / null group
filter / having · false	101 / null group	10 / won

checkCondition read $exists as !!target, and its $null arm tested only === true / === false, so any other value constrained nothing. After this PR, every non-boolean cell above is INVALID_FILTER / 400 on both drivers, and the true / false rows are unchanged (re-measured with the same scratch harness at the head).

Reach. POST /api/v1/data/:object/query never reached the defect. The route parses the request against the spec's query schema first, and that parse already refuses a non-boolean flag in where, in aggregations[i].filter and in having with 400 VALIDATION_FAILED. This was measured on the same base and is unchanged here. The engine refusal is the floor for every caller that reaches engine.aggregate without that parse: server-side code, a flow or hook, the analytics bridge that lowers a measure's filter into an aggregation filter, and a host calling applyInMemoryAggregation directly.

The change (packages/objectql/src/having-filter.ts)

• nonBooleanFlagComparandError(op, field, value, path) sits beside emptyFlagComparandError. assertConditionIsEvaluable calls it next to the $empty gate, so assertHavingIsEvaluable and assertAggregationFilterIsEvaluable (the gates engine.aggregate calls before any driver read) refuse at every depth, a branch a $or would short-circuit included. checkCondition raises it above the no-value exit as the per-row floor, as $empty does.
• The two arms now read the boolean itself: $exists is "has a value" and $null is its mirror. No truthiness reading and no drop.
• engine.ts and in-memory-aggregation.ts are not touched: both already route through the walker.
• Precedence: a { $field } in a flag's slot is now refused as a non-boolean first, as $empty's gate already does, rather than by the reference-position refusal. The code and status are the same (INVALID_FILTER / 400), and the two existing reference rows that assert only the path stay green, with a note added. A plain object or undefined is still refused first by the comparand-type door, in its own words. An array in the flag's slot reached the old !!target read and is now refused.

The words, and where they live

The text is driver-sql's own nonBooleanExistsComparandError / nonBooleanNullComparandError diagnostic, which is also the text driver-memory and driver-mongodb put on the wire. Its "this driver" clause is re-aimed at driver-sql, as driver-memory's copy re-aims it. No new wording.

The text has no importable home. Measured: @objectstack/core exports nothing for it. The spec's save door spells its own message (nonBooleanFlagComparandMessage, not exported, with different words). Every face spells its own copy: driver-sql, driver-memory, driver-mongodb, Turso's remote transport, service-analytics twice, and the spec save door. This package cannot depend on a driver. So this is a declared verbatim copy, not a silent one. The drivers' describeFilterOperand / safeShapePreview rendering is copied too (describeFlagOperand), and the copy is held to its source by packages/rest's new cell. That cell reads the where twin's withheld diagnostic off the error driver-sql throws (withheldFilterDiagnosticOf) and requires the engine's message to equal it under exactly two edits: the location re-rooted at the clause, and "this driver" re-aimed at driver-sql. A change to either side breaks that equality. Whether the sentence should get a shared @objectstack/core home (the shape #21007 gave its JSON-column sentence) is raised to the PM as an open question in the report on #20981, and is not decided here.

Disclosure: driver-sql withholds the field and the value from a where refusal, because a where can carry a merged read scope. A per-aggregation filter and a having never do, so this message names both, as this face's $empty and $icontains refusals already do.

Pins

• packages/objectql/src/engine-aggregate-flag-comparand-refusal.test.ts (new, 39 tests). Through engine.aggregate, over a driver of each having path's shape: rows, with no aggregate(), the face that evaluates aggregations[i].filter; and native, where the driver aggregates and the engine applies having. Each of "yes", 1, "false", 0, null (plus a list) under both flags is refused in the filter and in having, on an empty and a populated table, with no driver read. Each case asserts code INVALID_FILTER, status 400, the drivers' first three sentences (operator, field, the rendered comparand, the position, the declaration) and the reason. Four more positions are covered: $and, a $or branch after one that holds, $not, and beside a boolean sibling flag. A { $field } in the slot is refused as a non-boolean. The true / false control gives the sums and groups in the table above, on both paths. The published applyInMemoryAggregation(rows, ast, tz, fields), plus applyHaving and matchesAggregationFilter, refuse a row that reaches the flag, with and without a value.
• packages/rest/src/aggregation-flag-comparand-refusal.test.ts (new, 15 tests). This is the real SqlDriver (SQLite) cell, the declared domain:cli test-only touch. The refusal cases run on engine.aggregate, beside the where twin's diagnostic (the word parity above). One case records that the route refuses all three positions in its own parse first: 400 VALIDATION_FAILED, asserted by status, code and field path only. The true / false control runs through POST /api/v1/data/:object/query, and the filter's sum and the kept group equal the where twin's.
• driver-memory: this package does not depend on it, and the in-memory driver's test consumers are a closed census (check:driver-memory-census). Its cell is therefore the engine-level rows shape above, as [finding] a per-aggregation filter with $contains on a multiple lookup counts 0 on every driver while the same where finds the rows: the engine's aggregation evaluator never matches a stored array #20873 and [finding] a per-aggregation filter $nin on a multi-valued field counts the rows it was asked to exclude, and $in counts none, where the same where is refused 400: the aggregation evaluator has no JSON-column equality gate #21007 did, and the measurement table above was taken on the real InMemoryDriver.

Pin sweep. I ran a repo-wide grep for a non-boolean $exists / $null literal in tests and for both refusal sentences. Every other hit pins a driver's, the analytics door's or the save door's own face, and none asserts the engine's old answer. Two REFERENCE_REFUSED rows (engine-aggregate-filter.test.ts, engine-aggregate-having-comparand-shape.test.ts) now meet the flag gate first. They assert the path only and stay green, and each has a comment pointing at the new file. rest-4xx-message-truncation.test.ts carries a copy of driver-sql's $null text for a truncation fixture. It is not a pin on this face and is not touched.

Verification (final head 704a58ab2, after merging origin/main twice)

• pnpm --filter @objectstack/objectql test: 358 files, 7051 tests passed (post-merge, a876bc2af; the second merge brought no objectql, rest, core, formula, driver or spec data/ change). pnpm --filter @objectstack/objectql typecheck: exit 0. The new test file is in tsconfig.test.json's program (--listFiles: 1 hit; 0 in tsconfig.json's, which excludes tests), and check:test-typecheck is OK.
• At 704a58ab2: the four objectql files around this change (302 tests) and the rest cell (15 tests) passed, after rebuilding the closure. packages/rest: tsc --noEmit and check:test-typecheck OK. The 12 aggregation and flag-related rest files (aggregation-*, rest-aggregate-*, data-query-having-temporal-door, and the three other files naming a flag operator) passed at faafb51d3, before the merges: 200 passed, 158 skipped (the env-gated PostgreSQL / MySQL cells). The whole rest suite is declared to CI.
• Reverse verification, from the committed fix, through scripts/ablation-replace.mjs. Each mutation was proven on disk (anchor count and blob change), and each restore was proven by blob equal to HEAD with git diff HEAD empty:
  • leg A, both gates neutralised: objectql file 31 failed / 8 passed (the 8 controls stay green);
  • leg B, the one-time gate only: 29 failed / 10 passed (the engine cases go red; the per-row and control cases stay green);
  • leg C, the per-row floor only: 2 failed / 37 passed (only the published row-evaluator cases go red);
  • leg R, for the rest cell, which reads objectql through dist/: a string marker was planted, objectql rebuilt, and ablation-dist-preflight found the marker in 4 built files. Result: 10 failed / 5 passed (the 10 refusal cases). The leg was then restored and rebuilt; preflight --absent reported the marker absent from all 14 built files and a clean tree, and the cell passed 15/15.
• Driver conformance ledger (node scripts/check-driver-conformance.mjs): 50 covered, 0 DEBT, 0 exempt, before and after.
• Gates: node scripts/pm/dispatch-gates.mjs --commands after the final commit derived 63 families from the 6 changed paths. --ran reconciles them as 63 accounted for: 62 run and exit 0, 1 NOT MEASURED. The NOT MEASURED one is check:dual-build-cjs-loads (exit 3, prerequisite: it reads every package's dist/, and this tree builds only the closures). check:dts-closure first read red on two partially built packages, organizations and plugin-webhooks. Their partial dist/ files were written inside the two windows in which my own 560-second timeout killed check:type-check-debt. Both were rebuilt, and the gate then swept 61 built packages with 153/153 declaration files present. check:type-check-debt then ran to completion: exit 0, in 232 s.
• Lint, narrowed: eslint --no-inline-config --format json over the five touched TypeScript files reported 5 files, 0 errors and 0 warnings. These files are in the config's own population (files: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}'] minus NEVER_LINTED and packages/spec/**). The config never enables type-aware linting (no parserOptions.project), so this diff cannot move any untouched file's verdict. The full pnpm lint is CI's.

Acceptance notes

• The spec's three sentences now read true for every query face. These are filter.zod.ts ("Every query face refuses a non-boolean $null / $exists"), the save-door refusal text in filter-save-door-refusals.ts, and migration entry 18.filter-query-face-comparands-refused-at-save. No spec edit. @objectstack/formula's matchesFilterCondition, the RLS write-check evaluator, still answers a non-boolean flag (v === true ? … : …). By the repo's own vocabulary it is not a query face: its $empty docblock contrasts itself with "the spec's save door and every query face". Observation only, not filed: no public door was measured for it.
• Changeset: @objectstack/objectql minor, BREAKING, with the ADR-0087 disposition not-required (already-registered filter-query-face-comparands-refused-at-save). That registered entry's surface names a query having, the aggregate call's having and an aggregation filter, and its replacement is this change's whole migration. fix(driver-memory,driver-mongodb): refuse a non-boolean $exists comparand with INVALID_FILTER / 400, as $null's is refused (#20897) #20979 used the same disposition for the drivers' half.
• Tooling, observed while verifying. The turbo bump to 2.11.5 that origin/main brought in this window writes a managed "agent guidance" block into the root AGENTS.md before repo-scoped commands when it detects an agent. That happened here on every turbo run and on check:type-check-debt. It dirties every agent worktree (+11 lines), reddens check:pm-skill-ratchet locally (1119 lines against a ceiling of 1116), and puts a governed-surface edit one git add -A away. Here it was restored from HEAD each time and is not in this diff. Raised in the report on [finding] the aggregation filter and having read a non-boolean $exists by truthiness and DROP a non-boolean $null, on every driver: the engine evaluates both in-process, and its gate refuses only $empty #20981 for the seat to route.
• Not touched: engine.ts, in-memory-aggregation.ts, every driver, the spec and formula.

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

4 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 70dae533c58a60a7009d253cf73f87c3a5848d95 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 3a129916f67e4877849e656a043540b75b65c039 — the merge of head 704a58ab232e0c7afe425b5df41518cd4151e3a5 into base 70dae533c58a60a7009d253cf73f87c3a5848d95, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 3a129916f67e4877849e656a043540b75b65c039 && git checkout 3a129916f67e4877849e656a043540b75b65c039
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 70dae533c58a60a7009d253cf73f87c3a5848d95 704a58ab232e0c7afe425b5df41518cd4151e3a5 && git checkout -B drift-repro 70dae533c58a60a7009d253cf73f87c3a5848d95 && git merge --no-ff 704a58ab232e0c7afe425b5df41518cd4151e3a5

node scripts/docs-audit/affected-docs.mjs --json 70dae533c58a60a7009d253cf73f87c3a5848d95

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

[objectstack-fleet]

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 704a58ab232e0c7afe425b5df41518cd4151e3a5
Local-runs: none

① Derived judgments

Inputs: card #20981 (body; comments 5923295010, 5926372178, 5927784503, 5929772241, 5929818992), PR #21157 (body, the 6-file list, the net diff from merge-base e952cff57 of origin/main), and the 31 check-runs on the head. Read-only throughout: git fetch, diff, show and grep against the review ref, plus REST reads; nothing built, run or re-run.

1. One-time gate, both positions, every depth — right. assertConditionIsEvaluable throws nonBooleanFlagComparandError for $exists / $null when typeof target is not boolean, placed after the $empty gate and before the vocabulary check and the reference-position check. It is reached only through assertNodeIsEvaluable, which recurses every $and / $or branch and $not, so a branch the per-row $or would short-circuit past is judged. assertHavingIsEvaluable and assertAggregationFilterIsEvaluable both take that walk; engine.aggregate (untouched) calls the second in its per-aggregation loop, unconditionally, before getDriver, and the first once at the having seam before either applyHaving door is chosen. The refusal therefore precedes any driver read on the native and the rows path alike, and an empty table refuses too.

2. Per-row floor and the two arms — right. In checkCondition the same gate sits beside $empty's, above the no-value exit (both flags are in NO_VALUE_ANSWERED_BY_OPERATOR, so an absent column still reaches the arm). The arms now compare the boolean itself: $exists keeps a row iff has-a-value equals target; $null keeps it iff has-no-value equals target. No !!target, and no === true / === false pair that let a third value through. true / false keep their prior semantics.

3. The published applyInMemoryAggregation(rows, ast, tz, fields) — right, and named. It runs no one-time flag walk (it never did for $empty; the only filter-level door it runs is [finding] a per-aggregation filter $nin on a multi-valued field counts the rows it was asked to exclude, and $in counts none, where the same where is refused 400: the aggregation evaluator has no JSON-column equality gate #21007's JSON-column one), so for these flags it refuses per row through matchesAggregationFilter and checkCondition: a row that reaches the flag is refused, with or without fields; an empty rows array, or a row a $or branch settles first, is not judged there. That is the $empty precedent on main, consistent, and the changeset's BREAKING paragraph states exactly this reach. One phrase in the PR body ("both already route through the walker") is loose for this file — it routes through the per-row walk, not the one-time walker — but the changeset and the report are precise, and the new objectql file pins applyInMemoryAggregation with fields, applyHaving and matchesAggregationFilter on this.

4. The declared verbatim copy — byte-equal, read side by side. driver-sql's nonBooleanExistsComparandError / nonBooleanNullComparandError (sql-driver.ts) and the engine's nonBooleanFlagComparandError differ only where driver-sql writes "this driver" / "this driver's" and the engine writes "driver-sql" / "driver-sql's" (one occurrence per sentence), and in the location path. describeFlagOperand reproduces describeFilterOperand (null, array, typeof, constructor name unless Object) and safeShapePreview (JSON.stringify, typeof fallback, the 80-character cut at 77 plus "...") in the drivers' kind (preview) form. The copy is declared in its docblock, as the seat's amended fence requires.

5. The equality pin can fail. The rest cell reads withheldFilterDiagnosticOf(err) — the diagnostic driver-sql stores verbatim under its symbol, exported from @objectstack/driver-sql, a workspace dependency of packages/rest — asserts the twin's path is filter.name.OP, then requires toBe equality after exactly two edits (re-root the path; replaceAll "this driver" with "driver-sql"). A byte moved on either side, a twin not refused INVALID_FILTER, or a diagnostic no longer withheld each break it.

6. Precedence — right. A { $field } under $exists / $null now meets the flag gate before fieldReferencePositionError; same code and status. The two pre-existing REFERENCE_REFUSED rows (engine-aggregate-filter.test.ts, engine-aggregate-having-comparand-shape.test.ts) assert toContain(path) only, so they stay green; the diff adds a pointer comment to each. An array in the slot, formerly answered by !!target, is now refused; a plain object or undefined keeps the comparand-type door's earlier words, as the PR says.

7. Public surface. No export, signature or published type moves; index.ts, engine.ts, in-memory-aggregation.ts, every driver, the spec and formula are outside the diff; emptyFlagComparandError is untouched. The rest test file is the claim's declared domain:cli test-only touch.

8. The spec's three "every query face" sentences — true without a spec edit. formula's matchesFilterCondition keeps its v === true reading, and its own docblock (matches-filter.ts, the $empty arm) contrasts itself with "the spec's save door and every query face", so after this fix the sentences hold for the query faces. The card offered "name it, or narrow the sentence"; the PR names it in its acceptance notes.

② Semver level

@objectstack/objectql minor, with the BREAKING banner, the what-to-write-instead prose, and Clause-②: no (narrowing) in both the PR body and the changeset — the line the claim declared. Right: nothing new is accepted; shapes answered before are refused INVALID_FILTER / 400 now, and true / false answer as before. A narrowing is BREAKING and ships minor in the launch window (check-changeset-no-major.mjs; ADR-0087's level amendment). ADR-0087 marker: not-required (already-registered filter-query-face-comparands-refused-at-save). The entry pre-dates the merge base (present and registered at e952cff57); its surface names "the data-engine aggregate call's having, an aggregation filter", its reason states that every query face refuses a non-boolean $null / $exists, and its replacement is the boolean spelling this change prescribes — so it covers this migration rather than merely resolving. Only objectql publishes a change; the rest file is test-only. Check Changeset on the head: success.

③ Boundary flags

• open_questions[0] (where the sentence lives): answered in-seat, C (5929818992), with the claim's fence amended to one declared, equality-pinned copy. The diff is judged against that answer and matches it (①4, ①5). Not re-answered here.
• Breach note (the claim said "REST cell"): measured and right. FindDataRequestSchema.query is QueryWithTransportSchema, whose having and aggregations[i].filter are FilterConditionSchema with the spec: FilterConditionSchema (the save-time door) admits a non-boolean $null / $exists flag, so a stored dataset or widget filter carrying one saves clean and is refused 400 on every query face #20116 superRefine, so the route refuses all three positions 400 VALIDATION_FAILED before engine.aggregate is reached; the rest cell pins that in one case and drives the refusal cases through engine.aggregate over a real SqlDriver on SQLite — the pin the claim actually named ("through engine.aggregate ... on driver-sql"). The true / false control runs through the route.
• driver-memory pin: substituted by the engine-level rows-shape recorder, with the measurement table taken on the real InMemoryDriver (scratch, not committed), because driver-memory's test consumers are a closed census (check:driver-memory-census, root package.json) — the [finding] a per-aggregation filter with $contains on a multiple lookup counts 0 on every driver while the same where finds the rows: the engine's aggregation evaluator never matches a stored array #20873 / [finding] a per-aggregation filter $nin on a multi-valued field counts the rows it was asked to exclude, and $in counts none, where the same where is refused 400: the aggregation evaluator has no JSON-column equality gate #21007 precedent. Named in the report and in the file header; the refusal precedes every driver read, so the backend does not enter the verdict. Accepted as declared.
• out_of_scope_findings[0] (turbo 2.11.5 writes an agent-guidance block into AGENTS.md): not in this diff — AGENTS.md is absent from the file list. Escalated to the seat to route; nothing for this record to judge.
• out_of_scope_findings[1] (formula still reads a non-boolean flag): observation, verified (①8); no public door measured, not filed, not this card's.
• out_of_scope_findings[2] (this face's $empty wording differs from the drivers'): observation; emptyFlagComparandError is untouched here; the seat may carry it into the acceptance notes.
• Check-runs on the head. Completed, success: Auto Label; Check Changeset; Check PR Size; Check Documentation Links; Flag docs affected by code changes; Type Check · source gates; Type Check · debt ledger; Governed Surface Queue Guard; Dogfood Verify CLI; Dogfood Regression Gate (1/3); filter; and the four claim guards (card claims this branch, single-writer path, part-of, single-issue claim); the Vercel status is success. Skipped: Build Docs; Console Pin Gate; Packed-tarball smoke (opt-in). Still in_progress, so NOT verdicts here: Build Core; Test Core (1/6), (2/6), (3/6), (4/6), (5/6), (6/6); Type Check · workspace; Type Check · consumer gates; Lint & Repo Gates; Dogfood Regression Gate (2/3) and (3/3); Temporal Conformance (live PG + MySQL). The 39 objectql cases and the 15-case rest cell (the equality pin included) conclude inside Test Core; this record does not wait on them, and the queue lands only on their own conclusions.

Implemented-by: claude/issue-20981-agg-flag-comparands
Reviewed-by: session_01Ujdtvqs7ree7WyQmEDwEnG

VERDICT: PASS

---

Generated by Claude Code