fix(objectql)!: a per-aggregation filter refuses a scalar comparison on a declared JSON-stored field, in where's words - #21097
Conversation
… text move to @objectstack/core, byte for byte driver-sql's module-private JSON_COLUMN_INCOMPATIBLE_OPERATORS and the two texts of jsonColumnOperatorError now live in core's json-column-operator-refusal.ts, exported from the root; the driver imports both under the same names and keeps its own error constructor (the #8220 provenance seam). Pins: the set member for member and the texts by SHA-256 against what the driver printed at 8f78495, and the driver's thrown text against the shared one. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…on a declared JSON-stored field, in where's words $in / $nin / $eq / $ne / the orderings / $between and implicit equality on a field the object declares JSON-stored (a structured-JSON type or a multi-valued field) are refused INVALID_FILTER / 400 at assertAggregationFilterIsEvaluable, before any driver is asked, with the withheld text driver-sql's where refuses them in (now core's) and the diagnostic handed to the host log. Before, the in-memory evaluator compared the whole stored array against a scalar: $in counted 0 and $nin counted the rows it was asked to exclude. checkCondition carries the same refusal as the floor for a direct caller. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…re twin's body, on SQLite and live PostgreSQL Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…rals Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…inor, core minor, driver-sql patch) Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…gregation-filter-json-equality
…erasing them Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…gregation-filter-json-equality
📓 Docs Drift CheckThis PR changes 3 package(s): 6 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 1 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 38 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 92026124e73d6122f4ed4385c6f38dce2741b96a && git checkout 92026124e73d6122f4ed4385c6f38dce2741b96a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 8368f1c00551b289b536291e31de297823f64e0b f66bed95067d12b4d5ba627bcb0f7b9740519968 && git checkout -B drift-repro 8368f1c00551b289b536291e31de297823f64e0b && git merge --no-ff f66bed95067d12b4d5ba627bcb0f7b9740519968
node scripts/docs-audit/affected-docs.mjs --json 8368f1c00551b289b536291e31de297823f64e0b
|
Contract reviewServed-tier: Inputs, and nothing else: card #21007 (body and comments 5923286198, 5924145107, 5924194074, 5924484320, 5924546829, 5925997383); PR #21097 (body, the 10-file list, the net diff against ① Derived judgments
② Semver level
③ Boundary flagsDev flags (report 5925997383 and the PR body):
open_questions:
Check-runs on the head, read once:
No governed-surface path is in the file list. The PR is a draft. This at-tier record is the one the BREAKING changeset owes (claim 5924194074). Implemented-by: VERDICT: PASS |
…gregation-filter-json-equality
…n each filter before any row, as engine.aggregate does Given a field map, the published applyInMemoryAggregation reached the per-row backstop without the one-time gate, so an empty row set and a row the lowered filter's $null arm decided first answered 200 where engine.aggregate refuses 400. It now calls assertAggregationFilterSparesJsonStoredFields (exported from having-filter.ts, the same function) once per aggregations[i].filter, and takes an optional reportWithheld for the withheld diagnostic. The floor's docblock now says what it is: a backstop for a row that reaches the arm. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…ggregation's direct callers Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…gregation-filter-json-equality
Contract reviewServed-tier: Round-2 delta review on the at-tier PASS 5926186339 at ① Derived judgments
② Semver level
③ Boundary flagsRound-2 report 5927016404: Round-1 flags, carried forward:
New this round:
No governed-surface path is in the file list ( Implemented-by: VERDICT: PASS Generated by Claude Code |
Fixes #21007
Clause-②: yes (widening)
A per-aggregation
filternow refuses a scalar comparison on a declared JSON-stored field ($eq,$ne,$gt,$gte,$lt,$lte,$between,$in,$nin, implicit equality) withINVALID_FILTER/ 400, in the wordswhererefuses the same filter in. It no longer counts rows the stored arrays cannot support. The operator set and the refusal text move fromdriver-sqlto@objectstack/core, byte for byte, so both faces read one set and one sentence.Clause-② has two halves. It is
yes (widening)because@objectstack/core's root gains three exports (JSON_COLUMN_INCOMPATIBLE_OPERATORS,jsonColumnOperatorRefusalTextand its return typeJsonColumnOperatorRefusalText), andapplyInMemoryAggregationgains an optional trailingreportWithheldparameter. It also narrows:@objectstack/objectqlrefuses queries it used to answer 200, atengine.aggregateand at the publishedapplyInMemoryAggregationgiven a field map. The changeset therefore carriesminorfor objectql with a BREAKING banner and one ADR-0087 marker,minorfor core, andpatchfor driver-sql, whose output is unchanged. The seat answer on the card (## Seat answer — #21007, comment 5924546829) amended the claim to this surface and thisClause-②.What was wrong (measured before this change,
d1f8ce865)POST /api/v1/data/:object/queryon SQLite and a live PostgreSQL 16.14, over the card's six rows (ownersis amultiple: truelookup, andd1andd3holdu1). Both dialects answered identically:wheretwinmbeforeowners $in ['u1','u9'](the card)INVALID_FILTERowners $nin ['u1','u9'](the card)d1andd3countedowners $eq 'u1'/{ owners: 'u1' }owners $ne/$gt/$lte/$betweentags $eq 'red'['red']loosely=='red')meta(json)$eq/$inowners $contains 'u1'(the prescribed spelling)title $in/$nin/$eq(controls)What changed
@objectstack/core: a newsrc/utils/json-column-operator-refusal.ts, exported from the root besidetemporal-storage-form.js. Theexport *publishes three names:JSON_COLUMN_INCOMPATIBLE_OPERATORS(driver-sql's 22 spellings, member for member),jsonColumnOperatorRefusalText(field, op, bare), and its return typeJsonColumnOperatorRefusalText({ message, diagnostic }). These are the two stringsjsonColumnOperatorErrorbuilt, and nothing else. Each face keeps its own error constructor: driver-sql keeps its [A of #7929] a spec-declared provenance mark set at both read-scope merge boundaries, so the driver can restore the author-facing cross-field diagnostic without re-disclosing policy #8220 provenance seam, and objectql keeps its ADR-0112 envelope.driver-sql(sql-driver.ts): the module-private set and the two template strings are gone.jsonColumnOperatorErrorkeeps its name and signature, and now calls the core builder (hunk at:3298). One import line (with its comment) sits at:153–:156, after the top import block. It is outside the declared:3376–:3460region on purpose, so as not to touch the@objectstack/coreimport block that fix(plugin-security,driver-sql,driver-turso): lower type-blind at the RLS seam without a guard, then delete the F1/F2 whole-day and NOT-rewrite copies (#5930 step 4, group 2) #20988 and [finding]$contains/$notContainson a declared multi-valued or JSON-stored field still answer SUBSTRING on five faces, the analytics RLS read scope among them (u1admits a row storingu10) #20987 edit.assertOperatorAppliesToColumn(in fix(plugin-security,driver-sql,driver-turso): lower type-blind at the RLS seam without a guard, then delete the F1/F2 whole-day and NOT-rewrite copies (#5930 step 4, group 2) #20988's former region) is untouched: it reads the imported set under the same name.objectql(having-filter.ts):assertAggregationFilterIsEvaluablegainsassertAggregationFilterSparesJsonStoredFields. It runs once on the filter after the reference rule, againstdeclaredJsonStoredFields(declared.fields), and before any driver is asked for a row, so an empty table refuses too (objectql: a per-aggregationfilterrefuses an unknown operator only when rows exist —aggregations: [{ filter: { amount: { $median: 1 } } }]answers 400 on a populated table and 200 on an empty one #20122's rule). It walks$and/$or/$notand refuses implicit equality (reported as=, bare, as driver-sql does) and every operator in the shared set, whatever the comparand (nulland[]included). The withheld message is thrown, and the diagnostic, with the aggregation position, goes toreportWithheld, as objectql + REST: the per-aggregationfilterstill lacks four ofwhere's doors — a bad date, anaddDaysnumeric pair, an undeclared{ $field }and an unknown key answer200with every count 0 #20148 does.$contains/$notContains/$exists/$null/$emptykeep answering.checkConditioncarries the same refusal above its no-value exit, but only as a backstop for a row that reaches the arm. Its reach is the row's: an empty row set never gets there, and spec lowering rule 3 puts a$nullarm ahead of every negation that the walker's$orshort-circuit takes first. The gate is the complete door, and the docblocks now say so (round 2, from the review's ①.5).objectql(in-memory-aggregation.ts, round 2, the review's F10 (b)): the publishedapplyInMemoryAggregation(rows, ast, timezone, fields, reportWithheld?)now calls the sameassertAggregationFilterSparesJsonStoredFields(exported fromhaving-filter.ts, not from the package root) once peraggregations[i].filterwhen it is handedfields, before any row is judged. Before this, a direct caller reached only the backstop and got a row-dependent answer. This entry point holds no logger, so the closest seam is a new optional trailingreportWithheld(diagnostic), which receives the field, operator and position; without it the diagnostic is dropped and the 400 is unchanged.engine.aggregatepasses none, because it has already judged and logged the same filter. The gate's declaration parameter is narrowed to just thefieldsandreportWithheldmembers ofAggregationFilterDeclaration, sinceobjectis the reference rule's.objectql(engine.ts): one comment block and thereportWithheldlog line atassertAggregationFilterIsEvaluable's call site. The log line now reads "as it is for the same refusal in a where" instead of "…cross-field comparison…", since it carries two refusals now.Measured findings behind the shape (H1–H5)
SET_MEMBER_DESCRIPTION, the$in/$ninentries ofFILTER_OPERATORSand the$containsdocblock give no per-element reading. driver-sql'swhererefuses (it does not answer membership), so triage's "membership, as driver-sql does" misread it.in,=,nin) are refused earlier at both positions by the nested-relation door, so the evaluator only meets the$forms.@objectstack/core.aggregations[i].filter: every driver's aggregate face refuses a per-aggregation filter 501, and the analytics ObjectQL strategy hands measure filters toengine.aggregate.having. After fix(objectql)!: engine aggregate asks the field-type table for every row — min / max / avg over a refused type answer INVALID_FIELD / 400 on every driver #21037 (landed, merged here),min/maxover a multi-valued field is refusedINVALID_FIELDat the aggregate door. Measured on InMemoryDriver at the merged head:max(owners)with or withouthavinggives 400INVALID_FIELD. Sohavingcannot meet a JSON-stored column, and it is left alone.Tests
Round 1 numbers were read at
6e541b101; round 2 numbers are marked with the headf66bed950(main merged).@objectstack/corejson-column-operator-refusal.test.ts: 6 passed. It pins the set member for member, and the message and three diagnostics by SHA-256 and length against what driver-sql printed at8f784959c. Hashes avoid a third literal copy of the sentence.driver-sqlsql-driver-json-column-refusal-shared-text.test.ts, run beside the existing JSON-column, compile-refusal-seam and provenance suites: 248 passed. It checks everyFILTER_OPERATORSmember against the shared set: the driver's thrown message and withheld diagnostic equal the core builder's output.SqlDriverover SQLite) covered all 22 spellings plus bare equality, unmarked and author-marked, message and diagnostic, 46 entries. Before (8f784959c) and after:cmpidentical, sha256dbcf32f5…534b2on both. driver-sql'sdistno longer contains the sentence.objectqlengine-aggregate-filter-json-column-refusal.test.ts(engine-level cell over thefind()read shape): 74 passed. It covers 18 family cases on each ofowners,tagsandmeta(code,status, the$contains/$orprescription, the field absent from the message, field and operator in the logged diagnostic, and the driver never asked for a row), an empty table (pure and grouped), the logged position, 11 answered cases (membership, null predicates,titlecontrols) and the per-row floor.restaggregation-filter-json-column-refusal.test.ts: 52 per cell. SQLite passes and a live PostgreSQL 16.14 passes locally; MySQL is a named skip. Every family case asserts that the per-aggregation 400 body'serroris the same string as itswheretwin's. fix(objectql): a per-aggregation filter counts $contains on a multi-valued field by membership, as its where twin does #21004'saggregation-filter-array-membership.test.tsstill passes beside it.1a226419e: objectql local 6948 passed, rest local 5072 passed / 247 skipped, core 1809 passed. Read before the first merge: driver-sql 3285 passed / 188 skipped.typecheckpassed for core, driver-sql, objectql and rest. At head6e541b101: core, driver-sql (refusal suites), objectqlengine-aggregate*(571 passed) and restaggregation-filter*(150 passed with PostgreSQL) re-ran green.ablation-replace, plus a rebuilt objectqldist, plusablation-dist-preflight --absent):owners/tagscases still got a 400 from the per-row floor, without the logged diagnostic.metanegations ($ne,$nin,$nin [],$not $in, wheremetais null on every row) answered 200{ n: 6, m: 6 }; the mechanism was not traced. The empty table answered 200.git diff HEADempty, objectql rebuilt, preflight shows the marker present in 4 dist files with a clean tree, and both suites green again (74 and 104).engine-aggregate-filter-json-column-refusal.test.ts, 92 passed). Formeta(json)$ne,$ninand$not $in, each on four cells: an empty row set, an empty grouped row set,metanull in every row with the filter as speclowerFilterConditionlowers it (the shape that carries the$nullarm), and the same rows with the filter as written. Each must refuse 400INVALID_FILTERwith exactlyengine.aggregate's message, and hand the diagnostic (field, operator,At aggregations[1].filter.…) toreportWithheld. Also pinned: no reporter means the same refusal; no field map means nothing judged (m: 0, as before); and$containsstill answers.applyInMemoryAggregationcall deleted (ablation-replace, anchor 1 to 0, blobc65412761a90tof530761c0559): 13 of 92 red.c65412761a90,git diff HEADempty, 92 passed again.f66bed950: objectql local full suite 7008 passed (356 files), restaggregation-filter*150 passed / 61 skipped with a live PostgreSQL 16.14, core refusal pin 6 passed, driver-sql refusal pins 141 passed.Gates
node scripts/pm/dispatch-gates.mjs --commands(no paths) derived 70 families at6e541b101.check:adr-0087-registration,check:changeset-no-major,check:engine-double-contract,check:nul-bytes,check:doc-authoring,check:driver-conformance,check:driver-memory-census,check:query-options-erasureandcheck:test-source-alias.check:dual-build-cjs-loadsandcheck:type-check-debt. Both need the whole workspace built; two attempts at that build timed out in the shared verify-lock queue. CI's lint job builds first.--ranreconciliation: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun.f66bed950: re-derived with no paths, the same 70 families; 68 ran with exit 0 and the same 2 were NOT MEASURED (exit 3).--ran: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun.typecheckpassed for core and objectql. Narrowed lint: 10 changed.tsfiles, 0 errors, 0 warnings.e7bd7f667("type the shared-text pin's find options"):check:query-options-erasure's test surface grew 236 to 237 because of anas anyon afindoptions bag in the new driver-sql test.eslint --no-inline-config --format jsonover the 9 changed.tsfiles reports 9 files, 0 errors, 0 warnings.eslint.config.mjssets noparserOptions.projectand registers no typed rule, so linting is not type-aware and this diff cannot move a verdict on an untouched file. The fullpnpm lintis CI's.Acceptance notes
Round 2, from the at-tier review (5926186339).
applyInMemoryAggregationis gated (above).applyInMemoryAggregationdirect callers and the new optionalreportWithheld.The "flip the
m: 0/m: 6pins" step had nothing to flip. No suite onmainpinned a per-aggregation$in/$nincount on a JSON-stored field (fix(objectql): a per-aggregation filter counts $contains on a multi-valued field by membership, as its where twin does #21004's two suites pin only$contains/$notContains). The full objectql, rest and driver-sql runs found no other pin that this change turns. The refusal pins are new files beside fix(objectql): a per-aggregation filter counts $contains on a multi-valued field by membership, as its where twin does #21004's.A
{ $field }comparand on a JSON-stored field ({ owners: { $eq: { $field: 'title' } } }) is refused by this gate in the JSON-column words. driver-sql'swhererefuses it through its cross-field class rule, in that rule's words. Both answers areINVALID_FILTER/ 400 with the field withheld, so the two faces disagree only on which sentence they print.The REST envelope truncates the shared message at 500 characters on both faces, so it ends "…because the answ…". That is unchanged here by direction, and filed separately by the seat.
Findings for the seat, not filed here:
whereanswers the family per element on a multi-valued field, while the SQL family refuses it (engine-level measurement). The seat files it.$contains/$notContainson a declared multi-valued or JSON-stored field still answer SUBSTRING on five faces, the analytics RLS read scope among them (u1admits a row storingu10) #20987.Generated by Claude Code