fix(objectql): a per-aggregation filter counts $contains on a multi-valued field by membership, as its where twin does - #21004
Conversation
…ed multi-valued field by membership The engine's aggregation evaluator failed `$contains` on every value that was not a string, so a stored array never matched and `$notContains` matched every row. On a declared JSON-stored field (STRUCTURED_JSON_TYPES or isMultiValueField) both now ask membership, the reading `where` gives on every SQL dialect; a scalar string column keeps the substring test. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…lti-valued field beside its where twin Engine-level over the read shape find() presents on memory, SQLite and PostgreSQL; REST door on a real SqlDriver, SQLite always and the live PostgreSQL / MySQL cells where their URL is set, each row beside its where twin. having keeps the substring reading on a text projection. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…ship Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…mbership Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check8 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9b6a0192b0271d3310efbabdeca28e2801694fd8 && git checkout 9b6a0192b0271d3310efbabdeca28e2801694fd8
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a5bce408883b81a6e2382ebe709a03cc3a5b40b4 90ba78d9b150faf40e31f68841e9ac94aa1e13a1 && git checkout -B drift-repro a5bce408883b81a6e2382ebe709a03cc3a5b40b4 && git merge --no-ff 90ba78d9b150faf40e31f68841e9ac94aa1e13a1
node scripts/docs-audit/affected-docs.mjs --json a5bce408883b81a6e2382ebe709a03cc3a5b40b4 |
Contract reviewServed-tier: Inputs: card #20873 (body and all five comments: triage 5914962540, the blocked transition 5921472356, claim 5921990075, report 5922812043, claim amendment 5922857157); PR #21004 body, its five-file list, its four commits and the net diff against the merge base with ① Derived judgments
② Semver levelClause-②: no
③ Boundary flags
Implemented-by: VERDICT: PASS |
Fixes #20873
Clause-②: no
What changes
The per-aggregation
filter(engine.aggregate({ aggregations: [{ …, filter }] }), and soPOST /api/v1/data/:object/query) is evaluated by the engine's own walker,matchesAggregationFilterinpackages/objectql/src/having-filter.ts. Its$containsarm failed every value that was not a string, so a stored array never matched, and its$notContainsarm passed every such value, members included.On a DECLARED JSON-stored field both arms now ask MEMBERSHIP, the reading
FILTER_OPERATORS'$containsdocblock (@objectstack/spec) declares andwherealready gives on every SQL dialect (SqlDriver.applyJsonMembership):$contains: vholds whenvnames an element of the stored array. A member stored as a JSON number or boolean is named by its text ('1'names1,'1.50'names1.5,'true'namestrue,'null'namesnull). That is the candidate setdriver-sql'sjsonMembershipCandidatesbinds on every dialect. Array-only, as the SQL constructs are.$notContains: vis its exact complement, and a row with no value still satisfies it (非否定路径上的$ne/$nin/$notContains:driver-sql 排除 NULL 行,driver-memory / formula 返回它们(#5146 只裁定了$not) #5298), ascol IS NULL OR NOT (…)does in SQL.having.Files:
having-filter.ts(the two arms,storedArrayHasMember,declaredJsonStoredFields, and an optionaljsonStoredset threaded throughmatchesHaving/matchesAggregationFilter), plusin-memory-aggregation.ts. That file is the one place the engine hands the object's declared field map to the per-aggregation filter, so it reads the declared set once per call besidedeclaredFieldClasses.having-filter.tsis not on objectql's published entry points.in-memory-aggregation.tsis (applyInMemoryAggregationandbucketDateValue, fromindex.tsandcore.ts), and neither exported signature changes; the declared set is threaded through the internalaggregateBucketonly.The card's table, through the REST door, before and after
Measured with a real
SqlDriveron SQLite and on a live PostgreSQL 16.14. Rows:d1 ['u1','u2'],d2 ['u2'],d3 ['u3','u1'],d4 [],d5 ['u10'],d6 null. Base212d613c, head90ba78d9.wheretwinm, basem, headowners $contains 'u1'(the card)owners $contains 'u10'owners $notContains 'u1'tags $contains 'red'(d3holds['redwood'])tags $notContains 'red'$orof$containsu1 / u3 (the any-of spelling #7398's refusal prescribes)$notoverowners $contains 'u1'title $contains 'u1'(text, the control)On the in-memory driver the per-aggregation
mis the same evaluator's answer, also 2 now. Memory's ownwhereanswers 3 for the card at this base (d5too, by a per-element substring). That face belongs to #20874 (in flight), whose branch (10656601) moves it to membership and pinsd1, d3.The fork: by the DECLARED column (Zone 2 H2)
The fork reads the declaration (
STRUCTURED_JSON_TYPESorisMultiValueField), never the row. That is the contract's sentence: "One operator, two questions, selected by the COLUMN rather than by the caller". It is alsoSqlDriver.isJsonColumn's population (built from the same two spec sets) and #20874'sisJsonStoredField, character for character.Measured: on every fixture reachable through the public doors, the declared reading and a value-shape reading select the same rows.
$contains/$notContainson a declared structured-JSON field is refused before any row is read: the engine's text-operator declared-type door,INVALID_FILTER400, inwhereand in the per-aggregation filter alike, on all three backends.find()value is an array ornullon memory, SQLite and PostgreSQL alike. The write door wraps a scalar:'u1'is stored as['u1']on memory and SQLite.The two readings differ only on rows a direct caller hands the walker: a declared multi-valued column holding a scalar string, or an undeclared column holding an array. There the declared reading gives what SQL
wheregives (no member; the substring reading), and a value-shape reading would not. Both cases are pinned. Noopen_questionsfork results.Zone 2 hypotheses, measured
m: 0).$in/$ninleft as they are.where: { owners: { $in: ['u1','u9'] } }is refusedINVALID_FILTER400 on SQLite and PostgreSQL (the driver-sql: a declared operator on amultiple: true(JSON array) column silently answers wrong —$in/$eqalways zero rows,$ninreturns the rows it was asked to EXCLUDE #7398 JSON-column gate). Memory'swhereanswersd1, d3.m: 0for$inandm: 6for$nin, a 200 wherewhereis a 400. That is reported as an out-of-scope finding, not pinned.having.groupByon a multi-valued field is refused 400 on all three backends, and on a structured-JSON field too.min/maxover a multi-valued field. That is answered three ways: an array on memory, the serialized TEXT on SQLite's native aggregate,DATABASE_ERROR500 on PostgreSQL. So there is no singlewhereanswer to holdhavingto, andhavingis not handed the declared set.having$containson agroupBytext projection keeps substring, on SQLite and PostgreSQL at the REST door and on the engine level.$notContains 'u1'counted 6 wherewherecounts 4.applyJsonMembership's complement. No other claim holdshaving-filter.ts: #5930 step 4 (domain:engine): the engine-fed faces delete their hand-copied filter meaning (driver-sql, turso remote, memory query, mongodb, formula,having); the memory reference matcher retires (D6) #20822 group 3 is unclaimed, and [finding] the aggregationfilterandhavingread a non-boolean$existsby truthiness and DROP a non-boolean$null, on every driver: the engine evaluates both in-process, and its gate refuses only$empty#20981 is filed bare. Same gate family.d6is counted, as SQL'scol IS NULL OR NOT (…)counts it.in-memory-aggregation.tsand the two test files.driver-sql'sjsonMembershipCandidatesanddriver-memory'scontainsMemberCandidates(landed by PR fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face #20984 after this branch's merge base) are both module-private in driver packages, which objectql does not depend on. SostoredArrayHasMemberis the third copy of the rule onmain; see Acceptance notes.Compile-surface conclusions
driver-sqlapplyFilterCondition$contains/$notContainson a JSON column go throughapplyJsonMembership; thewheretwin numbers in the table above are this face, measured on SQLite and PostgreSQL 16.14.driver-sqlite-wasmanddriver-tursolocal inherit it (not measured separately).RemoteTransport.buildWhereSQL212d613c: its$contains/$notContainsarms gopushLike(substring over the stored text) with no JSON-column fork. Not measured (no remote libsql here). In the out-of-scope finding below.compileScopedFilterToSql212d613c:{ owners: { $contains: 'u1' } }compiles toinstr("t"."owners", ?) > 0on SQLite, which admits a row holding["u10"]. On PostgreSQL it compiles to"t"."owners" LIKE ? ESCAPE ?over a json column. In the out-of-scope finding below (an RLS read scope).lowerAnalyticsWherewhereto aFilterConditionand adds no$containsreading of its own. The ObjectQL strategy hands that to the driver (face 1). The native SQL strategy mapscontainsto the substring LIKE shape (native-sql-strategy.ts, read, not measured), in the same finding as face 3.formulamatchesFilterConditiontypeof actual === 'string' && typeof v === 'string' && actual.includes(v)(unchanged by #20972). Measured:['u1','u2']→ false,['u10']→ false,'u1 memo'→ true. In the out-of-scope finding below.having-filterapplyHaving/matchesHavingwithout a declared set unchanged (H4).driver-memory/driver-mongodbtranslateFieldOperatorscompiles$containsto a bare$regex, which MongoDB applies per array element (per-element substring). Read, not measured.Tests
pnpm --filter @objectstack/objectql exec vitest run --project local --maxWorkers=2 src/engine-aggregate-filter-array-membership.test.ts— 30 passed. The card's rows with the rows themselves, empty table, per group,havingcontrol, the member-text reading (number / exponent / boolean / null / non-JSON-number spellings / nested / object / scalar), the declared fork, the declared population.OS_TEST_POSTGRES_URL=… pnpm --filter @objectstack/rest exec vitest run --project local --maxWorkers=2 src/aggregation-filter-array-membership.test.ts— 18 passed (9 SQLite, 9 live PostgreSQL 16.14), 9 named skips (MySQL). Each row runs beside its livewheretwin, populated and empty.localproject on the merged head90ba78d9: 349 files, 6851 tests passed;repoproject 1 file / 5 passed.90ba78d9with the PostgreSQL cell live: 9 files, 106 passed, 34 skipped.pnpm --filter @objectstack/objectql typecheckandpnpm --filter @objectstack/rest typecheck: green. Both new test files are in their package's test program (tsc -p tsconfig.test.json --listFiles).Reverse verification
Each leg ran through
scripts/ablation-replace.mjs(anchor hits proven on disk; restore proven blob == HEAD andgit diff HEADempty), from the committed change.$containsarm put back to the substring test: 15 of 30 red (every membership$containsrow, the member-text rows, the declared-fork row); the$notContainsrows and controls green, as predicted.$notContainsarm put back: 3 red (its two rows and the complement row).dist/.declaredJsonStoredFieldswas emptied, objectql rebuilt, andablation-dist-preflight.mjsfound the marker in 4 built files. 12 red: the 6 membership rows on each of SQLite and PostgreSQL. Text control,havingand empty-table rows stayed green. Restore leg: rebuilt, marker absent from all 14 built files, tree clean, 18 passed.Driver conformance ledger
node scripts/check-driver-conformance.mjs: before (212d613c) "50 covered cell(s), 0 in the DEBT ledger, 0 exempt"; after (90ba78d9) the same.Gates
node scripts/pm/dispatch-gates.mjs --commandsre-derived with no paths at90ba78d9gives 63 commands, all run, exit codes recorded to disk.--ranreconciliation: 63 derived, 61 run (all exit 0), 2 NOT MEASURED, 0 unrun.check:dual-build-cjs-loadsandcheck:type-check-debt. Both are PREREQUISITE NOT MET (exit 3): they need the whole-workspace buildlint.ymlperforms first, and 42 / 5 packages have nodist/in this worktree.check-engine-split-ratiofirst refused on the shallow checkout. It was green after a deepen to its window (git fetch --shallow-since=2026-06-26 origin main).90ba78d9:eslint --no-inline-config --format jsonreports 4 files, 0 errors, 0 warnings.--print-config).eslint.config.mjsnever enables type-aware linting (noparserOptions.project, no typed rules; its own lines 327-328 say so), so this diff cannot move an untouched file's verdict.Acceptance notes
main.storedArrayHasMemberrestatesdriver-sql'sjsonMembershipCandidatesas a predicate.driver-memory'scontainsMemberCandidates(PR fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face #20984) is the other JS copy.@objectstack/spec/data, besideasciiCaseInsensitiveContainsandisEmptyFilterValue, the value-level filter rules every JS face already reads from there.$contains/$notContainson a declared multi-valued or JSON-stored field still answer SUBSTRING on five faces, the analytics RLS read scope among them (u1admits a row storingu10) #20987.FILTER_TEXT_CASEShas no array rows. The membership fixtures are literal per package:sql-driver-17590-json-column-membership.test.ts, [finding] driver-memory answers$containson a stored array by substring per element (u1matches a row storingu10), where the SQL drivers answer membership; the spec docblock records the gap against a card that answers 404 #20874'smemory-20874-contains-membership.test.ts, and the two files here. These use the sameu1/u10/redwooddisagreement rows.*_CASESkit besideFILTER_TEXT_CASESwould let all three faces be driven by one table.wheretwin.find()presents (measured identical on all three backends). A realInMemoryDriverconsumer would need a ruled entry in thecheck:driver-memory-censusledger.OS_TEST_POSTGRES_URL/OS_TEST_MYSQL_URLforpackages/rest, the same notedata-group-by-json-door.test.tscarries. The PostgreSQL cell's local run is above.$in/$ninon a multi-valued field answer 200 (m: 0/m: 6, the latter counting the rows it was asked to exclude) wherewhereis a 400 on the SQL family, and memory'swhereanswers membership;$containsmembership contract is not answered on the turso remote transport, the service-analytics SQL compilers (an RLS read scope over-reaches),driver-mongodbandformula;min/maxover a multi-valued field: three answers (array / serialized text / PostgreSQL 500);$startsWith/$icontainson a multi-valued field: PostgreSQLwhere500, SQLite over the serialized text, memory per element.Generated by Claude Code