fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face - #20984
Conversation
…ership, on every face A multi-valued or JSON-stored field now answers `$contains` / `$notContains` by whole-element membership (the SQL family's reading), on the query path's two spellings and on the analytics face's `$match` and its SQLite echo. A scalar column keeps the case-exact substring test. The fork is the field's declared storage shape, read from the declaration `syncSchema` recorded. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…ory face; one seam; docblock and sweep - memory-20874-contains-membership.test.ts: driver-sql's #17590 fixture row for row, plus the stored shapes SQLite decides (scalar, object, nested array, null/boolean/fractional members), through find() in both spellings, count(), the analytics query and its SQL echo executed on sql.js, and the nested-relation lowering's driver input. - driver-sql #17590 suite: a multi-valued lookup column and the u1/u10 case. - driver-memory: one public seam, filterContainsTest; the analytics echo reads its member set off the test the $match exit runs. - spec: the $contains docblock's stale driver-memory status and dead tracker link are replaced with the measured status. - the rest suite header and the pending #20802 changeset no longer state the in-memory gap. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 2 package(s): 4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 4 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 138 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin d5b60fc179b36ee6d41982a8c9bd69b30dc48485 && git checkout d5b60fc179b36ee6d41982a8c9bd69b30dc48485
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 05be3525961a4977ea49eda507e44a4482f87681 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17 && git checkout -B drift-repro 05be3525961a4977ea49eda507e44a4482f87681 && git merge --no-ff 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17
node scripts/docs-audit/affected-docs.mjs --json 05be3525961a4977ea49eda507e44a4482f87681
|
…ing) filterContainsTest is a new public method on the exported InMemoryDriver class, so the published surface widens; the entry stays minor. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: Inputs: card #20874 (body and all five comments: triage 5914978120, claim 5921357350, report 5922095993, claim amendment 5922144972, patch-round report 5922235042), PR #20984 (body, its 8-file list, the net diff of ① Derived judgments
② Semver level
Clause-②: yes (widening) — one new public method ③ Boundary flags
Implemented-by: VERDICT: PASS |
|
Fixes #20874
Clause-②: yes (widening)
What changes
driver-memorynow answers$containsand$notContainson a declared JSON-stored field by whole-element membership, the readingdriver-sqlcompiles on all three dialects. A field counts as JSON-stored when it ismultiple: true, amultiselect/checkboxes/tagsfield, or aSTRUCTURED_JSON_TYPESmember such asjson. A scalar text column keeps the case-exact substring test. The change covers every face of the package:$-spelling (normalizeFieldOperators) and its AST spelling (convertConditionToMongo):find,count, and every verb that goes throughconvertToMongoQuery;$match;generateSql), which now renders SQLite'sjson_eachmembership construct for such a column, so the echoed statement still returns the rows the chart was drawn from.One rule backs all of them:
InMemoryDriver.filterContainsTest(object, field, value). On a JSON-stored field it returns{ $elemMatch: { $in: members, $not: { $type: 'array' } } }, and on any other field{ $regex: filterSubstringPattern(value) }.$notContainsis$notover that test, on both faces. The members come from the comparand's text, read the waydriver-sql'sjsonMembershipCandidatesreads it:'1'names the string or the number 1,'1.50'names 1.5, and'true'/'false'/'null'name the string or the JSON literal.The
$containsdocblock inpackages/spec/src/data/filter.zod.tslost its stale "driver-memory DOES NOT ANSWER IT YET" bullet. Its dead tracker link went with it (that card answers 404). The replacement text states the measured status, and states that the other text operators over a stored array are not ruled by that section.Measured on
origin/mainf6ccca4abefore the change, and after (HEAD10656601)Fixture:
driver-sql's #17590 fixture plus a multi-valued lookupowners(['u1','u2'],['u10'],['u3','u1'],[]).wheredriver-sql){ owners: { $contains: 'u1' } }{ owners: { $notContains: 'u1' } }{ tags_: { $contains: 'red' } }{ nums: { $contains: '1' } }(members are numbers){ label: { $contains: 'red' } }(scalar control)engine.findon a realInMemoryDriverwith the nested-relation filter{ owners: { region: 'NA' } }(owneru1NA,u10EU; this is PR #20872's lowering):d1, d3, d5before andd1, d3after. Its$notgaved2, d4before andd2, d4, d5after, which matches the rest suite's SQLite rows. That was a one-off run (a throwaway probe file, not committed): this package may not import the engine and the engine's packages may not import this driver (check:driver-memory-census). What is pinned instead is the driver input the engine sends, an$orof one$containsper related id, which objectql'sengine-nested-relation-lowering.test.tsasserts.Mechanism hypotheses (order Zone 2)
H1 confirmed. The
$containsarm lowered to an escaped$regex, and mingo applies a$regexto each element of an array value. Reproduced on the fixture above before any edit.H2: the fork is by the DECLARED field, not the row's runtime shape. Memory has the declaration (
valueShapes, recorded bysyncSchemasince the$emptywork), and SQL forks on its JSON-column registry, which is also filled from the declaration. The population is the spec's JSON-stored classes:STRUCTURED_JSON_TYPESplusisMultiValueField, the two halvesdriver-sql's registry is built from. The two readings DO give different rows on one fixture, so the choice goes to the PM in the report'sopen_questions: a declaredjsonfield holding the scalar string'u1'. SQLite answers no member, because its constructs are array-only. A per-row-shape fork would answer it by substring. The declared fork matches SQLite, and the contract text says the question is "selected by the COLUMN", declared metadata. Undeclared fields (an object never passed throughsyncSchema) keep the substring reading, asSqlDriver.isJsonColumnanswersfalsefor a table it was never told about.H3 confirmed and pinned. Number members answered nothing;
'1','2','10','0'and'1.50'now give SQLite's rows.H4 confirmed:
$notContainsdiverged. It is the mirror arm of the same defect, changed under os-dev rule 3's bounded in-place exemption. All four conditions hold:col IS NULL OR NOT (…):$notover the test admits null and missing rows, pinned on both fixtures.memory-driver.tsis held by no other claim; the sibling$existscard is kept off it by its own claim.The claim's surface should be amended to include it.
H5 confirmed. The cube face borrowed
filterSubstringPatternand wrapped it in its own$regex, so it had the same defect. It now takes the driver's whole test. The echo reads its member set off that same test, so the chart and its echo cannot name different sets.H6 confirmed. See the
engine.findrun above.Compile-surface conclusions
driver-sqlapplyFilterCondition(driver-sqlite-wasm,driver-tursolocal inherit it)applyJsonMembershipemits membership on JSON columns. Evidence: the #17590 suite now carries a multi-valued lookup column and theu1/u10case, green on SQLite here. Its live PostgreSQL/MySQL cells are unprovisioned locally; the required live job runs the whole driver-sql suite. The two inheriting drivers were not separately run.RemoteTransport.buildWhereSQL$containsarm emitspushLike(a GLOB substring) on every column, JSON columns included, so remotelyu1would match["u10"]while the local transport answers membership. Not measured. Reported as a finding.read-scope-sqlcompileScopedFilterToSql{ owners: { $contains: 'u1' } }on a declaredlookup+multiple: truefield compiles toinstr("t"."owners", ?) > 0on SQLite andLIKE '%u1%'on PostgreSQL/MySQL, a substring over the stored JSON text. This is an RLS read-scope face, so it is reported as a security-relevant finding.filter-normalizerlowerAnalyticsWhere$containsto the cubecontainsoperator, which the native SQL strategy renders asLIKE, a substring. Reported.formulamatchesFilterConditiontypeof actual === 'string' && actual.includes(v), so a stored array never matches (fail-closed). Reported.having-filterdriver-memoryquery path and cube face (memory-analytics.ts), behindfilter-refusal.tsfilter-refusal.tsand the$existsarm are untouched (sibling card).driver-mongodbtranslateFieldOperators$containsto a native$regex, which MongoDB applies per array element: the same defect. Not measured. Reported.Why the shared pin is a mirrored literal table, not
FILTER_TEXT_CASES(order Zone 3, not taken)FILTER_TEXT_CASEShas no array column, so adding one changes the fixture of all five enrolled drivers.driver-mongodbimports every row of it and still carries the per-element defect, so its suite would go red. The table's own rule 2 says rows join a driver's suite in the PR that ends that driver's gap. Doing it here would also widen the spec touch beyond the one declared docblock. A new sibling case-set would add DEBT rows to a ledger that only goes down. So the new memory file mirrorsdriver-sql's #17590 fixture row for row and asserts the same literal row sets. That is how the #17590 file already holds its three dialect cells to one answer.Files
packages/drivers/driver-memory/src/memory-driver.ts: the population (isJsonStoredField), the members, the one test (filterContainsTest), both query-path spellings.packages/drivers/driver-memory/src/memory-analytics.ts: the$matchrows take the driver's test; the echo renders membership (sqliteMembershipPredicate).packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts(new): two fixtures. The first is the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 fixture plusowners. The second holds the stored shapes SQLite decides: a scalar, an object, a nested array,[null],[[null]],[true, 1.5],['']and a NULL row, each answer measured ondriver-sql/SQLite first. Every case runs onfindin both spellings, oncount, and on the cube's$match. The cube's echo is EXECUTED on sql.js over the same rows. The file also covers the nested-relation driver input and the fork.packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts: anownersmulti-valued lookup column and theu1/u10case (the claim's SQLite pin).packages/spec/src/data/filter.zod.ts: the one declared docblock..changeset/20874-memory-contains-membership.md:minor, not thepatchthe order suggested.filterContainsTestis a new public method on the exportedInMemoryDriver. It ships indist/index.d.ts(the existingfilterSubstringPatternappears there as the positive control), and the level ruling quoted inpr-automation.yml(WHICH LEVEL) grades an additive widening of a published package's public surface at leastminor.Clause-②: yes (widening): the new public method widens the published surface; no filter key or operator is added.packages/rest/src/data-nested-object-door.test.ts(a header comment that cited the gap and the removed docblock clause);.changeset/20802-nested-relation-filter-served.md, one sentence of a pending release note that said the in-memory driver still matches per element.Confirmation requested: a pending release note is corrected
This PR changes
.changeset/20802-nested-relation-filter-served.md, a pending release note this PR did not add. This is the DELIBERATE CORRECTION classcheck-empty-changeset.mjsnames, soCheck Changesetstays red on purpose. It is not a required context.skip-changesetmust not be applied. The note's last sentence said:This PR makes that false, so it now reads:
Please confirm the correction on this PR. If the release that consumes that note ships before this PR lands, the edit should be dropped on rebase.
Tests (HEAD
10656601)pnpm --filter @objectstack/driver-memory typecheckpasses, andvitest rungives 68 files / 1553 tests passed. Both typecheck programs include the new test file (--listFiles).pnpm --filter @objectstack/driver-sql typecheckpasses (its program includes the edited test).sql-driver-17590-json-column-membership.test.tsgives 22 passed, 2 skipped; the skips are the live PostgreSQL/MySQL cells, unprovisioned here, NOT MEASURED locally.pnpm --filter @objectstack/spec typecheckpasses.spec buildand thencheck:generatedreport "All 15 generated artifacts are up to date".Reverse verification. The code change was committed first. Then
memory-driver.tsandmemory-analytics.tswere checked out at BASE under an EXIT/INT/TERM restore trap. The landed mutation was verified on disk before the run:filterContainsTestcount 0, twoescapeRegex(val)arms back, and both files byte-equal to BASE.The predicted direction was that the cases where per-element substring and membership disagree go red and the agreeing ones stay green. Measured: 69 failed / 44 passed. The failures were every disagreeing case, on all four faces, plus the executed echo, the relation-lowering input and the fork pin. The
labelcontrols and theredwood,ab,u10and'0'cases stayed green. The restore was proven by blob hashes equal to HEAD (d376dadb…/9de9198b…), an emptygit diff HEADand clean porcelain.Guard ablation (
scripts/ablation-replace.mjs, anchor 1 to 0). Dropping$not: { $type: 'array' }turned exactly the nested-array cases red: 12, the json'u1', its complement and'null', on all four faces. The executed echo stayed green, since SQLite is the oracle there. The file was restored to the HEAD blob with an emptygit diff HEAD.Lint (narrowed, a measurement).
eslint.config.mjslints**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}minusNEVER_LINTED. The six changed TS files are all in it; the two changesets are not.eslint --no-inline-config --format jsonover those six files reports 6 files, 0 errors, 0 warnings at10656601.parserOptions.project, no typed rules; it says so itself), so this diff cannot move any untouched file's verdict.Driver conformance ledger:
node scripts/check-driver-conformance.mjsreads "50 covered cell(s), 0 in the DEBT ledger, 0 exempt" both atf6ccca4a(before) and at10656601(after).Gates:
dispatch-gates --commandswas re-derived at10656601. That gives 84 commands, six more than the dispatch list:engine-double-contract,objectql-double-limit,query-options-erasure,type-check-coverage,type-check-debtandwhere-matcher. The run also covered the eight roster gates flagged as sharing a directory with these paths. All exit 0 except:check-empty-changeset: exit 1, the deliberate correction above.check:dual-build-cjs-loadsandcheck:type-check-debt: exit 3,PREREQUISITE NOT MET(they need the whole workspace built). NOT MEASURED; CI's Build/Lint jobs own those.check:lean-entry-closure: exit 3 at first. It passed after building objectql.Acceptance notes (observed, not filed by this PR)
{ owners: { $startsWith: 'u1' } }givesd1, d3, d5on memory (per element) and no rows on SQLite (GLOB over the serialized text). The spec docblock now says so.engine.ts's delete-probe docblock (the$containspushdown) still says every backend answers a substring SUPERSET. That is stale fordriver-sqlsince driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 and for memory after this PR. The code stays correct because it narrows exactly throughstoredReferenceIncludes. Comment only, no carrier.driver-memoryholds no declaration for keeps the substring reading, and mingo still applies that$regexper element of an array value. This is the deliberate counterpart ofdriver-sql'sisJsonColumnreturning false for an unknown table.Generated by Claude Code