Skip to content

fix(driver-memory)!: refuse the equality and ordering family on a declared JSON-stored field, in the SQL family's words (#21066) - #21159

Merged
objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-21066-memory-json-column-family
Oct 1, 2026
Merged

objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-21066-memory-json-column-family

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #21066
Clause-②: yes (narrowing)

On a field the object declares JSON-stored (a multiple: true field, tags / multiselect / checkboxes, or a structured-JSON type such as json), driver-memory now refuses the scalar-comparison family that driver-sql's where refuses: $eq, $ne, $gt, $gte, $lt, $lte, $between, $in, $nin and implicit equality, whatever the comparand, at any depth. The answer is INVALID_FILTER / 400 with the same message. The operator set and the sentence are read from @objectstack/core (JSON_COLUMN_INCOMPATIBLE_OPERATORS, jsonColumnOperatorRefusalText, homed by PR #21097). There is no third copy. $contains / $notContains (membership), $null, $exists and $empty keep answering.

What was wrong (H1, measured at origin/main 670680e93 through engine.find)

A real ObjectQL over InMemoryDriver, #21004's six rows (owners is a multiple: true lookup, tags is a tags field). Every row reproduces the card:

where before now
owners $eq 'u1' d1, d3 (per element) 400 INVALID_FILTER
owners $in ['u1','u9'] d1, d3 400
owners $nin ['u1','u9'] d2, d4, d5, d6 400
owners $gt 'u1' d1, d2, d3, d5 400
tags $gt 'red' d3 400
also: bare { owners: 'u1' }, $ne, $gte, $lt, $lte, $between, tags $eq, { owners: null }, $eq null, $ne null, $in [], $nin [] rows, per element 400
controls: owners $contains 'u1' / $notContains / $null / $exists / $empty, title $in rows unchanged rows

The engine hands the driver the operators as written, except $ne / $nin. Those arrive inside the spec's null-safe lowering ($and of $or of $null: true and the operator). The gate walks $and / $or / $not, so that shape is refused too.

The analytics face (MemoryAnalyticsService) answered the same per-element rows. Its SQL echo rendered owners = 'u1', which matches no row over the JSON text the SQL family stores. It now refuses in query() and generateSql() alike.

What changed

  • filter-refusal.ts: the shape gate (assertFilterConditionShape) takes an optional FilterFieldDeclarations (isJsonStoredField, reportWithheld). It has two arms. Implicit equality on a declared JSON-stored field is refused as =, bare. Any operator in the shared set is refused AFTER the existing comparand-shape rules, which is driver-sql's order (comparand gate, then column-type gate). So an array under $eq or a one-element $between still gets its own refusal first. jsonStoredFieldOperatorError builds the error from the shared text: this package's unsupportedFilterError envelope, with the withheld diagnostic handed to reportWithheld (prefixed At PATH:) before the throw.
  • memory-driver.ts: convertToMongoQuery passes this.filterFieldDeclarations(object). The population is isJsonStoredField, the predicate $contains already forks on (STRUCTURED_JSON_TYPES or isMultiValueField). So the fields where $contains asks membership are exactly the fields where the family is refused. The diagnostic goes to the driver's logger at warn, the level driver-sql uses for its withheld filter diagnostics. That keeps the message's "the full diagnostic is in the server log" true here.
  • memory-analytics.ts: normalizeFilters takes the cube. It judges a where key (a cube member) by the field it maps to on the cube's table, the same (table, field path) pair filterContainsTest reads. Its diagnostic goes to the analytics service's own logger.
  • .changeset/21066-memory-json-column-family-refusal.md: @objectstack/driver-memory minor, BREAKING banner, Clause-②: yes (narrowing), one ADR-0087 marker not-required (no-migration-prescription). No registered id covers a filter operator on a JSON-stored column. The one migration-registry entry that mentions json columns (cel-predicate-one-value-comparand-refused) is the CEL list-comparand surface, not this one.

Hypotheses, measured

Pin sweep

  • The ONE per-element pin the package carried on a declared field flipped: memory-20444-empty-operator.test.ts had { tags: { $empty: true, $ne: null } } giving r2. It is now a refusal pin (code + status + the shared message). The composition ($empty beside a has-a-value sibling on one multi-value field) is kept through $null: false, which answers r2. driver-sql/SQLite answers that row too, and refuses the $ne: null spelling with the same body (measured on the built driver).
  • memory-matcher-scalar-comparand-array-value.test.ts pins per-element answers on a column declared text. Those cells still hold, and a header note now says the population there is a scalar-declared column.
  • Repo-wide: only two tests outside this package bind the real driver (packages/runtime's two ruled consumers). Neither filters a JSON-stored field. No other INVALID_FILTER pin moves.

Tests (final head 13407b76f)

  • pnpm --filter @objectstack/driver-memory exec vitest run --maxWorkers=2: 70 files, 1703 passed. The first run after the implementation, before any test edit: 69 files, 1 red of 1613, the per-element pin flipped above.
  • pnpm --filter @objectstack/driver-memory run typecheck: exit 0. tsc --listFiles includes both edited test files.
  • New memory-21066-json-column-family-refusal.test.ts (89 tests):
    • the card's five rows;
    • every family member on owners / tags / meta (json);
    • ten shapes per field (bare, bare null, $eq null, $ne null, the engine's $ne lowering, $in [], $nin [], under $not, an $or branch after a holding one, $eq beside $contains);
    • count / findOne / updateMany / deleteMany refusing with the table untouched;
    • the withheld message plus the logged At filter.$or[1].owners.$gte: diagnostic;
    • comparand-first ordering;
    • eleven answered controls;
    • the declaration boundary (undeclared object, scalar-declared column, the gate with and without declarations);
    • the analytics face, query() and generateSql(), including the cube.member spelling, plus its log line and a $contains control.
  • Ablations (node scripts/ablation-replace.mjs, wrap mode, run at de1fef341; the two later merges touched no file in this package; each restore proven blob == HEAD with git diff HEAD empty). The subjects are this package's src, imported relatively, so no dist is involved:
    • A filterFieldDeclarations' predicate forced to () => false: 73 red / 37 green of 110 across the new file and 20444. The 17 green in the new file are exactly the answered controls, the declaration-boundary trio, the premise, the comparand-first case and the analytics $contains control.
    • B the implicit-equality arm disabled: 7 red, exactly the six bare cases and the analytics bare case.
    • C the operator arm disabled: 67 red, every operator-based refusal pin, the direct-gate test and the 20444 flip.
  • Gate union, derived with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack (no paths; 7 changed paths, working tree clean) at 13407b76f: 60 derived, 60 run, every one exit 0. Reconciled with --ran: "60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit code and none of them is 3)". The same 60 also ran all-zero at the previous merge head 5b75fe461. At de1fef341, check:dual-build-cjs-loads exited 3 (PREREQUISITE NOT MET, no dist yet); it measured on both later heads.
  • Driver conformance ledger (node scripts/check-driver-conformance.mjs), before and after: byte-identical. 50 covered cells, 0 DEBT, 0 exempt. The shared matrix has no JSON-column or multi-value case-set, so this invariant is held by the per-package pins, not the matrix.

Acceptance notes


Generated by Claude Code

claude added 6 commits October 1, 2026 09:02
…lared JSON-stored field

The shape gate in front of the query path and the analytics face now
refuses a scalar comparison (the shared JSON_COLUMN_INCOMPATIBLE_OPERATORS
set, and implicit equality) aimed at a field declared JSON-stored, with
INVALID_FILTER / 400 and the shared refusal text from @objectstack/core.
The withheld diagnostic goes to the face's logger at warn.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
…write doors and the analytics face

Flip the one per-element pin the suite carried ($ne beside $empty on a
tags field) to a refusal pin, keep its composition through $null: false,
and note the declared population on the scalar-column array suite.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
…family refusal

Also tag filterFieldDeclarations @internal: it is reachable from the
analytics face, not a consumer contract.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/driver-memory, touching 12 documentable anchor(s).

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/data-modeling/drivers.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/permissions/authentication.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/plugins/packages.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/protocol/objectql/query-syntax.mdx (via InMemoryDriver (symbol, a top-level class))

⛔ 6 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/implementation-status.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v14.mdx (via generateSql (symbol, a method of class MemoryAnalyticsService))
  • content/docs/releases/v16.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-0.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-3.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-5.mdx (via generateSql (symbol, a method of class MemoryAnalyticsService))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 9 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from a53e51e3e00813bb5881a9194e1a82df6d53935b — the merge of head 2eab11e409f28a6d3be235dfabbd53e50bc124ef into base 2488b98b48f51e2a1bc5b5e50fc1c206c9565291, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin a53e51e3e00813bb5881a9194e1a82df6d53935b && git checkout a53e51e3e00813bb5881a9194e1a82df6d53935b
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 2eab11e409f28a6d3be235dfabbd53e50bc124ef && git checkout -B drift-repro 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 && git merge --no-ff 2eab11e409f28a6d3be235dfabbd53e50bc124ef

node scripts/docs-audit/affected-docs.mjs --json 2488b98b48f51e2a1bc5b5e50fc1c206c9565291

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

…narrowing)

The seat's answer on the card: InMemoryDriver.filterFieldDeclarations is in
the published .d.ts, so the declaration follows the precedent the analogous
filterContainsTest set. Level and ADR-0087 marker unchanged.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 2eab11e409f28a6d3be235dfabbd53e50bc124ef
Local-runs: none

PR #21159 on card #21066, judged against triage's direction 5925785069, the claim 5927990211, the report 5929893869 and the seat's answer 5929927010. Inputs: the card thread, the PR body and file list, the net diff against the merge-base with main (58a77dbde, 7 files, +565 / -10), and the check-runs on the head. Nothing built, run or re-run.

① Derived judgments

Every accept-set and public-surface change the diff implies, each named right or wrong:

  1. The query path narrows on a declared JSON-stored field — right. convertToMongoQuery hands this.filterFieldDeclarations(object) to the one shape gate, and that method is the filter entry of find, count, updateMany, deleteMany, aggregate and also distinct (six call sites on the head; findOne is pinned refusing in the new suite). The gate has two arms. Implicit equality (the !isFilterNode branch, so any scalar, null or Date comparand) is refused as bare =. An operator in the shared set is refused after every comparand-shape rule in the loop, with no continue ahead of it. The walk recurses $and / $or / $not, so the depth claim holds; the engine's null-safe lowering of $ne / $nin (an $or holding $null: true and the operator) reaches the operator arm on its second branch.
  2. The set and the sentence are read, not copied — right. filter-refusal.ts imports JSON_COLUMN_INCOMPATIBLE_OPERATORS and jsonColumnOperatorRefusalText from @objectstack/core; no literal of either exists in the package. Byte-identity with driver-sql: both drivers build the error as unsupportedFilterError(message) over the same core text (driver-sql's withheldFilterError is that call plus non-enumerable symbol carriers), so message, code INVALID_FILTER and status 400 are the same bytes. The new suite asserts the message equal to the imported text, and derives its FAMILY from the shared set intersected with SUPPORTED_FIELD_OPERATORS with a floor of the nine $-spellings — the forward pin for [finding] $startsWith / $icontains on a multi-valued lookup answer 500 on PostgreSQL and a wrong count on SQLite: the text operators other than $contains reach a JSON column unrefused and unruled #21009 the report describes.
  3. The predicate for "declared JSON-stored" — right, and the same reading as driver-sql's gate less two recorded members. Memory: isJsonStoredField is STRUCTURED_JSON_TYPES.has(type) || isMultiValueField(shape) over valueShapes, which only syncSchema fills (one write site). driver-sql: jsonFields is filled by JSON_COLUMN_TYPES.has(type) || isMultiValuedColumn(type, field), where JSON_COLUMN_TYPES is STRUCTURED_JSON_TYPES plus MULTI_OPTION_TYPES plus the driver-internal object / array aliases, and a single-value media column is asked per instance (mediaColumnIsJson). The two omissions are documented on the method and are not declarations an authored object can carry into this driver. An object never synced answers false and is not judged; SqlDriver.isJsonColumn answers false for a table with no jsonFields entry and assertOperatorAppliesToColumn returns on it. Same answer. A column declared text holding an array is not judged either; the memory-matcher-scalar-comparand-array-value pins hold on that population and its new header note says so.
  4. Order: comparand gate, then column-type gate — right, matching driver-sql at all three of its positions (the bare loop, the operator map and the reduction walk each run assertCompilableComparand before assertOperatorAppliesToColumn). Memory's loop refuses the array comparand, the malformed $between, the non-boolean $null / $exists / $empty, the array under a single-value operator and the $icontains / $like shapes first; pinned by the comparand-first test.
  5. The controls keep answering — right. $contains, $notContains, $startsWith, $endsWith, $icontains, $null, $exists and $empty are absent from the shared set by design; eleven answered controls are pinned on the query path and $contains on the analytics face. Spec FILTER_OPERATORS carries no equality or ordering alias beyond the nine $-spellings, and the set's bare infix spellings are refused by the vocabulary gate before this one (a refusal, in that gate's words; pre-existing).
  6. The analytics face narrows the same way — right. normalizeFilters is the face's only filter entry (no cube-style filters list is read anywhere in the file). The gate runs on the lowered where with declarations resolved through extractTableName(cube.sql) and resolveFieldPath(cube, member), the same (table, field path) pair filterContainsTest is asked, so the cube.member spelling is judged and pinned. Lowering keeps every family member reachable: rule 3 leaves $ne: null alone and wraps a non-null $ne / $nin in an $or; rule 1 splits $between into $gte / $lte, both in the set. query() and generateSql() share the entry, so the SQL echo refuses too. One corner, pre-existing and not of this diff: this face lowers type-blind, so rule 2 turns a lone $lte whose comparand is the last supported day (9999-12-31) into $null: false and answers it, where find() refuses $lte on the same field. A sentinel comparand on a list field; noted for the seat, not escalated.
  7. Disclosure — right, in the shared posture, with one recorded difference. The message withholds the field and the operator; the diagnostic, prefixed with the filter position, goes to the driver's or the service's own logger at warn, the level logWithheldFilterDiagnostic writes at in driver-sql, and the log line has the same shape. The seam is new to this package, as the report says. The difference: driver-sql resolves driver-sql: the #7929 withhold covers the cross-field family only — every other INVALID_FILTER refusal still names the target field, which is admin-authored on a read-scope predicate #8197 provenance and discloses the diagnostic on the wire for a subtree a boundary vouched author (plugin-security, plugin-sharing and service-analytics mark the caller's own where); this driver has no provenance read, withholds unconditionally, and carries no withheldFilterDiagnosticOf twin. The fail-closed direction, outside the direction's scope; noted.
  8. Public surface — one method, declared. InMemoryDriver.filterFieldDeclarations is public on the exported class and so in the published .d.ts; FilterFieldDeclarations is not re-exported from src/index.ts (checked). Declared in the changeset as yes (narrowing), as the seat ruled (5929927010) on [finding] driver-memory answers $contains on a stored array by substring per element (u1 matches a row storing u10), where the SQL drivers answer membership; the spec docblock records the gap against a card that answers 404 #20874's grading of filterContainsTest. Right.
  9. Edits outside the claim's file list — justified. memory-driver.ts (the declarations and the log sink) and memory-analytics.ts (the second gate caller) are the plumbing the gate needs to see a declaration; the single-writer path check on the head is success.
  10. The pin flip — flipped, not deleted. memory-20444-empty-operator.test.ts keeps the composition cell through $null: false giving r2, and adds a refusal pin for the $ne: null spelling asserting code, status and the shared message.
  11. Left alone, and rightly: the AST comparison-node door. No non-test emitter of a type: 'comparison' node exists outside driver-memory (swept on the head), so the per-element answer there is reachable by a direct driver call only, as the report says.
  12. A residual of the same class, found here. A no-operator object comparand on a declared JSON-stored field ({ meta: { k: 'a' } } on a json column, { owners: { k: 'a' } } on a multi lookup) is not judged by this gate: the bare arm sits inside !isFilterNode(spec) and the operator arm needs a $ key, so on a direct driver call this driver still answers it by deep equality, as it did before this PR, where driver-sql refuses the same input at its comparand gate. Through the engine, [finding] a no-operator object under a lookup, master_detail or json field answers per driver: the declared nested-relation filter returns no rows on memory and a 400 on SQL, and a json object comparand deep-equals on memory and is refused on SQL #20745's no-operator-object door refuses the json and relation kinds before any driver is asked, so public-door reach is none. Not in this card's family (the operators plus bare scalar equality); not a defect of this diff. For the seat to note beside the AST door, or file.

② Semver level

  • .changeset/21066-memory-json-column-family-refusal.md: @objectstack/driver-memory minor (from 17.5.0), a ! summary, a BREAKING banner naming every door that narrows, and exactly one ADR-0087 marker, not-required (no-migration-prescription), with its closing of the other categories. What the diff publishes is an accept-set narrowing on a released package plus one public method: (narrowing) is BREAKING under AGENTS.md's changeset rule, BREAKING ships minor in the launch window (check-changeset-no-major, the rule a dozen sibling changesets cite), and no authorable key, export or stored shape moves, so the marker's arm is right. Check Changeset, the job that runs check-adr-0087-registration against the merge-base, is success on the head.
  • Clause-②: yes (narrowing). The changeset's line and the PR body's line 2 both carry it, as the seat's answer prescribed. Right.
  • One defect, body-only: two sentences of the PR body still read no (narrowing) — the "What changed" bullet on the changeset, and the first "Acceptance notes" bullet ("carries the claim's no (narrowing) line verbatim and leaves the grading to the seat"). They contradict line 2 and the changeset. A PR-body edit, no new head; the seat makes it before the queue reads the body.

③ Boundary flags

Implemented-by: claude/issue-21066-memory-json-column-family
Reviewed-by: session_01Ujdtvqs7ree7WyQmEDwEnG

VERDICT: PASS


Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 1, 2026 15:07
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 1, 2026 15:08
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 45ce12a Oct 1, 2026
50 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-21066-memory-json-column-family branch October 1, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

2 participants