fix(objectql)!: a groupBy on a structured-JSON field is refused INVALID_FIELD / 400 at the engine aggregate door, on every driver (#20783) - #20804
Conversation
…gine's aggregate door A groupBy entry naming a json, composite, repeater, record, location, address or vector field is refused INVALID_FIELD / 400 before any driver is asked, in both entry spellings. Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
… door Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
… on SQLite and live SQL Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…d of erasing them Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 1 package(s): 14 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 3 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 8c3eed2f8b8b31473f732fd3acbe50bb1444b93d && git checkout 8c3eed2f8b8b31473f732fd3acbe50bb1444b93d
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 96e724475c476d018c2b6d13dcd117ade14d0aa0 43ae6c1fcc7b4770385757f8fc3b439ca3a80382 && git checkout -B drift-repro 96e724475c476d018c2b6d13dcd117ade14d0aa0 && git merge --no-ff 43ae6c1fcc7b4770385757f8fc3b439ca3a80382
node scripts/docs-audit/affected-docs.mjs --json 96e724475c476d018c2b6d13dcd117ade14d0aa0
|
Contract reviewServed-tier: PR #20804 (card #20783), isolated read-only review. The PR's head was still the sha above when read, twice. Inputs and nothing else: the card body and its three comments (triage 5905038653, the claim 5905436757, the ① Derived judgments
② Semver level
③ Boundary flagsEvery dev flag from the
Implemented-by: VERDICT: PASS Generated by Claude Code |
Fixes #20783
Clause-②: no (narrowing)
What this changes
A
groupByentry that names a declared structured-JSON field (json,composite,repeater,record,location,address,vector) is now refused byengine.aggregatewithINVALID_FIELD/ 400, in the engine's words, before any driver is asked. It holds on every driver and for every caller that reaches the engine: the REST query door, a flow or hook calling the engine in-process, and the analytics strategy that lowers a cube query ontoengine.aggregate. Both entry spellings are judged: the field name (groupBy[0]) and the{ field }object (groupBy[0].field), adateGranularitybucket included.The words, as
POST /api/v1/data/:object/queryreturns them (the route is inside the 500 characters the REST door keeps, and the REST pin asserts it there):The thrown error carries
code: 'INVALID_FIELD',statusandhttpStatus400,field,fields(every offending entry),objectandparam: 'groupBy'.Landing site, as the order expected:
packages/objectql's aggregate admission. No driver file, no serialization rule.packages/objectql/src/group-by-structured-json-door.ts(new):assertGroupByNamesNoStructuredJsonField(object, schema, groupBy). The class is the spec'sSTRUCTURED_JSON_TYPES(the set PR fix(objectql)!: a no-operator object beneath a relation, structured-JSON or undeclared id column is refused INVALID_FILTER / 400 on every driver (#20745) #20781's JSON arm judges), never a list minted here. Not judged: an undeclared name (the engine's registry-less tolerance; the REST ingress answers an unknown nameINVALID_FIELDfirst), a host with no field map, and every other type.packages/objectql/src/engine.ts: one call at the entry ofaggregate, right afterrejectCredentialAggregation(which reads the samegroupByentries), so a protected field keeps that refusal's words.aggregateis the only engine verb that takesgroupBy:findrefuses the key (ENGINE_FIND_OPTION_KEYS).packages/objectql/src/number-comparand-declared-type-door.ts: comment only. Its sentence "ajson… groupBy … is judged now" athavingbecame false for a json groupBy, which no longer reacheshaving.Why
INVALID_FIELD, an existing code. The verdict is about the named field's type at a position. That is the question the REST ingress answers withINVALID_FIELDfor an unknowngroupByname (assertGroupByFieldsExist), the search axis answers withINVALID_FIELDfor a field whose type it cannot scan, and the engine'sassertFilterIsMaterializableanswers withINVALID_FIELDfor a virtual field ("this verdict is about the NAME's type").INVALID_FILTERis the engine's value-shape envelope andgroupByis not a filter;INVALID_QUERYis the ingress's malformed-shape code, and the entry here is well formed.Before, measured on
origin/main7a09eee1b1Through
POST /api/v1/data/:object/query(the realRestServerroute overObjectStackProtocolImplementationandObjectQL) with{ groupBy: [FIELD], aggregations: [{ function: 'count', alias: 'n' }] }, and throughengine.aggregatedirectly (same answers). Drivers: InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and SqlDriver on a private PostgreSQL 16.13 started for this run. Three rows:titlex, x, y;meta{a:1},{a:2},{b:1}; and one differing value per row under every other structured-JSON field.groupBytitle(text, the control)x2 ·y1meta(json, the card){a:1},n3DATABASE_ERROR("could not identify an equality operator for type json")composite,repeater,record,location,addressn3vector[1,2]2 ·[3,4]1){ field: 'meta' }n3{ field: 'meta', dateGranularity: 'month' }nullbucket,n3nullbucket,n3['title', 'meta']After, the same run on this branch (
43ae6c1fcc)Every structured-JSON row above answers
400 INVALID_FIELDin the engine's words on all three drivers, naming the position (groupBy[0],groupBy[0].field,groupBy[1]for the mixed entry), the field and its declared type. No read of the object runs. Thetitlecontrol answersx2 ·y1 on all three, unchanged. The InMemoryDriver cells of this run came from a scratch script over the built packages. They are not committed:check:driver-memory-censusrefuses a new test consumer of that driver without a ruling, so the committed memory cell is the recording driver below, by construction.Hypotheses (zone 2): which held
H1: held.
aggregateresolvesgroupByentries against the declared field map before the driver inrejectCredentialAggregation(credential andinternalfields) and, forhaving, inaggregatedRowColumnTypes. The refusal sits beside the first, at the verb's entry, so it runs before the per-aggregationfilterandhavingdoors and before any driver is resolved. Code:INVALID_FIELD, reasons above. The REST ingress also resolvesgroupBynames (assertGroupByFieldsExistinmetadata-protocol), but only for the REST path. The engine is the one door every caller shares, as triage directed.H2: held, refined. Every member of
STRUCTURED_JSON_TYPESanswers per driver today, and none answers one way. Six members split exactly likejson(memory one merged group, SQLite per serialized document, PostgreSQL 500).vectorsplits differently on memory (one group per array, not one merged group) and still 500 on PostgreSQL, so it does not answer one way either. So the class is refused, notjsonalone.H3: held, and it was NOT already refused. A date-bucketed
{ field, dateGranularity }over ajsonfield passed the REST ingress (a known field, a valid granularity) and answered onenullbucket on memory and SQLite and 500 on PostgreSQL. It is refused now with the{ field }form's words, since no granularity makes a JSON document a date.H4: measured. The analytics face is partly the same door. Measured through
AnalyticsService.querywired withAnalyticsServicePlugin's own auto-bridges (executeAggregatetoengine.aggregate,executeRawSqltoengine.execute), with a cube dimensionsql: 'meta'on ajsonfield:dimensions: [meta], beforeengine.aggregate)DATABASE_ERROR(native SQL)dimensions: [meta], afterINVALID_FIELD(this door)timeDimensions: [{ meta, granularity: month }], beforenullbucketnullbucketINVALID_FIELD(the native strategy declines a granularity, soengine.aggregateserves it)So the ObjectQL strategy (any cube query on a driver without raw SQL, and any bucketed time dimension) reaches this door and is folded in with no
packages/servicesedit.NativeSQLStrategydoes not: on SQL drivers it compilesGROUP BY "meta"by hand and bypasses the engine. That half is a sibling card (reported to the PM below, not filed here). The dataset compiler only checks that a dimension's field is declared (assertDeclared), so a dataset dimension on ajsonfield compiles into the same cube dimension and splits the same way. The memory cube face (MemoryAnalyticsService) has zero production constructors (git grep "new MemoryAnalyticsService"outside tests: 0 hits at7a09eee1b1), so it reaches a driver only in tests. It was not edited and not measured for this shape.Producers measured (for "refuse, don't define")
Zero producers group by a structured-JSON field. Every
grouping/groupBy/groupByField/dimensionstarget inexamples/at7a09eee1b1is one ofstatus,priority,created_at,account,stage,total,region,sales_region,progress,issued_on,industry,category,signed_on,health,completed_date,close_date, and none is structured-JSON (the showcase's structured-JSON fields arefield-zoo'sf_*,account.hq,account.support_configandtask.location). The same count over the platform packages namesobject_name,user_id,id,action,topic,provider_id,organization_id,namespace,kind,actor_idandphone, plus two dynamic engine callers inservice-analytics(see the acceptance notes).Tests
packages/objectql/src/engine-group-by-json-door.test.ts(6 tests, recording driver, so the in-memory cell by construction). It covers every structured-JSON type with the full envelope (code,status,httpStatus,field,fields,object,param) and the position, and asserts zero reads. It covers the{ field }form, a date bucket, an entry after a scalar one and two offenders, and the REST door intofindData. Controls:text,number, amultiple: trueselect, an image field and an undeclared name all reach the driver, and a structured-JSON field as an aggregated column is unchanged. GUARDs: the judged types equalSTRUCTURED_JSON_TYPESover everyFieldType, and there is no verdict without a field map, for an undeclared name, or for an entry naming no field.packages/rest/src/data-group-by-json-door.test.ts: SQLite always, PostgreSQL / MySQL whereOS_TEST_POSTGRES_URL/OS_TEST_MYSQL_URLare set. Every structured-JSON type answers 400INVALID_FIELDwith the route in the REST body and zero reads. The object and bucket forms and the mixed entry answer the same 400. The controltitleanswersx2 ·y1 from the driver.@objectstack/rest, so the live cells run only locally. Local run with the private PostgreSQL 16.13: 6 passed (sqlite 3, live postgres 3) / 3 skipped (mysql, no URL).engine-nested-object-door.test.ts(PR fix(objectql)!: a no-operator object beneath a relation, structured-JSON or undeclared id column is refused INVALID_FILTER / 400 on every driver (#20745) #20781's pin): itshavingcase overgroupBy: ['meta']pinned the branch this PR closes, because a json groupBy never reacheshavingnow. It is replaced, not respelled. The JSON arm athavingis still reached through amaxof a json field (having: { top_meta: { a: 1 } }), which answers the sameINVALID_FILTERwhole-value words. No other fixture groups by a structured-JSON field. Measured by a grep over every test that both groups and declares a structured-JSON type; the downstream suites below stayed green.pnpm --filter @objectstack/objectql testat43ae6c1fcc: 342 files / 6739 passed.pnpm --filter @objectstack/rest testat43ae6c1fcc(with the live PostgreSQL URL set): 232 files / 4516 passed / 33 skipped.typecheckfor@objectstack/objectqland@objectstack/restat43ae6c1fcc: exit 0, including eachcheck:test-typecheck(objectql's ledger held at 40 files / 234 errors / 65 signatures; rest 0), so both new test files compile....@objectstack/objectqldirection), atcafaf885d8:@objectstack/service-analytics141 files / 3267 passed;@objectstack/metadata-protocol190 files passed, 3 skipped / 2792 passed, 19 skipped. The other consumers are declared to CI.Reverse verification (ablation), from the committed fix at
1eacad5296. It ran throughscripts/ablation-replace.mjsin WRAP mode, trap-restored. The engine's callassertGroupByNamesNoStructuredJsonField(object, this._registry.getObject(object), query.groupBy);was fed(query as { __ablated_20783__?: unknown }).__ablated_20783__(always undefined) instead ofquery.groupBy. On disk the anchor went 1 to 0 and the marker 0 to 1, with blob67198fc4a8fcto837fc9257e37. objectql was rebuilt, andablation-dist-preflightfound the marker in 4 built files.engine-nested-object-door.test.tsstayed green.titlecontrol stayed green.67198fc4a8fc),git diff HEADis empty, and whole-treegit status --porcelainis empty. After a rebuild, the--absentpreflight found the marker absent from all 14 built files, and the tree reading was clean. Then both pin sets were green again (21 passed; 6 passed + 3 skipped).Gates
node scripts/pm/dispatch-gates.mjs --commandsat43ae6c1fcc(the merge oforigin/main96e724475c, which touches none of this diff's packages) derived 65 commands over the 7 changed paths. All 65 were run on43ae6c1fcc, and--ranreconciles them: 65 derived, 65 run, 0 NOT-MEASURED, 0 UNRUN, all exit 0. Among them:check:adr-0087-registration --base origin/main:not-required (no-migration-prescription)accepted.check:changeset-no-major,check:empty-changeset,check:doc-authoring,check:nul-bytes,check:issue-citations.check:engine-double-contract,check:where-matcher,check:driver-memory-census,check:cross-package-test-inputs,check:test-source-alias,check:type-check-coverage,check:type-check-debt.check:query-options-erasure. It caught a first draft of the REST pin that erased anengine.aggregateoptions bag toany(test surface 236 to 237). Fixed by typing it (cafaf885d8), and the surface is back at 236.check:dual-build-cjs-loads, after a fullturbo run buildof./packages/*and./packages/*/*.Lint, narrowed and proven:
pnpm exec eslint --no-inline-config --format jsonover the 6 changed.tsfiles at43ae6c1fccfound 6 files, 0 errors, 0 warnings. Three facts make this narrowing a measurement:isPathIgnoredanswersfalsefor all 6.parserOptions.projectandprojectServicearenullfor every file, so type-aware linting is not enabled.Changeset
.changeset/20783-groupby-structured-json-refused.md:@objectstack/objectqlminor, a BREAKING banner,Clause-②: no (narrowing), and exactly one ADR-0087 marker,not-required (no-migration-prescription), in the form PR #20781's changeset uses. It carries no rewrite table and no arrow: the body states what an author sees now, why, who is affected and what is unchanged. No export or published type changes: the door module is internal, and@objectstack/objectql's root and./coreexports are unchanged.Acceptance notes
NativeSQLStrategyanswers one group per serialized document on SQLite and 500 on PostgreSQL. It never reaches this door. The fix lands inpackages/services/service-analytics, which this claim does not touch.groupByon amultiple: trueselect throughPOST /api/v1/data/:object/query: memory answers one group per array, SQLite one per serialized array, and PostgreSQL 500 (json equality). Not this card's class, and a multi-value field has a plausible other meaning (a bucket per member), so it is not refused here.count_distinctover ajsonfield through the same door: memory 3, SQLite 3, PostgreSQL 500.AGGREGATE_FIELD_TYPE_COMPATIBILITYacceptscount_distinctfor everyFieldTypeon the ground that every backend gives one answer. PostgreSQL does not for json.service-analyticshas two dynamicengine.aggregatecallers:resolveFkAttr(groupBy: ['id', attr], a cross-object dimension attribute) and the display-label pass (groupBy: ['id', displayField]). If that attribute or display field is structured-JSON, they now answer this 400. Before, by reading and not measured: memory and SQLite grouped underidfirst, so one row per record, and PostgreSQL would have answered the same json-equality 500. No example or platform dataset names such a dimension (the census above).groupBy[0]), not the cube member the caller wrote (c20783.meta). Carrier: the native-SQL sibling card, which touches the same face.countas a string ("2") where SQLite returned a number, in the stand-in wiring above. Observed only; not measured through the HTTP door.Generated by Claude Code