diff --git a/.changeset/20802-dotted-relation-route.md b/.changeset/20802-dotted-relation-route.md new file mode 100644 index 00000000000..eceb7262af3 --- /dev/null +++ b/.changeset/20802-dotted-relation-route.md @@ -0,0 +1,7 @@ +--- +"@objectstack/metadata-protocol": patch +--- + +fix(metadata-protocol): a dotted relation filter path names the nested-relation form as the route + +A filter key such as `account.industry` — a dotted path through a relation field — is still refused with `INVALID_FIELD` / 400 at the query parameter door. Its words no longer say a filter reaches only the object's own columns, which stopped being true when the engine began serving the nested-relation form in `where`: they now name that form, `{ "account": { "industry": VALUE } }`, beside the denormalise remedy, in the same words as the engine's own refusal. diff --git a/.changeset/20802-nested-relation-filter-served.md b/.changeset/20802-nested-relation-filter-served.md new file mode 100644 index 00000000000..ebb4387159d --- /dev/null +++ b/.changeset/20802-nested-relation-filter-served.md @@ -0,0 +1,28 @@ +--- +"@objectstack/objectql": minor +--- + +feat(objectql): the nested-relation filter `{ relation: { field: value } }` is served in `where`, lowered at the engine's filter seam — the drivers receive `$in` / `$contains` and are unchanged + +Clause-②: yes (widening) + +A condition on a related record's own fields, written beneath a relation field of the queried object — `{ "account": { "industry": "tech" } }` beneath a `lookup` — is now answered by the engine in `where`, on every verb that takes one (`find`, `findOne`, `count`, `aggregate`, `update`, `delete`) and by `judgeFilter`. It was refused with `INVALID_FILTER` / 400 until now; this supersedes the relation-field paragraph of the pending `20745-nested-object-door` entry. + +**How it is answered.** The engine reads the related object with the condition, then matches the relation field against the ids that read returns, and the drivers receive only that: `{ "account": { "$in": [ids] } }` on a single-valued relation, and on a multi-valued one (`multiple: true`) an `$or` of one `$contains` per id, so it matches on any member. The relation types are `lookup`, `master_detail`, `user` and `tree`. It composes as written inside `$and` / `$or` / `$not`, and the `FilterArray` sugar lowers to it too. No related record matching selects no rows; under `$not`, a record whose relation is empty satisfies the negation. + +**As the caller.** The related read is the engine's own `find` on the related object with the caller's execution context, so that object's access check, row scope and field permissions apply exactly as they do to a direct read of it. A condition on a field the caller cannot read is refused by the same check that refuses a direct filter on it (`PERMISSION_DENIED` / 403, naming the field), never answered with an empty list; a related record the caller cannot see matches no condition. + +**Bounded.** At most `RELATION_FILTER_ID_CAP` (1,000, exported) related ids feed one condition. A condition matching more is refused with `INVALID_FILTER` / 400, naming the cap, the related object and the two-step route — never run over a cut-off list. + +**Still refused, in the engine's words (`INVALID_FILTER` / 400, before any read):** a second level (a relation condition beneath the related object's own relation field, or a dotted key inside the condition), a key the related object does not declare, an empty condition `{}`, and a related object that is not registered. An aggregation's own `filter` and `having` keep refusing the form, and their words now name `where` as the place it is served. The dotted spelling `{ "account.industry": "tech" }` stays refused with `INVALID_FIELD` / 400, and its words now name the nested form to write instead. The structured-JSON and scalar-field refusals are unchanged. + +Measured through `POST /api/v1/data/:object/query` on SQLite and PostgreSQL 16 (owner `u1`, region NA, on `d1` and `d3`; `d4` has no owner): + +| `where` | before | now | +|:--|:--|:--| +| `{ owner: { region: "NA" } }` on a `lookup`, and its `master_detail` and multiple-lookup twins | `INVALID_FILTER` / 400 | `d1`, `d3` | +| `{ parent: { title: "a" } }` on a `tree` field | `INVALID_FILTER` / 400 | `d2`, `d3` | +| `{ $not: { owner: { region: "NA" } } }` | `INVALID_FILTER` / 400 | `d2`, `d4` | +| `{ owner: { region: "APAC" } }` (no owner matches) | `INVALID_FILTER` / 400 | no rows | + +On the in-memory driver, a multi-valued relation's `$contains` still matches a stored id by substring per element, so there an id that is a substring of another stored id (`u1` inside `u10`) also matches; SQLite and PostgreSQL match the element. diff --git a/.changeset/20802-nested-relation-prose.md b/.changeset/20802-nested-relation-prose.md new file mode 100644 index 00000000000..30c30a9cd7a --- /dev/null +++ b/.changeset/20802-nested-relation-prose.md @@ -0,0 +1,7 @@ +--- +"@objectstack/spec": patch +--- + +docs(spec): the `FilterCondition` docblock says the query engine serves the nested-relation form in `where` + +`FilterCondition`'s form 4, `{ relation: { field: value } }`, now states the served semantics: the engine reads the related object with the condition as the caller (its row scope and field permissions apply), matches the relation field against the ids it returns (`$in`, or any member on a multi-valued relation), reaches one level, and refuses a condition matching more related records than its cap rather than truncating. The `QueryFilter` example shows the form again, and the `Filter` nested arm's comment says the engine serves one level. The type and the schema are unchanged. diff --git a/content/docs/kernel/contracts/data-engine.mdx b/content/docs/kernel/contracts/data-engine.mdx index 2f8c4dd50e2..bb096e98177 100644 --- a/content/docs/kernel/contracts/data-engine.mdx +++ b/content/docs/kernel/contracts/data-engine.mdx @@ -178,18 +178,40 @@ where: { ], } -// A condition on a related record's fields: filter the related object first, -// then match the lookup against the ids it returns +// Nested relation filter +where: { + account: { industry: 'tech' }, +} +``` + +A nested relation filter is a condition on a related record's own fields, written +beneath a relation field (`lookup`, `master_detail`, `user` or `tree`, single or +multiple). The engine serves it in `where`, the same on every driver: it reads the +related object with the condition **as the caller** — that object's row scope and field +permissions apply, so a condition on a field the caller cannot read is refused +(`PERMISSION_DENIED` / 403), never answered with an empty list — and then matches the +relation field against the ids that read returns: `$in` on a single-valued relation, any +member on a multi-valued one (`multiple: true`). No related record matching selects no +rows; under `$not`, a record whose relation is empty satisfies the negation. + +It reaches **one level**: every key must be a field the related object declares +(`{ account: { owner: { region: 'NA' } } }` and the dotted `{ account: { 'owner.region': 'NA' } }` +are refused), and a condition matching more than 1,000 related records is refused with +`INVALID_FILTER` / 400 rather than run over a cut-off list. For either, run the two steps +yourself — filter the related object, then match its ids: + +```typescript const tech = await engine.find('account', { where: { industry: 'tech' }, fields: ['id'] }); where: { account: { $in: tech.map((a) => a.id) } } +// On a multi-valued lookup, one $contains per id: +where: { $or: tech.map((a) => ({ accounts: { $contains: a.id } })) } ``` -A plain object with no `$` operator beneath a field — `{ account: { industry: 'tech' } }` -under a lookup, `{ meta: { a: 1 } }` under a `json` field — is refused with -`INVALID_FILTER` / 400 on every driver: no driver follows a relation into the related -object, and a whole-value match on a JSON value means something different on each -backend. On a multi-valued lookup (`multiple: true`), match each id with `$contains` -(an `$or` of those for several ids) instead of `$in`. +An aggregation's own `filter` and `having` do not serve the nested form (put the +condition in `where`), and a plain object with no `$` operator beneath a `json` field — +`{ meta: { a: 1 } }`, a whole-value match — is refused with `INVALID_FILTER` / 400 on +every driver. A dotted path (`{ 'account.industry': 'tech' }`) is refused with +`INVALID_FIELD` / 400: write it nested instead. **Supported operators:** `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$between`, `$contains`, `$notContains`, `$startsWith`, `$endsWith`, `$null`, `$exists` diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index 8eeae62ea43..cdf1a44626c 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -255,6 +255,21 @@ const TYPE_TO_FORM: Readonly> = METADATA_FORM_REGISTRY; * spelling-tolerant lookup this comment has rejected since #4432, and it would * still persist the row under the plural `type`. */ +/** + * [#20802] The nested-relation spelling of a dotted relation path, as the + * dotted filter refusal names it: `'owner.region'` → `{ "owner": { "region": + * VALUE } }` — the form the engine serves at `where`. A path deeper than one + * relation is given the generic one-level shape. The engine door + * (`@objectstack/objectql`'s `nestedRelationRoute`) words it the same: + * `query-expression-conformance.test.ts` holds the two doors' routes equal. + */ +function nestedRelationRoute(dotted: string): string { + const [head, ...rest] = dotted.split('.'); + return rest.length === 1 + ? `{ "${head}": { "${rest[0]}": VALUE } }` + : `{ "${head}": { "FIELD": VALUE } }, one level deep`; +} + function canonicalMetaType(type: string): string { return canonicalMetaUrlType(type); } @@ -9878,10 +9893,15 @@ export class ObjectStackProtocolImplementation implements const headDef = gate.fields[head]; const headClass = classifyDottedFilterHead(headDef); const headType = String(headDef?.type ?? ''); + // [#20802] The relation head names the route the engine now + // SERVES — the condition nested beneath the relation field — in the + // engine door's words (`@objectstack/objectql`'s + // `assertFilterIsMaterializable`). One vocabulary across the doors. const body = headClass === 'relation' ? `filters on '${first}', which follows the relationship '${head}' into another ` - + `object — a filter reaches only columns of '${object}' itself, and '${head}' ` - + 'stores the related record\'s id, not an embedded document' + + `object as a dotted path, and '${head}' stores the related record's id, not an ` + + 'embedded document — to filter on the related record\'s fields, nest the condition ' + + `beneath the relation field: ${nestedRelationRoute(first)}` : headClass === 'virtual' ? `filters on '${first}', a dotted path whose head '${head}' is a virtual ` + `'${headType}' field on object '${object}' — its value is computed on read, ` diff --git a/packages/objectql/src/engine-nested-object-door.test.ts b/packages/objectql/src/engine-nested-object-door.test.ts index 603b3df64b4..f432f27e057 100644 --- a/packages/objectql/src/engine-nested-object-door.test.ts +++ b/packages/objectql/src/engine-nested-object-door.test.ts @@ -30,6 +30,14 @@ * measured (`d1`, `d3` for `$in` on a lookup and `$contains` on a multiple * lookup) and are not pinned in a new suite: that driver's test consumers are * a ruled, closed census (`check:driver-memory-census`). + * + * [#20802] The relation rows at `where` are SERVED now (maintainer ruling, + * letter A): the engine lowers the nested-relation form by reading the related + * object, and `engine-nested-relation-lowering.test.ts` pins it. What stays + * here is what still refuses: the structured-JSON and provisioned-`id` rows, + * `{}` beneath a relation (it names no field of the related object), and the + * relation rows at an aggregation's own `filter` and at `having`, whose words + * now say the form is served in `where`. */ import { describe, it, expect, beforeEach } from 'vitest'; @@ -74,15 +82,6 @@ const PROBE = { const OWNER_OBJECT = { name: OWNER, label: 'Owner', fields: { region: { name: 'region', type: 'text' } } }; -/** field · declared type · the related object the words name · whether the route is `$contains`. */ -const RELATIONS: ReadonlyArray = [ - ['owner', 'lookup', OWNER, false], - ['owners', 'lookup', OWNER, true], - ['boss', 'master_detail', OWNER, false], - ['assignee', 'user', 'sys_user', false], - ['parent', 'tree', OBJECT, false], -]; - /** field · declared type — structured-JSON columns. */ const JSONS: ReadonlyArray = [ ['meta', 'json'], @@ -153,25 +152,6 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p // ── where ──────────────────────────────────────────────────────────────── - it('refuses the nested-relation form beneath every relation type, single or multiple, naming the route that works — no read', async () => { - for (const [field, type, related, multiple] of RELATIONS) { - const err = await refusalOf(engine.find(OBJECT, { where: { [field]: { region: 'NA' } } as FilterCondition })); - expect(envelopeOf(err), field).toEqual(ENVELOPE); - expect(err!.httpStatus, field).toBe(400); - expect(err!.message, field).toMatch(/^find\('nested_object_probe'\): /); - expect(err!.message, field).toContain(`filter on '${field}'`); - expect(err!.message, field).toContain(`at where.${field},`); - expect(err!.message, field).toContain(`beneath the declared ${type} field '${field}'`); - expect(err!.message, field).toContain('nested-relation form'); - expect(err!.message, field).toContain('NOT applied'); - expect(err!.message, field).toContain(`Filter the related object '${related}' first`); - expect(err!.message, field).toContain( - multiple ? `{ "${field}": { "$contains": ID } }` : `{ "${field}": { "$in": [ID, …] } }`, - ); - } - expect(reads).toHaveLength(0); - }); - it('refuses a whole-value object beneath every structured-JSON type, naming what every driver answers alike — no read', async () => { for (const [field, type] of JSONS) { const err = await refusalOf(engine.find(OBJECT, { where: { [field]: { a: 1 } } as FilterCondition })); @@ -195,20 +175,23 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p }); it('refuses {} beneath a relation and a JSON column too, in the engine\'s words rather than each driver\'s', async () => { - for (const [where, words] of [ - [{ owner: {} }, 'nested-relation form'], - [{ meta: {} }, 'whole-value match'], + for (const [where, empty, words] of [ + // [#20802] Served at `where` otherwise — `{}` names no field of the related object. + [{ owner: {} }, '(no keys)', 'names no field of the related object'], + [{ meta: {} }, 'an empty object {}', 'whole-value match'], ] as const) { const err = await refusalOf(engine.find(OBJECT, { where: where as FilterCondition })); expect(envelopeOf(err), JSON.stringify(where)).toEqual(ENVELOPE); - expect(err!.message, JSON.stringify(where)).toContain('an empty object {}'); + expect(err!.message, JSON.stringify(where)).toContain(empty); expect(err!.message, JSON.stringify(where)).toContain(words); } expect(reads).toHaveLength(0); }); it('covers every engine verb that collects a filter — read and write sides — and the judge', async () => { - for (const where of [{ owner: { region: 'NA' } }, { meta: { a: 1 } }] as FilterCondition[]) { + // [#20802] The relation row is served at `where` on every verb now: + // `engine-nested-relation-lowering.test.ts`. + for (const where of [{ ship_to: { city: 'Paris' } }, { meta: { a: 1 } }] as FilterCondition[]) { const path = `at where.${Object.keys(where)[0]},`; for (const call of [ () => engine.find(OBJECT, { where }), @@ -230,9 +213,9 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p it('reaches inside $and / $or / $not, and answers the FilterArray sugar alike', async () => { const cases: ReadonlyArray = [ - [{ $and: [{ title: 'a' }, { owner: { region: 'NA' } }] }, 'where.$and[1].owner'], + [{ $and: [{ title: 'a' }, { ship_to: { city: 'Paris' } }] }, 'where.$and[1].ship_to'], [{ $or: [{ meta: { a: 1 } }, { amount: 30 }] }, 'where.$or[0].meta'], - [{ $not: { boss: { region: 'NA' } } }, 'where.$not.boss'], + [{ $not: { spec: { k: 1 } } }, 'where.$not.spec'], ]; for (const [where, path] of cases) { const err = await refusalOf(engine.find(OBJECT, { where })); @@ -240,10 +223,10 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p expect(err!.message, path).toContain(`at ${path},`); } const sugar = await refusalOf( - engine.find(OBJECT, { where: [['owner', '=', { region: 'NA' }]] } as unknown as EngineQueryOptions), + engine.find(OBJECT, { where: [['meta', '=', { a: 1 }]] } as unknown as EngineQueryOptions), ); expect(envelopeOf(sugar)).toEqual(ENVELOPE); - expect(sugar!.message).toContain('at where.owner,'); + expect(sugar!.message).toContain('at where.meta,'); expect(reads).toHaveLength(0); }); @@ -333,9 +316,9 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p }); const DOORS: ReadonlyArray<{ door: string; query: Record }> = [ - { door: 'where object', query: { where: { owner: { region: 'NA' } } } }, + { door: 'where object', query: { where: { ship_to: { city: 'Paris' } } } }, { door: '$filter string', query: { $filter: JSON.stringify({ meta: { a: 1 } }) } }, - { door: 'filter AST', query: { filter: [['owner', '=', { region: 'NA' }]] } }, + { door: 'filter AST', query: { filter: [['meta', '=', { a: 1 }]] } }, ]; it.each(DOORS)('the $door door refuses it', async ({ query }) => { diff --git a/packages/objectql/src/engine-nested-relation-lowering.test.ts b/packages/objectql/src/engine-nested-relation-lowering.test.ts new file mode 100644 index 00000000000..b8c300a2716 --- /dev/null +++ b/packages/objectql/src/engine-nested-relation-lowering.test.ts @@ -0,0 +1,363 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20802] The nested-relation form `{ relation: { field: value } }` is SERVED + * at `where`: the engine reads the related object with the condition, as the + * caller, and hands every driver the filter the two-step route would have + * written — `$in` on a single-valued relation, an `$or` of `$contains` per id + * on a multi-valued one. The drivers never see the nested form (ADR-0053 D-D1 + * item 5, D4 (b)). + * + * This suite pins the engine's half with a recording driver: what the related + * read asks for (the object, the condition, `id` only, one row past the cap, + * the caller's context), what the driver then receives — asserted EQUAL to what + * the engine sends for the hand-written two-step filter over the same ids, so + * the shared lowering (NULL-safe `$not`, …) is the one both spellings get — + * the cap, the structural refusals the first cut keeps (one level, declared + * keys, a registered related object), the two positions that keep the + * no-operator-object refusal, and the judge. The rows each driver answers are + * `@objectstack/rest`'s `data-nested-object-door.test.ts` (SQLite, PostgreSQL) + * and its `data-nested-relation-permission.test.ts` (the real security layer). + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import type { EngineAggregateOptions, EngineQueryOptions, FilterCondition } from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; +import { RELATION_FILTER_ID_CAP } from './relation-filter-lowering.js'; + +const OBJECT = 'nested_rel_ledger'; +const OWNER = 'nested_rel_owner'; +const USER = 'sys_user'; + +const OWNER_OBJECT = { + name: OWNER, + label: 'Owner', + fields: { + region: { name: 'region', type: 'text' }, + score: { name: 'score', type: 'number' }, + account: { name: 'account', type: 'lookup', reference: OBJECT }, + meta: { name: 'meta', type: 'json' }, + }, +}; + +/** A stand-in for the platform's user object: `user` fields point at `sys_user` by type. */ +const USER_OBJECT = { name: USER, label: 'User', fields: { region: { name: 'region', type: 'text' } } }; + +const LEDGER = { + name: OBJECT, + label: 'Ledger', + fields: { + title: { name: 'title', type: 'text' }, + owner: { name: 'owner', type: 'lookup', reference: OWNER }, + boss: { name: 'boss', type: 'master_detail', reference: OWNER }, + owners: { name: 'owners', type: 'lookup', reference: OWNER, multiple: true }, + assignee: { name: 'assignee', type: 'user' }, + parent: { name: 'parent', type: 'tree', reference: OBJECT }, + stray: { name: 'stray', type: 'lookup', reference: 'nested_rel_missing' }, + meta: { name: 'meta', type: 'json' }, + }, +}; + +/** field · declared type · the related object it reads · a condition on that object. */ +const SINGLE: ReadonlyArray]> = [ + ['owner', 'lookup', OWNER, { region: 'NA' }], + ['boss', 'master_detail', OWNER, { region: 'NA' }], + ['assignee', 'user', USER, { region: 'NA' }], + ['parent', 'tree', OBJECT, { title: 'a' }], +]; + +interface SeenRead { object: string; ast: any } + +/** + * A recording driver. A read of the object named in `answers` returns those + * rows (the related read's matches); every other read returns none. Writes + * are recorded, never applied. + */ +function makeRecordingDriver() { + const reads: SeenRead[] = []; + const writes: SeenRead[] = []; + const answers = new Map>>(); + const rowsOf = (o: string) => answers.get(o) ?? []; + const driver: any = { + name: 'recording', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(o: string, ast: any) { reads.push({ object: o, ast }); return rowsOf(o); }, + async findOne(o: string, ast: any) { reads.push({ object: o, ast }); return rowsOf(o)[0] ?? null; }, + async count(o: string, ast: any) { reads.push({ object: o, ast }); return rowsOf(o).length; }, + async create(_o: string, data: Record) { return { ...data, id: data.id ?? 'r_1' }; }, + async update(_o: string, id: string, data: Record) { return { ...data, id }; }, + async updateMany(o: string, ast: any) { writes.push({ object: o, ast }); return 0; }, + async delete() { return true; }, + async deleteMany(o: string, ast: any) { writes.push({ object: o, ast }); return 0; }, + async bulkCreate(o: string, batch: Record[]) { return Promise.all(batch.map((r) => this.create(o, r))); }, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, reads, writes, answers }; +} + +type Thrown = (Error & { code?: string; status?: number }) | null; +const refusalOf = async (p: Promise): Promise => p.then(() => null, (e: any) => e); +const envelopeOf = (err: Thrown) => ({ code: err?.code, status: err?.status }); +const INVALID_FILTER = { code: 'INVALID_FILTER', status: 400 }; + +const idRows = (...ids: string[]) => ids.map((id) => ({ id })); + +describe('[#20802] the nested-relation form is lowered at the engine\'s where seam', () => { + let engine: ObjectQL; + let rec: ReturnType; + + beforeEach(async () => { + rec = makeRecordingDriver(); + engine = new ObjectQL(); + engine.registerDriver(rec.driver, true); + await engine.init(); + engine.registry.registerObject(OWNER_OBJECT as any, 'test'); + engine.registry.registerObject(USER_OBJECT as any, 'test'); + engine.registry.registerObject(LEDGER as any, 'test'); + }); + + /** The `where` the driver received for the object's own read, after the related read. */ + const outerWhereOf = async (where: unknown, verb: 'find' = 'find') => { + rec.reads.length = 0; + await engine[verb](OBJECT, { where } as EngineQueryOptions); + const outer = rec.reads.filter((r) => r.object === OBJECT); + return outer[outer.length - 1]?.ast?.where; + }; + + /** What the driver receives for the hand-written two-step filter — the reference. */ + const twoStepWhereOf = async (where: FilterCondition) => { + const saved = new Map(rec.answers); + rec.answers.clear(); + rec.reads.length = 0; + await engine.find(OBJECT, { where }); + const outer = rec.reads.filter((r) => r.object === OBJECT); + for (const [k, v] of saved) rec.answers.set(k, v); + return outer[outer.length - 1]?.ast?.where; + }; + + it('every single-valued relation type: the related object is read as the condition says, then $in on its ids', async () => { + for (const [field, , target, condition] of SINGLE) { + rec.answers.clear(); + rec.answers.set(target, idRows('r1', 'r3')); + rec.reads.length = 0; + await engine.find(OBJECT, { where: { [field]: condition } as FilterCondition }); + const related = rec.reads.filter((r) => r.object === target && r.ast?.limit === RELATION_FILTER_ID_CAP + 1); + expect(related, field).toHaveLength(1); + expect(related[0].ast.where, field).toEqual(condition); + expect(related[0].ast.fields, field).toEqual(['id']); + const outer = rec.reads[rec.reads.length - 1]; + expect(outer.object, field).toBe(OBJECT); + expect(outer.ast.where, field).toEqual({ [field]: { $in: ['r1', 'r3'] } }); + } + }); + + it('a multi-valued relation matches on ANY member: an $or of $contains per id, the spec\'s any-of spelling', async () => { + rec.answers.set(OWNER, idRows('u1', 'u3')); + expect(await outerWhereOf({ owners: { region: 'NA' } })).toEqual({ + $and: [{ $or: [{ owners: { $contains: 'u1' } }, { owners: { $contains: 'u3' } }] }], + }); + // Beside the node's own `$and`, the lowered clause is appended, never overwriting it. + expect(await outerWhereOf({ $and: [{ title: 'a' }], owners: { region: 'NA' } })).toEqual({ + $and: [{ title: 'a' }, { $or: [{ owners: { $contains: 'u1' } }, { owners: { $contains: 'u3' } }] }], + }); + }); + + it('no related record matches: $in [] / $or [] — FALSE, never an absent predicate', async () => { + rec.answers.set(OWNER, []); + expect(await outerWhereOf({ owner: { region: 'APAC' } })).toEqual({ owner: { $in: [] } }); + expect(await outerWhereOf({ owners: { region: 'APAC' } })).toEqual({ $and: [{ $or: [] }] }); + }); + + it('composes as written inside $and / $or / $not, and the FilterArray sugar alike — exactly the two-step route\'s driver input', async () => { + rec.answers.set(OWNER, idRows('u1')); + const cases: ReadonlyArray = [ + [{ $or: [{ owner: { region: 'NA' } }, { title: 'a' }] }, { $or: [{ owner: { $in: ['u1'] } }, { title: 'a' }] }], + [{ $and: [{ title: 'c' }, { boss: { region: 'NA' } }] }, { $and: [{ title: 'c' }, { boss: { $in: ['u1'] } }] }], + [{ title: 'c', owner: { region: 'NA' } }, { title: 'c', owner: { $in: ['u1'] } }], + // `$not`: the lowered leaf takes the shared lowering's NULL-safe negation, + // so a row whose relation is empty satisfies it — the two-step route's reading. + [{ $not: { owner: { region: 'NA' } } }, { $not: { owner: { $in: ['u1'] } } }], + [{ $not: { owners: { region: 'NA' } } }, { $not: { $and: [{ $or: [{ owners: { $contains: 'u1' } }] }] } }], + [[['owner', '=', { region: 'NA' }]], { owner: { $in: ['u1'] } }], + ]; + for (const [nested, twoStep] of cases) { + const reference = await twoStepWhereOf(twoStep); + expect(await outerWhereOf(nested), JSON.stringify(nested)).toEqual(reference); + } + }); + + it('covers every verb that takes a where — the driver never receives the nested form — and the judge admits it', async () => { + rec.answers.set(OWNER, idRows('u1')); + const where = { owner: { region: 'NA' } } as FilterCondition; + const lowered = { owner: { $in: ['u1'] } }; + for (const [verb, call] of [ + ['find', () => engine.find(OBJECT, { where })], + ['findOne', () => engine.findOne(OBJECT, { where })], + ['count', () => engine.count(OBJECT, { where })], + ['aggregate', () => engine.aggregate(OBJECT, { where, aggregations: [{ function: 'count', alias: 'n' }] } as EngineAggregateOptions)], + ] as const) { + rec.reads.length = 0; + await call(); + const outer = rec.reads.filter((r) => r.object === OBJECT); + expect(outer.length, verb).toBeGreaterThan(0); + for (const read of outer) expect(read.ast.where, verb).toEqual(lowered); + } + for (const [verb, call] of [ + ['update', () => engine.update(OBJECT, { title: 'x' }, { where, multi: true })], + ['delete', () => engine.delete(OBJECT, { where, multi: true })], + ] as const) { + rec.writes.length = 0; + await call(); + expect(rec.writes, verb).toHaveLength(1); + expect(rec.writes[0].ast.where, verb).toEqual(lowered); + } + expect(engine.judgeFilter(OBJECT, where)).toEqual({ ok: true }); + expect(engine.judgeFilter(OBJECT, { owners: { region: 'NA' } })).toEqual({ ok: true }); + }); + + it('placeholders in the condition resolve against the caller before the related read', async () => { + rec.answers.set(OWNER, idRows('u9')); + rec.reads.length = 0; + await engine.find(OBJECT, { where: { owner: { id: '{current_user_id}' } }, context: { userId: 'u9' } } as EngineQueryOptions); + const related = rec.reads.find((r) => r.object === OWNER); + expect(related?.ast.where).toEqual({ id: 'u9' }); + }); + + it('the related read runs AS THE CALLER, through the middleware chain, and its refusal is the answer — loudly, no outer read', async () => { + const seen: Array<{ object: string; operation: string; context: any }> = []; + engine.registerMiddleware(async (op: any, next: () => Promise) => { + seen.push({ object: op.object, operation: op.operation, context: op.context }); + if (op.object === OWNER && op.context?.userId === 'u_denied') { + const denied = new Error(`query on '${OWNER}' references field(s) not readable by the caller: region`) as Error & { + code?: string; status?: number; + }; + denied.code = 'PERMISSION_DENIED'; + denied.status = 403; + throw denied; + } + await next(); + }); + rec.answers.set(OWNER, idRows('u1')); + + rec.reads.length = 0; + await engine.find(OBJECT, { where: { owner: { region: 'NA' } }, context: { userId: 'u_reader' } } as EngineQueryOptions); + const related = seen.find((s) => s.object === OWNER); + expect(related?.operation).toBe('find'); + expect(related?.context?.userId).toBe('u_reader'); + expect(related?.context?.isSystem).not.toBe(true); + + rec.reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { + where: { owner: { region: 'NA' } }, context: { userId: 'u_denied' }, + } as EngineQueryOptions)); + expect(envelopeOf(err)).toEqual({ code: 'PERMISSION_DENIED', status: 403 }); + expect(err!.message).toContain('region'); + expect(rec.reads.filter((r) => r.object === OBJECT), 'the outer read never ran').toHaveLength(0); + }); + + // ── the cap ────────────────────────────────────────────────────────────── + + it('the cap: past it the filter is REFUSED with the two-step route, never run over a cut-off list; at it, served whole', async () => { + const many = (n: number) => Array.from({ length: n }, (_, i) => ({ id: `u${i}` })); + rec.answers.set(OWNER, many(RELATION_FILTER_ID_CAP + 1)); + rec.reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { where: { owner: { region: 'NA' } } })); + expect(envelopeOf(err)).toEqual(INVALID_FILTER); + expect(err!.message).toMatch(/^find\('nested_rel_ledger'\): /); + expect(err!.message).toContain(`matched more than ${RELATION_FILTER_ID_CAP} records of the related object '${OWNER}'`); + expect(err!.message).toContain('the filter was NOT applied: a cut-off id list would silently drop matching rows'); + expect(err!.message).toContain('{ "owner": { "$in": [ID, …] } }'); + expect(rec.reads.filter((r) => r.object === OBJECT), 'no outer read').toHaveLength(0); + + const multi = await refusalOf(engine.find(OBJECT, { where: { owners: { region: 'NA' } } })); + expect(envelopeOf(multi)).toEqual(INVALID_FILTER); + expect(multi!.message).toContain('{ "owners": { "$contains": ID } }'); + + rec.answers.set(OWNER, many(RELATION_FILTER_ID_CAP)); + const where = await outerWhereOf({ owner: { region: 'NA' } }); + expect((where as any).owner.$in).toHaveLength(RELATION_FILTER_ID_CAP); + }); + + // ── what the first cut keeps refusing ──────────────────────────────────── + + it('keeps refusing, in the engine\'s words and before any read: a second level, a dotted key, an undeclared key, {}, an unregistered related object', async () => { + const cases: ReadonlyArray = [ + [{ parent: { owner: { region: 'NA' } } }, 'where.parent', "'owner' is itself a lookup field of the related object 'nested_rel_ledger'"], + [{ owner: { account: { title: 'a' } } }, 'where.owner', "'account' is itself a lookup field of the related object 'nested_rel_owner'"], + [{ owner: { 'account.title': 'a' } }, 'where.owner', "'account.title' is a dotted path: the condition reaches one level only"], + [{ owner: { regio: 'NA' } }, 'where.owner', "'regio' is not a field of the related object 'nested_rel_owner'"], + [{ owner: {} }, 'where.owner', 'names no field of the related object'], + [{ stray: { region: 'NA' } }, 'where.stray', "no object 'nested_rel_missing' is registered here"], + [{ $or: [{ title: 'a' }, { owner: { regio: 'NA' } }] }, 'where.$or[1].owner', "'regio' is not a field"], + ]; + for (const [where, path, words] of cases) { + rec.reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { where })); + expect(envelopeOf(err), JSON.stringify(where)).toEqual(INVALID_FILTER); + expect(err!.message, JSON.stringify(where)).toContain('puts a nested-relation condition ('); + expect(err!.message, JSON.stringify(where)).toContain(`at ${path}, beneath the declared`); + expect(err!.message, JSON.stringify(where)).toContain(words); + expect(err!.message, JSON.stringify(where)).toContain('The filter was NOT applied.'); + expect(rec.reads, JSON.stringify(where)).toHaveLength(0); + // The judge gives execution's verdict, word for word. + expect(engine.judgeFilter(OBJECT, where), JSON.stringify(where)).toEqual({ ok: false, ...INVALID_FILTER, message: err!.message }); + } + }); + + it('keeps refusing a json field\'s object comparand and the dotted path — the latter now naming the nested route', async () => { + const json = await refusalOf(engine.find(OBJECT, { where: { meta: { a: 1 } } })); + expect(envelopeOf(json)).toEqual(INVALID_FILTER); + expect(json!.message).toContain('whole-value match'); + const dotted = await refusalOf(engine.find(OBJECT, { where: { 'owner.region': 'NA' } })); + expect(envelopeOf(dotted)).toEqual({ code: 'INVALID_FIELD', status: 400 }); + expect(dotted!.message).toContain('nest the condition beneath the relation field: { "owner": { "region": VALUE } }'); + expect(rec.reads).toHaveLength(0); + }); + + it('the related object\'s own doors judge the condition\'s comparands, and the judge asks them too', async () => { + const where = { owner: { score: { $gt: 'abc' } } } as FilterCondition; + const err = await refusalOf(engine.find(OBJECT, { where })); + expect(envelopeOf(err)).toEqual(INVALID_FILTER); + expect(err!.message).toMatch(/^find\('nested_rel_owner'\): filter on 'score'/); + expect(engine.judgeFilter(OBJECT, where)).toEqual({ ok: false, ...INVALID_FILTER, message: err!.message }); + const json = await refusalOf(engine.find(OBJECT, { where: { owner: { meta: { a: 1 } } } })); + expect(envelopeOf(json)).toEqual(INVALID_FILTER); + expect(json!.message).toContain("find('nested_rel_owner'): filter on 'meta'"); + }); + + it('an aggregation\'s own filter and having keep the refusal, and say the form is served in where', async () => { + const perAggregation = await refusalOf(engine.aggregate(OBJECT, { + aggregations: [{ function: 'count', alias: 'all' }, { function: 'count', alias: 'n', filter: { owner: { region: 'NA' } } }], + } as EngineAggregateOptions)); + expect(envelopeOf(perAggregation)).toEqual(INVALID_FILTER); + expect(perAggregation!.message).toContain("the nested-relation form, which the engine serves in 'where' and not in an aggregation's 'filter'"); + expect(perAggregation!.message).toContain('{ "owner": { "$in": [ID, …] } }'); + const multi = await refusalOf(engine.aggregate(OBJECT, { + aggregations: [{ function: 'count', alias: 'all' }, { function: 'count', alias: 'n', filter: { owners: { region: 'NA' } } }], + } as EngineAggregateOptions)); + expect(multi!.message).toContain("Put the condition in 'where' instead: here the stored list of ids is compared as one value"); + expect(multi!.message).not.toContain('$contains'); + const having = await refusalOf(engine.aggregate(OBJECT, { + groupBy: ['owner'], aggregations: [{ function: 'count', alias: 'n' }], having: { owner: { region: 'NA' } }, + } as EngineAggregateOptions)); + expect(envelopeOf(having)).toEqual(INVALID_FILTER); + expect(having!.message).toContain("which the engine serves in 'where' and not in 'having'"); + expect(rec.reads).toHaveLength(0); + }); + + it('CONTROL a filter with no nested-relation condition reaches the driver by reference and triggers no related read', async () => { + for (const where of [ + { owner: { $in: ['u1'] } }, + { owners: { $contains: 'u1' } }, + { owner: 'u1', title: 'a' }, + { meta: { $null: false } }, + ] as FilterCondition[]) { + rec.reads.length = 0; + await engine.find(OBJECT, { where }); + expect(rec.reads, JSON.stringify(where)).toHaveLength(1); + expect(rec.reads[0].object).toBe(OBJECT); + expect(rec.reads[0].ast.where, JSON.stringify(where)).toEqual(where); + } + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 036856ead49..2b09ab7b657 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -39,7 +39,7 @@ import { // FilterCondition` lowering, run once per filter position after the doors and // after token resolution (`resolveThenLowerWhere`), so every driver and the // in-process `having` / per-aggregation evaluator receive the lowered filter. -import { lowerFilterCondition, type FilterLoweringOptions } from '@objectstack/spec/data'; +import { lowerFilterCondition, type FilterCondition, type FilterLoweringOptions } from '@objectstack/spec/data'; // [#5574] D6, executable. The ceiling and the refusal message live in // `packages/spec/src/data/bulk-write-hook-conformance.ts` so BOTH phases and // both verbs enforce one definition; the engine raises, the contract decides. @@ -53,7 +53,18 @@ import { assertTemporalComparandsInterpretable, } from './temporal-comparand-door.js'; import { assertTextOperatorTargetsAreStringCapable } from './text-operator-declared-type-door.js'; -import { narrowHavingNumberComparands, narrowNumberComparands } from './number-comparand-declared-type-door.js'; +import { + mapRelationConditions, + narrowHavingNumberComparands, + narrowNumberComparands, +} from './number-comparand-declared-type-door.js'; +import { + lowerRelationSite, + RELATION_FILTER_ID_CAP, + relationFilterCapError, + type RelationFilterSite, + type RelationReplacement, +} from './relation-filter-lowering.js'; // Seek pagination for the walks that must read EVERY row — the autonumber seed // scan is one (#6249). Shared with `summary-backfill` rather than re-rolled: // the cursor merge is the part that is easy to get subtly wrong. @@ -956,6 +967,7 @@ function lowerWhereFilterArray( operation: string, bag: T, schema?: unknown, + schemaOf?: (name: string) => unknown, ): T { if (!bag) return bag; const where = (bag as Record).where; @@ -1035,7 +1047,13 @@ function lowerWhereFilterArray( // [#20745] …and beneath a relation column (the nested-relation form no // driver serves), a structured-JSON column (a whole-value match the // drivers share no meaning for) and an undeclared `id`, in words per kind. - const numeric = narrowNumberComparands(object, operation, schema, where); + // [#20802] …except the nested-relation form, which this position SERVES: + // beneath a relation column it is admitted against the related object's + // declarations (`schemaOf`, the registry) and kept as written — the engine + // lowers it once it can read (`ObjectQL.lowerRelationConditions`) — or + // refused in words of its own (a key the related object does not declare, + // a second level, a related object that is not registered). + const numeric = narrowNumberComparands(object, operation, schema, where, 'where', { schemaOf }); // [#7872] The comparand-type door, on the OBJECT form. `parseFilterAST` // runs the same walk on everything it lowers or passes through, but // NEITHER door routes an object-form filter through it — Door 1 gates on @@ -1123,7 +1141,7 @@ function lowerWhereFilterArray( // array sugar (`[['amount','>','abc']]`) names numeric fields too. [#20546] // …and lowers `['amount', '=', { a: 1 }]` to the no-operator object its // second arm refuses. - lowered.where = narrowNumberComparands(object, operation, schema, condition); + lowered.where = narrowNumberComparands(object, operation, schema, condition, 'where', { schemaOf }); return lowered as T; } @@ -1241,6 +1259,13 @@ function admissionRefusalOf( * condition. * 2. {@link resolveWhereFilterTokens}: the placeholder resolver. * + * [#20802] Execution then lowers each nested-relation condition by READING + * the related object (`ObjectQL.lowerRelationConditions`); that read admits the + * condition through the related object's own two stages. The judge runs those + * two stages on each condition too — as a `find` on the related object — and + * stops there: which ids the read finds, the cap and the caller's permissions + * on the related object need data and a caller, and are execution's alone. + * * What differs by verb sits BETWEEN or AROUND those stages and judges * something other than `where`: option-key folding and refusal, the driver * lookup (`getDriver`, before stage 1 on the writes and between the stages on @@ -1266,10 +1291,31 @@ function judgeWhereAdmission( where: unknown, schema: unknown, context: Parameters[0], + schemaOf?: (name: string) => unknown, ): EngineFilterJudgement { try { - const admitted = lowerWhereFilterArray(object, operation, { where }, schema); - resolveThenLowerWhere(admitted.where, context, declaredDatetimeLowering(schema)); + const admitted = lowerWhereFilterArray(object, operation, { where }, schema, schemaOf); + const resolved = resolveWhereFilterTokens(admitted.where, context); + // [#20802] Each nested-relation condition the door admitted: execution + // reads the related object with it (`ObjectQL.lowerRelationConditions`), + // and that read admits it through the related object's own doors and + // resolver — so the judge asks the same of it here, as a `find` on the + // related object. What the read then finds (the ids, the cap, the caller's + // permissions) needs data and a caller, and is execution's alone. The rest + // of the filter is lowered with each condition standing for the empty id + // set, which the shared lowering treats as it treats any `$in` / `$or`. + const sites = relationSitesOf(object, operation, resolved, schema, schemaOf); + for (const site of sites) { + const inner = judgeWhereAdmission(site.target, 'find', site.condition, schemaOf?.(site.target), context, schemaOf); + if (!inner.ok) return inner; + } + const related = sites.length === 0 + ? resolved + : mapRelationConditions(object, operation, schema, resolved, { + schemaOf: schemaOf ?? (() => undefined), + replace: (site) => lowerRelationSite(site, []), + }); + lowerFilterCondition(related, declaredDatetimeLowering(schema)); return { ok: true }; } catch (thrown) { const refusal = admissionRefusalOf(thrown); @@ -1278,6 +1324,31 @@ function judgeWhereAdmission( } } +/** + * [#20802] The nested-relation conditions a `where` holds, in walk order: the + * collecting pass of {@link mapRelationConditions} (each admitted site is + * recorded and kept as written, so nothing is rewritten). Empty for a filter + * with none, and for a registry-less host (no field map, no relation column). + */ +function relationSitesOf( + object: string, + operation: string, + where: unknown, + schema: unknown, + schemaOf: ((name: string) => unknown) | undefined, +): RelationFilterSite[] { + const sites: RelationFilterSite[] = []; + if (where == null) return sites; + mapRelationConditions(object, operation, schema, where, { + schemaOf: schemaOf ?? (() => undefined), + replace: (site) => { + sites.push(site); + return { kind: 'value', value: site.condition }; + }, + }); + return sites; +} + /** * [#20082] One operation's effective-permission resolution — see * `ObjectQL.permissionResolution`. `get()` asks the registered resolver at most @@ -8943,6 +9014,7 @@ export class ObjectQL implements IObjectQLEngine { where, this._registry.getObject(object), options?.context, + this.relatedSchemaOf, ); } @@ -11159,15 +11231,114 @@ export class ObjectQL implements IObjectQLEngine { * reason — the position's declared-type reader (`where`: the object's * fields; `having`: the aggregated row's columns). */ - private resolveWhereTokens( + private async resolveWhereTokens( ast: QueryAST | undefined, execCtx: ExecutionContext | undefined, lowering: FilterLoweringOptions, position: 'where' | 'having' = 'where', - ): void { + operation = 'find', + ): Promise { if (!ast || ast[position] == null) return; // [#20157] Through the stage function the judge also calls. - ast[position] = resolveThenLowerWhere(ast[position], execCtx, lowering); + if (position === 'having') { + ast[position] = resolveThenLowerWhere(ast[position], execCtx, lowering); + return; + } + // [#20802] `where` serves the nested-relation form: resolve, then lower + // each relation condition (a read of the related object, as the caller), + // then the shared lowering — {@link resolveRelateThenLowerWhere}. + ast[position] = await this.resolveRelateThenLowerWhere( + ast.object, operation, ast[position], execCtx, lowering, + ); + } + + /** + * [#20802] Stage 2 of `where` admission on every verb, whole: resolve the + * placeholders, then LOWER EACH NESTED-RELATION CONDITION + * ({@link lowerRelationConditions}), then run the shared lowering + * ({@link resolveThenLowerWhere}'s second half, `lowerFilterCondition`). + * + * The order is ADR-0053 D-D1 item 3's, extended by one step: tokens first, + * so a condition on the related object carries the resolved values (one + * instant for the whole filter) into the related read; the relation step + * before the shared lowering, so that lowering reads the `$in` / `$contains` + * the relation step produced — including the NULL-safe `$not` over it — and + * never a nested object whose columns belong to another object. + */ + private async resolveRelateThenLowerWhere( + object: string, + operation: string, + where: W, + execCtx: ExecutionContext | undefined, + lowering: FilterLoweringOptions, + ): Promise { + const resolved = resolveWhereFilterTokens(where, execCtx); + const related = await this.lowerRelationConditions(object, operation, resolved, execCtx); + return lowerFilterCondition(related, lowering); + } + + /** + * [#20802] The registry's declared schema of a related object, by name — + * what the `where` door admits a nested-relation condition's keys against. + */ + private readonly relatedSchemaOf = (name: string): unknown => this._registry.getObject(name); + + /** + * [#20802] Lower every nested-relation condition in one `where` — + * `{ owner: { region: 'NA' } }` beneath a relation field — into a filter + * every driver answers, by READING the related object: the ids of the + * records the condition matches become `$in` on a single-valued relation, or + * an `$or` of `$contains` per id on a multi-valued one + * (`relation-filter-lowering.ts` holds the forms, the cap and the words). + * + * **As the caller.** The read is this engine's own `find` on the related + * object with the caller's execution context — not a driver call and not a + * system read — so the related object's CRUD gate, row scope and field + * permissions apply exactly as they do to a direct read of it, and its own + * doors judge the condition's comparands against its own declarations. A + * refusal from any of them is the answer, loudly; nothing is swallowed. + * + * **Bounded.** The read asks for one id more than + * {@link RELATION_FILTER_ID_CAP}; receiving it, the filter is refused + * (`INVALID_FILTER` / 400, the two-step route in the words), never run over + * a cut-off list. + * + * Returns `where` by reference when it holds no condition — the common path + * reads nothing and allocates nothing beyond one walk. + */ + private async lowerRelationConditions( + object: string, + operation: string, + where: W, + execCtx: ExecutionContext | undefined, + ): Promise { + if (where == null) return where; + const schema = this._registry.getObject(object); + const sites = relationSitesOf(object, operation, where, schema, this.relatedSchemaOf); + if (sites.length === 0) return where; + const context = `${operation}('${object}')`; + const replacements: RelationReplacement[] = []; + for (const site of sites) { + const rows = await this.find(site.target, { + where: site.condition as FilterCondition, + fields: ['id'], + limit: RELATION_FILTER_ID_CAP + 1, + ...(execCtx ? { context: execCtx } : {}), + }); + const matched = Array.isArray(rows) ? rows : []; + if (matched.length > RELATION_FILTER_ID_CAP) { + throw relationFilterCapError(site, context, RELATION_FILTER_ID_CAP); + } + replacements.push(lowerRelationSite( + site, + matched.map((row) => (row as { id?: unknown } | null)?.id).filter((id) => id !== undefined && id !== null), + )); + } + let next = 0; + return mapRelationConditions(object, operation, schema, where, { + schemaOf: this.relatedSchemaOf, + replace: () => replacements[next++], + }); } /** @@ -11186,13 +11357,16 @@ export class ObjectQL implements IObjectQLEngine { * The lowering is copy-on-write too, so a `where` it rewrites lands on the * copy, never on the caller's object. */ - private withResolvedWhere( + private async withResolvedWhere( + object: string, + operation: string, options: T, lowering: FilterLoweringOptions, - ): T { + ): Promise { if (!options || options.where == null) return options; - // [#20157] Through the stage function the judge also calls. - const resolved = resolveThenLowerWhere(options.where, options.context, lowering); + // [#20157] Through the stage function the judge also calls. [#20802] …with + // the nested-relation step between resolution and the shared lowering. + const resolved = await this.resolveRelateThenLowerWhere(object, operation, options.where, options.context, lowering); return resolved === options.where ? options : ({ ...options, where: resolved } as T); } @@ -11405,7 +11579,7 @@ export class ObjectQL implements IObjectQLEngine { // (#4371, three shipped instances in #4370). query = foldEngineOptionAliases(object, 'find', query, ENGINE_QUERY_SLOTS, ENGINE_WIRE_ONLY_SLOTS); rejectUnknownEngineOptions(object, 'find', query, ENGINE_FIND_OPTION_KEYS); - query = lowerWhereFilterArray(object, 'find', query, this._registry.getObject(object)); + query = lowerWhereFilterArray(object, 'find', query, this._registry.getObject(object), this.relatedSchemaOf); this.logger.debug('Find operation starting', { object, query }); const driver = this.getDriver(object); // `object` LAST: the resolved name must win. Spread-first used to let a @@ -11485,7 +11659,7 @@ export class ObjectQL implements IObjectQLEngine { }; // [ADR-0053 D-D1, amended — #5930] Resolve, then lower (the shared // lowering), against the object's declared field types. - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findSchema)); + await this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findSchema), 'where', 'find'); await this.executeWithMiddleware(opCtx, async () => { const hookContext: HookContext = { @@ -11703,7 +11877,7 @@ export class ObjectQL implements IObjectQLEngine { // matters here too: findOne({ sort }) means "first row of THIS order". query = foldEngineOptionAliases(objectName, 'findOne', query, ENGINE_QUERY_SLOTS, ENGINE_WIRE_ONLY_SLOTS); rejectUnknownEngineOptions(objectName, 'findOne', query, ENGINE_FIND_OPTION_KEYS); - query = lowerWhereFilterArray(objectName, 'findOne', query, this._registry.getObject(objectName)); + query = lowerWhereFilterArray(objectName, 'findOne', query, this._registry.getObject(objectName), this.relatedSchemaOf); this.logger.debug('FindOne operation', { objectName }); const driver = this.getDriver(objectName); // `object` after the spread for the same reason as find(); `limit: 1` @@ -11758,7 +11932,7 @@ export class ObjectQL implements IObjectQLEngine { context: mergeReadContext(query?.context, options?.context), }; // [ADR-0053 D-D1, amended — #5930] Resolve, then lower. - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findOneSchema)); + await this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findOneSchema), 'where', 'findOne'); await this.executeWithMiddleware(opCtx, async () => { // [#3195] `findOne` fires the SAME `beforeFind`/`afterFind` hooks as @@ -13366,7 +13540,7 @@ export class ObjectQL implements IObjectQLEngine { // [#5158] Lower before the by-id extraction below reads `where.id`: on an // array that read is `undefined` whatever the caller wrote, so an // `update({ where: [['id','=',x]] })` used to route to the multi-row path. - options = lowerWhereFilterArray(object, 'update', options, this._registry.getObject(object)); + options = lowerWhereFilterArray(object, 'update', options, this._registry.getObject(object), this.relatedSchemaOf); // Expand `{filter-placeholder}` values BEFORE the id is extracted (#3810). // The read path resolves them; without the same call here the SAME filter @@ -13381,7 +13555,7 @@ export class ObjectQL implements IObjectQLEngine { // itself. Resolve first, then extract. // [ADR-0053 D-D1, amended — #5930] …and lowered in the same stage, before // the by-id extraction below reads the result. - options = this.withResolvedWhere(options, declaredDatetimeLowering(this._registry.getObject(object))); + options = await this.withResolvedWhere(object, 'update', options, declaredDatetimeLowering(this._registry.getObject(object))); // [#20308] The insert door's rule, same place: a blank on a // non-string-typed column is `null` before the middleware, the @@ -16045,12 +16219,12 @@ export class ObjectQL implements IObjectQLEngine { rejectUnknownEngineOptions(object, 'delete', options, ENGINE_DELETE_OPTION_KEYS); // [#5158] Same ordering reason as update(): the dispatch decision below // reads `where.id`, which an unlowered array never carries. - options = lowerWhereFilterArray(object, 'delete', options, this._registry.getObject(object)); + options = lowerWhereFilterArray(object, 'delete', options, this._registry.getObject(object), this.relatedSchemaOf); // Expand `{filter-placeholder}` values before the id is extracted — same // reasoning as update() above (#3810). // [ADR-0053 D-D1, amended — #5930] …and lowered in the same stage. - options = this.withResolvedWhere(options, declaredDatetimeLowering(this._registry.getObject(object))); + options = await this.withResolvedWhere(object, 'delete', options, declaredDatetimeLowering(this._registry.getObject(object))); // Extract ID logic mirroring update(): only a SCALAR `where.id` means // "delete one row by primary key". An operator object ({ $in: [...] }, …) @@ -16565,7 +16739,7 @@ export class ObjectQL implements IObjectQLEngine { // `query.where` only, so an unfolded `{ filter }` counted the whole table. query = foldEngineOptionAliases(object, 'count', query, ENGINE_WHERE_SLOTS); rejectUnknownEngineOptions(object, 'count', query, ENGINE_COUNT_OPTION_KEYS); - query = lowerWhereFilterArray(object, 'count', query, this._registry.getObject(object)); + query = lowerWhereFilterArray(object, 'count', query, this._registry.getObject(object), this.relatedSchemaOf); const driver = this.getDriver(object); // The AST must ride on the opCtx so the security/sharing middlewares can @@ -16581,11 +16755,14 @@ export class ObjectQL implements IObjectQLEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; - // [ADR-0053 D-D1, amended — #5930] Resolve, then lower. - this.resolveWhereTokens( + // [ADR-0053 D-D1, amended — #5930] Resolve, then lower. [#20802] …with + // the nested-relation step between the two. + await this.resolveWhereTokens( opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(this._registry.getObject(object)), + 'where', + 'count', ); // The caller's own `where`, placeholders expanded — captured BEFORE the // middleware chain scopes `opCtx.ast.where`, so the find() fallback below @@ -16674,7 +16851,7 @@ export class ObjectQL implements IObjectQLEngine { // `query.where` only, so an unfolded `{ filter }` aggregated every row. query = foldEngineOptionAliases(object, 'aggregate', query, ENGINE_WHERE_SLOTS); rejectUnknownEngineOptions(object, 'aggregate', query, ENGINE_AGGREGATE_OPTION_KEYS); - query = lowerWhereFilterArray(object, 'aggregate', query, this._registry.getObject(object)); + query = lowerWhereFilterArray(object, 'aggregate', query, this._registry.getObject(object), this.relatedSchemaOf); // ADR-0061 `search` → the rows are searched BEFORE they are grouped, by // the one expander `find` runs, at the same point in the sequence (after // the `where` doors above, before the AST is built and tokens resolve). @@ -16958,7 +17135,7 @@ export class ObjectQL implements IObjectQLEngine { // reads each aggregated column's type (`min` / `max` of a `datetime` // field is a `datetime`; a `count` is a number). const rowLowering = declaredDatetimeLowering(this._registry.getObject(object)); - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, rowLowering); + await this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, rowLowering, 'where', 'aggregate'); // [#10576] Filter tokens (`{userId}`-style placeholders, #3810) resolve // in per-aggregation filters exactly as they do in `where` — a filter // position is a filter position, and an unresolved placeholder would be @@ -16988,7 +17165,7 @@ export class ObjectQL implements IObjectQLEngine { // and the other doors judged a string that resolves to a string. { const havingColumnTypes = aggregatedRowColumnTypes(query.groupBy, query.aggregations, declaredFields); - this.resolveWhereTokens( + await this.resolveWhereTokens( opCtx.ast as QueryAST, opCtx.context, { isDatetimeColumn: (column) => havingColumnTypes.get(column) === 'datetime' }, diff --git a/packages/objectql/src/filter-comparand-shape.ts b/packages/objectql/src/filter-comparand-shape.ts index 0eab708c29c..224f06f38a5 100644 --- a/packages/objectql/src/filter-comparand-shape.ts +++ b/packages/objectql/src/filter-comparand-shape.ts @@ -188,6 +188,19 @@ export function assertListComparandShapes( * agreement pin in `query-expression-conformance.test.ts` is what keeps the * duplication honest. */ +/** + * [#20802] The nested-relation spelling of a dotted relation path, as the + * dotted refusal names it: `'owner.region'` → `{ "owner": { "region": VALUE } }`. + * A path deeper than one relation names no field it could nest (the nested + * form reaches one level), so it is given the generic shape and says so. + */ +export function nestedRelationRoute(dotted: string): string { + const [head, ...rest] = dotted.split('.'); + return rest.length === 1 + ? `{ "${head}": { "${rest[0]}": VALUE } }` + : `{ "${head}": { "FIELD": VALUE } }, one level deep`; +} + export function assertFilterIsMaterializable( object: string, operation: string, @@ -221,10 +234,16 @@ export function assertFilterIsMaterializable( const headDef = fields[head] as { type?: unknown } | undefined; const headClass = classifyDottedFilterHead(headDef as never); const headType = String(headDef?.type ?? ''); + // [#20802] The relation head names the route the engine now SERVES: the + // same condition nested beneath the relation field (one level), which + // `where` lowers by reading the related object. The dotted SPELLING stays + // refused — no backend serves the path — and the denormalise remedy stays + // the shared tail, word for word with the ingress door. const body = headClass === 'relation' - ? `filters on '${first}', which follows the relationship '${head}' into another object — ` - + `a filter reaches only columns of '${object}' itself, and '${head}' stores the related ` - + 'record\'s id, not an embedded document' + ? `filters on '${first}', which follows the relationship '${head}' into another object as a ` + + `dotted path, and '${head}' stores the related record's id, not an embedded document — to ` + + 'filter on the related record\'s fields, nest the condition beneath the relation field: ' + + nestedRelationRoute(first) : headClass === 'virtual' ? `filters on '${first}', a dotted path whose head '${head}' is a virtual ${headType} ` + `field on '${object}' — its value is computed on read, so no driver materialises a ` diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index fe857b42167..503647a30f8 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -141,6 +141,11 @@ export type { AdmittedValueShapeViolationTally } from './engine.js'; // type of `ObjectQL.listDatasourceDefs()`. Exported so a consumer sweeping for // `sys_secret` references can name the shape it reads instead of re-declaring it. export type { DatasourceDef } from './engine.js'; +// [#20802] The cap on the related ids one nested-relation condition may feed +// (`{ owner: { region: 'NA' } }`, served at `where`): past it the engine +// refuses the filter rather than truncate it, and its words name this number. +// Exported so a caller running the two-step route itself can page by it. +export { RELATION_FILTER_ID_CAP } from './relation-filter-lowering.js'; // [#16159] `SUMMARY_RECOMPUTE_CODE` joins the class it names. The refusal's own // docblock tells a caller to identify it by `code` rather than `instanceof` // (the two-realm split #14936 measured: this package declares BOTH realms in diff --git a/packages/objectql/src/no-operator-object-door.ts b/packages/objectql/src/no-operator-object-door.ts index 248297a9d5b..aa0361703bb 100644 --- a/packages/objectql/src/no-operator-object-door.ts +++ b/packages/objectql/src/no-operator-object-door.ts @@ -124,6 +124,17 @@ * undeclared key that is not platform-provisioned (the engine's registry-less * tolerance: no second opinion about a name). * + * ## [#20802] The relation kind at `where`: served, not refused + * + * The maintainer ruled the nested-relation form served (#20802, letter A), + * lowered at the engine's filter seam with the drivers untouched. At `where` + * the walk therefore hands a no-operator object beneath a relation column to + * `relation-filter-lowering.ts` instead of refusing it: admitted against the + * related object's declarations, then lowered by reading the related object as + * the caller. The relation words below are raised only at the two positions + * the engine evaluates itself — an aggregation's `filter` and `having` — and + * say so. The `scalar` and `json` kinds are unchanged. + * * @see https://github.com/objectstack-ai/objectstack/issues/20546 * @see https://github.com/objectstack-ai/objectstack/issues/20745 */ @@ -276,11 +287,16 @@ function scalarWords(refusal: NoOperatorObjectRefusal): string { } /** - * [#20745] The relation kind's words. The route is the one every data-path - * driver serves today (measured): the related object's own query, then its - * ids — `$in` on a single-valued column, `$contains` per id on a multi-valued - * one. An aggregated column names no declaration to read either from, so it - * is given the single-valued spelling. + * [#20745] The relation kind's words. [#20802] The nested-relation form is + * SERVED at `where` (`relation-filter-lowering.ts`), so this arm raises these + * words only where it is not: an aggregation's own `filter` and `having`, + * which the engine evaluates itself over rows it already holds. The words send + * the condition to `where`, and name the ids route that works at THIS + * position: `$in` on a single-valued column. A multi-valued column has no + * member test here — the engine's evaluator compares the stored list as one + * value, so neither `$in` nor `$contains` matches a member of it — so `where` + * is its only route. An aggregated column (`having`) names no declaration to + * read either from, so it is given the single-valued spelling. */ function relationWords(refusal: NoOperatorObjectRefusal): string { const { field, column } = refusal; @@ -288,15 +304,17 @@ function relationWords(refusal: NoOperatorObjectRefusal): string { const target = def === undefined ? undefined : referenceTargetOf(def); const related = target === undefined ? 'the related object' : `the related object '${target}'`; const multiple = def !== undefined && isMultiValueField(def); - const match = multiple - ? `{ "${field}": { "$contains": ID } } for one id, an $or of those for several` - : `{ "${field}": { "$in": [ID, …] } }`; + const position = refusal.aggregated ? "'having'" : "an aggregation's 'filter'"; + const route = multiple + ? `Put the condition in 'where' instead: here the stored list of ids is compared as one value, so no ` + + 'operator matches one member of it.' + : `Put the condition in 'where' instead, or filter ${related} first and match '${field}' against ` + + `the ids it returns: { "${field}": { "$in": [ID, …] } }.`; return ( - `beneath ${describeColumn(refusal)} — the nested-relation form, which the engine does not serve. ` - + `The filter was NOT applied. Filter ${related} first, then match '${field}' against the ids it ` - + `returns: ${match}. An object with no "$" operator is filter structure, not a value: '${field}' ` - + 'stores the related record\'s id, no driver follows it into the related object, and an empty ' - + 'answer would read exactly like a real one.' + `beneath ${describeColumn(refusal)} — the nested-relation form, which the engine serves in 'where' ` + + `and not in ${position}. The filter was NOT applied. ${route} An object with no "$" operator is ` + + `filter structure, not a value, here: '${field}' holds the related record's id, and an empty answer ` + + 'would read exactly like a real one.' ); } diff --git a/packages/objectql/src/number-comparand-declared-type-door.ts b/packages/objectql/src/number-comparand-declared-type-door.ts index 6c489c283cb..251644d8b07 100644 --- a/packages/objectql/src/number-comparand-declared-type-door.ts +++ b/packages/objectql/src/number-comparand-declared-type-door.ts @@ -141,6 +141,18 @@ * no meaning for), and a platform-provisioned column the declared map omits * (`id`, …): the same walk and the same three positions, words per kind. * + * ## [#20802] …and the nested-relation arm, at `where` only + * + * The nested-relation form is SERVED at `where` (`relation-filter-lowering.ts`). + * There a no-operator object beneath a relation column is not refused by the + * arm above: it is admitted against the related object's declarations + * (`admitRelationCondition` — one level, declared keys, a registered related + * object) or refused in words of its own. The engine then walks the admitted + * `where` again with this SAME walk ({@link mapRelationConditions}) to collect + * each condition and, after reading the related object, to replace it — so the + * door and the lowering find a condition at the same boundaries by + * construction. The per-aggregation `filter` and `having` keep the refusal. + * * @see numberComparandDoorVerdict — the pure verdict (lane 1, `@objectstack/spec`). * @see https://github.com/objectstack-ai/objectstack/issues/20336 (the contract) * @see https://github.com/objectstack-ai/objectstack/issues/20351 (this door) @@ -167,6 +179,13 @@ import { type NoOperatorObjectColumn, type NoOperatorObjectRefusal, } from './no-operator-object-door.js'; +import { + admitRelationCondition, + relationConditionRefusalMessage, + type RelationConditionRefusal, + type RelationFilterSite, + type RelationReplacement, +} from './relation-filter-lowering.js'; /** The operators whose one comparand is judged — the contract's list, never a re-listing. */ const SCALAR_OPERATORS: ReadonlySet = new Set(NUMBER_COMPARAND_DOOR_SCALAR_OPERATORS); @@ -209,19 +228,57 @@ type FactsOf = (key: string) => KeyFacts | null; interface RefusalSiteContext { readonly aggregated: boolean; readonly boundByDriver: boolean; + /** + * [#20802] This position serves the nested-relation form: a no-operator + * object beneath a relation column is a condition on the related object, + * admitted structurally ({@link admitRelationCondition}) and lowered by the + * engine. Only `where` does; the per-aggregation `filter` and `having` keep + * the no-operator-object refusal (see `relation-filter-lowering.ts`). + */ + readonly servesRelations: boolean; } /** `where`, both spellings: a real declared field, and the driver binds it. */ -const WHERE_SITE: RefusalSiteContext = { aggregated: false, boundByDriver: true }; +const WHERE_SITE: RefusalSiteContext = { aggregated: false, boundByDriver: true, servesRelations: true }; /** The per-aggregation `filter`: a real declared field, but the engine evaluates it itself. */ -const AGGREGATION_FILTER_SITE: RefusalSiteContext = { aggregated: false, boundByDriver: false }; +const AGGREGATION_FILTER_SITE: RefusalSiteContext = { aggregated: false, boundByDriver: false, servesRelations: false }; /** `having`: an aggregated-row column, evaluated by the engine, never bound. */ -const HAVING_SITE: RefusalSiteContext = { aggregated: true, boundByDriver: false }; +const HAVING_SITE: RefusalSiteContext = { aggregated: true, boundByDriver: false, servesRelations: false }; + +/** + * [#20802] What the walk needs, beyond one position's facts, to serve the + * nested-relation form at `where`. + */ +export interface RelationArm { + /** + * The related object's declared schema, by name — the engine's registry. + * Absent, or answering nothing, and a condition beneath a relation is + * refused: its keys cannot be judged against a field map nobody supplied. + */ + readonly schemaOf?: (name: string) => unknown; + /** + * LOWERING only: what replaces an admitted condition. Absent at the door, + * where an admitted condition is kept as written for the engine to lower. + */ + readonly replace?: (site: RelationFilterSite) => RelationReplacement; +} + +/** One walk's context: the position, and — at `where` — the relation arm. */ +interface WalkContext extends RefusalSiteContext { + readonly relations?: RelationArm; + /** + * [#20802] The lowering pass: only admitted relation conditions are + * rewritten. The number arm is not asked again — the door already narrowed + * every comparand it judges, and a second pass must not narrow twice. + */ + readonly lowerOnly?: boolean; +} /** The first refusal the walk met, and which arm raised it. */ type Refusal = | { readonly arm: 'number'; readonly site: NonNumericComparand } - | { readonly arm: 'no-operator-object'; readonly site: NoOperatorObjectRefusal }; + | { readonly arm: 'no-operator-object'; readonly site: NoOperatorObjectRefusal } + | { readonly arm: 'relation'; readonly site: RelationConditionRefusal }; /** The walk's answer: the (possibly narrowed) node, or the first refusal. */ type Outcome = @@ -341,7 +398,9 @@ function judgeFieldSpec( * [#20546] It carries TWO arms, asked in order at every field key: the * no-operator-object arm (`no-operator-object-door.ts` — a plain object with * no `$` key where a scalar column's value belongs), then the number arm - * ({@link judgeFieldSpec}). One traversal, one set of boundaries (the depth + * ({@link judgeFieldSpec}). [#20802] At `where` the first arm hands a relation + * column's object to the nested-relation admission instead, and the lowering + * pass ({@link WalkContext.lowerOnly}) asks that question alone. One traversal, one set of boundaries (the depth * bound, the combinators descended, the `$` and dotted keys skipped), two * questions — the shape the spec's save-door walk takes for its own arms * (`checkFilterConditionComparands`: "One walk, one set of boundaries, `n` @@ -354,9 +413,12 @@ function judgeFieldSpec( * fields beneath it ungated — a hole, not a false 400), and a dotted key names * a path this door does not judge. Copy-on-write throughout. */ -function walkCondition(factsOf: FactsOf, node: unknown, path: string, depth: number, ctx: RefusalSiteContext): Outcome { +function walkCondition(factsOf: FactsOf, node: unknown, path: string, depth: number, ctx: WalkContext): Outcome { if (depth > 32 || !isFilterNode(node)) return kept(node); let out: Record | undefined; + // [#20802] Lowered multi-valued relation conditions, AND-ed into this node + // in place of their field entries once every key has been walked. + let clauses: FilterClause[] | undefined; for (const [key, value] of Object.entries(node)) { const here = `${path}.${key}`; let judged: Outcome; @@ -382,27 +444,64 @@ function walkCondition(factsOf: FactsOf, node: unknown, path: string, depth: num // the nested-relation form, and none shares a meaning for a whole-value // match. if (facts.column !== null && isNoOperatorObject(value)) { - return { - ok: false, - refusal: { - arm: 'no-operator-object', - site: { field: key, column: facts.column, path: here, keys: Object.keys(value), aggregated: ctx.aggregated }, - }, - }; + // [#20802] …except beneath a relation column at a position that serves + // the nested-relation form (`where`): there it is a condition on the + // related object, admitted by its declarations — or refused, loudly, + // in words of its own — and, in the lowering pass, rewritten. + if (ctx.servesRelations && facts.column.kind === 'relation' && facts.column.def !== undefined) { + const admitted = admitRelationCondition(key, facts.column.def, value, here, ctx.relations?.schemaOf); + if (!admitted.ok) return { ok: false, refusal: { arm: 'relation', site: admitted.refusal } }; + const replace = ctx.relations?.replace; + if (!replace) continue; + const replacement = replace(admitted.site); + if (replacement.kind === 'clause') { + (clauses ??= []).push(replacement.clause); + delete (out ??= { ...node })[key]; + continue; + } + judged = kept(replacement.value); + } else { + return { + ok: false, + refusal: { + arm: 'no-operator-object', + site: { field: key, column: facts.column, path: here, keys: Object.keys(value), aggregated: ctx.aggregated }, + }, + }; + } + } else { + if (ctx.lowerOnly) continue; + const meta = facts.number; + // Only a judged field can refuse or narrow a comparand; a `formula` + // whose return type is unreadable is `deferred`, and everything else is + // `not-judged` — the spec's verdict, never a list here. + if (!meta || numberComparandFieldVerdict(meta) !== 'judged') continue; + judged = judgeFieldSpec(meta, key, value, here, ctx); } - const meta = facts.number; - // Only a judged field can refuse or narrow a comparand; a `formula` - // whose return type is unreadable is `deferred`, and everything else is - // `not-judged` — the spec's verdict, never a list here. - if (!meta || numberComparandFieldVerdict(meta) !== 'judged') continue; - judged = judgeFieldSpec(meta, key, value, here, ctx); } if (!judged.ok) return judged; if (judged.value !== value) (out ??= { ...node })[key] = judged.value; } + if (clauses) return kept(withClauses(out ?? { ...node }, clauses)); return kept(out ?? node); } +/** A condition the lowering AND-s into a node in place of a field entry. */ +type FilterClause = Record; + +/** + * [#20802] AND `clauses` into `node` (a copy the walk owns): appended to the + * node's own `$and` when it has one, else as a new `$and`. A node whose `$and` + * is not a list — a shape the doors refuse, kept for them — is wrapped instead, + * so nothing it carries is overwritten. + */ +function withClauses(node: Record, clauses: readonly FilterClause[]): Record { + if (!Object.prototype.hasOwnProperty.call(node, '$and')) return { ...node, $and: [...clauses] }; + const existing = node.$and; + if (Array.isArray(existing)) return { ...node, $and: [...existing, ...clauses] }; + return { $and: [node, ...clauses] }; +} + /** The judged fields of a `where` or a per-aggregation `filter`: the object's declared map. */ function declaredFactsOf(schema: unknown): FactsOf | null { // A registry-less host must not invent a verdict about a field map it cannot @@ -450,7 +549,9 @@ function refuse(context: string, refusal: Refusal): never { throw invalidFilterError( refusal.arm === 'number' ? numberComparandRefusalMessage(refusal.site, context) - : noOperatorObjectRefusalMessage(refusal.site, context), + : refusal.arm === 'relation' + ? relationConditionRefusalMessage(refusal.site, context) + : noOperatorObjectRefusalMessage(refusal.site, context), ); } @@ -477,10 +578,52 @@ export function narrowNumberComparands( schema: unknown, where: W, path = 'where', + relations?: Pick, ): W { const factsOf = declaredFactsOf(schema); if (!factsOf) return where; - const walked = walkCondition(factsOf, where, path, 0, path === 'where' ? WHERE_SITE : AGGREGATION_FILTER_SITE); + // [#20802] At `where`, a condition beneath a relation column is ADMITTED + // here, against the related object's declarations (`relations.schemaOf`), + // and kept as written: the engine lowers it once it can read + // ({@link mapRelationConditions}). Without the related object's schema it is + // refused — the door cannot judge keys against a field map it was not given. + const ctx: WalkContext = path === 'where' + ? { ...WHERE_SITE, relations: relations?.schemaOf ? { schemaOf: relations.schemaOf } : undefined } + : AGGREGATION_FILTER_SITE; + const walked = walkCondition(factsOf, where, path, 0, ctx); + if (!walked.ok) refuse(`${operation}('${object}')`, walked.refusal); + return walked.value as W; +} + +/** + * [#20802] The LOWERING pass over a `where` the door already admitted: the same + * walk, at the same boundaries, asking one question — is this a nested-relation + * condition — and handing each one it admits to `relations.replace`, whose + * answer takes its place (a new value for the field, or a clause AND-ed into + * its node). Nothing else is judged or narrowed again. Copy-on-write: returns + * `where` by reference when it holds no condition. + * + * The engine calls it twice per filter position that holds a condition: once + * to COLLECT the admitted conditions (a `replace` that records each site and + * returns it unchanged, whose output is discarded), and — after the related + * reads — once to REPLACE them, in the same order. One walk, so the two passes + * and the door cannot disagree about where a condition is. + * + * A condition this pass cannot admit is refused in the door's words: the door + * ran the same admission on the same declarations, so that is unreachable + * unless the filter changed between the two (a hook, a middleware) — and then + * the refusal is the right answer rather than a nested form reaching a driver. + */ +export function mapRelationConditions( + object: string, + operation: string, + schema: unknown, + where: W, + relations: Required, +): W { + const factsOf = declaredFactsOf(schema); + if (!factsOf) return where; + const walked = walkCondition(factsOf, where, 'where', 0, { ...WHERE_SITE, relations, lowerOnly: true }); if (!walked.ok) refuse(`${operation}('${object}')`, walked.refusal); return walked.value as W; } diff --git a/packages/objectql/src/protocol-explicit-filter-field-gate.test.ts b/packages/objectql/src/protocol-explicit-filter-field-gate.test.ts index c52552679ee..295003a1229 100644 --- a/packages/objectql/src/protocol-explicit-filter-field-gate.test.ts +++ b/packages/objectql/src/protocol-explicit-filter-field-gate.test.ts @@ -393,13 +393,15 @@ describe('#7534 — unknown field on the EXPLICIT filter axes (real ObjectQL eng // `{owner_id: {region: 'NA'}}` names `owner_id` on THIS object and // `region` on the related one. Judging `region` against this object's // field map would refuse a legitimate relation filter. - // [#20745] The engine refuses the nested-relation form itself (no - // driver serves it) with INVALID_FILTER in its own words. This gate - // still never descends: the answer is not its INVALID_FIELD about - // `region`. + // [#20745] The engine refused the nested-relation form itself for a + // while. [#20802] It serves it now, judging the condition's keys + // against the RELATED object's declarations — here `owner_id` points + // at `sys_user`, which this harness does not register, so the engine + // answers INVALID_FILTER in its own words. This gate still never + // descends: the answer is not its INVALID_FIELD about `region`. const err: any = await find({ where: { owner_id: { region: 'NA' } } }).then(() => null, (e: any) => e); expect({ code: err?.code, status: err?.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); - expect(err.message).toContain('nested-relation form'); + expect(err.message).toContain("puts a nested-relation condition (keys \"region\") at where.owner_id, beneath the declared lookup field 'owner_id'"); expect(err.field).toBeUndefined(); }); diff --git a/packages/objectql/src/query-expression-conformance.test.ts b/packages/objectql/src/query-expression-conformance.test.ts index f482d3daf66..6751c8fc48b 100644 --- a/packages/objectql/src/query-expression-conformance.test.ts +++ b/packages/objectql/src/query-expression-conformance.test.ts @@ -1226,17 +1226,37 @@ describe('#4226 — sort / select / expand on the list path (real ObjectQL engin // inner keys belong to ANOTHER object — the exact shape the collectors // refuse to descend into. If this control answers the dotted verdict's // INVALID_FIELD, that verdict has started judging comparand VALUES, - // which is a different (and wrong) gate. [#20745] The engine refuses - // the form itself — no driver serves it — so both doors answer the - // no-operator-object arm's INVALID_FILTER, in its own words. - for (const run of [ - () => protocol.findData({ object: 'showcase_task', query: { where: { project_id: { name: 'Apollo' } } } }), - () => engine.find('showcase_task', { where: { project_id: { name: 'Apollo' } } }), - ]) { - const err: any = await run().then(() => null, (e: any) => e); - expect({ code: err?.code, status: err?.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); - expect(err.message).toContain('nested-relation form'); - } + // which is a different (and wrong) gate. [#20745] The engine refused + // the form itself for a while — no driver served it. [#20802] It is + // SERVED now: the engine reads the related object with the condition + // and hands the driver `$in` on its ids, so both doors answer the rows + // the undotted `{ project_id: 'p1' }` control answers. + const byFk: any = await protocol.findData({ object: 'showcase_task', query: { where: { project_id: 'p1' } } }); + const wanted = byFk.records.map((r: any) => r.id).sort(); + expect(wanted).toHaveLength(5); + const viaProtocol: any = await protocol.findData({ + object: 'showcase_task', query: { where: { project_id: { name: 'Apollo' } } }, + }); + expect(viaProtocol.records.map((r: any) => r.id).sort()).toEqual(wanted); + const viaEngine = await engine.find('showcase_task', { where: { project_id: { name: 'Apollo' } } }); + expect(viaEngine.map((r: any) => r.id).sort()).toEqual(wanted); + const none = await engine.find('showcase_task', { where: { project_id: { name: 'Gemini' } } }); + expect(none).toEqual([]); + }); + + it('[#20802] both doors\' dotted relation refusals name the SAME served nested route', async () => { + const ingressErr: any = await protocol + .findData({ object: 'showcase_task', query: { where: { 'project_id.name': 'Apollo' } } }) + .then(() => null, (e: unknown) => e); + const engineErr: any = await engine + .find('showcase_task', { where: { 'project_id.name': 'Apollo' } }) + .then(() => null, (e: unknown) => e); + const route = 'nest the condition beneath the relation field: { "project_id": { "name": VALUE } }'; + expect(String(ingressErr?.message)).toContain(route); + expect(String(engineErr?.message)).toContain(route); + // The claim this change made false is gone from both doors. + expect(String(ingressErr?.message)).not.toContain('a filter reaches only columns'); + expect(String(engineErr?.message)).not.toContain('a filter reaches only columns'); }); it('⛔ CONTROL — the structured/JSON head stays UNJUDGED at both doors: the ruled carve-out', async () => { diff --git a/packages/objectql/src/relation-filter-lowering.ts b/packages/objectql/src/relation-filter-lowering.ts new file mode 100644 index 00000000000..868138675b4 --- /dev/null +++ b/packages/objectql/src/relation-filter-lowering.ts @@ -0,0 +1,304 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20802] The NESTED-RELATION form, SERVED: `{ relation: { field: value } }` — + * a plain object with no `$`-operator key beneath a relation field of the + * queried object, whose keys are fields of the related object — is lowered at + * the engine's filter seam into a filter every driver already answers, and the + * drivers never see the nested form (ADR-0053 D-D1 item 5, the #5930 ruling's + * D4 (b): drivers receive lowered input and are not changed). + * + * ## The cut, closed (maintainer ruling on #20802, letter A) + * + * - **Forward, one level.** The condition names a relation field of the + * queried object and fields of the related object. A second relation beneath + * it (`{ owner: { account: { name: 'x' } } }`, or the dotted + * `{ owner: { 'account.name': 'x' } }`) is refused, loudly. The reverse form + * (a parent filtered by its children) is not served here at all. + * - **Where.** `where`, on every verb that takes one (`find`, `findOne`, + * `count`, `aggregate`, `update`, `delete`) and on the judge-only + * `judgeFilter`. An aggregation's own `filter` and `having` keep the + * no-operator-object refusal: the engine evaluates both itself, over rows it + * already holds, and a multi-valued relation has no member test there. + * - **As the caller.** The related object is read through the engine's own + * `find`, under the CALLER's execution context — the related object's CRUD + * gate, row scope and field permissions apply exactly as they do to a direct + * read of it. A condition on a field the caller cannot read is refused by the + * one field-predicate check that refuses it on a direct read (the security + * layer's filter-oracle guard, a loud `PERMISSION_DENIED`), never answered + * with an empty result. There is no second copy of that rule here. + * - **Bounded, loudly.** At most {@link RELATION_FILTER_ID_CAP} related ids + * feed one condition. The read asks for one more than the cap; when it gets + * it, the filter is refused `INVALID_FILTER` / 400 with the two-step route in + * the words — ⛔ never a truncated match. + * - **A multi-valued relation matches on any member.** It is lowered to the + * spec's own any-of spelling — an `$or` of one `$contains` per id (the + * membership reading, `FILTER_OPERATORS`' `$contains` docblock) — because the + * SQL family refuses `$in` over the JSON column a multi-valued relation is + * stored in. A single-valued relation is lowered to `$in`. + * + * ## What the lowered form means where no related record matches + * + * - **No related record matches:** `$in: []` (single-valued) and `{ $or: [] }` + * (multi-valued) — FALSE on every driver (the #5322 identities), so the + * condition selects no row. ⛔ Never an absent predicate, which would select + * every row. + * - **Under `$not`**, the lowered clause is an ordinary `$in` / `$contains` + * leaf, so the shared lowering's NULL-safe `$not` (#5146) applies to it: a + * row whose relation is empty, or points at a record the condition does not + * match or the caller cannot see, satisfies the negation. + * + * ## Where each part lives + * + * This module holds the parts that are not a walk: the cap, the structural + * admission of one condition ({@link admitRelationCondition}), the lowered + * form ({@link lowerRelationSite}) and the words. The ONE filter walk that + * finds a condition — with the column's declaration in hand, at the same + * boundaries as every other arm — is `number-comparand-declared-type-door.ts`'s + * `walkCondition`; the read that turns a condition into ids is the engine's + * (`ObjectQL.lowerRelationConditions`). + * + * @see https://github.com/objectstack-ai/objectstack/issues/20802 + */ + +import { + isMultiValueField, + REFERENCE_VALUE_TYPES, + referenceTargetOf, + type FilterCondition, +} from '@objectstack/spec/data'; +import { invalidFilterError } from './filter-comparand-shape.js'; +import { isNoOperatorObject, provisionedNoOperatorObjectColumn } from './no-operator-object-door.js'; + +/** + * The most related ids one nested-relation condition may feed into the lowered + * filter. Over it, the condition is refused ({@link relationFilterCapError}), + * never truncated. + * + * One named number for every driver: it sits far below the bound-parameter + * ceiling of every SQL dialect the platform runs (a single-valued relation + * binds one parameter per id; a multi-valued one binds one per `$contains` + * arm), and it is the size past which a caller is better served reading the + * related object in pages themselves — the route the refusal names. + */ +export const RELATION_FILTER_ID_CAP = 1000; + +/** One nested-relation condition the walk found and admitted. */ +export interface RelationFilterSite { + /** The relation field of the queried object — the filter key. */ + readonly field: string; + /** The field's declared type (`lookup`, `master_detail`, `user`, `tree`). */ + readonly type: string; + /** The related object the field points at. */ + readonly target: string; + /** The field stores a list of ids (`multiple: true`). */ + readonly multiple: boolean; + /** The condition on the related object: the object beneath the field. */ + readonly condition: Record; + /** The key path the condition sits at (`where.owner`, `where.$or[1].owner`, …). */ + readonly path: string; +} + +/** Why a nested-relation condition cannot be served as written. */ +export type RelationConditionRefusalReason = + | 'unregistered-target' + | 'empty' + | 'dotted-key' + | 'undeclared-key' + | 'second-level'; + +/** A condition the admission refused: the site, the reason, and the key at fault. */ +export interface RelationConditionRefusal { + readonly reason: RelationConditionRefusalReason; + readonly field: string; + readonly type: string; + /** `undefined` when the field declares no related object at all. */ + readonly target: string | undefined; + readonly multiple: boolean; + readonly path: string; + /** The condition's own keys, in order. */ + readonly keys: readonly string[]; + /** The key the refusal is about (`dotted-key`, `undeclared-key`, `second-level`). */ + readonly key?: string; + /** `second-level`: the declared type of the related object's relation field. */ + readonly keyType?: string; +} + +/** The admission's answer for one condition. */ +export type RelationConditionVerdict = + | { readonly ok: true; readonly site: RelationFilterSite } + | { readonly ok: false; readonly refusal: RelationConditionRefusal }; + +/** What replaces one admitted condition in the lowered filter. */ +export type RelationReplacement = + /** The field's new value (`{ $in: [...] }`), in place. */ + | { readonly kind: 'value'; readonly value: Record } + /** A clause AND-ed into the node in place of the field entry (`{ $or: [...] }`). */ + | { readonly kind: 'clause'; readonly clause: FilterCondition }; + +function declaredFieldsOf(schema: unknown): Record | null { + const fields = (schema as { fields?: unknown } | undefined)?.fields; + return fields !== null && typeof fields === 'object' ? (fields as Record) : null; +} + +function declaredTypeOf(def: unknown): string | undefined { + const type = (def as { type?: unknown } | null | undefined)?.type; + return typeof type === 'string' ? type : undefined; +} + +/** + * Admit one nested-relation condition as written, or say why it cannot be + * served — structurally, from declarations alone: no data is read here, so the + * judge (`judgeFilter`) and execution give the same verdict. + * + * The condition is served when the relation's related object is registered + * (`schemaOf` answers its declared field map), the condition names at least + * one field, and every key is a field the related object declares — or a + * column the platform provisions on every record (`id`, `created_at`, + * `updated_at`) — undotted, and not itself a relation holding a condition of + * its own (one level). What each key is compared WITH is the related object's + * own doors' question, asked when the related object is read. + */ +export function admitRelationCondition( + field: string, + def: unknown, + condition: Record, + path: string, + schemaOf: ((name: string) => unknown) | undefined, +): RelationConditionVerdict { + const type = declaredTypeOf(def) ?? 'lookup'; + const target = referenceTargetOf(def); + const multiple = isMultiValueField(def as Parameters[0]); + const keys = Object.keys(condition); + const refuse = ( + reason: RelationConditionRefusalReason, + key?: string, + keyType?: string, + ): RelationConditionVerdict => ({ + ok: false, + refusal: { + reason, field, type, target, multiple, path, keys, + ...(key === undefined ? {} : { key }), + ...(keyType === undefined ? {} : { keyType }), + }, + }); + const relatedFields = target === undefined || schemaOf === undefined ? null : declaredFieldsOf(schemaOf(target)); + if (target === undefined || relatedFields === null) return refuse('unregistered-target'); + if (keys.length === 0) return refuse('empty'); + for (const key of keys) { + if (key.includes('.')) return refuse('dotted-key', key); + const declared = Object.prototype.hasOwnProperty.call(relatedFields, key) ? relatedFields[key] : undefined; + if (declared === undefined) { + if (provisionedNoOperatorObjectColumn(key) !== null) continue; + return refuse('undeclared-key', key); + } + const keyType = declaredTypeOf(declared); + if (keyType !== undefined && REFERENCE_VALUE_TYPES.has(keyType) && isNoOperatorObject(condition[key])) { + return refuse('second-level', key, keyType); + } + } + return { ok: true, site: { field, type, target, multiple, condition, path } }; +} + +/** + * The lowered form of one admitted condition, given the ids of the related + * records it matched: `$in` on a single-valued relation; on a multi-valued one, + * an `$or` of one `$contains` per id — the spec's any-of spelling over a stored + * list, and the only one the SQL family answers (it refuses `$in` over the JSON + * column). Each id is compared as its text, the `$contains` comparand contract. + * + * No ids yield `{ $in: [] }` / `{ $or: [] }`: FALSE, so the condition selects + * no row — never an absent predicate. + */ +export function lowerRelationSite(site: RelationFilterSite, ids: readonly unknown[]): RelationReplacement { + if (!site.multiple) return { kind: 'value', value: { $in: [...ids] } }; + return { + kind: 'clause', + clause: { $or: ids.map((id) => ({ [site.field]: { $contains: String(id) } })) }, + }; +} + +/** How the words name the condition: its keys, never its values. */ +function describeKeys(keys: readonly string[]): string { + if (keys.length === 0) return 'no keys'; + const shown = keys.slice(0, 3).map((key) => JSON.stringify(key)).join(', '); + const more = keys.length > 3 ? `, and ${keys.length - 3} more` : ''; + return `keys ${shown}${more}`; +} + +/** The ids route every driver serves: `$in` on a single-valued relation, `$contains` per id on a multi-valued one. */ +function idsRoute(field: string, multiple: boolean): string { + return multiple + ? `{ "${field}": { "$contains": ID } } for one id, an $or of those for several` + : `{ "${field}": { "$in": [ID, …] } }`; +} + +/** + * The words of a structural refusal ({@link admitRelationCondition}): the + * position, the verdict and the route first — the REST door bounds a 4xx + * message at 500 characters by truncation — and nothing after the route that a + * caller needs. + */ +export function relationConditionRefusalMessage(refusal: RelationConditionRefusal, context: string): string { + const { field, type, target, multiple, path, key } = refusal; + const related = target === undefined ? 'the related object' : `the related object '${target}'`; + const named = target === undefined ? 'the related object' : `'${target}'`; + const head = + `${context}: filter on '${field}' puts a nested-relation condition (${describeKeys(refusal.keys)}) at ` + + `${path}, beneath the declared ${type} field '${field}'`; + const verdict = 'The filter was NOT applied.'; + const twoStep = `then match '${field}' against its ids: ${idsRoute(field, multiple)}.`; + switch (refusal.reason) { + case 'unregistered-target': + return ( + head + + (target === undefined + ? ', and the field declares no related object to read. ' + : `, and no object '${target}' is registered here to read. `) + + `${verdict} Match '${field}' against ids you hold: ${idsRoute(field, multiple)}.` + ); + case 'empty': + return ( + `${head}, and it names no field of ${related}. ${verdict} Name one ` + + `({ "${field}": { "FIELD": VALUE } }), or test that '${field}' has a value: ` + + `{ "${field}": { "$null": false } }.` + ); + case 'undeclared-key': + return ( + `${head}, and '${key}' is not a field of ${related}. ${verdict} Name a field it declares: ` + + `{ "${field}": { "FIELD": VALUE } }. A key it does not declare would read as a real empty answer.` + ); + case 'dotted-key': + return ( + `${head}, and '${key}' is a dotted path: the condition reaches one level only. ${verdict} ` + + `Read ${named} with that condition yourself, ${twoStep}` + ); + case 'second-level': + default: + return ( + `${head}, and '${key}' is itself a ${refusal.keyType ?? 'relation'} field of ${related} holding a ` + + `condition of its own: one level only. ${verdict} Read ${named} with { "${key}": { … } } yourself, ` + + twoStep + ); + } +} + +/** The structural refusal, in the engine's `INVALID_FILTER` / 400 envelope. */ +export function relationConditionError(refusal: RelationConditionRefusal, context: string): Error { + return invalidFilterError(relationConditionRefusalMessage(refusal, context)); +} + +/** + * The cap refusal: the condition matched more related records than + * {@link RELATION_FILTER_ID_CAP}, so the filter is refused — `INVALID_FILTER` / + * 400 — rather than run over a cut-off id list that would silently drop + * matching rows. The words name the cap, the relation and the two-step route. + */ +export function relationFilterCapError(site: RelationFilterSite, context: string, cap: number): Error { + return invalidFilterError( + `${context}: the nested-relation condition at ${site.path} matched more than ${cap} records of ` + + `the related object '${site.target}' (the engine's cap), so the filter was NOT applied: a cut-off ` + + `id list would silently drop matching rows. Narrow the condition, or read '${site.target}' yourself ` + + `page by page and match '${site.field}' against its ids: ${idsRoute(site.field, site.multiple)}.`, + ); +} diff --git a/packages/rest/src/data-nested-object-door.test.ts b/packages/rest/src/data-nested-object-door.test.ts index b14f5d5a9cd..de467f2ba46 100644 --- a/packages/rest/src/data-nested-object-door.test.ts +++ b/packages/rest/src/data-nested-object-door.test.ts @@ -1,33 +1,34 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * [#20745] A plain object with no `$`-operator key beneath a relation field - * (the nested-relation form), a structured-JSON field (a whole-value match) or - * the platform-provisioned `id` column is refused at the public door — - * `POST /api/v1/data/:object/query` answers `400 INVALID_FILTER` in the - * engine's words, naming the field and the path, before any read — over a - * real `SqlDriver`; and the route the refusal names answers the rows the - * nested form meant. + * [#20802] The nested-relation form `{ relation: { field: value } }` is SERVED + * at the public door — `POST /api/v1/data/:object/query` answers the rows the + * form means, over a real `SqlDriver` — and what the first cut keeps refusing + * is still refused, `400 INVALID_FILTER` in the engine's words, before any read. * - * Measured on the base (`origin/main` `a51920f5fb`) through this door, three - * rows (owner `u1`, region NA, on `d1` and `d3`): + * #20745's table, re-read on this branch through this door (owner `u1`, region + * NA, on `d1` and `d3`; `d4` has no owner): * - * | `where` | InMemoryDriver | SQLite | PostgreSQL 16 | - * |:--|:--|:--|:--| - * | `{ owner: { region: 'NA' } }` (lookup; master-detail, multiple lookup, user, tree alike) | 200, no rows | 400 `INVALID_FILTER`, the driver's words | same as SQLite | - * | `{ meta: { a: 1 } }` (json; address, composite alike) | 200, the deep-equal rows | 400, the driver's words | same | - * | `{ id: { a: 1 } }` | 200, no rows | 400, the driver's words | same | - * | route `{ owner: { $in: ['u1'] } }` | `d1`, `d3` | `d1`, `d3` | `d1`, `d3` | - * | route `{ owners: { $contains: 'u1' } }` (multiple lookup) | `d1`, `d3` | `d1`, `d3` | `d1`, `d3` | - * | `{ owners: { $in: ['u1'] } }` (multiple lookup) | `d1`, `d3` | 400, the driver's JSON-column words | same | + * | `where` | before (PR #20781) | now: SQLite · PostgreSQL 16 | + * |:--|:--|:--| + * | `{ owner: { region: 'NA' } }` (lookup; master-detail and multiple lookup alike) | 400 `INVALID_FILTER` | `d1`, `d3` | + * | `{ parent: { title: 'a' } }` (tree) | 400 | `d2`, `d3` | + * | `{ $not: { owner: { region: 'NA' } } }` | 400 | `d2`, `d4` — the row with no owner satisfies the negation | + * | `{ $or: [{ owner: { region: 'EU' } }, { title: 'a' }] }` | 400 | `d1`, `d2` | + * | `{ owner: { region: 'APAC' } }` (no related record matches) | 400 | no rows | + * | `{ meta: { a: 1 } }` (json; address alike), `{ id: { a: 1 } }` | 400 | 400, unchanged | + * | a second level, an undeclared key, `{}`, an unregistered related object | 400 | 400, in words of their own | + * | `aggregations[1].filter` / `having` `{ owner: { region: 'NA' } }` | 400 | 400, the words say `where` serves it | * - * The arm sits in the engine, in front of every driver, so one verdict holds - * on each cell. InMemoryDriver's refusal row is `@objectstack/objectql`'s - * `engine-nested-object-door.test.ts` by construction (the arm answers before - * a driver is resolved). Its route readings above were measured, not pinned + * The InMemoryDriver cells were measured on this branch and are not pinned * here: this package does not depend on the in-memory driver, and that * driver's test consumers are a ruled, closed census - * (`check:driver-memory-census`). + * (`check:driver-memory-census`). They answered every row above alike, except + * the multi-valued arm where one stored id contains another as a substring + * (`u1` / `u10`): the in-memory driver matches `$contains` per element by + * substring — the gap `FILTER_OPERATORS`' `$contains` docblock records for it. + * `@objectstack/objectql`'s `engine-nested-relation-lowering.test.ts` pins the + * driver input the engine sends, which is the same on every driver. * * ## The dialect axis of THIS file * @@ -43,7 +44,7 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import type { EngineAggregateOptions, FilterCondition } from '@objectstack/spec/data'; -import { ObjectQL } from '@objectstack/objectql'; +import { ObjectQL, RELATION_FILTER_ID_CAP } from '@objectstack/objectql'; import { SqlDriver } from '@objectstack/driver-sql'; import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; import { RestServer } from './rest-server'; @@ -79,6 +80,8 @@ const ROWS = [ { id: 'd1', title: 'a', owner: 'u1', boss: 'u1', owners: ['u1'], assignee: 'u1', meta: { a: 1 }, ship_to: { city: 'Paris' } }, { id: 'd2', title: 'b', owner: 'u2', boss: 'u2', owners: ['u2'], assignee: 'u2', parent: 'd1', meta: { a: 2 }, ship_to: { city: 'Rome' } }, { id: 'd3', title: 'c', owner: 'u1', boss: 'u1', owners: ['u1', 'u2'], assignee: 'u1', parent: 'd1', meta: { b: 1 }, ship_to: { city: 'Paris' } }, + // No relation at all: the row the negation's NULL polarity is about. + { id: 'd4', title: 'd' }, ]; interface Cell { @@ -104,26 +107,38 @@ const CELLS: readonly Cell[] = [ }, ]; +/** name · the `where` · the rows it means. */ +const SERVED: ReadonlyArray = [ + ['a lookup (the card)', { owner: { region: 'NA' } }, ['d1', 'd3']], + ['a master-detail', { boss: { region: 'NA' } }, ['d1', 'd3']], + ['a multiple lookup — any member', { owners: { region: 'NA' } }, ['d1', 'd3']], + ['a multiple lookup, the other member', { owners: { region: 'EU' } }, ['d2', 'd3']], + ['a tree field', { parent: { title: 'a' } }, ['d2', 'd3']], + ['beside a column of the object', { title: 'c', owner: { region: 'NA' } }, ['d3']], + ['inside $or', { $or: [{ owner: { region: 'EU' } }, { title: 'a' }] }, ['d1', 'd2']], + ['under $not — no owner satisfies it', { $not: { owner: { region: 'NA' } } }, ['d2', 'd4']], + ['under $not, a multiple lookup', { $not: { owners: { region: 'NA' } } }, ['d2', 'd4']], + ['no related record matches', { owner: { region: 'APAC' } }, []], + ['no related record matches, a multiple lookup', { owners: { region: 'APAC' } }, []], + ['under $not, no related record matches', { $not: { owner: { region: 'APAC' } } }, ['d1', 'd2', 'd3', 'd4']], + ['an operator in the condition', { owner: { region: { $in: ['EU'] } } }, ['d2']], +]; + /** - * name · the `where` · the field · the path · words only this kind's refusal - * prints · the route it names. Both are asserted on the REST body, so the - * route is pinned to land inside the door's 500-character message bound. + * name · the `where` · the path · words only this refusal prints · the route + * it names. Both are asserted on the REST body, so the route is pinned to land + * inside the door's 500-character message bound. */ -const REFUSED: ReadonlyArray = [ - ['a lookup (the card)', { owner: { region: 'NA' } }, 'owner', 'where.owner', 'nested-relation form', `Filter the related object '${OWNER}' first, then match 'owner' against the ids it returns: { "owner": { "$in": [ID, …] } }.`], - ['a master-detail', { boss: { region: 'NA' } }, 'boss', 'where.boss', 'nested-relation form', '{ "boss": { "$in": [ID, …] } }'], - ['a multiple lookup', { owners: { region: 'NA' } }, 'owners', 'where.owners', 'nested-relation form', '{ "owners": { "$contains": ID } } for one id, an $or of those for several'], - ['a user field', { assignee: { region: 'NA' } }, 'assignee', 'where.assignee', 'nested-relation form', `Filter the related object 'sys_user' first`], - ['a tree field', { parent: { title: 'a' } }, 'parent', 'where.parent', 'nested-relation form', `Filter the related object '${OBJECT}' first`], - ['a json field (the card)', { meta: { a: 1 } }, 'meta', 'where.meta', 'whole-value match', '{ "meta": { "$null": false } }, or store the part you filter on in a field of its own'], - ['an address field', { ship_to: { city: 'Paris' } }, 'ship_to', 'where.ship_to', 'whole-value match', '{ "ship_to": { "$null": false } }'], - ['the id column (the card)', { id: { a: 1 } }, 'id', 'where.id', "the platform-provisioned text column 'id'", `Compare 'id' with a value ({ "id": VALUE })`], - ['inside $not', { $not: { owner: { region: 'NA' } } }, 'owner', 'where.$not.owner', 'nested-relation form', '{ "owner": { "$in": [ID, …] } }'], +const REFUSED: ReadonlyArray = [ + ['a json field (the card)', { meta: { a: 1 } }, 'where.meta', 'whole-value match', '{ "meta": { "$null": false } }, or store the part you filter on in a field of its own'], + ['an address field', { ship_to: { city: 'Paris' } }, 'where.ship_to', 'whole-value match', '{ "ship_to": { "$null": false } }'], + ['the id column (the card)', { id: { a: 1 } }, 'where.id', "the platform-provisioned text column 'id'", `Compare 'id' with a value ({ "id": VALUE })`], + ['a second level', { parent: { owner: { region: 'NA' } } }, 'where.parent', `'owner' is itself a lookup field of the related object '${OBJECT}'`, '{ "parent": { "$in": [ID, …] } }'], + ['a key the related object does not declare', { owner: { regio: 'NA' } }, 'where.owner', `'regio' is not a field of the related object '${OWNER}'`, '{ "owner": { "FIELD": VALUE } }'], + ['an empty condition', { owner: {} }, 'where.owner', 'names no field of the related object', '{ "owner": { "$null": false } }'], + ['a related object not registered here', { assignee: { region: 'NA' } }, 'where.assignee', "no object 'sys_user' is registered here", '{ "assignee": { "$in": [ID, …] } }'], ]; -/** The arm's own words, in every refusal it raises — a control must never be answered in them. */ -const ARM_WORDS = 'is filter structure, not a value'; - function createMockServer() { const noop = () => {}; return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; @@ -144,7 +159,7 @@ const idsOf = (body: any): string[] => (body?.records ?? []).map((r: any) => r.i for (const cell of CELLS) { const config = cell.config(); describe.skipIf(!config)( - `[#20745] a no-operator object beneath a relation, JSON or id column at the public door — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, + `[#20802] the nested-relation form at the public door — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, () => { let engine: ObjectQL; let driver: any; @@ -169,10 +184,10 @@ for (const cell of CELLS) { for (const row of OWNERS) await engine.insert(OWNER, { ...row } as any); for (const row of ROWS) await engine.insert(OBJECT, { ...row } as any); - // Reads of THIS object — the protocol's own metadata traffic is not the question. + // Reads of the two objects — the protocol's own metadata traffic is not the question. for (const verb of ['find', 'findOne', 'count', 'aggregate'] as const) { const real = driver[verb].bind(driver); - driver[verb] = (o: string, ...rest: unknown[]) => { if (o === OBJECT) reads.n += 1; return real(o, ...rest); }; + driver[verb] = (o: string, ...rest: unknown[]) => { if (o === OBJECT || o === OWNER) reads.n += 1; return real(o, ...rest); }; } const protocol = new ObjectStackProtocolImplementation(engine as any); @@ -194,25 +209,51 @@ for (const cell of CELLS) { try { await engine?.destroy(); } catch { /* noop */ } }); - it('where: every row of the card answers 400 INVALID_FILTER in the engine\'s words, naming the field and the path — no read', async () => { + it('where: #20745\'s table answers the rows the form means — on every relation type, composed as written', async () => { + for (const [name, where, rows] of SERVED) { + const res = await query({ where }); + expect(res.status, `${name}: ${JSON.stringify(res.body)}`).toBe(200); + expect(idsOf(res.body), name).toEqual([...rows].sort()); + const direct = await engine.find(OBJECT, { where }); + expect(direct.map((r: any) => r.id).sort(), `engine.find, ${name}`).toEqual([...rows].sort()); + } + }); + + it('the nested form answers exactly what the two-step route answers', async () => { + const na = await query({ where: { region: 'NA' } }, OWNER); + const ids = idsOf(na.body); + expect(ids).toEqual(['u1']); + for (const [nested, twoStep] of [ + [{ owner: { region: 'NA' } }, { owner: { $in: ids } }], + [{ owners: { region: 'NA' } }, { $or: ids.map((id) => ({ owners: { $contains: id } })) }], + [{ $not: { owner: { region: 'NA' } } }, { $not: { owner: { $in: ids } } }], + ] as Array<[FilterCondition, FilterCondition]>) { + expect(idsOf((await query({ where: nested })).body), JSON.stringify(nested)) + .toEqual(idsOf((await query({ where: twoStep })).body)); + } + }); + + it('where: what the first cut keeps refusing answers 400 INVALID_FILTER in the engine\'s words — no read', async () => { const before = reads.n; - for (const [name, where, field, path, words, route] of REFUSED) { + for (const [name, where, path, words, route] of REFUSED) { const res = await query({ where }); expect(res.status, `${name}: ${JSON.stringify(res.body)}`).toBe(400); expect(res.body.code, name).toBe('INVALID_FILTER'); - expect(res.body.error, name).toContain(`filter on '${field}'`); expect(res.body.error, name).toContain(`at ${path},`); expect(res.body.error, name).toContain('The filter was NOT applied.'); expect(res.body.error, name).toContain(words); expect(res.body.error, name).toContain(route); const err = await engine.find(OBJECT, { where }).then(() => null, (e: any) => e); expect({ code: err?.code, status: err?.status }, `engine.find, ${name}`).toEqual({ code: 'INVALID_FILTER', status: 400 }); - expect(err?.message, `engine.find, ${name}`).toContain(ARM_WORDS); } - expect(reads.n - before, 'no read of the object — every refusal precedes the driver').toBe(0); + const dotted = await query({ where: { 'owner.region': 'NA' } }); + expect(dotted.status, JSON.stringify(dotted.body)).toBe(400); + expect(dotted.body.code).toBe('INVALID_FIELD'); + expect(JSON.stringify(dotted.body)).toContain('nest the condition beneath the relation field'); + expect(reads.n - before, 'no read of either object — every refusal precedes the driver').toBe(0); }); - it('the per-aggregation filter and having: 400 INVALID_FILTER at their own positions — no read', async () => { + it('the per-aggregation filter and having: 400 INVALID_FILTER at their own positions, naming where — no read', async () => { const before = reads.n; const filter = await query({ aggregations: [{ function: 'count', alias: 'n' }, { function: 'count', alias: 'm', filter: { owner: { region: 'NA' } } }], @@ -220,6 +261,7 @@ for (const cell of CELLS) { expect(filter.status, JSON.stringify(filter.body)).toBe(400); expect(filter.body.code).toBe('INVALID_FILTER'); expect(filter.body.error).toContain('at aggregations[1].filter.owner,'); + expect(filter.body.error).toContain("which the engine serves in 'where'"); const having = await query({ groupBy: ['owner'], aggregations: [{ function: 'count', alias: 'n' }], @@ -228,35 +270,57 @@ for (const cell of CELLS) { expect(having.status, JSON.stringify(having.body)).toBe(400); expect(having.body.code).toBe('INVALID_FILTER'); expect(having.body.error).toContain('at having.owner,'); - expect(reads.n - before, 'no read of the object — every refusal precedes the driver').toBe(0); + expect(reads.n - before, 'no read — every refusal precedes the driver').toBe(0); + // …and in `where`, the same condition narrows the aggregate. + const counted = await query({ + where: { owner: { region: 'NA' } }, + aggregations: [{ function: 'count', alias: 'n' }], + } satisfies EngineAggregateOptions as Record); + expect(counted.status, JSON.stringify(counted.body)).toBe(200); + expect(JSON.stringify(counted.body)).toContain('"n":2'); + }); + + it('the cap: a condition matching more related records than the cap is REFUSED, never truncated; at the cap it is served whole', async () => { + const many = Array.from({ length: RELATION_FILTER_ID_CAP }, (_, i) => ({ id: `cap_${i}`, region: 'CAP' })); + // In batches: one multi-row INSERT of a thousand rows passes SQLite's + // compound-SELECT limit. + for (let i = 0; i < many.length; i += 100) await engine.insert(OWNER, many.slice(i, i + 100) as any); + await engine.insert(OWNER, { id: 'cap_extra', region: 'CAP_EXTRA' } as any); + await engine.insert(OBJECT, { id: 'd_cap', title: 'cap', owner: 'cap_extra' } as any); + try { + const over = await query({ where: { owner: { region: { $in: ['CAP', 'CAP_EXTRA'] } } } }); + expect(over.status, JSON.stringify(over.body)).toBe(400); + expect(over.body.code).toBe('INVALID_FILTER'); + expect(over.body.error).toContain(`matched more than ${RELATION_FILTER_ID_CAP} records of the related object '${OWNER}'`); + expect(over.body.error).toContain('the filter was NOT applied'); + expect(over.body.error).toContain('{ "owner": { "$in": [ID, …] } }'); + // Exactly the cap: served, every id in play (no ledger row points at one). + const at = await query({ where: { owner: { region: 'CAP' } } }); + expect(at.status, JSON.stringify(at.body)).toBe(200); + expect(idsOf(at.body)).toEqual([]); + const extra = await query({ where: { owner: { region: 'CAP_EXTRA' } } }); + expect(idsOf(extra.body)).toEqual(['d_cap']); + } finally { + await engine.delete(OBJECT, { where: { id: 'd_cap' } } as any); + await engine.delete(OWNER, { where: { region: { $in: ['CAP', 'CAP_EXTRA'] } }, multi: true } as any); + } }); - it('the named route answers the rows the nested form meant: the related object\'s ids, then $in (single) or $contains (multiple)', async () => { + it('CONTROL the routes still answer the rows, and a file field\'s object reaches the driver unjudged', async () => { const na = await query({ where: { region: 'NA' } }, OWNER); - expect(na.status, JSON.stringify(na.body)).toBe(200); const ids = idsOf(na.body); - expect(ids).toEqual(['u1']); for (const field of ['owner', 'boss', 'assignee']) { const res = await query({ where: { [field]: { $in: ids } } }); - expect(res.status, `${field}: ${JSON.stringify(res.body)}`).toBe(200); expect(idsOf(res.body), field).toEqual(['d1', 'd3']); } const multiple = await query({ where: { owners: { $contains: ids[0] } } }); - expect(multiple.status, JSON.stringify(multiple.body)).toBe(200); expect(idsOf(multiple.body)).toEqual(['d1', 'd3']); - const anyOf = await query({ where: { $or: [{ owners: { $contains: 'u1' } }, { owners: { $contains: 'u9' } }] } }); - expect(idsOf(anyOf.body)).toEqual(['d1', 'd3']); - const tree = await query({ where: { parent: { $in: ['d1'] } } }); - expect(idsOf(tree.body)).toEqual(['d2', 'd3']); const present = await query({ where: { meta: { $null: false } } }); expect(idsOf(present.body)).toEqual(['d1', 'd2', 'd3']); - }); - - it('CONTROL a file field\'s object reaches the driver, never the arm\'s refusal', async () => { const before = reads.n; const res = await query({ where: { photo: { url: 'x' } } }); expect(reads.n - before, 'the driver was asked').toBe(1); - expect(JSON.stringify(res.body)).not.toContain(ARM_WORDS); + expect(JSON.stringify(res.body)).not.toContain('is filter structure, not a value'); }); }, ); diff --git a/packages/rest/src/data-nested-relation-permission.test.ts b/packages/rest/src/data-nested-relation-permission.test.ts new file mode 100644 index 00000000000..747b5c3f16c --- /dev/null +++ b/packages/rest/src/data-nested-relation-permission.test.ts @@ -0,0 +1,202 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20802] The nested-relation form reads the related object AS THE CALLER: + * the related object's row scope and field permissions apply to the condition + * exactly as they apply to a direct read of it — with the REAL security layer + * (`SecurityPlugin` on a real `ObjectQL` over a real `SqlDriver`), through + * `POST /api/v1/data/:object/query`. + * + * The ruling's permission axis, measured: + * + * - **A field the caller cannot read.** A direct filter on it is refused today + * — `403 PERMISSION_DENIED`, the security layer's filter-oracle guard + * (`assertReadableQueryFields`), naming the field. The nested form reaches + * that SAME check through the related read, so it answers the same refusal, + * loudly — ⛔ never a `200` with no rows, and ⛔ never the rows a system read + * would have matched (which is filtering by a value the caller may not see). + * There is no second copy of the rule in the engine. + * - **The related object's row scope.** A related record the caller's RLS + * hides matches no condition: the rows it would have selected are not + * selected, and a system caller still gets them. + * + * SQLite only: the checks are the security layer's, before any driver + * dialect is involved; `data-nested-object-door.test.ts` carries the dialect + * axis of the lowering itself. + */ + +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import type { FilterCondition } from '@objectstack/spec/data'; +import { PermissionSetSchema } from '@objectstack/spec/security'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { SecurityPlugin } from '@objectstack/plugin-security'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { RestServer } from './rest-server'; + +const OBJECT = 'rest_nested_perm_ledger'; +const OWNER = 'rest_nested_perm_owner'; + +const SYS_CTX = { isSystem: true, userId: 'usr_system' }; + +const MEMBER_SET = PermissionSetSchema.parse({ + name: 'member_default', + label: 'Member', + objects: { '*': { allowRead: true, allowCreate: true, allowEdit: true, allowDelete: true } }, + // The field the caller may not read, on the RELATED object. + fields: { [`${OWNER}.secret`]: { readable: false, editable: false } }, + // The related object's row scope: the caller sees owners outside region HIDDEN only. + rowLevelSecurity: [{ name: 'owner_scope', object: OWNER, operation: 'all', using: "record.region != 'HIDDEN'" }], +}); + +const MEMBER_CTX = { userId: 'usr_member', positions: [], permissions: [MEMBER_SET.name], posture: 'MEMBER' }; + +const OWNERS = [ + { id: 'u1', region: 'NA', secret: 's1' }, + { id: 'u2', region: 'EU', secret: 's2' }, + { id: 'u3', region: 'HIDDEN', secret: 's3' }, +]; +const ROWS = [ + { id: 'd1', title: 'a', owner: 'u1' }, + { id: 'd2', title: 'b', owner: 'u2' }, + { id: 'd3', title: 'c', owner: 'u1' }, + { id: 'd4', title: 'd', owner: 'u3' }, +]; + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +function makeRes() { + const res: any = { + write: () => true, end: () => {}, + header: () => res, + status: (code: number) => { res._status = code; return res; }, + json: (body: any) => { res._json = body; return res; }, + }; + return res; +} + +const idsOf = (rows: any): string[] => (Array.isArray(rows) ? rows : rows?.records ?? []).map((r: any) => r.id).sort(); +const envelopeOf = (e: any) => ({ code: e?.code, status: e?.statusCode ?? e?.status }); + +describe('[#20802] the nested-relation form reads the related object as the caller — the real security layer', () => { + let engine: ObjectQL; + let query: (where: FilterCondition, object?: string) => Promise<{ status: number; body: any }>; + + beforeAll(async () => { + engine = new ObjectQL(); + engine.registerDriver( + new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as any), + true, + ); + await engine.init(); + engine.registerApp({ + id: 'com.objectstack.qa.nested-relation-permission-20802', + name: 'Nested relation permission', + version: '1.0.0', + type: 'plugin', + scope: 'system', + objects: [ + { + name: OWNER, + label: 'Owner', + sharingModel: 'public_read_write', + fields: { + region: { name: 'region', type: 'text' }, + secret: { name: 'secret', type: 'text' }, + }, + }, + { + name: OBJECT, + label: 'Ledger', + sharingModel: 'public_read_write', + fields: { + title: { name: 'title', type: 'text' }, + owner: { name: 'owner', type: 'lookup', reference: OWNER }, + }, + }, + ], + } as never); + await engine.syncSchemas(); + + const services: Record = { + manifest: { register: vi.fn() }, + objectql: engine, + metadata: { + get: async (_type: string, name: string) => engine.getSchema(name) ?? null, + list: async () => [MEMBER_SET], + }, + }; + const ctx = { + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, + registerService: vi.fn(), + getService: (name: string) => { + if (!(name in services)) throw new Error(`service not registered: ${name}`); + return services[name]; + }, + }; + const plugin = new SecurityPlugin({ fallbackPermissionSet: 'member_default' }); + await plugin.init(ctx as never); + await plugin.start(ctx as never); + vi.spyOn((engine as unknown as { logger: { warn: () => void } }).logger, 'warn').mockImplementation(() => undefined); + + await engine.insert(OWNER, OWNERS.map((r) => ({ ...r })), { context: SYS_CTX } as never); + await engine.insert(OBJECT, ROWS.map((r) => ({ ...r })), { context: SYS_CTX } as never); + + const protocol = new ObjectStackProtocolImplementation(engine as any); + const rest = new RestServer(createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any); + (rest as any).resolveExecCtx = async () => MEMBER_CTX; + rest.registerRoutes(); + const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/data/:object/query'); + expect(route).toBeDefined(); + query = async (where, object = OBJECT) => { + const res = makeRes(); + await route!.handler({ params: { object }, body: JSON.parse(JSON.stringify({ where })), query: {}, headers: {} } as any, res); + return { status: res._status ?? 200, body: res._json }; + }; + }); + + afterAll(async () => { + try { await engine?.destroy(); } catch { /* noop */ } + }); + + it('CONTROL a condition on a field the caller can read is served under the security layer', async () => { + const res = await query({ owner: { region: 'NA' } }); + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(idsOf(res.body)).toEqual(['d1', 'd3']); + }); + + it('a field the caller cannot read: the direct filter is refused today, and the nested form answers the SAME refusal — never an empty result', async () => { + // The one check, measured on a direct read of the related object. + const direct = await engine.find(OWNER, { where: { secret: 's1' }, context: MEMBER_CTX } as never).then(() => null, (e: any) => e); + expect(envelopeOf(direct)).toEqual({ code: 'PERMISSION_DENIED', status: 403 }); + expect(String(direct?.message)).toContain('secret'); + + // The nested form, in-process and at the public door. + const nested = await engine.find(OBJECT, { where: { owner: { secret: 's1' } }, context: MEMBER_CTX } as never) + .then((rows) => ({ rows }), (e: any) => e); + expect(envelopeOf(nested)).toEqual({ code: 'PERMISSION_DENIED', status: 403 }); + expect(String(nested?.message)).toContain('secret'); + expect(String(nested?.message)).toContain(OWNER); + + const res = await query({ owner: { secret: 's1' } }); + expect(res.status, JSON.stringify(res.body)).toBe(403); + expect(JSON.stringify(res.body)).toContain('PERMISSION_DENIED'); + expect(res.body?.records, 'no rows are served beside the refusal').toBeUndefined(); + + // What the refusal withholds: a system read CAN filter by the value. + const system = await engine.find(OBJECT, { where: { owner: { secret: 's1' } }, context: SYS_CTX } as never); + expect(idsOf(system)).toEqual(['d1', 'd3']); + }); + + it('the related object\'s row scope applies: a related record the caller cannot see matches no condition', async () => { + const res = await query({ owner: { region: 'HIDDEN' } }); + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(idsOf(res.body)).toEqual([]); + // The record is there, and a system caller's condition finds it. + const system = await engine.find(OBJECT, { where: { owner: { region: 'HIDDEN' } }, context: SYS_CTX } as never); + expect(idsOf(system)).toEqual(['d4']); + }); +}); diff --git a/packages/rest/src/data-no-operator-object-door.test.ts b/packages/rest/src/data-no-operator-object-door.test.ts index 834b33b0fa5..f715073679d 100644 --- a/packages/rest/src/data-no-operator-object-door.test.ts +++ b/packages/rest/src/data-no-operator-object-door.test.ts @@ -7,8 +7,9 @@ * the field and the path, before any read — over a real `SqlDriver`, with a * file field's object reaching the driver as written. The two controls triage * first named here — a `lookup` field's nested relation filter and a `json` - * field's object comparand — are refused by the same arm since #20745, in - * words of their own (`data-nested-object-door.test.ts` pins them). + * field's object comparand — were refused by the same arm since #20745, in + * words of their own; the nested relation filter is SERVED at `where` since + * #20802 (`data-nested-object-door.test.ts` pins both). * * Measured on the base (`origin/main` `fbec216e2d`) through this door and the * engine, three rows: @@ -16,7 +17,7 @@ * | `where` | InMemoryDriver | SQLite | PostgreSQL 16 | * |:--|:--|:--|:--| * | `{ amount: { a: 1 } }` (number), `{ title: { a: 1 } }` (text) | 200, no rows | 400 `INVALID_FILTER`, the driver's words | same as SQLite | - * | `{ owner: { region: 'NA' } }` (lookup), `{ meta: { a: 1 } }` (json) | 200, no rows / one row | 400, the driver's words | same — refused since #20745 | + * | `{ owner: { region: 'NA' } }` (lookup), `{ meta: { a: 1 } }` (json) | 200, no rows / one row | 400, the driver's words | same — refused since #20745; the lookup row served since #20802 | * | `aggregations[1].filter` `{ amount: { a: 1 } }` / `having` `{ total: { a: 1 } }` | count 0 / no group | same | same | * * The arm sits in the engine, in front of every driver, so one verdict holds diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index 3d2d4d4b2fa..db70c63be64 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -1922,8 +1922,9 @@ function checkFilterConditionComparands( const hasOperatorKeys = Object.keys(value).some((k) => k.startsWith('$')); if (!hasOperatorKeys) { // Nested relation / deep equality — the schema does not re-parse these, - // so the walk descends itself. (The query engine refuses both at query - // time, `INVALID_FILTER`; see form 4 on {@link FilterCondition}.) + // so the walk descends itself. (At query time the engine serves the + // nested relation in `where` and refuses a whole-value match, + // `INVALID_FILTER`; see form 4 on {@link FilterCondition}.) checkFilterConditionComparands(value, ctx, [...path, key], depth + 1); continue; } @@ -1980,21 +1981,28 @@ function checkFilterConditionComparands( * 1. Implicit equality: { field: value } * 2. Explicit operators: { field: { $op: value } } * 3. Logical combinations: { $and: [...], $or: [...], $not: {...} } - * 4. Nested relations: { relation: { field: value } } — the type and the schema - * accept this form, and the query engine REFUSES it: a plain object with no - * `$` operator beneath a relation field answers `INVALID_FILTER` / 400 on - * every driver, because no data-path driver follows a relation into the - * related object (the field stores the related record's id). Filter the - * related object first, then match the relation field against the ids it - * returns: `{ relation: { $in: [id, …] } }`, or `{ relation: { $contains: id } }` - * per id on a multi-valued relation. The same object beneath a JSON-valued - * field (a whole-value match) and beneath a scalar field is refused too. + * 4. Nested relations: { relation: { field: value } } — a condition on the + * related record's own fields, beneath a relation field (`lookup`, + * `master_detail`, `user`, `tree`; single or multiple). The query engine + * SERVES it in `where`, the same on every driver: it reads the related + * object with the condition AS THE CALLER — that object's row scope and + * field permissions apply, so a condition on a field the caller cannot read + * is refused, never answered empty — and matches the relation field against + * the ids it returns (`$in` on a single-valued relation; any member, an + * `$or` of `$contains` per id, on a multi-valued one). One level: every key + * must be a field the related object declares, and a relation condition + * beneath it, or a dotted key, is refused. A condition matching more related + * records than the engine's cap is refused rather than truncated; filter the + * related object yourself then, and match its ids the same way. An + * aggregation's own `filter` and `having` do not serve the form. The same + * object beneath a JSON-valued field (a whole-value match) and beneath a + * scalar field is refused. */ export type FilterCondition = { [key: string]: | any // Implicit equality: key: value | z.infer // Explicit operators: key: { $op: value } - | FilterCondition; // Nested relation: key: { nested: ... } — accepted here, refused by the engine (form 4 above) + | FilterCondition; // Nested relation: key: { nested: ... } — served by the engine in `where`, one level (form 4 above) } & { /** Logical AND - combines all conditions that must be true */ $and?: FilterCondition[]; @@ -2130,7 +2138,10 @@ export const FilterConditionSchema: z.ZodType * $or: [ // Logical combination * { role: "admin" }, * { email: { $contains: "@company.com" } } - * ] + * ], + * account: { // Nested relation (form 4): a field + * industry: "tech" // of the related record, one level + * } * } * } * ``` @@ -2220,7 +2231,7 @@ export type Filter = { $null?: boolean; $exists?: boolean; } - | (T[K] extends object ? Filter : never); // Nested relation — typed here, refused by the engine (see FilterCondition) + | (T[K] extends object ? Filter : never); // Nested relation — typed at any depth here; the engine serves one level, in `where` (see FilterCondition) } & { $and?: Filter[]; $or?: Filter[]; diff --git a/scripts/adr-anchors/packages__objectql__src__relation-filter-lowering.ts.json b/scripts/adr-anchors/packages__objectql__src__relation-filter-lowering.ts.json new file mode 100644 index 00000000000..44e3d3ee32f --- /dev/null +++ b/scripts/adr-anchors/packages__objectql__src__relation-filter-lowering.ts.json @@ -0,0 +1,7 @@ +{ + "file": "packages/objectql/src/relation-filter-lowering.ts", + "adrs": [ + "ADR-0053" + ], + "invariant": "ADR-0053 D-D1 (amended 2026-09-30) item 5: drivers receive the lowered filter. The nested-relation form `{ relation: { field: value } }` is lowered at the engine's `where` seam — between token resolution and the shared `lowerFilterCondition` — by reading the related object with the engine's own `find` under the CALLER's context, into `$in` (single-valued) or an `$or` of `$contains` per id (multi-valued); no driver ever sees the nested form and no driver is changed. The id set is bounded by `RELATION_FILTER_ID_CAP` and refused past it, never truncated. Reading the related object as the system, swallowing the related read's refusal, truncating the id set, or teaching a driver the nested form reverts the ruling that served it." +}