feat(objectql): serve the nested-relation filter in where — lowered at the engine seam, the related object read as the caller, a loud cap, drivers untouched (#20802)

.changeset/20802-dotted-relation-route.md

@@ -0,0 +1,7 @@
+---
+"@objectstack/metadata-protocol": patch
+---
+
+fix(metadata-protocol): a dotted relation filter path names the nested-relation form as the route
+
+A filter key such as `account.industry` — a dotted path through a relation field — is still refused with `INVALID_FIELD` / 400 at the query parameter door. Its words no longer say a filter reaches only the object's own columns, which stopped being true when the engine began serving the nested-relation form in `where`: they now name that form, `{ "account": { "industry": VALUE } }`, beside the denormalise remedy, in the same words as the engine's own refusal.

.changeset/20802-nested-relation-filter-served.md

@@ -0,0 +1,28 @@
+---
+"@objectstack/objectql": minor
+---
+
+feat(objectql): the nested-relation filter `{ relation: { field: value } }` is served in `where`, lowered at the engine's filter seam — the drivers receive `$in` / `$contains` and are unchanged
+
+Clause-②: yes (widening)
+
+A condition on a related record's own fields, written beneath a relation field of the queried object — `{ "account": { "industry": "tech" } }` beneath a `lookup` — is now answered by the engine in `where`, on every verb that takes one (`find`, `findOne`, `count`, `aggregate`, `update`, `delete`) and by `judgeFilter`. It was refused with `INVALID_FILTER` / 400 until now; this supersedes the relation-field paragraph of the pending `20745-nested-object-door` entry.
+
+**How it is answered.** The engine reads the related object with the condition, then matches the relation field against the ids that read returns, and the drivers receive only that: `{ "account": { "$in": [ids] } }` on a single-valued relation, and on a multi-valued one (`multiple: true`) an `$or` of one `$contains` per id, so it matches on any member. The relation types are `lookup`, `master_detail`, `user` and `tree`. It composes as written inside `$and` / `$or` / `$not`, and the `FilterArray` sugar lowers to it too. No related record matching selects no rows; under `$not`, a record whose relation is empty satisfies the negation.
+
+**As the caller.** The related read is the engine's own `find` on the related object with the caller's execution context, so that object's access check, row scope and field permissions apply exactly as they do to a direct read of it. A condition on a field the caller cannot read is refused by the same check that refuses a direct filter on it (`PERMISSION_DENIED` / 403, naming the field), never answered with an empty list; a related record the caller cannot see matches no condition.
+
+**Bounded.** At most `RELATION_FILTER_ID_CAP` (1,000, exported) related ids feed one condition. A condition matching more is refused with `INVALID_FILTER` / 400, naming the cap, the related object and the two-step route — never run over a cut-off list.
+
+**Still refused, in the engine's words (`INVALID_FILTER` / 400, before any read):** a second level (a relation condition beneath the related object's own relation field, or a dotted key inside the condition), a key the related object does not declare, an empty condition `{}`, and a related object that is not registered. An aggregation's own `filter` and `having` keep refusing the form, and their words now name `where` as the place it is served. The dotted spelling `{ "account.industry": "tech" }` stays refused with `INVALID_FIELD` / 400, and its words now name the nested form to write instead. The structured-JSON and scalar-field refusals are unchanged.
+
+Measured through `POST /api/v1/data/:object/query` on SQLite and PostgreSQL 16 (owner `u1`, region NA, on `d1` and `d3`; `d4` has no owner):
+
+| `where` | before | now |
+|:--|:--|:--|
+| `{ owner: { region: "NA" } }` on a `lookup`, and its `master_detail` and multiple-lookup twins | `INVALID_FILTER` / 400 | `d1`, `d3` |
+| `{ parent: { title: "a" } }` on a `tree` field | `INVALID_FILTER` / 400 | `d2`, `d3` |
+| `{ $not: { owner: { region: "NA" } } }` | `INVALID_FILTER` / 400 | `d2`, `d4` |
+| `{ owner: { region: "APAC" } }` (no owner matches) | `INVALID_FILTER` / 400 | no rows |
+
+On the in-memory driver, a multi-valued relation's `$contains` still matches a stored id by substring per element, so there an id that is a substring of another stored id (`u1` inside `u10`) also matches; SQLite and PostgreSQL match the element.

.changeset/20802-nested-relation-prose.md

@@ -0,0 +1,7 @@
+---
+"@objectstack/spec": patch
+---
+
+docs(spec): the `FilterCondition` docblock says the query engine serves the nested-relation form in `where`
+
+`FilterCondition`'s form 4, `{ relation: { field: value } }`, now states the served semantics: the engine reads the related object with the condition as the caller (its row scope and field permissions apply), matches the relation field against the ids it returns (`$in`, or any member on a multi-valued relation), reaches one level, and refuses a condition matching more related records than its cap rather than truncating. The `QueryFilter` example shows the form again, and the `Filter<T>` nested arm's comment says the engine serves one level. The type and the schema are unchanged.

content/docs/kernel/contracts/data-engine.mdx

@@ -178,18 +178,40 @@ where: {
   ],
 }
 
-// A condition on a related record's fields: filter the related object first,
-// then match the lookup against the ids it returns
+// Nested relation filter
+where: {
+  account: { industry: 'tech' },
+}
+```
+
+A nested relation filter is a condition on a related record's own fields, written
+beneath a relation field (`lookup`, `master_detail`, `user` or `tree`, single or
+multiple). The engine serves it in `where`, the same on every driver: it reads the
+related object with the condition **as the caller** — that object's row scope and field
+permissions apply, so a condition on a field the caller cannot read is refused
+(`PERMISSION_DENIED` / 403), never answered with an empty list — and then matches the
+relation field against the ids that read returns: `$in` on a single-valued relation, any
+member on a multi-valued one (`multiple: true`). No related record matching selects no
+rows; under `$not`, a record whose relation is empty satisfies the negation.
+
+It reaches **one level**: every key must be a field the related object declares
+(`{ account: { owner: { region: 'NA' } } }` and the dotted `{ account: { 'owner.region': 'NA' } }`
+are refused), and a condition matching more than 1,000 related records is refused with
+`INVALID_FILTER` / 400 rather than run over a cut-off list. For either, run the two steps
+yourself — filter the related object, then match its ids:
+
+```typescript
 const tech = await engine.find('account', { where: { industry: 'tech' }, fields: ['id'] });
 where: { account: { $in: tech.map((a) => a.id) } }
+// On a multi-valued lookup, one $contains per id:
+where: { $or: tech.map((a) => ({ accounts: { $contains: a.id } })) }
 ```
 
-A plain object with no `$` operator beneath a field — `{ account: { industry: 'tech' } }`
-under a lookup, `{ meta: { a: 1 } }` under a `json` field — is refused with
-`INVALID_FILTER` / 400 on every driver: no driver follows a relation into the related
-object, and a whole-value match on a JSON value means something different on each
-backend. On a multi-valued lookup (`multiple: true`), match each id with `$contains`
-(an `$or` of those for several ids) instead of `$in`.
+An aggregation's own `filter` and `having` do not serve the nested form (put the
+condition in `where`), and a plain object with no `$` operator beneath a `json` field —
+`{ meta: { a: 1 } }`, a whole-value match — is refused with `INVALID_FILTER` / 400 on
+every driver. A dotted path (`{ 'account.industry': 'tech' }`) is refused with
+`INVALID_FIELD` / 400: write it nested instead.
 
 **Supported operators:** `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$between`, `$contains`, `$notContains`, `$startsWith`, `$endsWith`, `$null`, `$exists`
 

packages/metadata-protocol/src/protocol.ts

@@ -255,6 +255,21 @@ const TYPE_TO_FORM: Readonly<Record<string, FormView>> = METADATA_FORM_REGISTRY;
  * spelling-tolerant lookup this comment has rejected since #4432, and it would
  * still persist the row under the plural `type`.
  */
+/**
+ * [#20802] The nested-relation spelling of a dotted relation path, as the
+ * dotted filter refusal names it: `'owner.region'` → `{ "owner": { "region":
+ * VALUE } }` — the form the engine serves at `where`. A path deeper than one
+ * relation is given the generic one-level shape. The engine door
+ * (`@objectstack/objectql`'s `nestedRelationRoute`) words it the same:
+ * `query-expression-conformance.test.ts` holds the two doors' routes equal.
+ */
+function nestedRelationRoute(dotted: string): string {
+    const [head, ...rest] = dotted.split('.');
+    return rest.length === 1
+        ? `{ "${head}": { "${rest[0]}": VALUE } }`
+        : `{ "${head}": { "FIELD": VALUE } }, one level deep`;
+}
+
 function canonicalMetaType(type: string): string {
     return canonicalMetaUrlType(type);
 }
@@ -9878,10 +9893,15 @@ export class ObjectStackProtocolImplementation implements
             const headDef = gate.fields[head];
             const headClass = classifyDottedFilterHead(headDef);
             const headType = String(headDef?.type ?? '');
+            // [#20802] The relation head names the route the engine now
+            // SERVES — the condition nested beneath the relation field — in the
+            // engine door's words (`@objectstack/objectql`'s
+            // `assertFilterIsMaterializable`). One vocabulary across the doors.
             const body = headClass === 'relation'
                 ? `filters on '${first}', which follows the relationship '${head}' into another `
-                  + `object — a filter reaches only columns of '${object}' itself, and '${head}' `
-                  + 'stores the related record\'s id, not an embedded document'
+                  + `object as a dotted path, and '${head}' stores the related record's id, not an `
+                  + 'embedded document — to filter on the related record\'s fields, nest the condition '
+                  + `beneath the relation field: ${nestedRelationRoute(first)}`
                 : headClass === 'virtual'
                     ? `filters on '${first}', a dotted path whose head '${head}' is a virtual `
                       + `'${headType}' field on object '${object}' — its value is computed on read, `

packages/objectql/src/engine-nested-object-door.test.ts

@@ -30,6 +30,14 @@
  * measured (`d1`, `d3` for `$in` on a lookup and `$contains` on a multiple
  * lookup) and are not pinned in a new suite: that driver's test consumers are
  * a ruled, closed census (`check:driver-memory-census`).
+ *
+ * [#20802] The relation rows at `where` are SERVED now (maintainer ruling,
+ * letter A): the engine lowers the nested-relation form by reading the related
+ * object, and `engine-nested-relation-lowering.test.ts` pins it. What stays
+ * here is what still refuses: the structured-JSON and provisioned-`id` rows,
+ * `{}` beneath a relation (it names no field of the related object), and the
+ * relation rows at an aggregation's own `filter` and at `having`, whose words
+ * now say the form is served in `where`.
  */
 
 import { describe, it, expect, beforeEach } from 'vitest';
@@ -74,15 +82,6 @@ const PROBE = {
 
 const OWNER_OBJECT = { name: OWNER, label: 'Owner', fields: { region: { name: 'region', type: 'text' } } };
 
-/** field · declared type · the related object the words name · whether the route is `$contains`. */
-const RELATIONS: ReadonlyArray<readonly [string, string, string, boolean]> = [
-  ['owner', 'lookup', OWNER, false],
-  ['owners', 'lookup', OWNER, true],
-  ['boss', 'master_detail', OWNER, false],
-  ['assignee', 'user', 'sys_user', false],
-  ['parent', 'tree', OBJECT, false],
-];
-
 /** field · declared type — structured-JSON columns. */
 const JSONS: ReadonlyArray<readonly [string, string]> = [
   ['meta', 'json'],
@@ -153,25 +152,6 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p
 
   // ── where ────────────────────────────────────────────────────────────────
 
-  it('refuses the nested-relation form beneath every relation type, single or multiple, naming the route that works — no read', async () => {
-    for (const [field, type, related, multiple] of RELATIONS) {
-      const err = await refusalOf(engine.find(OBJECT, { where: { [field]: { region: 'NA' } } as FilterCondition }));
-      expect(envelopeOf(err), field).toEqual(ENVELOPE);
-      expect(err!.httpStatus, field).toBe(400);
-      expect(err!.message, field).toMatch(/^find\('nested_object_probe'\): /);
-      expect(err!.message, field).toContain(`filter on '${field}'`);
-      expect(err!.message, field).toContain(`at where.${field},`);
-      expect(err!.message, field).toContain(`beneath the declared ${type} field '${field}'`);
-      expect(err!.message, field).toContain('nested-relation form');
-      expect(err!.message, field).toContain('NOT applied');
-      expect(err!.message, field).toContain(`Filter the related object '${related}' first`);
-      expect(err!.message, field).toContain(
-        multiple ? `{ "${field}": { "$contains": ID } }` : `{ "${field}": { "$in": [ID, …] } }`,
-      );
-    }
-    expect(reads).toHaveLength(0);
-  });
-
   it('refuses a whole-value object beneath every structured-JSON type, naming what every driver answers alike — no read', async () => {
     for (const [field, type] of JSONS) {
       const err = await refusalOf(engine.find(OBJECT, { where: { [field]: { a: 1 } } as FilterCondition }));
@@ -195,20 +175,23 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p
   });
 
   it('refuses {} beneath a relation and a JSON column too, in the engine\'s words rather than each driver\'s', async () => {
-    for (const [where, words] of [
-      [{ owner: {} }, 'nested-relation form'],
-      [{ meta: {} }, 'whole-value match'],
+    for (const [where, empty, words] of [
+      // [#20802] Served at `where` otherwise — `{}` names no field of the related object.
+      [{ owner: {} }, '(no keys)', 'names no field of the related object'],
+      [{ meta: {} }, 'an empty object {}', 'whole-value match'],
     ] as const) {
       const err = await refusalOf(engine.find(OBJECT, { where: where as FilterCondition }));
       expect(envelopeOf(err), JSON.stringify(where)).toEqual(ENVELOPE);
-      expect(err!.message, JSON.stringify(where)).toContain('an empty object {}');
+      expect(err!.message, JSON.stringify(where)).toContain(empty);
       expect(err!.message, JSON.stringify(where)).toContain(words);
     }
     expect(reads).toHaveLength(0);
   });
 
   it('covers every engine verb that collects a filter — read and write sides — and the judge', async () => {
-    for (const where of [{ owner: { region: 'NA' } }, { meta: { a: 1 } }] as FilterCondition[]) {
+    // [#20802] The relation row is served at `where` on every verb now:
+    // `engine-nested-relation-lowering.test.ts`.
+    for (const where of [{ ship_to: { city: 'Paris' } }, { meta: { a: 1 } }] as FilterCondition[]) {
       const path = `at where.${Object.keys(where)[0]},`;
       for (const call of [
         () => engine.find(OBJECT, { where }),
@@ -230,20 +213,20 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p
 
   it('reaches inside $and / $or / $not, and answers the FilterArray sugar alike', async () => {
     const cases: ReadonlyArray<readonly [FilterCondition, string]> = [
-      [{ $and: [{ title: 'a' }, { owner: { region: 'NA' } }] }, 'where.$and[1].owner'],
+      [{ $and: [{ title: 'a' }, { ship_to: { city: 'Paris' } }] }, 'where.$and[1].ship_to'],
       [{ $or: [{ meta: { a: 1 } }, { amount: 30 }] }, 'where.$or[0].meta'],
-      [{ $not: { boss: { region: 'NA' } } }, 'where.$not.boss'],
+      [{ $not: { spec: { k: 1 } } }, 'where.$not.spec'],
     ];
     for (const [where, path] of cases) {
       const err = await refusalOf(engine.find(OBJECT, { where }));
       expect(envelopeOf(err), path).toEqual(ENVELOPE);
       expect(err!.message, path).toContain(`at ${path},`);
     }
     const sugar = await refusalOf(
-      engine.find(OBJECT, { where: [['owner', '=', { region: 'NA' }]] } as unknown as EngineQueryOptions),
+      engine.find(OBJECT, { where: [['meta', '=', { a: 1 }]] } as unknown as EngineQueryOptions),
     );
     expect(envelopeOf(sugar)).toEqual(ENVELOPE);
-    expect(sugar!.message).toContain('at where.owner,');
+    expect(sugar!.message).toContain('at where.meta,');
     expect(reads).toHaveLength(0);
   });
 
@@ -333,9 +316,9 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p
     });
 
     const DOORS: ReadonlyArray<{ door: string; query: Record<string, unknown> }> = [
-      { door: 'where object', query: { where: { owner: { region: 'NA' } } } },
+      { door: 'where object', query: { where: { ship_to: { city: 'Paris' } } } },
       { door: '$filter string', query: { $filter: JSON.stringify({ meta: { a: 1 } }) } },
-      { door: 'filter AST', query: { filter: [['owner', '=', { region: 'NA' }]] } },
+      { door: 'filter AST', query: { filter: [['meta', '=', { a: 1 }]] } },
     ];
 
     it.each(DOORS)('the $door door refuses it', async ({ query }) => {