From a4c5777e412dce1e1d9d74f4b2733fea665148d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:01:17 +0000 Subject: [PATCH 1/7] wip(spec): shared FilterCondition lowering module (ADR-0053 D-D1 amended) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- packages/spec/src/data/filter-lowering.ts | 464 ++++++++++++++++++++++ 1 file changed, 464 insertions(+) create mode 100644 packages/spec/src/data/filter-lowering.ts diff --git a/packages/spec/src/data/filter-lowering.ts b/packages/spec/src/data/filter-lowering.ts new file mode 100644 index 00000000000..04ab3eec135 --- /dev/null +++ b/packages/spec/src/data/filter-lowering.ts @@ -0,0 +1,464 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The shared `FilterCondition → FilterCondition` lowering (ADR-0053 D-D1, as + * amended 2026-09-30 by the maintainer's ruling on #5930). + * + * ## What it is + * + * One pure rewrite of a filter tree, applied ONCE at the seams that already run + * the shared comparand doors (`assertListComparandShapes`, + * `normalizeFilterComparandTypes`), after those doors AND after filter-token + * resolution (the amendment's item 3). Every face downstream of a seam — a + * driver's compiler, an in-process evaluator — receives the lowered filter and + * compiles what it is handed (item 5). Until a face's deletion card lands it + * keeps its own copy of the same rules; each copy is idempotent on lowered + * input (item 9), so a rule is applied at most once in effect. + * + * It holds three rules, all of them compositions of leaves every face already + * compiles (`$and`, `$or`, `$lt`, `$gte`, `$lte`, `$null`): + * + * 1. **`$between`** becomes two conjuncts — `$gte` its minimum and `$lte` its + * maximum — which rule 2 then reads. + * 2. **The whole-day upper bound.** A `$lte` whose comparand is a bare + * `YYYY-MM-DD` becomes `$lt` {@link nextUtcCalendarDay}(day), in the + * calendar-STRING domain: the lowering emits a calendar string, never a + * storage form, so `temporalFilterValue` stays operator-blind and each driver + * converts the bound as it converts any comparand (ADR-0053 D-A1). Because + * the lowering runs before any face converts, D-E3's order — widen the day + * first, convert the bound second — holds by construction on every seam-fed + * face. On the last supported day ({@link UNBOUNDED_ABOVE}) the upper bound + * is dropped: a lone `$lte` keeps only `{ $null: false }`, and a `$between` + * keeps its minimum. An instant or a `Date` comparand is never widened, and + * `$gte` / `$gt` / `$lt` keep their midnight anchor. + * 3. **NULL polarity** — the rulings recorded in code by the four hand copies + * (`driver-sql`, `driver-turso`'s transport, the analytics read scope and its + * `where` normalizer), lowered leaf by leaf exactly as they compile them: + * - a negative-polarity leaf (`$ne` a non-null value, `$nin`, `$notContains`) + * is satisfied by a row with no value: `{ $or: [{ f: { $null: true } }, + * { f: { op: v } }] }` (#5298); + * - every leaf of a `$not` operand is made TOTAL in the direction its own + * operators answer for a missing value (#5146): a leaf no missing value + * satisfies takes a `{ f: { $null: false } }` conjunct, a leaf every + * missing value satisfies takes the `$or` escape above, and a leaf that is + * already total (`$null`, `$exists`, `$empty`, a null `$eq` / `$ne`, a + * `{ $field }` comparison) is left alone. A nested `$not` totalises its own + * operand. + * + * ## Column-type scope (item 7) + * + * Rules 1 and 2 are meant for a `datetime` column. A seam that can read the + * declared field types passes {@link FilterLoweringOptions.isDatetimeColumn}; + * then a `date`, `time` or non-temporal column lowers byte-identical for those + * two rules — the scope `SqlDriver` holds. A seam that cannot omits it and the + * rules apply type-blind, as the type-blind emitters do (sound on `Field.date` + * text, where `< next-day` orders exactly as `<= day`). Rule 3 is not about the + * column's type and applies on every column. + * + * ## Contract + * + * - **Copy-on-write.** The input is never mutated; a subtree nothing rewrote is + * returned by reference, so a filter with nothing to lower comes back as the + * SAME object (the engine seam's allocation contract). + * - **Idempotent.** `lowerFilterCondition(lowerFilterCondition(x))` deep-equals + * `lowerFilterCondition(x)`: rule 2 emits `$lt`, which nothing rewrites + * again; rule 1 leaves no `$between`; and rule 3 recognises the two guards it + * emits (and the same shapes an author wrote) as already total, so it does + * not stack a second guard. + * - **Never refuses.** It is not a door: a shape it does not understand (a + * malformed `$between`, a comparand that is not a bare day, an unknown `$` + * key) is passed through exactly as written for the face that owns its + * refusal. The doors run before it at every seam. + * - **Provenance travels.** A rewritten node carries the filter-subtree + * provenance mark of the node it replaces ({@link markFilterSubtreeProvenance}), + * so a refusal raised downstream resolves to the same author / policy + * attribution it resolved to before the rewrite (#8220). + * - **Closed output vocabulary.** It introduces only `$and`, `$or`, `$lt`, + * `$gte`, `$lte` and `$null` — nothing a seam-fed face cannot already compile. + * + * A pure function of the filter: no I/O, no state, no dialect. It lives beside + * {@link nextUtcCalendarDay} and the comparand doors because what a bare day + * denotes as a bound is protocol (ADR-0053 D-D2), and every seam's package + * already depends on `@objectstack/spec` — never on the package root entry. + */ + +import { isUnboundedAbove, nextUtcCalendarDay } from './calendar-day'; +import { filterSubtreeProvenanceOf, markFilterSubtreeProvenance } from './filter-subtree-provenance'; + +/** How one seam reads the columns it lowers. */ +export interface FilterLoweringOptions { + /** + * A TYPED seam's declared-type reader: does `column` hold a declared + * `datetime`? When given, rules 1 and 2 (the `$between` split and the + * whole-day upper bound) rewrite only the columns it answers `true` for, and + * every other column lowers byte-identical for those two rules. Omit it only + * on a seam that cannot read declarations at all; the two rules then apply + * type-blind (ADR-0053 D-D1, amended, item 7). + */ + readonly isDatetimeColumn?: (column: string) => boolean; +} + +/** + * Lower one `FilterCondition` (see the module note). A value that is not a + * filter node — `undefined`, `null`, anything the doors would have refused — + * is returned as it is. + */ +export function lowerFilterCondition(filter: T, options: FilterLoweringOptions = {}): T { + return lowerNode(filter, options, false, EMPTY_FIELD_SET) as T; +} + +// ── Internals ──────────────────────────────────────────────────────────────── + +const EMPTY_FIELD_SET: ReadonlySet = new Set(); + +/** + * A plain object — the shape every `FilterCondition` node and operator map + * has. The prototype check is load-bearing, as in the comparand doors: a + * `Date`, a `Map` or a class instance is data, not structure. + */ +function isFilterNode(value: unknown): value is Record { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return false; + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + +/** `{ $field: 'other_column' }` — a column reference, never a literal. */ +function isFieldReference(value: unknown): boolean { + return isFilterNode(value) && typeof value.$field === 'string'; +} + +/** An operator map: a plain object with at least one `$` key. */ +function isOperatorMap(spec: unknown): spec is Record { + return isFilterNode(spec) && !isFieldReference(spec) && Object.keys(spec).some((k) => k.startsWith('$')); +} + +/** Copy the provenance mark of `from` onto `to` (a new object replacing it). */ +function carryProvenance(from: unknown, to: T): T { + const mark = filterSubtreeProvenanceOf(from); + return mark === null ? to : markFilterSubtreeProvenance(to, mark); +} + +/** Is `spec` exactly `{ $null: }` — the guard conjunct rule 3 emits? */ +function isNullFlag(spec: unknown, flag: boolean): boolean { + return isFilterNode(spec) && Object.keys(spec).length === 1 && spec.$null === flag; +} + +/** The field of a single-key node `{ f: spec }` whose key is a column, else `null`. */ +function soleField(node: unknown): string | null { + if (!isFilterNode(node)) return null; + const keys = Object.keys(node); + return keys.length === 1 && !keys[0].startsWith('$') ? keys[0] : null; +} + +/** + * Is this `$or` array the NULL escape rule 3 emits — `[{ f: { $null: true } }, + * { f: }]` — on one column? Its second arm is then already total (TRUE + * for a row with no value) and is not guarded a second time. + */ +function nullEscapeField(branches: readonly unknown[]): string | null { + if (branches.length !== 2) return null; + const field = soleField(branches[0]); + if (field === null || soleField(branches[1]) !== field) return null; + return isNullFlag((branches[0] as Record)[field], true) ? field : null; +} + +/** + * The columns a conjunction already requires to hold a value: a key or an + * `$and` arm that is exactly `{ f: { $null: false } }`. A leaf beside such a + * conjunct is FALSE for a row with no value whatever the leaf answers, so a + * `$not` operand needs no second requirement for that column. + */ +function requiredFields(node: Record, inherited: ReadonlySet): ReadonlySet { + let out: Set | undefined; + const add = (field: string): void => { + if (inherited.has(field)) return; + out ??= new Set(inherited); + out.add(field); + }; + for (const [key, value] of Object.entries(node)) { + if (!key.startsWith('$') && isNullFlag(value, false)) add(key); + } + if (Array.isArray(node.$and)) { + for (const arm of node.$and) { + const field = soleField(arm); + if (field !== null && isNullFlag((arm as Record)[field], false)) add(field); + } + } + return out ?? inherited; +} + +/** + * Lower one node. `negated` is true inside a `$not` operand (rule 3 makes its + * leaves total); `required` is the set of columns the enclosing conjunction + * already requires to hold a value. + */ +function lowerNode( + node: unknown, + options: FilterLoweringOptions, + negated: boolean, + inheritedRequired: ReadonlySet, +): unknown { + if (!isFilterNode(node)) return node; + const required = negated ? requiredFields(node, inheritedRequired) : EMPTY_FIELD_SET; + let out: Record | undefined; + const conjuncts: unknown[] = []; + const replace = (key: string, value: unknown): void => { + out ??= { ...node }; + out[key] = value; + }; + const drop = (key: string): void => { + out ??= { ...node }; + delete out[key]; + }; + + for (const [key, value] of Object.entries(node)) { + if (key === '$and' || key === '$or') { + if (!Array.isArray(value)) continue; + // A NULL escape this lowering (or an author) already wrote: its second + // arm is total, so it is lowered for rules 1-2 only. + const escaped = key === '$or' ? nullEscapeField(value) : null; + let copy: unknown[] | undefined; + value.forEach((child, index) => { + const lowered = escaped !== null && index === 1 + ? lowerLeafNode(child as Record, escaped, options) + : lowerNode(child, options, negated, key === '$and' ? required : EMPTY_FIELD_SET); + if (lowered !== child) { + copy ??= [...value]; + copy[index] = lowered; + } + }); + if (copy) replace(key, carryProvenance(value, copy)); + continue; + } + if (key === '$not') { + const lowered = lowerNode(value, options, true, EMPTY_FIELD_SET); + if (lowered !== value) replace(key, lowered); + continue; + } + // Any other `$` key is not a column: it is passed through as written, for + // the face that owns its refusal. + if (key.startsWith('$')) continue; + + const bounded = lowerBounds(key, value, options); + const guarded = guardNullPolarity(key, bounded.spec, negated, required); + if (guarded.conjuncts.length === 0 && bounded.conjuncts.length === 0 && guarded.spec === value) continue; + if (guarded.spec === undefined) drop(key); + else replace(key, guarded.spec); + conjuncts.push(...bounded.conjuncts.flatMap((spec) => guardNullPolarity(key, spec, negated, required).all), ...guarded.conjuncts); + } + + if (conjuncts.length > 0) { + const existing = out?.$and ?? node.$and; + replace('$and', Array.isArray(existing) ? [...existing, ...conjuncts] : conjuncts); + } + return out ? carryProvenance(node, out) : node; +} + +/** + * The second arm of a NULL escape, `{ f: }`: rules 1-2 only (its NULL + * polarity is already the escape's). + */ +function lowerLeafNode( + node: Record, + field: string, + options: FilterLoweringOptions, +): unknown { + const spec = node[field]; + const bounded = lowerBounds(field, spec, options); + if (bounded.spec === spec && bounded.conjuncts.length === 0) return node; + const out: Record = {}; + if (bounded.spec !== undefined) out[field] = bounded.spec; + if (bounded.conjuncts.length > 0) out.$and = bounded.conjuncts.map((c) => ({ [field]: c })); + return carryProvenance(node, out); +} + +/** + * Rules 1 and 2 on one column's spec. Returns the rewritten spec (the SAME + * reference when nothing applied; `undefined` never — a spec always keeps at + * least one operator) and any operator that could not join the map because an + * operator of that name is already in it (the one-operator-per-conjunct rule: + * a lowered key never clobbers an author's own). + */ +function lowerBounds( + field: string, + spec: unknown, + options: FilterLoweringOptions, +): { spec: unknown; conjuncts: Record[] } { + const none = { spec, conjuncts: [] as Record[] }; + if (!isOperatorMap(spec)) return none; + if (!('$lte' in spec) && !('$between' in spec)) return none; + if (options.isDatetimeColumn && !options.isDatetimeColumn(field)) return none; + + const emitted: [string, unknown][] = []; + let changed = false; + for (const [op, comparand] of Object.entries(spec)) { + if (op === '$lte') { + const upper = upperBound(comparand); + if (upper !== null) changed = true; + emitted.push(...(upper ?? [['$lte', comparand] as [string, unknown]])); + continue; + } + if (op === '$between' && isLiteralRange(comparand)) { + changed = true; + emitted.push(['$gte', comparand[0]]); + // On the last supported day the range keeps its minimum alone: `$gte` + // already asks for a value, so no `$null: false` is added beside it. + const upper = upperBound(comparand[1]); + if (upper === null) emitted.push(['$lte', comparand[1]]); + else if (upper[0][0] === '$lt') emitted.push(...upper); + continue; + } + emitted.push([op, comparand]); + } + if (!changed) return none; + + const map: Record = {}; + const conjuncts: Record[] = []; + for (const [op, comparand] of emitted) { + if (Object.prototype.hasOwnProperty.call(map, op)) conjuncts.push({ [op]: comparand }); + else map[op] = comparand; + } + return { spec: carryProvenance(spec, map), conjuncts }; +} + +/** + * Rule 2 on one upper bound: `[['$lt', nextDay]]` for a bare day, + * `[['$null', false]]` on the last supported day, `null` when the bound is not + * a bare calendar day (an instant, a `Date`, a `{ $field }`, anything else) + * and is kept as written. + */ +function upperBound(comparand: unknown): [string, unknown][] | null { + const next = nextUtcCalendarDay(comparand); + if (next === null) return null; + return isUnboundedAbove(next) ? [['$null', false]] : [['$lt', next]]; +} + +/** + * A `$between` rule 1 splits: exactly two LITERAL endpoints. A range holding a + * `{ $field }` reference or other structure is left whole, so the face that + * refuses it still refuses it — splitting it would hand `$gte` a column + * reference, which is a comparison some faces accept. + */ +function isLiteralRange(comparand: unknown): comparand is [unknown, unknown] { + return Array.isArray(comparand) + && comparand.length === 2 + && comparand.every((end) => end === null || typeof end !== 'object' || end instanceof Date); +} + +// ── Rule 3: NULL polarity ──────────────────────────────────────────────────── + +/** How one column constraint is made total for a row with no value. */ +type NullGuard = 'none' | 'requireValue' | 'allowNull'; + +/** + * Does a row with no value satisfy this one operator, in the two-valued + * reading the JS faces give it? The table of the four hand copies + * (`driver-sql`'s `nullValueSatisfiesOperator`), cell for cell. + */ +function nullValueSatisfiesOperator(op: string, value: unknown): boolean { + switch (op) { + case '$eq': return value === null; + case '$ne': return value !== null; + case '$null': return value === true; + case '$exists': return value === false; + case '$empty': return value === true; + case '$nin': return true; + case '$notContains': return true; + default: return false; + } +} + +/** The six scalar comparisons a `{ $field }` comparand compiles as a column-to-column test. */ +const CROSS_FIELD_COMPARISON_OPERATORS: ReadonlySet = new Set([ + '$eq', '$ne', '$gt', '$gte', '$lt', '$lte', +]); + +/** + * Is this operator already TOTAL for a row with no value — TRUE or FALSE, never + * UNKNOWN? The copies' `operatorIsNullTotal`, cell for cell. + */ +function operatorIsNullTotal(op: string, value: unknown): boolean { + if (CROSS_FIELD_COMPARISON_OPERATORS.has(op) && isFieldReference(value)) return true; + switch (op) { + case '$null': + case '$exists': + case '$empty': + return true; + case '$eq': + case '$ne': + return value === null; + default: + return false; + } +} + +/** + * The guard one column constraint needs inside a `$not` operand: total when + * every operator is, satisfied by no value only when every operator is. The + * copies' `nullGuardForFieldSpec`, cell for cell. + */ +function nullGuardForFieldSpec(spec: unknown): NullGuard { + if (spec === null) return 'none'; + if (!isFilterNode(spec)) return 'requireValue'; + let total = true; + let nullSatisfies = true; + for (const [op, value] of Object.entries(spec)) { + if (!operatorIsNullTotal(op, value)) total = false; + if (!nullValueSatisfiesOperator(op, value)) nullSatisfies = false; + } + if (total) return 'none'; + return nullSatisfies ? 'allowNull' : 'requireValue'; +} + +/** + * A negative-polarity operator a row with no value satisfies OUTSIDE a `$not` + * (#5298): `$ne` a value, `$nin`, `$notContains`. A `$ne: null` is the total + * `IS NOT NULL`, and a `$ne: { $field }` is the total column comparison. + */ +function isNegativePolarityOperator(op: string, value: unknown): boolean { + if (op === '$ne') return value !== null && !isFieldReference(value); + return op === '$nin' || op === '$notContains'; +} + +/** `{ $or: [{ f: { $null: true } }, { f: spec }] }` — TRUE for a row with no value. */ +function nullEscape(field: string, spec: unknown): Record { + return { $or: [{ [field]: { $null: true } }, { [field]: spec }] }; +} + +/** + * Rule 3 on one column's spec. Returns the spec that stays under the column's + * key (`undefined` when the whole spec moved into a guard) and the conjuncts + * the guard adds to the enclosing node's `$and`. `all` is the spec and the + * conjuncts as one list of conjuncts, for a spec that is itself a conjunct. + */ +function guardNullPolarity( + field: string, + spec: unknown, + negated: boolean, + required: ReadonlySet, +): { spec: unknown; conjuncts: unknown[]; all: unknown[] } { + if (negated) { + const guard = nullGuardForFieldSpec(spec); + if (guard === 'none' || (guard === 'requireValue' && required.has(field))) { + return { spec, conjuncts: [], all: [{ [field]: spec }] }; + } + const conjuncts = guard === 'requireValue' + ? [{ [field]: { $null: false } }, { [field]: spec }] + : [nullEscape(field, spec)]; + return { spec: undefined, conjuncts, all: conjuncts }; + } + if (!isOperatorMap(spec)) return { spec, conjuncts: [], all: [{ [field]: spec }] }; + const kept: Record = {}; + const conjuncts: unknown[] = []; + for (const [op, value] of Object.entries(spec)) { + if (isNegativePolarityOperator(op, value)) conjuncts.push(nullEscape(field, { [op]: value })); + else kept[op] = value; + } + if (conjuncts.length === 0) return { spec, conjuncts: [], all: [{ [field]: spec }] }; + const rest = Object.keys(kept).length > 0 ? carryProvenance(spec, kept) : undefined; + return { + spec: rest, + conjuncts, + all: rest === undefined ? conjuncts : [{ [field]: rest }, ...conjuncts], + }; +} From 57b357cddf348fa02430a8ab58893d1479d84114 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:03:15 +0000 Subject: [PATCH 2/7] wip(objectql): run the shared lowering after token resolution on every filter position Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- packages/objectql/src/engine.ts | 124 ++++++++++++++++++++++++++++---- packages/spec/src/data/index.ts | 7 ++ 2 files changed, 118 insertions(+), 13 deletions(-) diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 4614dc910a7..240f8c9e269 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -35,6 +35,11 @@ import { normalizeFilterComparandTypes, VALID_AST_OPERATORS, } from '@objectstack/spec/data'; +// [ADR-0053 D-D1, amended 2026-09-30 — #5930] The shared `FilterCondition → +// FilterCondition` lowering, run once per filter position after the doors and +// after token resolution (`resolveThenLowerWhere`), so every driver and the +// in-process `having` / per-aggregation evaluator receive the lowered filter. +import { lowerFilterCondition, type FilterLoweringOptions } from '@objectstack/spec/data'; // [#5574] D6, executable. The ceiling and the refusal message live in // `packages/spec/src/data/bulk-write-hook-conformance.ts` so BOTH phases and // both verbs enforce one definition; the engine raises, the contract decides. @@ -1139,6 +1144,61 @@ function resolveWhereFilterTokens( return resolveFilterTokens(where, filterTokenContextFrom(context)); } +/** + * [ADR-0053 D-D1, amended 2026-09-30 — #5930] Stage 2 of every filter + * position's admission, whole: resolve the placeholders + * ({@link resolveWhereFilterTokens}), THEN run the shared lowering + * (`lowerFilterCondition`, `@objectstack/spec/data`) — the `$between` split, + * the whole-day upper bound and the NULL-polarity guards. + * + * The order is the amendment's item 3: the lowering reads the comparand the + * comparison will run with, so a date macro (`{today}`, `{current_month_end}`) + * has already become the bare day the whole-day rule widens. Run beside the + * doors instead, it would meet `{today}` unresolved and leave it bare. + * + * ONE function for every position, so no verb can resolve without lowering: + * `where` on the five read / write verbs (`ObjectQL.resolveWhereTokens`, + * `ObjectQL.withResolvedWhere`), `aggregations[i].filter` and `having` on + * `aggregate`, and the judge ({@link judgeWhereAdmission}), which runs the + * same stage so its verdict is execution's. The lowering never refuses, so it + * adds no verdict of its own. + * + * `lowering` is the position's declared-type reader: this seam reads the + * object's declarations, so the whole-day rule rewrites a declared + * `datetime` column only ({@link declaredDatetimeLowering}) — the scope + * `SqlDriver` holds, so a `date`, `time` or non-temporal column reaches every + * driver byte-identical to before (the amendment's item 7). + * + * Returns the input by reference when nothing resolved and nothing lowered. + */ +function resolveThenLowerWhere( + where: W, + context: Parameters[0], + lowering: FilterLoweringOptions, +): W { + return lowerFilterCondition(resolveWhereFilterTokens(where, context), lowering); +} + +/** + * [ADR-0053 D-D1 item 7 — #5930] The typed reading of one object's filter + * positions: a column is `datetime` exactly when the object's declared field + * map says `type: 'datetime'` — the same test `SqlDriver` indexes its + * `datetimeFields` by, so the columns this seam widens are a subset of the + * columns every face widens today. No field map (a registry-less host) reads + * no column as `datetime`: the whole-day rule is then left to the faces, as it + * was, rather than applied type-blind to columns no driver would widen. + */ +function declaredDatetimeLowering(schema: unknown): FilterLoweringOptions { + const fields = (schema as { fields?: unknown } | undefined)?.fields; + return { + isDatetimeColumn: (column) => + fields !== null + && typeof fields === 'object' + && Object.prototype.hasOwnProperty.call(fields, column) + && ((fields as Record)[column]?.type === 'datetime'), + }; +} + /** * [#20157] Is this thrown value a door's DIAGNOSTIC (the ADR-0112 envelope: * a string `code` and a numeric `status`), or something else? @@ -1205,7 +1265,7 @@ function judgeWhereAdmission( ): EngineFilterJudgement { try { const admitted = lowerWhereFilterArray(object, operation, { where }, schema); - resolveWhereFilterTokens(admitted.where, context); + resolveThenLowerWhere(admitted.where, context, declaredDatetimeLowering(schema)); return { ok: true }; } catch (thrown) { const refusal = admissionRefusalOf(thrown); @@ -11002,15 +11062,22 @@ export class ObjectQL implements IObjectQLEngine { * is `FILTER_TOKEN_UNKNOWN` / 400. The resolver needs no field type — it walks * values, never keys — so a `having` keyed by aggregate aliases resolves as a * `where` keyed by fields does. + * + * [ADR-0053 D-D1, amended — #5930] …and then LOWERS the position, through + * {@link resolveThenLowerWhere}: resolution and lowering are one stage, so a + * verb cannot run one without the other. `lowering` is required for that + * reason — the position's declared-type reader (`where`: the object's + * fields; `having`: the aggregated row's columns). */ private resolveWhereTokens( ast: QueryAST | undefined, - execCtx?: ExecutionContext, + execCtx: ExecutionContext | undefined, + lowering: FilterLoweringOptions, position: 'where' | 'having' = 'where', ): void { if (!ast || ast[position] == null) return; // [#20157] Through the stage function the judge also calls. - ast[position] = resolveWhereFilterTokens(ast[position], execCtx); + ast[position] = resolveThenLowerWhere(ast[position], execCtx, lowering); } /** @@ -11023,13 +11090,19 @@ export class ObjectQL implements IObjectQLEngine { * made rather than assigning through: `options` belongs to the caller, and * writing back would bake one request's user id into a filter object the * caller may reuse (view metadata and flow node config both get reused). + * + * [ADR-0053 D-D1, amended — #5930] …and lowered, in the same stage + * ({@link resolveThenLowerWhere}), with the object's declared-type reader. + * The lowering is copy-on-write too, so a `where` it rewrites lands on the + * copy, never on the caller's object. */ private withResolvedWhere( options: T, + lowering: FilterLoweringOptions, ): T { if (!options || options.where == null) return options; // [#20157] Through the stage function the judge also calls. - const resolved = resolveWhereFilterTokens(options.where, options.context); + const resolved = resolveThenLowerWhere(options.where, options.context, lowering); return resolved === options.where ? options : ({ ...options, where: resolved } as T); } @@ -11320,7 +11393,9 @@ export class ObjectQL implements IObjectQLEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); + // [ADR-0053 D-D1, amended — #5930] Resolve, then lower (the shared + // lowering), against the object's declared field types. + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findSchema)); await this.executeWithMiddleware(opCtx, async () => { const hookContext: HookContext = { @@ -11592,7 +11667,8 @@ export class ObjectQL implements IObjectQLEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); + // [ADR-0053 D-D1, amended — #5930] Resolve, then lower. + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, declaredDatetimeLowering(_findOneSchema)); await this.executeWithMiddleware(opCtx, async () => { // [#3195] `findOne` fires the SAME `beforeFind`/`afterFind` hooks as @@ -13014,7 +13090,9 @@ export class ObjectQL implements IObjectQLEngine { // Ordering matters: a scalar `where.id` becomes the by-id fast path below, // so an unresolved `{current_user_id}` would be bound as the primary key // itself. Resolve first, then extract. - options = this.withResolvedWhere(options); + // [ADR-0053 D-D1, amended — #5930] …and lowered in the same stage, before + // the by-id extraction below reads the result. + options = this.withResolvedWhere(options, declaredDatetimeLowering(this._registry.getObject(object))); // [#20308] The insert door's rule, same place: a blank on a // non-string-typed column is `null` before the middleware, the @@ -15653,7 +15731,8 @@ export class ObjectQL implements IObjectQLEngine { // Expand `{filter-placeholder}` values before the id is extracted — same // reasoning as update() above (#3810). - options = this.withResolvedWhere(options); + // [ADR-0053 D-D1, amended — #5930] …and lowered in the same stage. + options = this.withResolvedWhere(options, declaredDatetimeLowering(this._registry.getObject(object))); // Extract ID logic mirroring update(): only a SCALAR `where.id` means // "delete one row by primary key". An operator object ({ $in: [...] }, …) @@ -16182,7 +16261,12 @@ export class ObjectQL implements IObjectQLEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); + // [ADR-0053 D-D1, amended — #5930] Resolve, then lower. + this.resolveWhereTokens( + opCtx.ast as QueryAST, + opCtx.context, + declaredDatetimeLowering(this._registry.getObject(object)), + ); // The caller's own `where`, placeholders expanded — captured BEFORE the // middleware chain scopes `opCtx.ast.where`, so the find() fallback below // still passes the unscoped filter (find() applies the read filters itself). @@ -16535,7 +16619,14 @@ export class ObjectQL implements IObjectQLEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); + // [ADR-0053 D-D1, amended — #5930] Each of this verb's three filter + // positions resolves, then lowers, through `resolveThenLowerWhere`: `where` + // and `aggregations[i].filter` narrow the object's raw rows, so they read + // its declared field types; `having` narrows the aggregated row, so it + // reads each aggregated column's type (`min` / `max` of a `datetime` + // field is a `datetime`; a `count` is a number). + const rowLowering = declaredDatetimeLowering(this._registry.getObject(object)); + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, rowLowering); // [#10576] Filter tokens (`{userId}`-style placeholders, #3810) resolve // in per-aggregation filters exactly as they do in `where` — a filter // position is a filter position, and an unresolved placeholder would be @@ -16546,11 +16637,10 @@ export class ObjectQL implements IObjectQLEngine { { const astAggs = (opCtx.ast as QueryAST).aggregations; if (Array.isArray(astAggs) && astAggs.some((a) => (a as { filter?: unknown })?.filter != null)) { - const tokenCtx = filterTokenContextFrom(opCtx.context); (opCtx.ast as QueryAST).aggregations = astAggs.map((a) => { const f = (a as { filter?: unknown })?.filter; if (f == null) return a; - const resolved = resolveFilterTokens(f as any, tokenCtx); + const resolved = resolveThenLowerWhere(f as any, opCtx.context, rowLowering); return resolved === f ? a : { ...(a as object), filter: resolved } as typeof a; }); } @@ -16564,7 +16654,15 @@ export class ObjectQL implements IObjectQLEngine { // doors: the temporal door steps around a `{placeholder}` exactly as // `where`'s does (so, as there, the resolved value is not judged again), // and the other doors judged a string that resolves to a string. - this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context, 'having'); + { + const havingColumnTypes = aggregatedRowColumnTypes(query.groupBy, query.aggregations, declaredFields); + this.resolveWhereTokens( + opCtx.ast as QueryAST, + opCtx.context, + { isDatetimeColumn: (column) => havingColumnTypes.get(column) === 'datetime' }, + 'having', + ); + } await this.executeWithMiddleware(opCtx, async () => { const ast = opCtx.ast as QueryAST; diff --git a/packages/spec/src/data/index.ts b/packages/spec/src/data/index.ts index b0de45a0838..d3125954577 100644 --- a/packages/spec/src/data/index.ts +++ b/packages/spec/src/data/index.ts @@ -131,6 +131,13 @@ export * from './date-macros.zod'; // rule (#8690 C half). `ui/dashboard.zod.ts` re-exports the vocabulary. export * from './date-range-presets'; export * from './calendar-day'; +// [ADR-0053 D-D1, amended 2026-09-30 — #5930] The shared `FilterCondition → +// FilterCondition` lowering the seams run once, after the comparand doors and +// after filter-token resolution: the `$between` split, the whole-day upper +// bound in the calendar-string domain, and the NULL-polarity guards. Beside +// `calendar-day` because what a bare day denotes as a bound is protocol, and +// on this subpath only — never the package root entry (the ruling's D3). +export * from './filter-lowering'; // Session-scoped filter placeholders ({current_user_id} / {current_org_id}) — // the sibling vocabulary to date macros. Presentation scope only; RLS is the // enforcement boundary. See context-tokens.zod.ts. From 2352e2f35c33a226d8e5f9a64a78be4b4c7351ee Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:08:43 +0000 Subject: [PATCH 3/7] wip(plugin-security): run the shared lowering at the RLS compile seam, typed by the declared datetime columns Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .../plugin-security/src/rls-compiler.ts | 51 +++++++++++++++++-- .../plugin-security/src/security-plugin.ts | 19 ++++++- 2 files changed, 64 insertions(+), 6 deletions(-) diff --git a/packages/plugins/plugin-security/src/rls-compiler.ts b/packages/plugins/plugin-security/src/rls-compiler.ts index fbc9822d80f..4e5e8181d77 100644 --- a/packages/plugins/plugin-security/src/rls-compiler.ts +++ b/packages/plugins/plugin-security/src/rls-compiler.ts @@ -18,6 +18,11 @@ import type { CelFilterFailReason } from '@objectstack/formula'; // engine's `where` seam and analytics' read-scope guard call. Called on every // compiled policy filter, never restated (see `judgeCompiledComparands`). import { assertListComparandShapes, normalizeFilterComparandTypes } from '@objectstack/spec/data'; +// [ADR-0053 D-D1, amended 2026-09-30 — #5930] The shared lowering, run on every +// compiled policy filter right after the two faces above +// (`judgeCompiledComparands`), so `using` and `check` hand their consumers the +// same lowered filter. +import { lowerFilterCondition, type FilterLoweringOptions } from '@objectstack/spec/data'; /** * Why a policy's predicate produced no filter — the compiler's OWN answer, @@ -117,6 +122,18 @@ interface RLSUserContext { export interface RlsFieldGuard { /** Every column the object declares, exactly as `getObjectFieldNames` builds it. */ declared: ReadonlySet; + /** + * [ADR-0053 D-D1 item 7 — #5930] The columns the object declares as + * `datetime`, read from the same declaration as {@link declared}. The shared + * lowering's whole-day rule (a bare-day `$lte`, a `$between`) rewrites these + * columns only, the scope `SqlDriver` holds — so a `date`, `time` or + * non-temporal column reaches `using`'s drivers and `check`'s evaluator + * byte-identical to before. Absent (no guard, or a caller that did not read + * the types) reads NO column as `datetime`: the rule is left to each face's + * own copy, as it was, never applied type-blind to a column no driver + * widens. The NULL-polarity guards do not depend on it. + */ + datetime?: ReadonlySet; } /** {@link judgeCompiledFields}' answer. */ @@ -277,14 +294,30 @@ type RlsComparandVerdict = * it, and there a refusal could only be a 400 the caller cannot act on. * * Returns the faces' own filter (the type face narrows an exact-range `bigint` - * copy-on-write and returns the same reference otherwise). A thrown value + * copy-on-write and returns the same reference otherwise), then LOWERED by the + * shared `lowerFilterCondition` (ADR-0053 D-D1, amended — #5930): this is the + * RLS compile seam the amendment names, so the `$between` split, the whole-day + * upper bound on the guard's `datetime` columns and the NULL-polarity guards + * are applied here once, for `using` and `check` alike. A thrown value * without the ADR-0112 envelope (a string `code` and a numeric `status`) is not * a verdict about the filter, so it is re-thrown rather than dressed up as one. */ -function judgeCompiledComparands(filter: Record): RlsComparandVerdict { +function judgeCompiledComparands( + filter: Record, + lowering: FilterLoweringOptions, +): RlsComparandVerdict { try { assertListComparandShapes(filter); - return { ok: true, filter: normalizeFilterComparandTypes(filter) }; + // [ADR-0053 D-D1, amended — #5930] The RLS compile seam's lowering, AFTER + // both faces (the amendment's items 2-3). No token resolution precedes it + // because none exists on either clause: the engine resolves the caller's + // `where` before its middleware composes this filter, and the write check + // evaluates it directly — measured, a policy comparand `'{today}'` compiles + // and reaches both consumers verbatim, so the lowering reads it as the + // non-day string it is and leaves it as written, as every face does today. + // Never refuses: a shape it does not lower passes through for the face that + // owns its refusal. + return { ok: true, filter: lowerFilterCondition(normalizeFilterComparandTypes(filter), lowering) }; } catch (thrown) { const { code, status } = (thrown ?? {}) as { code?: unknown; status?: unknown }; if (!(thrown instanceof Error) || typeof code !== 'string' || typeof status !== 'number') throw thrown; @@ -301,6 +334,16 @@ function judgeCompiledComparands(filter: Record): RlsComparandV } } +/** + * [ADR-0053 D-D1 item 7 — #5930] The RLS seam's declared-type reader: a column + * is `datetime` when the caller's guard says so ({@link RlsFieldGuard.datetime}), + * and no column is when it carries no type set. + */ +function rlsLowering(fieldGuard: RlsFieldGuard | undefined): FilterLoweringOptions { + const datetime = fieldGuard?.datetime; + return { isDatetimeColumn: (column) => datetime?.has(column) === true }; +} + /** * Sentinel filter used when applicable RLS policies exist but none can * be compiled against the current execution context (typically because a @@ -644,7 +687,7 @@ export class RLSCompiler { // refusal joins `deniedBy` like the rows above — the per-request // fail-closed route a list under `==` already takes — so `using` and // `check` refuse together and a granting sibling still grants. - const comparands = judgeCompiledComparands(outcome.filter); + const comparands = judgeCompiledComparands(outcome.filter, rlsLowering(fieldGuard)); if (comparands.ok) { filters.push(comparands.filter); POLICY_OF_COMPILED_FILTER.set(comparands.filter, (policy as { name?: string }).name ?? '(unnamed)'); diff --git a/packages/plugins/plugin-security/src/security-plugin.ts b/packages/plugins/plugin-security/src/security-plugin.ts index 5313307af7c..688588e1dd1 100644 --- a/packages/plugins/plugin-security/src/security-plugin.ts +++ b/packages/plugins/plugin-security/src/security-plugin.ts @@ -1003,6 +1003,14 @@ export class SecurityPlugin implements Plugin { * invalidation — a kernel restart drops the cache. */ private readonly fieldNamesCache = new Map | null>(); + /** + * [ADR-0053 D-D1 item 7 — #5930] Each object's declared `datetime` columns, + * read in the SAME pass as {@link fieldNamesCache} (`loadObjectFieldNames`) + * and invalidated with it. Handed to the RLS compile seam as + * `RlsFieldGuard.datetime`, so the shared lowering's whole-day rule rewrites + * a policy's `datetime` columns only — the scope every driver holds. + */ + private readonly datetimeFieldNamesCache = new Map>(); /** * Per-object cache of tenancy opt-out. `true` means the schema * explicitly disabled multi-tenancy (`tenancy.enabled === false` or @@ -1382,6 +1390,7 @@ export class SecurityPlugin implements Plugin { if (typeof md?.watch === 'function') { this.metadataWatch = md.watch('*', () => { this.fieldNamesCache.clear(); + this.datetimeFieldNamesCache.clear(); this.tenancyDisabledCache.clear(); this.cbpRelCache.clear(); this.objectSecurityMetaCache.clear(); @@ -7203,7 +7212,7 @@ export class SecurityPlugin implements Plugin { compilable, context, 'using', - objectFields ? { declared: objectFields } : undefined, + objectFields ? { declared: objectFields, datetime: this.datetimeFieldNamesCache.get(object) } : undefined, ); // Every applicable policy dropped for a missing field → deny sentinel. if (layer1 == null && dropped > 0) { @@ -7469,7 +7478,7 @@ export class SecurityPlugin implements Plugin { withCheck, context, 'check', - objectFields ? { declared: objectFields } : undefined, + objectFields ? { declared: objectFields, datetime: this.datetimeFieldNamesCache.get(object) } : undefined, ); } @@ -8890,19 +8899,25 @@ export class SecurityPlugin implements Plugin { (obj as any)?.systemFields?.tenant === false; this.tenancyDisabledCache.set(objectName, !!tenancyDisabled); const set = new Set(['id']); + // [ADR-0053 D-D1 item 7 — #5930] The `datetime` columns, from the same + // declaration (see `datetimeFieldNamesCache`). + const datetime = new Set(); if (Array.isArray(obj.fields)) { for (const f of obj.fields) { if (f?.name) set.add(String(f.name)); + if (f?.name && f.type === 'datetime') datetime.add(String(f.name)); } } else if (typeof obj.fields === 'object') { for (const key of Object.keys(obj.fields)) { set.add(key); const v = (obj.fields as Record)[key]; if (v && typeof v === 'object' && v.name) set.add(String(v.name)); + if (v && typeof v === 'object' && v.type === 'datetime') datetime.add(key); } } else { return null; } + this.datetimeFieldNamesCache.set(objectName, datetime); return set; } catch { return null; From 2dda79b9afebda9072ec5d7460ff76be87d20af1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:21:44 +0000 Subject: [PATCH 4/7] wip: lowering unit table, engine seam pins, door pins read the lowered driver input Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .../src/engine-filter-array-lowering.test.ts | 12 +- ...umber-comparand-declared-type-door.test.ts | 20 +- ...engine-shared-filter-lowering-seam.test.ts | 172 ++++++++++++ ...e-text-operator-declared-type-door.test.ts | 18 +- .../spec/src/data/filter-lowering.test.ts | 248 ++++++++++++++++++ 5 files changed, 462 insertions(+), 8 deletions(-) create mode 100644 packages/objectql/src/engine-shared-filter-lowering-seam.test.ts create mode 100644 packages/spec/src/data/filter-lowering.test.ts diff --git a/packages/objectql/src/engine-filter-array-lowering.test.ts b/packages/objectql/src/engine-filter-array-lowering.test.ts index aa2660571b2..94f7c641244 100644 --- a/packages/objectql/src/engine-filter-array-lowering.test.ts +++ b/packages/objectql/src/engine-filter-array-lowering.test.ts @@ -673,8 +673,11 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158) await engine.find('deal', asFilterArrayQuery([['stage', 'in', ['won', 'lost']]])); expect(lastWhere()).toEqual({ stage: { $in: ['won', 'lost'] } }); + // [ADR-0053 D-D1, amended — #5930] `$nin` reaches the driver with the + // shared lowering's NULL escape around it, the #5298 reading every face + // already gives it; the list itself is untouched. await engine.find('deal', asFilterArrayQuery([['stage', 'not_in', ['lost']]])); - expect(lastWhere()).toEqual({ stage: { $nin: ['lost'] } }); + expect(lastWhere()).toEqual({ $and: [{ $or: [{ stage: { $null: true } }, { stage: { $nin: ['lost'] } }] }] }); await engine.find('deal', asFilterArrayQuery([['amount', 'between', [5, 25]]])); expect(lastWhere()).toEqual({ amount: { $between: [5, 25] } }); @@ -686,7 +689,8 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158) await engine.find('deal', { where: { stage: { $in: [] } } }); expect(lastWhere()).toEqual({ stage: { $in: [] } }); await engine.find('deal', { where: { stage: { $nin: [] } } }); - expect(lastWhere()).toEqual({ stage: { $nin: [] } }); + // [ADR-0053 D-D1, amended — #5930] …inside the shared lowering's NULL escape. + expect(lastWhere()).toEqual({ $and: [{ $or: [{ stage: { $null: true } }, { stage: { $nin: [] } }] }] }); }); it('the gate does not re-judge list MEMBERS — that is #5234, on another face', async () => { @@ -728,8 +732,10 @@ describe('Door 2 lowers FilterArray to FilterCondition before the driver (#5158) }); it('a scalar on a NON-collection operator is untouched', async () => { + // [ADR-0053 D-D1, amended — #5930] `$ne` reaches the driver inside the + // shared lowering's NULL escape; the scalar comparand is untouched. await engine.find('deal', asFilterArrayQuery([['stage', '!=', 'won']])); - expect(lastWhere()).toEqual({ stage: { $ne: 'won' } }); + expect(lastWhere()).toEqual({ $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] }); // String bounds on a range comparison stay legal, and since #5685 the // declaration agrees: `FieldOperatorsSchema` now declares `$gt` as // number|Date|string|FieldReference, matching the ISO strings the showcase diff --git a/packages/objectql/src/engine-number-comparand-declared-type-door.test.ts b/packages/objectql/src/engine-number-comparand-declared-type-door.test.ts index 06b3e1d68db..f00585caf64 100644 --- a/packages/objectql/src/engine-number-comparand-declared-type-door.test.ts +++ b/packages/objectql/src/engine-number-comparand-declared-type-door.test.ts @@ -52,6 +52,7 @@ import { NUMBER_COMPARAND_DOOR_FIXTURE_OBJECT, NUMBER_COMPARAND_DOOR_LIST_OPERATORS, NUMBER_COMPARAND_DOOR_SCALAR_OPERATORS, + lowerFilterCondition, type EngineAggregateOptions, type EngineQueryOptions, type FilterCondition, @@ -68,6 +69,19 @@ import { const OBJECT = NUMBER_COMPARAND_DOOR_FIXTURE_OBJECT; +/** + * [ADR-0053 D-D1, amended — #5930] What a driver receives is the door's output + * after the engine's shared lowering (the NULL-polarity guards on `$ne` / `$nin` + * / `$notContains`, the whole-day rule on a declared `datetime`), which runs + * after this door on every verb. The door adds nothing beyond that, so a pin on + * the driver's input compares against the same lowering of the door's answer. + */ +const lowered = (where: unknown): unknown => + lowerFilterCondition(where, { + isDatetimeColumn: (column) => + (NUMBER_COMPARAND_DOOR_FIXTURE.fields as Record)[column]?.type === 'datetime', + }); + interface SeenRead { ast: any } /** Minimal recording driver — the same witness shape as the sibling door suites. */ @@ -195,7 +209,7 @@ describe('[#20351] the number-comparand declared-type door at the engine collect const asWritten = JSON.stringify(filter); await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); - expect(reads[0]?.ast?.where, `${c.name}: the driver must receive the number`).toEqual(c.expectedFilter()); + expect(reads[0]?.ast?.where, `${c.name}: the driver must receive the number`).toEqual(lowered(c.expectedFilter())); // Copy-on-write: the filter belongs to the caller (view metadata, flow config). expect(JSON.stringify(filter), `${c.name}: the caller's filter must not be edited`).toBe(asWritten); } @@ -207,7 +221,7 @@ describe('[#20351] the number-comparand declared-type door at the engine collect const filter = c.filter(); await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); - expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(filter); + expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(lowered(filter)); } }); @@ -291,7 +305,7 @@ describe('[#20351] the number-comparand declared-type door at the engine collect expect(reads).toHaveLength(0); } await engine.find(OBJECT, { where: { $or: [{ f_text: 'a' }, { $not: { f_number: { $in: ['12', 5] } } }] } }); - expect(reads[0]?.ast?.where).toEqual({ $or: [{ f_text: 'a' }, { $not: { f_number: { $in: [12, 5] } } }] }); + expect(reads[0]?.ast?.where).toEqual(lowered({ $or: [{ f_text: 'a' }, { $not: { f_number: { $in: [12, 5] } } }] })); }); it('refuses a {placeholder} against a number field UNRESOLVED — before the token resolver, in the door\'s words', async () => { diff --git a/packages/objectql/src/engine-shared-filter-lowering-seam.test.ts b/packages/objectql/src/engine-shared-filter-lowering-seam.test.ts new file mode 100644 index 00000000000..de52ae05502 --- /dev/null +++ b/packages/objectql/src/engine-shared-filter-lowering-seam.test.ts @@ -0,0 +1,172 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [ADR-0053 D-D1, amended 2026-09-30 — #5930] The engine's placement of the + * shared `FilterCondition → FilterCondition` lowering (`lowerFilterCondition`, + * `@objectstack/spec/data`): once per filter position, AFTER the comparand doors + * and AFTER filter-token resolution (the amendment's item 3), on every verb that + * takes a filter — `find`, `findOne`, `count`, `update`, `delete`, and + * `aggregate`'s three positions (`where`, `aggregations[i].filter`, `having`). + * + * The witness is what leaves the engine: the recording driver's `where` for the + * verbs a driver compiles, and the operation context a middleware reads for the + * two positions the engine evaluates itself. Row answers are the faces' suites' + * business; these pins say WHERE the rule runs, so reverting any one seam call + * turns its pin red. + * + * Fixture: `closed_at` is the declared `datetime` the whole-day rule is for, + * `due_on` a `date` the typed seam must leave byte-identical (item 7). + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { EngineAggregateOptions, EngineQueryOptions, FilterCondition } from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; + +const OBJECT = 'lowering_probe'; + +const SCHEMA = { + name: OBJECT, + label: 'Lowering probe', + fields: { + id: { name: 'id', type: 'text' }, + stage: { name: 'stage', type: 'text' }, + amount: { name: 'amount', type: 'number' }, + closed_at: { name: 'closed_at', type: 'datetime' }, + due_on: { name: 'due_on', type: 'date' }, + }, +}; + +interface Seen { verb: string; ast: any } + +function makeRecordingDriver() { + const seen: Seen[] = []; + const driver: any = { + name: 'recording', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(_o: string, ast: any) { seen.push({ verb: 'find', ast }); return []; }, + async findOne(_o: string, ast: any) { seen.push({ verb: 'findOne', ast }); return null; }, + async count(_o: string, ast: any) { seen.push({ verb: 'count', ast }); return 0; }, + async create(_o: string, data: Record) { return { ...data, id: 'r1' }; }, + async update(_o: string, id: string, data: Record) { return { ...data, id }; }, + async updateMany(_o: string, ast: any) { seen.push({ verb: 'updateMany', ast }); return 0; }, + async delete() { return true; }, + async deleteMany(_o: string, ast: any) { seen.push({ verb: 'deleteMany', ast }); return 0; }, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, seen }; +} + +/** A bare-day upper bound on the datetime column, and what the lowering makes of it. */ +const WHOLE_DAY_IN = { closed_at: { $lte: '2026-07-28' } }; +const WHOLE_DAY_OUT = { closed_at: { $lt: '2026-07-29' } }; +/** A negative-polarity leaf, and its NULL escape. */ +const NEGATIVE_IN = { stage: { $ne: 'won' } }; +const NEGATIVE_OUT = { $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] }; + +describe('[ADR-0053 D-D1 amended — #5930] the engine runs the shared lowering once per filter position', () => { + let engine: ObjectQL; + let seen: Seen[]; + + beforeEach(async () => { + const rec = makeRecordingDriver(); + seen = rec.seen; + engine = new ObjectQL(); + engine.registerDriver(rec.driver, true); + await engine.init(); + engine.registry.registerObject(SCHEMA as any, 'test'); + seen.length = 0; + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + const lastWhere = (verb: string) => [...seen].reverse().find((s) => s.verb === verb)?.ast?.where; + + it('find: the driver receives the lowered where — whole-day bound and NULL escape', async () => { + await engine.find(OBJECT, { where: WHOLE_DAY_IN }); + expect(lastWhere('find')).toEqual(WHOLE_DAY_OUT); + await engine.find(OBJECT, { where: NEGATIVE_IN }); + expect(lastWhere('find')).toEqual(NEGATIVE_OUT); + }); + + it('findOne: the driver receives the lowered where', async () => { + await engine.findOne(OBJECT, { where: WHOLE_DAY_IN }); + expect(lastWhere('findOne') ?? lastWhere('find')).toEqual(WHOLE_DAY_OUT); + }); + + it('count: the driver receives the lowered where', async () => { + await engine.count(OBJECT, { where: NEGATIVE_IN }); + expect(lastWhere('count')).toEqual(NEGATIVE_OUT); + }); + + it('update (multi): the driver receives the lowered where', async () => { + await engine.update(OBJECT, { stage: 'x' }, { where: WHOLE_DAY_IN, multi: true } as any); + expect(lastWhere('updateMany')).toEqual(WHOLE_DAY_OUT); + }); + + it('delete (multi): the driver receives the lowered where', async () => { + await engine.delete(OBJECT, { where: NEGATIVE_IN, multi: true } as any); + expect(lastWhere('deleteMany')).toEqual(NEGATIVE_OUT); + }); + + it('aggregate: `where`, `aggregations[i].filter` and `having` are each lowered, before the middleware chain', async () => { + let atMiddleware: any; + engine.registerMiddleware(async (opCtx: any, next: any) => { + if (opCtx.operation === 'aggregate') atMiddleware = structuredClone(opCtx.ast); + return next(); + }); + await engine.aggregate(OBJECT, { + where: WHOLE_DAY_IN, + groupBy: ['stage'], + aggregations: [ + { function: 'count', alias: 'n' }, + { function: 'count', alias: 'open_n', filter: NEGATIVE_IN }, + { function: 'max', field: 'closed_at', alias: 'last_closed' }, + ], + having: { last_closed: { $between: ['2026-07-01', '2026-07-28'] } }, + } as EngineAggregateOptions); + expect(atMiddleware.where).toEqual(WHOLE_DAY_OUT); + expect(atMiddleware.aggregations[0].filter).toBeUndefined(); + expect(atMiddleware.aggregations[1].filter).toEqual(NEGATIVE_OUT); + // `max(closed_at)` is a `datetime` column of the aggregated row, so the + // whole-day rule reaches it; the aggregated row's type is what `having` reads. + expect(atMiddleware.having).toEqual({ last_closed: { $gte: '2026-07-01', $lt: '2026-07-29' } }); + // The rows path asks the driver for rows with the lowered `where` too. + expect(lastWhere('find')).toEqual(WHOLE_DAY_OUT); + }); + + it('item 3: the lowering runs AFTER token resolution — `{today}` resolves to the day, then widens', async () => { + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-03-10T12:00:00.000Z')); + await engine.find(OBJECT, { where: { closed_at: { $lte: '{today}' } } } as EngineQueryOptions); + expect(lastWhere('find')).toEqual({ closed_at: { $lt: '2026-03-11' } }); + await engine.update(OBJECT, { stage: 'x' }, { where: { closed_at: { $lte: '{today}' } }, multi: true } as any); + expect(lastWhere('updateMany')).toEqual({ closed_at: { $lt: '2026-03-11' } }); + }); + + it('item 7: a typed seam leaves a `date` column, and a `$between` on a number, byte-identical', async () => { + const where = { due_on: { $lte: '2026-07-28' }, amount: { $between: [5, 25] } }; + await engine.find(OBJECT, { where }); + expect(lastWhere('find')).toBe(where); + }); + + it('the last supported day: `$lte` keeps only { $null: false }', async () => { + await engine.find(OBJECT, { where: { closed_at: { $lte: '9999-12-31' } } }); + expect(lastWhere('find')).toEqual({ closed_at: { $null: false } }); + }); + + it('copy-on-write: the caller\'s filter object is never edited', async () => { + const where: FilterCondition = { closed_at: { $lte: '2026-07-28' }, stage: { $ne: 'won' } }; + const asWritten = JSON.stringify(where); + await engine.find(OBJECT, { where }); + await engine.update(OBJECT, { stage: 'x' }, { where, multi: true } as any); + expect(JSON.stringify(where)).toBe(asWritten); + }); + + it('the judge runs the same stage and gains no verdict from it', () => { + expect(engine.judgeFilter(OBJECT, { closed_at: { $between: ['2026-07-01', '2026-07-28'] } })).toEqual({ ok: true }); + expect(engine.judgeFilter(OBJECT, { $not: { stage: { $ne: 'won' } } })).toEqual({ ok: true }); + }); +}); diff --git a/packages/objectql/src/engine-text-operator-declared-type-door.test.ts b/packages/objectql/src/engine-text-operator-declared-type-door.test.ts index 440cf85c86b..38d40a47897 100644 --- a/packages/objectql/src/engine-text-operator-declared-type-door.test.ts +++ b/packages/objectql/src/engine-text-operator-declared-type-door.test.ts @@ -68,6 +68,7 @@ import { TEXT_OPERATOR_DOOR_FIXTURE, TEXT_OPERATOR_DOOR_FIXTURE_OBJECT, TEXT_OPERATOR_DOOR_TYPE_CLASSES, + lowerFilterCondition, type FilterTextCase, type FilterTextRowsCase, type TextOperatorDoorCase, @@ -81,6 +82,19 @@ import { findTextOperatorOverNonTextField } from './text-operator-declared-type- const OBJECT = TEXT_OPERATOR_DOOR_FIXTURE_OBJECT; +/** + * [ADR-0053 D-D1, amended — #5930] What a driver receives is the door's output + * after the engine's shared lowering (the NULL-polarity guards on `$ne` / `$nin` + * / `$notContains`, the whole-day rule on a declared `datetime`), which runs + * after this door on every verb. The door rewrites nothing, so a pin on the + * driver's input compares against the same lowering of the caller's filter. + */ +const lowered = (where: unknown): unknown => + lowerFilterCondition(where, { + isDatetimeColumn: (column) => + (TEXT_OPERATOR_DOOR_FIXTURE.fields as Record)[column]?.type === 'datetime', + }); + interface SeenRead { ast: any } /** Minimal recording driver — the same witness shape as the #7872 door suite. */ @@ -187,7 +201,7 @@ describe('[#15773] the text-operator declared-type door at the engine collection const filter = c.filter(); await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); - expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(filter); + expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(lowered(filter)); } }); @@ -197,7 +211,7 @@ describe('[#15773] the text-operator declared-type door at the engine collection const filter = c.filter(); await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); - expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(filter); + expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(lowered(filter)); } }); diff --git a/packages/spec/src/data/filter-lowering.test.ts b/packages/spec/src/data/filter-lowering.test.ts new file mode 100644 index 00000000000..9be3b3bcea4 --- /dev/null +++ b/packages/spec/src/data/filter-lowering.test.ts @@ -0,0 +1,248 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The shared `FilterCondition → FilterCondition` lowering's own unit table + * (ADR-0053 D-D1, amended 2026-09-30 — #5930): each rule input → output, the + * column-type scope, idempotence, the closure of the output vocabulary, + * copy-on-write, provenance, and the shapes it deliberately passes through. + * An ordinary suite, not a gate. The seams that call it pin their own + * placement (`@objectstack/objectql`, `@objectstack/plugin-security`). + */ + +import { describe, expect, it } from 'vitest'; +import { lowerFilterCondition, type FilterLoweringOptions } from './filter-lowering'; +import { filterSubtreeProvenanceOf, markFilterSubtreeProvenance } from './filter-subtree-provenance'; +import { FILTER_LOGIC_CASES } from './filter-logic-conformance'; +import { TEMPORAL_CASES } from './temporal-conformance'; + +/** A typed seam whose `at` column is the one declared `datetime`. */ +const TYPED: FilterLoweringOptions = { isDatetimeColumn: (column) => column === 'at' }; + +interface Row { + readonly name: string; + readonly input: unknown; + readonly output: unknown; + readonly options?: FilterLoweringOptions; +} + +/** Rules 1 and 2 — the `$between` split and the whole-day upper bound. */ +const BOUND_ROWS: readonly Row[] = [ + { name: '$lte on a bare day → $lt the next day, in the calendar-string domain', + input: { at: { $lte: '2026-07-28' } }, output: { at: { $lt: '2026-07-29' } }, options: TYPED }, + { name: 'month and year rollover', + input: { at: { $lte: '2026-12-31' } }, output: { at: { $lt: '2027-01-01' } }, options: TYPED }, + { name: 'leap day', + input: { at: { $lte: '2024-02-28' } }, output: { at: { $lt: '2024-02-29' } }, options: TYPED }, + { name: '$lte on the last supported day keeps only { $null: false }', + input: { at: { $lte: '9999-12-31' } }, output: { at: { $null: false } }, options: TYPED }, + { name: '$between → $gte its minimum, and the whole-day rule on its maximum', + input: { at: { $between: ['2026-07-01', '2026-07-28'] } }, output: { at: { $gte: '2026-07-01', $lt: '2026-07-29' } }, options: TYPED }, + { name: '$between whose maximum is the last supported day keeps its minimum alone', + input: { at: { $between: ['2026-07-01', '9999-12-31'] } }, output: { at: { $gte: '2026-07-01' } }, options: TYPED }, + { name: '$between whose maximum is an instant splits into $gte / $lte, the maximum kept as written', + input: { at: { $between: ['2026-07-01', '2026-07-28T10:00:00.000Z'] } }, + output: { at: { $gte: '2026-07-01', $lte: '2026-07-28T10:00:00.000Z' } }, options: TYPED }, + { name: 'an instant $lte is never widened', + input: { at: { $lte: '2026-07-28T10:00:00.000Z' } }, output: { at: { $lte: '2026-07-28T10:00:00.000Z' } }, options: TYPED }, + { name: 'a Date $lte is never widened', + input: { at: { $lte: new Date('2026-07-28T00:00:00.000Z') } }, output: { at: { $lte: new Date('2026-07-28T00:00:00.000Z') } }, options: TYPED }, + { name: 'an impossible day is not a calendar day and is kept as written', + input: { at: { $lte: '2026-02-30' } }, output: { at: { $lte: '2026-02-30' } }, options: TYPED }, + { name: '$gte / $gt / $lt keep their midnight anchor', + input: { at: { $gte: '2026-07-01', $gt: '2026-07-02', $lt: '2026-07-28' } }, + output: { at: { $gte: '2026-07-01', $gt: '2026-07-02', $lt: '2026-07-28' } }, options: TYPED }, + { name: 'the dashboard window { $gte, $lte } keeps its lower bound and widens its upper', + input: { at: { $gte: '2026-07-01', $lte: '2026-07-28' } }, output: { at: { $gte: '2026-07-01', $lt: '2026-07-29' } }, options: TYPED }, + { name: 'a lowered key never clobbers an author\'s own: the collision becomes its own conjunct', + input: { at: { $lt: '2026-07-20', $lte: '2026-07-28' } }, + output: { at: { $lt: '2026-07-20' }, $and: [{ at: { $lt: '2026-07-29' } }] }, options: TYPED }, + { name: 'the rule reaches every depth: $and, $or and $not', + input: { $or: [{ at: { $lte: '2026-07-28' } }, { $and: [{ at: { $between: ['2026-01-01', '2026-01-31'] } }] }] }, + output: { $or: [{ at: { $lt: '2026-07-29' } }, { $and: [{ at: { $gte: '2026-01-01', $lt: '2026-02-01' } }] }] }, + options: TYPED }, +]; + +/** Item 7 — the column-type scope. */ +const SCOPE_ROWS: readonly Row[] = [ + { name: 'typed seam: a date column lowers byte-identical', + input: { on: { $lte: '2026-07-28' } }, output: { on: { $lte: '2026-07-28' } }, options: TYPED }, + { name: 'typed seam: a $between on a non-datetime column is left whole', + input: { amount: { $between: [5, 25] } }, output: { amount: { $between: [5, 25] } }, options: TYPED }, + { name: 'typed seam: a text column holding a day is not widened', + input: { code: { $lte: '2026-07-28' } }, output: { code: { $lte: '2026-07-28' } }, options: TYPED }, + { name: 'type-blind seam: every column takes the rule (sound on date text: < next-day orders as <= day)', + input: { on: { $lte: '2026-07-28' } }, output: { on: { $lt: '2026-07-29' } } }, + { name: 'type-blind seam: $between splits on any column', + input: { amount: { $between: [5, 25] } }, output: { amount: { $gte: 5, $lte: 25 } } }, +]; + +/** Rule 3 — NULL polarity, leaf by leaf, exactly as the four hand copies compile it. */ +const NULL_ROWS: readonly Row[] = [ + { name: '$ne a value: a row with no value satisfies it (#5298)', + input: { stage: { $ne: 'won' } }, output: { $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] } }, + { name: '$nin: a row with no value satisfies it (#5298)', + input: { stage: { $nin: ['won'] } }, output: { $and: [{ $or: [{ stage: { $null: true } }, { stage: { $nin: ['won'] } }] }] } }, + { name: '$notContains: a row with no value satisfies it (#5298)', + input: { name: { $notContains: 'x' } }, output: { $and: [{ $or: [{ name: { $null: true } }, { name: { $notContains: 'x' } }] }] } }, + { name: '$ne: null is the total IS NOT NULL and is left alone', + input: { stage: { $ne: null } }, output: { stage: { $ne: null } } }, + { name: '$ne a { $field } is a total column comparison and is left alone', + input: { a: { $ne: { $field: 'b' } } }, output: { a: { $ne: { $field: 'b' } } } }, + { name: 'only the negative operator moves; its siblings stay under the key', + input: { amount: { $gt: 1, $ne: 5 } }, output: { amount: { $gt: 1 }, $and: [{ $or: [{ amount: { $null: true } }, { amount: { $ne: 5 } }] }] } }, + { name: '$not over a positive leaf requires a value (#5146)', + input: { $not: { stage: 'won' } }, output: { $not: { $and: [{ stage: { $null: false } }, { stage: 'won' }] } } }, + { name: '$not over a negative leaf keeps the NULL escape (#5146)', + input: { $not: { stage: { $ne: 'won' } } }, output: { $not: { $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] } } }, + { name: '$not over a total leaf is left alone', + input: { $not: { stage: { $null: true } } }, output: { $not: { stage: { $null: true } } } }, + { name: '$not totalises leaf by leaf through $or (De Morgan stays sound)', + input: { $not: { $or: [{ a: 1 }, { b: { $nin: [1] } }] } }, + output: { $not: { $or: [{ $and: [{ a: { $null: false } }, { a: 1 }] }, { $and: [{ $or: [{ b: { $null: true } }, { b: { $nin: [1] } }] }] }] } } }, + { name: 'the whole-day rule runs first, then the $not guard reads the lowered leaf', + input: { $not: { at: { $lte: '2026-07-28' } } }, output: { $not: { $and: [{ at: { $null: false } }, { at: { $lt: '2026-07-29' } }] } }, options: TYPED }, + { name: 'a positive leaf outside $not is left alone (UNKNOWN already reads as no match there)', + input: { a: 1, b: { $gt: 2 }, c: { $in: [1] } }, output: { a: 1, b: { $gt: 2 }, c: { $in: [1] } } }, +]; + +/** What the lowering deliberately passes through — it is not a door. */ +const PASS_THROUGH_ROWS: readonly Row[] = [ + { name: 'a malformed $between (not a pair) is left for the face that refuses it', + input: { at: { $between: ['2026-07-01'] } }, output: { at: { $between: ['2026-07-01'] } }, options: TYPED }, + { name: 'a $between holding a { $field } is left whole — split, $gte would accept a reference $between refuses', + input: { at: { $between: [{ $field: 'opened_at' }, '2026-07-28'] } }, + output: { at: { $between: [{ $field: 'opened_at' }, '2026-07-28'] } }, options: TYPED }, + { name: 'an unresolved placeholder is not a calendar day', + input: { at: { $lte: '{today}' } }, output: { at: { $lte: '{today}' } }, options: TYPED }, + { name: 'an unknown $ key at node level is passed through', + input: { $where: 'x', at: { $gte: '2026-07-01' } }, output: { $where: 'x', at: { $gte: '2026-07-01' } }, options: TYPED }, + { name: 'implicit equality on a datetime column is untouched', + input: { at: '2026-07-28' }, output: { at: '2026-07-28' }, options: TYPED }, +]; + +const ALL_ROWS = [...BOUND_ROWS, ...SCOPE_ROWS, ...NULL_ROWS, ...PASS_THROUGH_ROWS]; + +/** Every operator key (`$`-prefixed) anywhere in a filter. */ +function operatorKeys(value: unknown, out = new Set()): Set { + if (Array.isArray(value)) { + for (const item of value) operatorKeys(item, out); + } else if (value !== null && typeof value === 'object' && !(value instanceof Date)) { + for (const [key, child] of Object.entries(value)) { + if (key.startsWith('$')) out.add(key); + operatorKeys(child, out); + } + } + return out; +} + +/** The vocabulary the lowering may introduce (the design's §3.4 closed output set, T1 subset). */ +const INTRODUCED = new Set(['$and', '$or', '$lt', '$gte', '$lte', '$null']); + +function deepFreeze(value: T): T { + if (value !== null && typeof value === 'object' && !(value instanceof Date)) { + for (const child of Object.values(value as object)) deepFreeze(child); + Object.freeze(value); + } + return value; +} + +describe('lowerFilterCondition — the rule table', () => { + for (const row of ALL_ROWS) { + it(row.name, () => { + expect(lowerFilterCondition(row.input, row.options)).toEqual(row.output); + }); + } +}); + +describe('lowerFilterCondition — idempotence (a bound is lowered at most once)', () => { + it('lower(lower(x)) deep-equals lower(x) for every row of the table', () => { + for (const row of ALL_ROWS) { + const once = lowerFilterCondition(row.input, row.options); + expect(lowerFilterCondition(once, row.options), row.name).toEqual(once); + } + }); + + it('…and for every FILTER_LOGIC_CASES and TEMPORAL_CASES filter, typed and type-blind', () => { + const filters = [...FILTER_LOGIC_CASES.map((c) => c.filter), ...TEMPORAL_CASES.map((c) => c.filter)]; + expect(filters.length).toBeGreaterThan(40); + for (const options of [TYPED, undefined]) { + for (const filter of filters) { + const once = lowerFilterCondition(filter, options); + expect(lowerFilterCondition(once, options), JSON.stringify(filter)).toEqual(once); + } + } + }); + + it('a filter already in lowered form is returned by reference (the NULL escape and the requirement are recognised)', () => { + for (const lowered of [ + { at: { $lt: '2026-07-29' } }, + { $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] }, + { $not: { $and: [{ stage: { $null: false } }, { stage: 'won' }] } }, + ]) { + expect(lowerFilterCondition(lowered, TYPED)).toBe(lowered); + } + }); +}); + +describe('lowerFilterCondition — the output vocabulary is closed', () => { + it('introduces no operator outside $and / $or / $lt / $gte / $lte / $null, and leaves no $between it applies to', () => { + const filters = [ + ...ALL_ROWS.map((r) => [r.input, r.options] as const), + ...FILTER_LOGIC_CASES.map((c) => [c.filter, undefined] as const), + ...TEMPORAL_CASES.map((c) => [c.filter, undefined] as const), + ]; + for (const [input, options] of filters) { + const before = operatorKeys(input); + const after = operatorKeys(lowerFilterCondition(input, options)); + for (const op of after) { + expect(before.has(op) || INTRODUCED.has(op), `${op} in ${JSON.stringify(input)}`).toBe(true); + } + } + }); + + it('every TEMPORAL_CASES datetime bound leaves the typed seam with no bare-day $lte and no $between', () => { + const typed: FilterLoweringOptions = { isDatetimeColumn: (column) => column === 'at' }; + for (const c of TEMPORAL_CASES.filter((t) => t.kind === 'datetime')) { + const text = JSON.stringify(lowerFilterCondition(c.filter, typed)); + expect(text, c.name).not.toMatch(/"\$lte":"\d{4}-\d{2}-\d{2}"/); + expect(text, c.name).not.toContain('"$between"'); + } + }); +}); + +describe('lowerFilterCondition — copy-on-write, provenance, non-nodes', () => { + it('never edits its input', () => { + for (const row of ALL_ROWS) { + const input = deepFreeze(structuredClone(row.input)); + expect(() => lowerFilterCondition(input, row.options), row.name).not.toThrow(); + } + }); + + it('returns the SAME reference when nothing lowers', () => { + const where = { a: 1, b: { $gt: 2 }, $or: [{ c: { $in: [1] } }] }; + expect(lowerFilterCondition(where)).toBe(where); + }); + + it('keeps unchanged sibling subtrees by reference', () => { + const untouched = { c: { $in: [1] } }; + const lowered = lowerFilterCondition({ $or: [untouched, { at: { $lte: '2026-07-28' } }] }, TYPED) as { + $or: unknown[]; + }; + expect(lowered.$or[0]).toBe(untouched); + }); + + it('carries the provenance mark of every node it replaces (#8220)', () => { + const author = markFilterSubtreeProvenance({ at: { $lte: '2026-07-28' } }, 'author'); + const policy = markFilterSubtreeProvenance({ stage: { $ne: 'won' } }, 'policy'); + const lowered = lowerFilterCondition({ $and: [author, policy] }, TYPED) as { $and: unknown[] }; + expect(lowered.$and[0]).not.toBe(author); + expect(filterSubtreeProvenanceOf(lowered.$and[0])).toBe('author'); + expect(filterSubtreeProvenanceOf(lowered.$and[1])).toBe('policy'); + }); + + it('hands a value that is not a filter node back as it is', () => { + for (const value of [undefined, null, 5, 'x', [], new Date(0)]) { + expect(lowerFilterCondition(value)).toBe(value); + } + }); +}); From ccf6a4492f9fdd006ee7fca8a0f1baefd6a2c2a2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:25:05 +0000 Subject: [PATCH 5/7] wip: RLS seam pins, changeset, ADR-0053 anchor for the lowering module Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .changeset/5930-shared-filter-lowering.md | 26 ++++ .../src/rls-compiled-comparand-faces.test.ts | 7 +- .../src/rls-empty-membership-polarity.test.ts | 5 +- .../src/rls-shared-lowering-seam.test.ts | 134 ++++++++++++++++++ ...__spec__src__data__filter-lowering.ts.json | 7 + 5 files changed, 177 insertions(+), 2 deletions(-) create mode 100644 .changeset/5930-shared-filter-lowering.md create mode 100644 packages/plugins/plugin-security/src/rls-shared-lowering-seam.test.ts create mode 100644 scripts/adr-anchors/packages__spec__src__data__filter-lowering.ts.json diff --git a/.changeset/5930-shared-filter-lowering.md b/.changeset/5930-shared-filter-lowering.md new file mode 100644 index 00000000000..5bcc297d151 --- /dev/null +++ b/.changeset/5930-shared-filter-lowering.md @@ -0,0 +1,26 @@ +--- +'@objectstack/spec': minor +'@objectstack/objectql': patch +'@objectstack/plugin-security': patch +--- + +feat(spec, objectql, plugin-security): one shared filter lowering, run once at the engine and RLS seams (ADR-0053 D-D1, amended) + +Clause-②: yes + +`@objectstack/spec/data` exports `lowerFilterCondition(filter, options?)` and its `FilterLoweringOptions` type. It is not exported from the package root entry. It is a pure `FilterCondition → FilterCondition` rewrite that applies three rules once: + +- `$between` becomes `$gte` its minimum and `$lte` its maximum. +- A `$lte` whose comparand is a bare `YYYY-MM-DD` day becomes `$lt` the next day, in the calendar-string domain. On the last supported day (`9999-12-31`) a lone `$lte` becomes `{ $null: false }`, and a `$between` keeps only its minimum. +- The NULL-polarity guards the drivers already compile. A `$ne` of a value, a `$nin` or a `$notContains` holds for a row with no value. Every leaf of a `$not` operand is made total. + +The rewrite is copy-on-write, idempotent and never refuses. A node it rewrites keeps its filter-subtree provenance mark. With `options.isDatetimeColumn` (a typed seam), the first two rules change only a declared `datetime` column. Without it they apply to every column. + +As ADR-0053 D-D1 (amended 2026-09-30) requires, the seams now run it once, after the comparand doors and after filter-token resolution: + +- **`@objectstack/objectql`** runs it on every filter position, typed by the object's declared fields. That covers `where` on `find`, `findOne`, `count`, `update` and `delete`, and `aggregate`'s `where`, `aggregations[i].filter` and `having`. `having` is typed by the aggregated row's columns, so `max` of a `datetime` field counts as a `datetime`. Drivers receive the lowered filter. A date macro such as `{today}` is resolved before the lowering reads it. +- **`@objectstack/plugin-security`** runs it on every compiled RLS policy filter (`using` and `check`), right after the two comparand faces. `SecurityPlugin` now hands the compile seam the object's declared `datetime` columns (`RlsFieldGuard.datetime`). A guard without that set treats no column as `datetime`. + +Row answers stay the same on every driver. Each driver keeps its own copy of these rules, and every copy gives the same answer on lowered input. One result changes. The engine evaluates `aggregate`'s `aggregations[i].filter` and `having` itself, and that evaluator now treats a row or group with no value the way every driver's `where` already does. It no longer counts such a row in a `$between` on a `datetime` column. It now keeps such a row under a `$not` over an ordering such as `$lt`. + +Nothing is removed or renamed, and there is nothing to migrate. diff --git a/packages/plugins/plugin-security/src/rls-compiled-comparand-faces.test.ts b/packages/plugins/plugin-security/src/rls-compiled-comparand-faces.test.ts index 797da6ff784..d007ef9a0a5 100644 --- a/packages/plugins/plugin-security/src/rls-compiled-comparand-faces.test.ts +++ b/packages/plugins/plugin-security/src/rls-compiled-comparand-faces.test.ts @@ -20,6 +20,7 @@ import { describe, it, expect, vi } from 'vitest'; import type { RowLevelSecurityPolicy } from '@objectstack/spec/security'; import { compileCelToFilter } from '@objectstack/formula'; +import { lowerFilterCondition } from '@objectstack/spec/data'; import { RLSCompiler, RLS_DENY_FILTER } from './rls-compiler.js'; @@ -152,7 +153,11 @@ describe('[#20212] CONTROL — a compiled filter the faces accept passes through const filter = compiler.compileFilter([policy(clause, predicate)], CTX, clause, GUARD); - expect(filter).toEqual(compiled.ok ? compiled.filter : undefined); + // [ADR-0053 D-D1, amended — #5930] The faces pass the compiled filter + // through unchanged; what the seam hands on is the shared lowering of + // it (the NULL-polarity guards — this GUARD declares no `datetime` + // column, so the whole-day rule reads none), and nothing else. + expect(filter).toEqual(compiled.ok ? lowerFilterCondition(compiled.filter, { isDatetimeColumn: () => false }) : undefined); expect(warn).not.toHaveBeenCalled(); }); } diff --git a/packages/plugins/plugin-security/src/rls-empty-membership-polarity.test.ts b/packages/plugins/plugin-security/src/rls-empty-membership-polarity.test.ts index f0195bfc256..63d24a86550 100644 --- a/packages/plugins/plugin-security/src/rls-empty-membership-polarity.test.ts +++ b/packages/plugins/plugin-security/src/rls-empty-membership-polarity.test.ts @@ -99,7 +99,10 @@ describe('[#13552] emptied membership under negation — the guard must fire (de it('NON-empty membership under `$not` keeps working — the `not in` feature', () => { const ctx: any = { userId: 'u_me', tenantId: 'org-1', positions: [], org_user_ids: ['u_other', 'u_third'] }; const filter = compiler.compileFilter([policy('!(owner in current_user.org_user_ids)')], ctx); - expect(filter).toEqual({ $not: { owner: { $in: ['u_other', 'u_third'] } } }); + // [ADR-0053 D-D1, amended — #5930] The `$not` operand leaves the RLS seam + // totalised by the shared lowering (#5146): the positive `$in` leaf takes a + // `$null: false` requirement. The admitted rows below do not move. + expect(filter).toEqual({ $not: { $and: [{ owner: { $null: false } }, { owner: { $in: ['u_other', 'u_third'] } }] } }); // r1 (u_me), r4 (null owner — $in over null is false, $not inverts), r5 (u_fourth). expect(admitted(filter as Record)).toBe(3); }); diff --git a/packages/plugins/plugin-security/src/rls-shared-lowering-seam.test.ts b/packages/plugins/plugin-security/src/rls-shared-lowering-seam.test.ts new file mode 100644 index 00000000000..63c328bc219 --- /dev/null +++ b/packages/plugins/plugin-security/src/rls-shared-lowering-seam.test.ts @@ -0,0 +1,134 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [ADR-0053 D-D1, amended 2026-09-30 — #5930] The RLS compile seam's placement + * of the shared `FilterCondition → FilterCondition` lowering + * (`lowerFilterCondition`, `@objectstack/spec/data`): run once on every compiled + * policy filter, right after the two comparand faces (`judgeCompiledComparands`), + * for `using` and `check` alike — so the read scope's drivers and the write + * check's evaluator receive one lowered filter. + * + * Column-type scope (the amendment's item 7): the seam reads the declared + * `datetime` columns its caller hands it (`RlsFieldGuard.datetime`), so the + * whole-day rule rewrites those only — the scope every driver holds — and a + * guard without types reads no column as `datetime`. The NULL-polarity guards + * do not depend on the type. + * + * Token order (item 3): nothing resolves a placeholder on either RLS clause — + * measured: a policy comparand `'{today}'` compiles and reaches both consumers + * verbatim. The lowering reads it as the non-day string it is and leaves it as + * written, which is what every face does with it today. + */ + +import { describe, it, expect, vi } from 'vitest'; +import type { PermissionSet, RowLevelSecurityPolicy } from '@objectstack/spec/security'; +import { RLSCompiler } from './rls-compiler.js'; +import { SecurityPlugin } from './security-plugin.js'; + +const CTX = { userId: 'u1', tenantId: 'org-1', positions: [] } as any; + +const DECLARED = new Set(['id', 'stage', 'signed_on', 'due_on']); +/** The typed guard: `signed_on` is the one declared `datetime`. */ +const TYPED_GUARD = { declared: DECLARED, datetime: new Set(['signed_on']) }; + +const policy = (clause: 'using' | 'check', predicate: string): RowLevelSecurityPolicy => + ({ name: 'p', object: 'contract', operation: 'all', [clause]: predicate }) as unknown as RowLevelSecurityPolicy; + +const compile = (clause: 'using' | 'check', predicate: string, guard: any = TYPED_GUARD) => + new RLSCompiler().compileFilter([policy(clause, predicate)], CTX, clause, guard); + +describe('[ADR-0053 D-D1 amended — #5930] the RLS compile seam lowers every compiled policy filter', () => { + for (const clause of ['using', 'check'] as const) { + it(`${clause}: a bare-day upper bound on a declared datetime becomes $lt the next day`, () => { + expect(compile(clause, "record.signed_on <= '2026-01-05'")).toEqual({ signed_on: { $lt: '2026-01-06' } }); + }); + + it(`${clause}: the last supported day keeps only { $null: false }`, () => { + expect(compile(clause, "record.signed_on <= '9999-12-31'")).toEqual({ signed_on: { $null: false } }); + }); + + it(`${clause}: a column the guard does not type as datetime is left byte-identical`, () => { + expect(compile(clause, "record.due_on <= '2026-01-05'")).toEqual({ due_on: { $lte: '2026-01-05' } }); + }); + + it(`${clause}: a guard with no types reads no column as datetime`, () => { + expect(compile(clause, "record.signed_on <= '2026-01-05'", { declared: DECLARED })) + .toEqual({ signed_on: { $lte: '2026-01-05' } }); + }); + + it(`${clause}: the NULL-polarity guards apply whatever the type`, () => { + expect(compile(clause, "record.stage != 'won'")) + .toEqual({ $and: [{ $or: [{ stage: { $null: true } }, { stage: { $ne: 'won' } }] }] }); + expect(compile(clause, "!(record.stage == 'won')")) + .toEqual({ $not: { $and: [{ stage: { $null: false } }, { stage: 'won' }] } }); + }); + + it(`${clause}: an unresolved '{today}' reaches the seam verbatim and is left as written`, () => { + expect(compile(clause, "record.signed_on <= '{today}'")).toEqual({ signed_on: { $lte: '{today}' } }); + }); + } +}); + +const CONTRACT_SCHEMA = { + name: 'contract', + fields: { + stage: { type: 'text' }, + signed_on: { type: 'datetime' }, + due_on: { type: 'date' }, + }, +}; + +const MEMBER_WITH_POLICY: PermissionSet = { + name: 'member_default', + label: 'Member', + objects: { '*': { allowRead: true, allowCreate: true, allowEdit: true, allowDelete: true } }, + rowLevelSecurity: [ + { + name: 'signed_through_jan_5', + object: 'contract', + operation: 'all', + using: "record.signed_on <= '2026-01-05' && record.due_on <= '2026-01-05'", + check: "record.signed_on <= '2026-01-05' && record.due_on <= '2026-01-05'", + }, + ], +} as unknown as PermissionSet; + +async function bootWithSchema(schema: Record) { + const services: Record = { + manifest: { register: vi.fn() }, + objectql: { registerMiddleware: vi.fn(), getSchema: () => schema, findOne: vi.fn(async () => null) }, + metadata: { get: async () => schema, list: async () => [MEMBER_WITH_POLICY] }, + 'org-scoping': { name: 'com.objectstack.org-scoping' }, + }; + const ctx: Record = { + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, + registerService: vi.fn(), + getService: (name: string) => { + if (!(name in services)) throw new Error(`service not registered: ${name}`); + return services[name]; + }, + }; + const plugin = new SecurityPlugin({ fallbackPermissionSet: 'member_default' }); + await plugin.init(ctx as any); + await plugin.start(ctx as any); + return plugin; +} + +/** The lowered policy: the datetime column widened, the date column untouched. */ +const LOWERED_POLICY = { + $and: [{ signed_on: { $lt: '2026-01-06' } }, { due_on: { $lte: '2026-01-05' } }], +}; + +describe('[ADR-0053 D-D1 amended — #5930] SecurityPlugin hands the seam the declared datetime columns', () => { + it('using (the read scope): the datetime column is lowered, the date column is not', async () => { + const plugin = await bootWithSchema(CONTRACT_SCHEMA); + const filter = await (plugin as any).getReadFilter('contract', CTX); + expect(JSON.stringify(filter)).toContain(JSON.stringify(LOWERED_POLICY)); + }); + + it('check (the write gate): the same lowered filter', async () => { + const plugin = await bootWithSchema(CONTRACT_SCHEMA); + const filter = await (plugin as any).computeWriteCheckFilter([MEMBER_WITH_POLICY], 'contract', 'insert', CTX); + expect(filter).toEqual(LOWERED_POLICY); + }); +}); diff --git a/scripts/adr-anchors/packages__spec__src__data__filter-lowering.ts.json b/scripts/adr-anchors/packages__spec__src__data__filter-lowering.ts.json new file mode 100644 index 00000000000..77d25639f45 --- /dev/null +++ b/scripts/adr-anchors/packages__spec__src__data__filter-lowering.ts.json @@ -0,0 +1,7 @@ +{ + "file": "packages/spec/src/data/filter-lowering.ts", + "adrs": [ + "ADR-0053" + ], + "invariant": "ADR-0053 D-D1 (amended 2026-09-30): the bare-day upper bound, the `$between` split and the NULL-polarity guards are applied ONCE, by this one shared `FilterCondition → FilterCondition` lowering, at the seams that already run the comparand doors — after the doors and after filter-token resolution — and drivers receive the lowered filter. It emits a calendar string, never a storage form (D-A1), and runs before any face converts (D-E3, structural). It lives on the `@objectstack/spec/data` subpath and never on the package root entry. Moving the rule back into an emitter, running it before token resolution, or widening a column no declared `datetime` names at a typed seam reverts the amendment." +} From 5ccdd85a4c83392aeb0883351415da9faec7993d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 05:28:11 +0000 Subject: [PATCH 6/7] chore(spec): regenerate api-surface and export-origins for the lowering export Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- packages/spec/api-surface/data.json | 2 ++ packages/spec/export-origins/data.json | 2 ++ 2 files changed, 4 insertions(+) diff --git a/packages/spec/api-surface/data.json b/packages/spec/api-surface/data.json index cfe2929ec49..4900a7de75f 100644 --- a/packages/spec/api-surface/data.json +++ b/packages/spec/api-surface/data.json @@ -336,6 +336,7 @@ "FilterConditionSchema (const)", "FilterLogicCase (interface)", "FilterLogicRow (interface)", + "FilterLoweringOptions (interface)", "FilterOperatorKey (type)", "FilterSubtreeProvenance (type)", "FilterTextCase (type)", @@ -860,6 +861,7 @@ "likePatternToRegExp (function)", "likePatternToRegexSource (function)", "lintAuthoredRecordKeys (function)", + "lowerFilterCondition (function)", "markFilterSubtreeProvenance (function)", "matchesLikePattern (function)", "missingFieldValues (function)", diff --git a/packages/spec/export-origins/data.json b/packages/spec/export-origins/data.json index e2eb504b6f3..bcafcdced6d 100644 --- a/packages/spec/export-origins/data.json +++ b/packages/spec/export-origins/data.json @@ -327,6 +327,7 @@ "FilterConditionSchema": "src/data/filter.zod.ts#FilterConditionSchema (const)", "FilterLogicCase": "src/data/filter-logic-conformance.ts#FilterLogicCase (interface)", "FilterLogicRow": "src/data/filter-logic-conformance.ts#FilterLogicRow (interface)", + "FilterLoweringOptions": "src/data/filter-lowering.ts#FilterLoweringOptions (interface)", "FilterOperatorKey": "src/data/filter.zod.ts#FilterOperatorKey (type)", "FilterSubtreeProvenance": "src/data/filter-subtree-provenance.ts#FilterSubtreeProvenance (type)", "FilterTextCase": "src/data/filter-text-conformance.ts#FilterTextCase (type)", @@ -847,6 +848,7 @@ "likePatternToRegExp": "src/data/filter.zod.ts#likePatternToRegExp (function)", "likePatternToRegexSource": "src/data/filter.zod.ts#likePatternToRegexSource (function)", "lintAuthoredRecordKeys": "src/data/authoring-key-lint.ts#lintAuthoredRecordKeys (function)", + "lowerFilterCondition": "src/data/filter-lowering.ts#lowerFilterCondition (function)", "markFilterSubtreeProvenance": "src/data/filter-subtree-provenance.ts#markFilterSubtreeProvenance (function)", "matchesLikePattern": "src/data/filter.zod.ts#matchesLikePattern (function)", "missingFieldValues": "src/data/autonumber-format.ts#missingFieldValues (function)", From 36e5ce8828a20a32ba415108d429c6f2141ca738 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 06:39:35 +0000 Subject: [PATCH 7/7] =?UTF-8?q?chore(changeset):=20grade=20@objectstack/pl?= =?UTF-8?q?ugin-security=20minor=20=E2=80=94=20RlsFieldGuard.datetime=20wi?= =?UTF-8?q?dens=20a=20published=20accept=20set?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .changeset/5930-shared-filter-lowering.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/5930-shared-filter-lowering.md b/.changeset/5930-shared-filter-lowering.md index 5bcc297d151..2083d4b500c 100644 --- a/.changeset/5930-shared-filter-lowering.md +++ b/.changeset/5930-shared-filter-lowering.md @@ -1,7 +1,7 @@ --- '@objectstack/spec': minor '@objectstack/objectql': patch -'@objectstack/plugin-security': patch +'@objectstack/plugin-security': minor --- feat(spec, objectql, plugin-security): one shared filter lowering, run once at the engine and RLS seams (ADR-0053 D-D1, amended)