diff --git a/.changeset/19377-between-field-endpoint-runtime-door.md b/.changeset/19377-between-field-endpoint-runtime-door.md new file mode 100644 index 0000000000..f5e1519724 --- /dev/null +++ b/.changeset/19377-between-field-endpoint-runtime-door.md @@ -0,0 +1,42 @@ +--- +'@objectstack/spec': minor +--- + +**BREAKING for callers** — `parseFilterAST` now refuses a `{ $field }` column reference as a `$between` endpoint, exactly as the authoring schema has since 2026-08-11. A reference at either bound is refused with `INVALID_FILTER` / 400, and the refusal names the side — MIN or MAX, plus the index (#19377). + +Clause-②: yes + +## What changed, and why it is the implementation catching up rather than a new rule + +`RANGE_ENDPOINT_DESCRIPTION` — the published endpoint contract shared by both of `$between`'s bounds — has stated verbatim since 2026-08-11 that "A { $field } reference is NOT an endpoint shape: no backend resolves one inside a list". The ruling that wrote it (ADR-0049 enforce-or-remove) removed `FieldReferenceSchema` from both endpoint unions, and it shipped at the authoring schema alone. The runtime door disagreed with it: `parseFilterAST({ f: { $between: [{ $field: 'a' }, 'M'] } })` returned the filter unchanged, same object reference, measured on `origin/main` before this change and re-measured after. + +One published sentence therefore had two truth values, decided by which door a caller came through — and the door that passed it is the one an embedder reaches by handing a lowered filter straight to a driver. There, nothing resolves the reference: the in-memory matchers compare the raw reference OBJECT and the range silently matches nothing, while both SQL faces refuse the position. A filter that names a window and answers no rows, or 400s one layer down, is what a caller got instead of a refusal they could act on. + +``` +FROM parseFilterAST({ close_date: { $between: [{ $field: 'contract.start' }, '2026-12-31'] } }) + -> the same object, unchanged, straight on to the driver + +TO throws INVALID_FILTER / 400: + 'Operator "$between" on field "close_date" does not accept a { "$field": … } + reference as an endpoint (at where.close_date.$between[0], the MIN bound). …' +``` + +## Migration — FROM → TO + +| You wrote | Write instead | +| --- | --- | +| `{ $between: [{ $field: 'contract.start' }, '2026-12-31'] }` | `{ $between: ['2026-01-01', '2026-12-31'] }` — the literal bound the range was meant to stop at | +| a range that was genuinely meant to be column-to-column | `{ "$gte": { "$field": "a" }, "$lte": { "$field": "b" } }` — two scalar bounds, the position that compiles on every face | + +**The one-line fix: write the literal bound, or — if the range really was column-to-column — drop `$between` and write the two bounds separately as `$gte` / `$lte`.** Nothing was evaluating the old filter, so treat the replacement as a new one and test it: at every backend the reference range either matched nothing or was refused. + +## What does NOT change + +- **A `{ $field }` reference as the WHOLE comparand of `$eq` / `$ne` / `$gt` / `$gte` / `$lt` / `$lte`.** That is #5222's shipped column-to-column capability, it is the alternative this refusal prescribes, and it lowers exactly as before — pinned by a lit control in the same file. +- **Every legal range.** Numbers, Dates, ISO days, UTC instants, clock times and non-temporal text all lower byte-identically, same object reference. +- **The three older endpoint carve-outs.** Arity, `null` (2026-08-31) and blank (2026-09-17) are checked first, so a pair carrying one of those keeps the message and the prescription it already had — an author who wrote `null` is still sent to the null predicate, not to a scalar comparison. +- **A plain object that is not a reference** keeps the comparand-TYPE door's own sentence, one step further on. +- **`$in` / `$nin` members.** The same 2026-08-11 decision rules a reference out of those positions too and `SET_MEMBER_DESCRIPTION` publishes it, but that is a second split over a different published sentence; it is measured and filed separately, and this change deliberately does not move it. +- **The published export surface.** No export is added, removed or renamed; the refusal rides the existing `$between` arm of the shared comparand-shape door, so the engine's delegating wrapper inherits it unchanged. + + diff --git a/packages/spec/src/data/filter-comparand-shape.test.ts b/packages/spec/src/data/filter-comparand-shape.test.ts index ce1a6c3b71..b901bd281a 100644 --- a/packages/spec/src/data/filter-comparand-shape.test.ts +++ b/packages/spec/src/data/filter-comparand-shape.test.ts @@ -245,6 +245,18 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922 // is the empty string, `filter.test.ts` pins the schema side of this very // row, and a trim here would re-open the split in the other direction. ["[' ', 'M']", [' ', 'M']], + // The 2026-08-11 `{ $field }` carve-out (#7596), reaching this door + // under #19377 — the same two-door question, one endpoint spelling over. + ["[{ $field }, 'M']", [{ $field: 'a' }, 'M']], + ["['A', { $field }]", ['A', { $field: 'b' }]], + ['[{ $field }, { $field }]', [{ $field: 'a' }, { $field: 'b' }]], + // A non-string referent is still the SHAPE the author wrote, at both + // doors: the schema door reads `'$field' in value` and so does this one. + ['[{ $field: 42 }, M]', [{ $field: 42 }, 'M']], + // A plain object that is NOT a reference was already refused at both + // doors, each with its own wording — the row is here so the loop cannot + // be read as "every object endpoint is the #7596 refusal now". + ["[{ nope: 1 }, 'M']", [{ nope: 1 }, 'M']], ]; const refused: string[] = []; for (const [label, pair] of endpointPairs) { @@ -259,9 +271,134 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922 if (schemaRefuses) refused.push(label); } // Guards the loop from passing vacuously in either direction. - expect(refused).toHaveLength(7); + expect(refused).toHaveLength(12); }); + // ── the `{ $field }` endpoint carve-out, ruled 2026-08-11 (#7596) ────── + // + // The ruling is the oldest of the four and the last to reach this door: it + // removed `FieldReferenceSchema` from both endpoint unions and published the + // sentence "A { $field } reference is NOT an endpoint shape", while this door + // went on lowering such a range unchanged (#19377). + + it.each([ + ['a reference as the MIN bound', { at: { $between: [{ $field: 'a' }, 'M'] } }], + ['a reference as the MAX bound', { at: { $between: ['A', { $field: 'b' }] } }], + ['both bounds references', { at: { $between: [{ $field: 'a' }, { $field: 'b' }] } }], + ['a reference with a non-string referent', { at: { $between: [{ $field: 42 }, 'M'] } }], + ['the lowered array form', [['at', 'between', [{ $field: 'a' }, 'M']]]], + ])('refuses a { $field } $between ENDPOINT — %s', (_label, where) => { + const err = refusalOf(() => parseFilterAST(where)); + expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER); + expect(err.status).toBe(400); + }); + + it('the reference refusal names the SIDE and prescribes the two-bound spelling', () => { + // The author who wrote a reference was reaching for a column-to-column + // comparison, which the platform HAS one operator over — so the remedy is + // a different filter, not a different literal, and the message says so. + const err = refusalOf(() => parseFilterAST({ close_date: { $between: [{ $field: 'a' }, 'M'] } })); + expect(err.message).toMatch( + /^Operator "\$between" on field "close_date" does not accept a \{ "\$field": … \} reference/, + ); + expect(err.message).toContain('at where.close_date.$between[0], the MIN bound'); + expect(err.message).toContain('{"$gte": {"$field": "a"}, "$lte": {"$field": "b"}}'); + expect(err.message).toMatch(/Authoring spellings: between\./); + expect(err.message).toMatch(/UNFILTERED result set/); + expect(refusalOf(() => parseFilterAST({ close_date: { $between: ['A', { $field: 'b' }] } })).message) + .toContain('at where.close_date.$between[1], the MAX bound'); + // ⛔ The in-memory evaluator is NOT offered as an escape: it does not + // resolve a list member either, it fails silently instead of loudly, so + // naming it would send an author to the one path that answers a wrong row + // set rather than an error. The schema door's twin holds the same line. + expect(err.message).not.toContain('matchesFilter'); + }); + + it('a reference bound is refused at its own path inside $and / $or / $not too', () => { + expect(refusalOf(() => parseFilterAST({ $not: { at: { $between: [{ $field: 'a' }, 'M'] } } })).message) + .toContain('where.$not.at.$between[0]'); + expect(refusalOf(() => parseFilterAST({ $or: [{ at: { $between: ['A', { $field: 'b' }] } }] })).message) + .toContain('where.$or[0].at.$between[1]'); + }); + + it('the three older endpoint carve-outs keep their own messages', () => { + // This check is LAST inside the arm, so every pair that already carried a + // refusal keeps the one it had. A red here means a pair was re-routed and + // an author who wrote `null` is now being sent to a scalar comparison. + expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, null] } })).message) + .toContain('requires two non-null bounds'); + expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, ''] } })).message) + .toContain('requires two non-blank bounds'); + expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }] } })).message) + .toContain('requires a [min, max] value array'); + // And a plain object that is not a reference keeps the comparand-TYPE + // door's own sentence, one step further on. + expect(refusalOf(() => parseFilterAST({ at: { $between: [{ nope: 1 }, 'M'] } })).message) + .toContain('plain object'); + }); + + it('LIT CONTROL — legal ranges and the reference\'s own ORDERING slots still pass', () => { + // Without these the refusal could be a blanket rejection of `$between`, or + // of the reference itself, and every assertion above would still be green. + expect(parseFilterAST({ at: { $between: ['A', 'M'] } })) + .toEqual({ at: { $between: ['A', 'M'] } }); + expect(parseFilterAST({ n: { $between: [0, 100] } })).toEqual({ n: { $between: [0, 100] } }); + expect(parseFilterAST({ at: { $between: ['2026-01-01', '2026-12-31'] } })) + .toEqual({ at: { $between: ['2026-01-01', '2026-12-31'] } }); + // #5222's shipped capability: the reference IS a whole comparand of the + // four ordering operators, which is what this refusal prescribes. Refusing + // it there would delete the escape the message names. + for (const op of ['$gt', '$gte', '$lt', '$lte']) { + expect(parseFilterAST({ a: { [op]: { $field: 'b' } } })) + .toEqual({ a: { [op]: { $field: 'b' } } }); + } + // The two-bound spelling the message prescribes must itself lower. + expect(parseFilterAST({ a: { $gte: { $field: 'b' }, $lte: { $field: 'c' } } })) + .toEqual({ a: { $gte: { $field: 'b' }, $lte: { $field: 'c' } } }); + }); + + it('re-routes the pairs the comparand-TYPE door used to answer — convergence, pinned', () => { + // ⚠️ This is the one part of the delta that is NOT "was accepted, is now + // refused". A pair whose OTHER element the comparand-TYPE door would have + // refused now meets this door first, so it answers with the reference + // sentence instead of the type one: same code, same status, and the schema + // door already answered every one of these with the reference message, so + // the two doors CONVERGE rather than diverge. Pinned rather than left to + // the reader, because "nothing else changed" is not true of these rows. + const mixed = refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, { nope: 1 }] } })); + expect(mixed.code).toBe(StandardErrorCode.enum.INVALID_FILTER); + expect(mixed.status).toBe(400); + expect(mixed.message).toContain('does not accept a { "$field": … } reference'); + // …and it names the REFERENCE's index, not the other element's. + expect(mixed.message).toContain('where.at.$between[0]'); + // A non-string referent, and a reference reached through the PROTOTYPE, are + // the shape the author wrote — at both doors. The TYPE door would have + // called each of them a plain object and prescribed a literal, which is the + // wrong remedy for someone who was reaching for a column. + for (const bound of [{ $field: 42 }, Object.create({ $field: 'a' }) as object]) { + expect(refusalOf(() => parseFilterAST({ at: { $between: [bound, 'M'] } })).message) + .toContain('does not accept a { "$field": … } reference'); + } + // A pair carrying NO reference is untouched: it still reaches the TYPE door + // and still answers with the TYPE door's own sentence. + expect(refusalOf(() => parseFilterAST({ at: { $between: [{ nope: 1 }, 'M'] } })).message) + .toContain('plain object'); + }); + + it.todo( + 'the $in / $nin MEMBER positions of the same 2026-08-11 ruling still DISAGREE across the two ' + + 'doors — FieldOperatorsSchema refuses a { $field } member, parseFilterAST lowers it ' + + 'unchanged (measured on this branch). Filed separately; ⛔ not pinned green here, because a ' + + 'green pin would read as a ruling nobody made', + ); + + // ⚠️ `$in` / `$nin` MEMBERS carrying a `{ $field }` reference are the SAME + // #7596 ruling one position over, published by `SET_MEMBER_DESCRIPTION`, and + // this door still lowers them unchanged — measured under #19377 and filed + // separately. ⛔ Deliberately NOT pinned here in either direction: pinning a + // measured defect green reads as a ruling nobody made, and refusing it would + // be a narrowing of a published face this card was never given. + // ── the ordering carve-out, ruled 2026-09-01 (#14080) ────────────────── it.each([ @@ -373,6 +510,10 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922 { close_date: { $between: ['2026-07-01', ''] } }, { close_date: { $between: ['', '2026-07-31'] } }, { close_date: { $between: [5, undefined] } }, + // The 2026-08-11 reference carve-out (#7596): the two-bound prescription + // is the longest thing this arm assembles, inside the same bound. + { close_date: { $between: [{ $field: 'a' }, 'M'] } }, + { close_date: { $between: ['A', { $field: 'b' }] } }, // The 2026-09-01 ordering carve-out (#14080): `$gte` / `$lte` carry the // longest spelling lists, so they are the tallest of the four. { close_date: { $gte: null } }, diff --git a/packages/spec/src/data/filter-comparand-shape.ts b/packages/spec/src/data/filter-comparand-shape.ts index e82206fa07..d958b5e72a 100644 --- a/packages/spec/src/data/filter-comparand-shape.ts +++ b/packages/spec/src/data/filter-comparand-shape.ts @@ -166,6 +166,49 @@ * falsy `$in` / `$nin` MEMBER stays a value — #13357's rows stand, because * this ruling is about range ENDPOINTS and those are about VALUES. * + * ## Refused BY RULING, 2026-08-11: a `{ $field }` `$between` ENDPOINT (#7596) + * + * The OLDEST of the endpoint rulings and the last to reach this door. #7596 + * removed `FieldReferenceSchema` from both endpoint unions under ADR-0049 + * enforce-or-remove, because no backend ever resolved one in a list position: + * `matches-filter.ts` leaves the list unresolved and orders against the raw + * reference OBJECT, so it silently matches nothing, and both SQL faces refuse + * the position with `INVALID_FILTER` / 400. The published endpoint contract + * has said so verbatim ever since — "A { $field } reference is NOT an endpoint + * shape" (`RANGE_ENDPOINT_DESCRIPTION`, `./filter.zod.ts`). + * + * It shipped at the SCHEMA door alone. This door went on lowering + * `{ $between: [{ $field: 'a' }, 'M'] }` unchanged — one published sentence + * with two truth values, decided by which door a caller came through, and the + * door that passed it is the one an embedder reaches by handing a lowered + * filter straight to a driver. Measured again under #19377 before the change; + * closed here the way #19071 closed the blank spelling one endpoint over. + * + * The scope is the `$between` ENDPOINT position and nothing wider: + * + * - **Recognised by SHAPE** — a non-array object carrying a `$field` key, + * which is the schema door's own `isFieldReferenceShape` test, so the two + * doors cannot drift over what counts as a reference. ⛔ Not the comparand + * TYPE door's stricter `typeof value.$field === 'string'`: that door steps + * around references on purpose and refuses `{ $field: 42 }` as a plain + * object, and what this refusal has to name is the shape the author WROTE. + * - **The reference stays legal in the four ORDERING slots** — #5222's + * shipped capability, and the alternative this refusal prescribes: a + * column-to-column range is its two bounds written separately, + * `{ $gte: { $field: 'a' }, $lte: { $field: 'b' } }`, which every face + * already answers. + * - **`$in` / `$nin` MEMBERS are NOT judged here.** #7596 rules a reference + * out of those positions too and `SET_MEMBER_DESCRIPTION` publishes that + * rule, but this door still lowers such a member unchanged. That is a + * SECOND split over a different published sentence, needing its own wording + * and its own ruling; ⛔ absorbing it silently here is the exact move this + * card's family exists to refuse. + * - **Checked LAST among the endpoint carve-outs**, after arity, `null` and + * blank. Every pair that already carried a refusal keeps the message it + * had — `[{ $field: 'a' }, null]` still answers with the 2026-08-31 null + * prescription — so this check changes the verdict only for pairs this door + * accepts today. + * * ## Refusal envelope * * Every refusal carries `code: 'INVALID_FILTER'` and `status: 400` (ADR-0112 @@ -238,6 +281,28 @@ function isFilterNode(value: unknown): value is Record { ); } +/** + * Is `value` shaped like a `{ $field: … }` reference? — the #7596 endpoint + * carve-out's recogniser. + * + * SHAPE only, and the referenced NAME is never consulted, not even its type: + * the point is to recognise what the author WROTE so the refusal can name it, + * which has to happen for any `{ $field: … }` and not only for one whose + * referent would have resolved. + * + * Spelled exactly as the schema door's `isFieldReferenceShape` + * (`./filter.zod.ts`) spells it, rather than imported — `filter.zod.ts` + * imports THIS module and the reverse edge would be a cycle, the same argument + * {@link LIST_COMPARAND_OPERATORS} records. ⛔ Deliberately NOT routed through + * {@link isFilterNode}, whose `Date` arm this predicate does not carry: the + * two doors then answer identically by construction rather than by argument. + * `filter-comparand-shape.test.ts` reads both doors on one input set so they + * cannot drift apart anyway. + */ +function isFieldReferenceShape(value: unknown): boolean { + return !!value && typeof value === 'object' && !Array.isArray(value) && '$field' in value; +} + /** `string` / `number` / `null` / `object` … — the word the message uses. */ function describeOperand(value: unknown): string { if (value === null) return 'null'; @@ -444,6 +509,47 @@ function blankRangeBoundError( ); } +/** + * A `$between` bound that is a `{ $field }` REFERENCE — refused BY RULING, + * 2026-08-11 (#7596), implemented at this door under #19377; see the module + * note's fourth "Refused BY RULING" section. + * + * Its own message rather than an arm of any of the three above: those + * prescribe a VALUE (a bound the author did not type, or the null predicate), + * and the author who wrote a reference was reaching for a column-to-column + * comparison — a capability the platform HAS, one operator over. The remedy is + * therefore a different filter, not a different literal, and the message + * spends its budget saying so. + * + * The received reference is NOT previewed. Its position is already named to + * the index and the side, and the 500-char client bound (#5423) buys more as + * the two-bound spelling than as an echo of what the author is looking at. + * + * The schema door's twin is `listPositionFieldReferenceMessage` + * (`./filter.zod.ts`), reconciled by pin, as the null and blank pairs are — + * and like that one it ⛔ does NOT offer the in-memory evaluator as an escape: + * `matchesFilter` does not resolve a list member either, it fails silently + * instead of loudly, so naming it would send an author to the one path whose + * answer is a wrong row set rather than an error. + */ +function fieldReferenceRangeBoundError( + context: string | undefined, + field: string, + index: number, + path: string, +): Error { + const spellings = LIST_COMPARAND_OPERATORS.get('$between') ?? []; + return invalidFilterComparandError( + context, + `Operator "$between" on field "${field}" does not accept a { "$field": … } reference as an ` + + `endpoint (at ${path}[${index}], the ${index === 0 ? 'MIN' : 'MAX'} bound). No evaluation ` + + `path resolves one inside a list. Write a literal bound, or range column-to-column as two ` + + `bounds: {"$gte": {"$field": "a"}, "$lte": {"$field": "b"}}. Authoring spellings: ` + + `${spellings.join(', ')}. The filter was NOT applied, and an unapplied filter would have ` + + `returned the UNFILTERED result set.`, + ); +} + /** * A `null` comparand of `$gt` / `$gte` / `$lt` / `$lte` — refused BY RULING, * 2026-09-01 (#14080); see the module note's second "Refused BY RULING" @@ -579,6 +685,16 @@ function assertFieldListComparands( context, field, comparand[blankBound], blankBound, `${path}.${op}`, ); } + // Then the `{ $field }` REFERENCE carve-out (2026-08-11 ruling, #7596, + // reaching this door under #19377) — LAST, so every pair that already + // carried a refusal keeps the message it had, and only a pair this door + // accepts today can reach it. Shape, not value: `{ $field: 42 }` is the + // shape the author wrote and is named as such, one step before the TYPE + // door would have called it a plain object. + const referenceBound = comparand.findIndex(isFieldReferenceShape); + if (referenceBound !== -1) { + throw fieldReferenceRangeBoundError(context, field, referenceBound, `${path}.${op}`); + } continue; } if (!Array.isArray(comparand)) { diff --git a/packages/spec/src/migrations/entries/semantic/18.filter-between-field-reference-endpoint-refused.ts b/packages/spec/src/migrations/entries/semantic/18.filter-between-field-reference-endpoint-refused.ts new file mode 100644 index 0000000000..5607c81188 --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/18.filter-between-field-reference-endpoint-refused.ts @@ -0,0 +1,104 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +// The ledger's FIRST record of the 2026-08-11 removal, registered when the +// RUNTIME door finally enforced it. The schema-door half shipped without an +// entry, so this one covers both doors rather than only the second. +export const entry: SemanticMigration = { + id: 'filter-between-field-reference-endpoint-refused', + // No backticks in `surface` — build-upgrade-guide.ts renders it inside a + // code span already, and a nested backtick would close it. + surface: + 'either endpoint of a $between range, authored as a { $field } column reference, in any filter ' + + 'this platform stores or executes. The carriers split in two, because they are DETECTED ' + + 'differently and only one half answers on save. (a) REFUSED AT SAVE — a view, page or ' + + 'component filter RULE (ViewFilterRuleSchema, whose value is shaped by the operator) and the ' + + 'NormalizedFilter AST the query faces validate against, plus the enforced FieldOperatorsSchema ' + + 'copy itself. (b) NOT JUDGED AT SAVE — a dashboard widget filter, a dataset filter, a report ' + + 'runtimeFilter, a rollup filter and a relatedListFilter: every one of those slots is typed ' + + 'FilterConditionSchema, a loose record intersected with the $and / $or / $not shape that never ' + + 'judges an operator map, so such a document parses GREEN and the endpoint is refused only when ' + + 'the filter is EXECUTED, at the runtime lowering door this change closes. ARITY is not what ' + + 'changed: a reference endpoint is a well-formed TWO-element range one of whose elements no ' + + 'backend resolves', + replacement: + 'a literal bound — the value the range was meant to stop at, written out. If the range was ' + + 'genuinely meant to be COLUMN-TO-COLUMN, that is not a $between at all: write the two ' + + 'bounds separately as scalar comparisons, {"$gte": {"$field": "a"}} for the lower bound and ' + + '{"$lte": {"$field": "b"}} for the upper one, which is the position #5222 compiles on every ' + + 'face. ⛔ There is no replacement that can be DERIVED from what was written: the literal a ' + + 'reference stood for is not recoverable, and dropping the operator would delete a ' + + 'constraint the author wrote and WIDEN the result set silently. A reference remains legal, ' + + 'unchanged, as the WHOLE comparand of $eq / $ne / $gt / $gte / $lt / $lte', + reason: + 'Maintainer ruling of 2026-08-11 on #7596, ADR-0049 enforce-or-remove: REMOVE. Both $between ' + + 'endpoint unions carried FieldReferenceSchema and no backend ever resolved one in a list ' + + 'position — matches-filter.ts leaves the list unresolved and orders against the raw ' + + 'reference OBJECT, so the range silently matches nothing, and both SQL faces refuse the ' + + 'position with INVALID_FILTER / 400. The published endpoint contract has stated the rule ' + + 'verbatim since that day: "A { $field } reference is NOT an endpoint shape" ' + + '(RANGE_ENDPOINT_DESCRIPTION, packages/spec/src/data/filter.zod.ts). ' + + '⚠️ That ruling shipped at the AUTHORING SCHEMA door alone, and no ledger entry was written ' + + 'for it — measured before this change: no semantic entry, no retired key, no spec-changes ' + + 'row and no upgrade-guide line named the shape. That was not an omission, and this entry ' + + 'SUPERSEDES a recorded answer rather than filling a silence: the 2026-08-11 changeset (PR ' + + '#7713) carried the disposition not-required (no-migration-prescription), reviewed and ' + + 'accepted on #7596 and shipped in the published CHANGELOG. What changed is the fact that ' + + 'disposition rested on. It was claimed for a removal whose reach was believed to be the ' + + 'authoring schema alone; the runtime half now ships with a migration prescription of its own ' + + '(below), and a body carrying a prescription is exactly what that category refuses. So the ' + + 'transition is registered here, covering BOTH doors, and the earlier not-required reading is ' + + 'retired by this record. The runtime lowering door disagreed with ' + + 'the declaration for the whole of that window: parseFilterAST({ f: { $between: [{ $field: ' + + '"a" }, "M"] } }) returned the filter unchanged, same object reference, measured on ' + + 'origin/main immediately before the change and re-measured after. One published sentence, ' + + 'two truth values, decided by which door a caller came through — and the door that passed ' + + 'it is the one an embedder reaches by handing a lowered filter straight to a driver. This ' + + 'entry therefore registers the transition for BOTH doors, not only the second, which is ' + + 'why it is filed under #19377 rather than as an already-registered rider. ' + + '⚠️ No D2 conversion and no stored-metadata rewrite, and the load path was MEASURED rather ' + + 'than assumed: applyConversionsToStoredItem — the one primitive every stored-row ' + + 'rehydration seam calls — never throws and never validates, and replays only the ' + + 'positively-recognised lossless transforms in the conversion registry; measured on this ' + + 'branch, a stored view carrying { close_date: { $between: [{ $field: "contract.start" }, ' + + '"2026-12-31" ] } } comes back as the SAME object reference. Rewriting is not available in ' + + 'principle here, not merely declined: the literal the author meant is not recoverable from ' + + 'a reference, and the column-to-column reading has a different OPERATOR SHAPE (two scalar ' + + 'bounds), so producing it would be the platform rewriting one filter into another. That is ' + + 'the same ground the two nearest narrowings of this surface set stand on — ' + + 'filter-between-blank-endpoint-refused and filter-preset-ordering-comparand-refused. The ' + + 'read path does not re-validate stored rows, so no stored view becomes unreadable; what ' + + 'changes is that RE-SAVING one is refused, at the endpoint\'s own path, with the side ' + + 'named. Ships at once, no grace window and no dual spelling (2026-08-27 maintainer ruling ' + + '「短期不考虑渐进」). ADR-0049 / ADR-0087 / ADR-0112.', + acceptanceCriteria: + 'Grep every authored $between array — view and dashboard widget filters, dataset filters, ' + + 'report runtimeFilters, page and component filters, rollup filters, saved AST filters, SDK ' + + 'and MCP callers — and read BOTH of its elements for a { $field } key. A range with two ' + + 'literal endpoints parses byte-identically to before, numbers, Dates, ISO days, UTC ' + + 'instants, clock times and non-temporal text included; nothing is trimmed, defaulted or ' + + 'copied from its neighbour, so an accepted range arrives byte-identical to what was ' + + 'written. ⚠️ THE DETECTOR IS NOT THE SAME ON EVERY CARRIER, and re-saving the document is ' + + 'mechanical on only one half of them. Re-saving DOES answer for a view, page or component ' + + 'filter RULE and for the NormalizedFilter AST: a reference endpoint reports an issue at the ' + + 'rule\'s own value path (for a list view, list.filter.N.value) or at the endpoint\'s own AST ' + + 'path ($between.0 / $between.1) — note the schema door names the INDEX, while the runtime ' + + 'door additionally names the side, MIN or MAX. ⛔ Re-saving surfaces NOTHING for a dashboard ' + + 'widget filter, a dataset filter, a report runtimeFilter, a rollup filter or a ' + + 'relatedListFilter: those slots are FilterConditionSchema and such a document parses green — ' + + 'measured, not assumed. For those carriers the detectors are the GREP above and EXECUTING the ' + + 'surface, where the runtime lowering door now refuses with INVALID_FILTER / 400 naming the ' + + 'index and the side. ⛔ Do not read a clean re-save of a dashboard as a completed sweep. A ' + + 'range whose BOTH endpoints are references reports both positions at the schema door; the ' + + 'runtime door throws on the first. ⚠️ Do not assume such a range was showing the window it ' + + 'named: at every backend it either matched NOTHING (the in-memory matchers) or was refused ' + + '(both SQL faces), so a surface carrying one was never answering the query its filter ' + + 'claimed — decide the window from what the surface was SUPPOSED to show. If the intent was ' + + 'column-to-column, the replacement is the two-bound spelling and it needs testing as a NEW ' + + 'filter, because nothing was ever evaluating the old one. Endpoints that are null, blank or ' + + 'of the wrong type keep their own refusals and their own entries. $in / $nin MEMBERS ' + + 'carrying a reference are ruled out by the same 2026-08-11 decision and refused at the ' + + 'authoring schema door (SET_MEMBER_DESCRIPTION); they are outside THIS entry\'s transition ' + + 'and are worth sweeping in the same pass.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 8aaebe7654..93dc4cea15 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -8290,6 +8290,106 @@ const step18: MigrationStep = { + 'one side was ever meant, write it as `$gte` / `$lte` rather than inventing a second bound. ' + '`null` bounds are unaffected by this entry and keep their own refusal and prescription.', }, + // The ledger's FIRST record of the 2026-08-11 removal, registered when the + // RUNTIME door finally enforced it. The schema-door half shipped without an + // entry, so this one covers both doors rather than only the second. + { + id: 'filter-between-field-reference-endpoint-refused', + // No backticks in `surface` — build-upgrade-guide.ts renders it inside a + // code span already, and a nested backtick would close it. + surface: + 'either endpoint of a $between range, authored as a { $field } column reference, in any filter ' + + 'this platform stores or executes. The carriers split in two, because they are DETECTED ' + + 'differently and only one half answers on save. (a) REFUSED AT SAVE — a view, page or ' + + 'component filter RULE (ViewFilterRuleSchema, whose value is shaped by the operator) and the ' + + 'NormalizedFilter AST the query faces validate against, plus the enforced FieldOperatorsSchema ' + + 'copy itself. (b) NOT JUDGED AT SAVE — a dashboard widget filter, a dataset filter, a report ' + + 'runtimeFilter, a rollup filter and a relatedListFilter: every one of those slots is typed ' + + 'FilterConditionSchema, a loose record intersected with the $and / $or / $not shape that never ' + + 'judges an operator map, so such a document parses GREEN and the endpoint is refused only when ' + + 'the filter is EXECUTED, at the runtime lowering door this change closes. ARITY is not what ' + + 'changed: a reference endpoint is a well-formed TWO-element range one of whose elements no ' + + 'backend resolves', + replacement: + 'a literal bound — the value the range was meant to stop at, written out. If the range was ' + + 'genuinely meant to be COLUMN-TO-COLUMN, that is not a $between at all: write the two ' + + 'bounds separately as scalar comparisons, {"$gte": {"$field": "a"}} for the lower bound and ' + + '{"$lte": {"$field": "b"}} for the upper one, which is the position #5222 compiles on every ' + + 'face. ⛔ There is no replacement that can be DERIVED from what was written: the literal a ' + + 'reference stood for is not recoverable, and dropping the operator would delete a ' + + 'constraint the author wrote and WIDEN the result set silently. A reference remains legal, ' + + 'unchanged, as the WHOLE comparand of $eq / $ne / $gt / $gte / $lt / $lte', + reason: + 'Maintainer ruling of 2026-08-11 on #7596, ADR-0049 enforce-or-remove: REMOVE. Both $between ' + + 'endpoint unions carried FieldReferenceSchema and no backend ever resolved one in a list ' + + 'position — matches-filter.ts leaves the list unresolved and orders against the raw ' + + 'reference OBJECT, so the range silently matches nothing, and both SQL faces refuse the ' + + 'position with INVALID_FILTER / 400. The published endpoint contract has stated the rule ' + + 'verbatim since that day: "A { $field } reference is NOT an endpoint shape" ' + + '(RANGE_ENDPOINT_DESCRIPTION, packages/spec/src/data/filter.zod.ts). ' + + '⚠️ That ruling shipped at the AUTHORING SCHEMA door alone, and no ledger entry was written ' + + 'for it — measured before this change: no semantic entry, no retired key, no spec-changes ' + + 'row and no upgrade-guide line named the shape. That was not an omission, and this entry ' + + 'SUPERSEDES a recorded answer rather than filling a silence: the 2026-08-11 changeset (PR ' + + '#7713) carried the disposition not-required (no-migration-prescription), reviewed and ' + + 'accepted on #7596 and shipped in the published CHANGELOG. What changed is the fact that ' + + 'disposition rested on. It was claimed for a removal whose reach was believed to be the ' + + 'authoring schema alone; the runtime half now ships with a migration prescription of its own ' + + '(below), and a body carrying a prescription is exactly what that category refuses. So the ' + + 'transition is registered here, covering BOTH doors, and the earlier not-required reading is ' + + 'retired by this record. The runtime lowering door disagreed with ' + + 'the declaration for the whole of that window: parseFilterAST({ f: { $between: [{ $field: ' + + '"a" }, "M"] } }) returned the filter unchanged, same object reference, measured on ' + + 'origin/main immediately before the change and re-measured after. One published sentence, ' + + 'two truth values, decided by which door a caller came through — and the door that passed ' + + 'it is the one an embedder reaches by handing a lowered filter straight to a driver. This ' + + 'entry therefore registers the transition for BOTH doors, not only the second, which is ' + + 'why it is filed under #19377 rather than as an already-registered rider. ' + + '⚠️ No D2 conversion and no stored-metadata rewrite, and the load path was MEASURED rather ' + + 'than assumed: applyConversionsToStoredItem — the one primitive every stored-row ' + + 'rehydration seam calls — never throws and never validates, and replays only the ' + + 'positively-recognised lossless transforms in the conversion registry; measured on this ' + + 'branch, a stored view carrying { close_date: { $between: [{ $field: "contract.start" }, ' + + '"2026-12-31" ] } } comes back as the SAME object reference. Rewriting is not available in ' + + 'principle here, not merely declined: the literal the author meant is not recoverable from ' + + 'a reference, and the column-to-column reading has a different OPERATOR SHAPE (two scalar ' + + 'bounds), so producing it would be the platform rewriting one filter into another. That is ' + + 'the same ground the two nearest narrowings of this surface set stand on — ' + + 'filter-between-blank-endpoint-refused and filter-preset-ordering-comparand-refused. The ' + + 'read path does not re-validate stored rows, so no stored view becomes unreadable; what ' + + 'changes is that RE-SAVING one is refused, at the endpoint\'s own path, with the side ' + + 'named. Ships at once, no grace window and no dual spelling (2026-08-27 maintainer ruling ' + + '「短期不考虑渐进」). ADR-0049 / ADR-0087 / ADR-0112.', + acceptanceCriteria: + 'Grep every authored $between array — view and dashboard widget filters, dataset filters, ' + + 'report runtimeFilters, page and component filters, rollup filters, saved AST filters, SDK ' + + 'and MCP callers — and read BOTH of its elements for a { $field } key. A range with two ' + + 'literal endpoints parses byte-identically to before, numbers, Dates, ISO days, UTC ' + + 'instants, clock times and non-temporal text included; nothing is trimmed, defaulted or ' + + 'copied from its neighbour, so an accepted range arrives byte-identical to what was ' + + 'written. ⚠️ THE DETECTOR IS NOT THE SAME ON EVERY CARRIER, and re-saving the document is ' + + 'mechanical on only one half of them. Re-saving DOES answer for a view, page or component ' + + 'filter RULE and for the NormalizedFilter AST: a reference endpoint reports an issue at the ' + + 'rule\'s own value path (for a list view, list.filter.N.value) or at the endpoint\'s own AST ' + + 'path ($between.0 / $between.1) — note the schema door names the INDEX, while the runtime ' + + 'door additionally names the side, MIN or MAX. ⛔ Re-saving surfaces NOTHING for a dashboard ' + + 'widget filter, a dataset filter, a report runtimeFilter, a rollup filter or a ' + + 'relatedListFilter: those slots are FilterConditionSchema and such a document parses green — ' + + 'measured, not assumed. For those carriers the detectors are the GREP above and EXECUTING the ' + + 'surface, where the runtime lowering door now refuses with INVALID_FILTER / 400 naming the ' + + 'index and the side. ⛔ Do not read a clean re-save of a dashboard as a completed sweep. A ' + + 'range whose BOTH endpoints are references reports both positions at the schema door; the ' + + 'runtime door throws on the first. ⚠️ Do not assume such a range was showing the window it ' + + 'named: at every backend it either matched NOTHING (the in-memory matchers) or was refused ' + + '(both SQL faces), so a surface carrying one was never answering the query its filter ' + + 'claimed — decide the window from what the surface was SUPPOSED to show. If the intent was ' + + 'column-to-column, the replacement is the two-bound spelling and it needs testing as a NEW ' + + 'filter, because nothing was ever evaluating the old one. Endpoints that are null, blank or ' + + 'of the wrong type keep their own refusals and their own entries. $in / $nin MEMBERS ' + + 'carrying a reference are ruled out by the same 2026-08-11 decision and refused at the ' + + 'authoring schema door (SET_MEMBER_DESCRIPTION); they are outside THIS entry\'s transition ' + + 'and are worth sweeping in the same pass.', + }, { id: 'filter-preset-ordering-comparand-refused', // No backticks in `surface` — build-upgrade-guide.ts renders it inside a