fix(spec): refuse a { $field } reference as a $between endpoint at the runtime filter door

.changeset/19377-between-field-endpoint-runtime-door.md

@@ -0,0 +1,42 @@
+---
+'@objectstack/spec': minor
+---
+
+**BREAKING for callers** — `parseFilterAST` now refuses a `{ $field }` column reference as a `$between` endpoint, exactly as the authoring schema has since 2026-08-11. A reference at either bound is refused with `INVALID_FILTER` / 400, and the refusal names the side — MIN or MAX, plus the index (#19377).
+
+Clause-②: yes
+
+## What changed, and why it is the implementation catching up rather than a new rule
+
+`RANGE_ENDPOINT_DESCRIPTION` — the published endpoint contract shared by both of `$between`'s bounds — has stated verbatim since 2026-08-11 that "A { $field } reference is NOT an endpoint shape: no backend resolves one inside a list". The ruling that wrote it (ADR-0049 enforce-or-remove) removed `FieldReferenceSchema` from both endpoint unions, and it shipped at the authoring schema alone. The runtime door disagreed with it: `parseFilterAST({ f: { $between: [{ $field: 'a' }, 'M'] } })` returned the filter unchanged, same object reference, measured on `origin/main` before this change and re-measured after.
+
+One published sentence therefore had two truth values, decided by which door a caller came through — and the door that passed it is the one an embedder reaches by handing a lowered filter straight to a driver. There, nothing resolves the reference: the in-memory matchers compare the raw reference OBJECT and the range silently matches nothing, while both SQL faces refuse the position. A filter that names a window and answers no rows, or 400s one layer down, is what a caller got instead of a refusal they could act on.
+
+```
+FROM  parseFilterAST({ close_date: { $between: [{ $field: 'contract.start' }, '2026-12-31'] } })
+      -> the same object, unchanged, straight on to the driver
+
+TO    throws INVALID_FILTER / 400:
+      'Operator "$between" on field "close_date" does not accept a { "$field": … }
+       reference as an endpoint (at where.close_date.$between[0], the MIN bound). …'
+```
+
+## Migration — FROM → TO
+
+| You wrote | Write instead |
+| --- | --- |
+| `{ $between: [{ $field: 'contract.start' }, '2026-12-31'] }` | `{ $between: ['2026-01-01', '2026-12-31'] }` — the literal bound the range was meant to stop at |
+| a range that was genuinely meant to be column-to-column | `{ "$gte": { "$field": "a" }, "$lte": { "$field": "b" } }` — two scalar bounds, the position that compiles on every face |
+
+**The one-line fix: write the literal bound, or — if the range really was column-to-column — drop `$between` and write the two bounds separately as `$gte` / `$lte`.** Nothing was evaluating the old filter, so treat the replacement as a new one and test it: at every backend the reference range either matched nothing or was refused.
+
+## What does NOT change
+
+- **A `{ $field }` reference as the WHOLE comparand of `$eq` / `$ne` / `$gt` / `$gte` / `$lt` / `$lte`.** That is #5222's shipped column-to-column capability, it is the alternative this refusal prescribes, and it lowers exactly as before — pinned by a lit control in the same file.
+- **Every legal range.** Numbers, Dates, ISO days, UTC instants, clock times and non-temporal text all lower byte-identically, same object reference.
+- **The three older endpoint carve-outs.** Arity, `null` (2026-08-31) and blank (2026-09-17) are checked first, so a pair carrying one of those keeps the message and the prescription it already had — an author who wrote `null` is still sent to the null predicate, not to a scalar comparison.
+- **A plain object that is not a reference** keeps the comparand-TYPE door's own sentence, one step further on.
+- **`$in` / `$nin` members.** The same 2026-08-11 decision rules a reference out of those positions too and `SET_MEMBER_DESCRIPTION` publishes it, but that is a second split over a different published sentence; it is measured and filed separately, and this change deliberately does not move it.
+- **The published export surface.** No export is added, removed or renamed; the refusal rides the existing `$between` arm of the shared comparand-shape door, so the engine's delegating wrapper inherits it unchanged.
+
+<!-- adr-0087: registered filter-between-field-reference-endpoint-refused -->

packages/spec/src/data/filter-comparand-shape.test.ts

@@ -245,6 +245,18 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
       // is the empty string, `filter.test.ts` pins the schema side of this very
       // row, and a trim here would re-open the split in the other direction.
       ["['   ', 'M']", ['   ', 'M']],
+      // The 2026-08-11 `{ $field }` carve-out (#7596), reaching this door
+      // under #19377 — the same two-door question, one endpoint spelling over.
+      ["[{ $field }, 'M']", [{ $field: 'a' }, 'M']],
+      ["['A', { $field }]", ['A', { $field: 'b' }]],
+      ['[{ $field }, { $field }]', [{ $field: 'a' }, { $field: 'b' }]],
+      // A non-string referent is still the SHAPE the author wrote, at both
+      // doors: the schema door reads `'$field' in value` and so does this one.
+      ['[{ $field: 42 }, M]', [{ $field: 42 }, 'M']],
+      // A plain object that is NOT a reference was already refused at both
+      // doors, each with its own wording — the row is here so the loop cannot
+      // be read as "every object endpoint is the #7596 refusal now".
+      ["[{ nope: 1 }, 'M']", [{ nope: 1 }, 'M']],
     ];
     const refused: string[] = [];
     for (const [label, pair] of endpointPairs) {
@@ -259,9 +271,134 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
       if (schemaRefuses) refused.push(label);
     }
     // Guards the loop from passing vacuously in either direction.
-    expect(refused).toHaveLength(7);
+    expect(refused).toHaveLength(12);
   });
 
+  // ── the `{ $field }` endpoint carve-out, ruled 2026-08-11 (#7596) ──────
+  //
+  // The ruling is the oldest of the four and the last to reach this door: it
+  // removed `FieldReferenceSchema` from both endpoint unions and published the
+  // sentence "A { $field } reference is NOT an endpoint shape", while this door
+  // went on lowering such a range unchanged (#19377).
+
+  it.each([
+    ['a reference as the MIN bound', { at: { $between: [{ $field: 'a' }, 'M'] } }],
+    ['a reference as the MAX bound', { at: { $between: ['A', { $field: 'b' }] } }],
+    ['both bounds references', { at: { $between: [{ $field: 'a' }, { $field: 'b' }] } }],
+    ['a reference with a non-string referent', { at: { $between: [{ $field: 42 }, 'M'] } }],
+    ['the lowered array form', [['at', 'between', [{ $field: 'a' }, 'M']]]],
+  ])('refuses a { $field } $between ENDPOINT — %s', (_label, where) => {
+    const err = refusalOf(() => parseFilterAST(where));
+    expect(err.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+    expect(err.status).toBe(400);
+  });
+
+  it('the reference refusal names the SIDE and prescribes the two-bound spelling', () => {
+    // The author who wrote a reference was reaching for a column-to-column
+    // comparison, which the platform HAS one operator over — so the remedy is
+    // a different filter, not a different literal, and the message says so.
+    const err = refusalOf(() => parseFilterAST({ close_date: { $between: [{ $field: 'a' }, 'M'] } }));
+    expect(err.message).toMatch(
+      /^Operator "\$between" on field "close_date" does not accept a \{ "\$field": … \} reference/,
+    );
+    expect(err.message).toContain('at where.close_date.$between[0], the MIN bound');
+    expect(err.message).toContain('{"$gte": {"$field": "a"}, "$lte": {"$field": "b"}}');
+    expect(err.message).toMatch(/Authoring spellings: between\./);
+    expect(err.message).toMatch(/UNFILTERED result set/);
+    expect(refusalOf(() => parseFilterAST({ close_date: { $between: ['A', { $field: 'b' }] } })).message)
+      .toContain('at where.close_date.$between[1], the MAX bound');
+    // ⛔ The in-memory evaluator is NOT offered as an escape: it does not
+    // resolve a list member either, it fails silently instead of loudly, so
+    // naming it would send an author to the one path that answers a wrong row
+    // set rather than an error. The schema door's twin holds the same line.
+    expect(err.message).not.toContain('matchesFilter');
+  });
+
+  it('a reference bound is refused at its own path inside $and / $or / $not too', () => {
+    expect(refusalOf(() => parseFilterAST({ $not: { at: { $between: [{ $field: 'a' }, 'M'] } } })).message)
+      .toContain('where.$not.at.$between[0]');
+    expect(refusalOf(() => parseFilterAST({ $or: [{ at: { $between: ['A', { $field: 'b' }] } }] })).message)
+      .toContain('where.$or[0].at.$between[1]');
+  });
+
+  it('the three older endpoint carve-outs keep their own messages', () => {
+    // This check is LAST inside the arm, so every pair that already carried a
+    // refusal keeps the one it had. A red here means a pair was re-routed and
+    // an author who wrote `null` is now being sent to a scalar comparison.
+    expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, null] } })).message)
+      .toContain('requires two non-null bounds');
+    expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, ''] } })).message)
+      .toContain('requires two non-blank bounds');
+    expect(refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }] } })).message)
+      .toContain('requires a [min, max] value array');
+    // And a plain object that is not a reference keeps the comparand-TYPE
+    // door's own sentence, one step further on.
+    expect(refusalOf(() => parseFilterAST({ at: { $between: [{ nope: 1 }, 'M'] } })).message)
+      .toContain('plain object');
+  });
+
+  it('LIT CONTROL — legal ranges and the reference\'s own ORDERING slots still pass', () => {
+    // Without these the refusal could be a blanket rejection of `$between`, or
+    // of the reference itself, and every assertion above would still be green.
+    expect(parseFilterAST({ at: { $between: ['A', 'M'] } }))
+      .toEqual({ at: { $between: ['A', 'M'] } });
+    expect(parseFilterAST({ n: { $between: [0, 100] } })).toEqual({ n: { $between: [0, 100] } });
+    expect(parseFilterAST({ at: { $between: ['2026-01-01', '2026-12-31'] } }))
+      .toEqual({ at: { $between: ['2026-01-01', '2026-12-31'] } });
+    // #5222's shipped capability: the reference IS a whole comparand of the
+    // four ordering operators, which is what this refusal prescribes. Refusing
+    // it there would delete the escape the message names.
+    for (const op of ['$gt', '$gte', '$lt', '$lte']) {
+      expect(parseFilterAST({ a: { [op]: { $field: 'b' } } }))
+        .toEqual({ a: { [op]: { $field: 'b' } } });
+    }
+    // The two-bound spelling the message prescribes must itself lower.
+    expect(parseFilterAST({ a: { $gte: { $field: 'b' }, $lte: { $field: 'c' } } }))
+      .toEqual({ a: { $gte: { $field: 'b' }, $lte: { $field: 'c' } } });
+  });
+
+  it('re-routes the pairs the comparand-TYPE door used to answer — convergence, pinned', () => {
+    // ⚠️ This is the one part of the delta that is NOT "was accepted, is now
+    // refused". A pair whose OTHER element the comparand-TYPE door would have
+    // refused now meets this door first, so it answers with the reference
+    // sentence instead of the type one: same code, same status, and the schema
+    // door already answered every one of these with the reference message, so
+    // the two doors CONVERGE rather than diverge. Pinned rather than left to
+    // the reader, because "nothing else changed" is not true of these rows.
+    const mixed = refusalOf(() => parseFilterAST({ at: { $between: [{ $field: 'a' }, { nope: 1 }] } }));
+    expect(mixed.code).toBe(StandardErrorCode.enum.INVALID_FILTER);
+    expect(mixed.status).toBe(400);
+    expect(mixed.message).toContain('does not accept a { "$field": … } reference');
+    // …and it names the REFERENCE's index, not the other element's.
+    expect(mixed.message).toContain('where.at.$between[0]');
+    // A non-string referent, and a reference reached through the PROTOTYPE, are
+    // the shape the author wrote — at both doors. The TYPE door would have
+    // called each of them a plain object and prescribed a literal, which is the
+    // wrong remedy for someone who was reaching for a column.
+    for (const bound of [{ $field: 42 }, Object.create({ $field: 'a' }) as object]) {
+      expect(refusalOf(() => parseFilterAST({ at: { $between: [bound, 'M'] } })).message)
+        .toContain('does not accept a { "$field": … } reference');
+    }
+    // A pair carrying NO reference is untouched: it still reaches the TYPE door
+    // and still answers with the TYPE door's own sentence.
+    expect(refusalOf(() => parseFilterAST({ at: { $between: [{ nope: 1 }, 'M'] } })).message)
+      .toContain('plain object');
+  });
+
+  it.todo(
+    'the $in / $nin MEMBER positions of the same 2026-08-11 ruling still DISAGREE across the two '
+    + 'doors — FieldOperatorsSchema refuses a { $field } member, parseFilterAST lowers it '
+    + 'unchanged (measured on this branch). Filed separately; ⛔ not pinned green here, because a '
+    + 'green pin would read as a ruling nobody made',
+  );
+
+  // ⚠️ `$in` / `$nin` MEMBERS carrying a `{ $field }` reference are the SAME
+  // #7596 ruling one position over, published by `SET_MEMBER_DESCRIPTION`, and
+  // this door still lowers them unchanged — measured under #19377 and filed
+  // separately. ⛔ Deliberately NOT pinned here in either direction: pinning a
+  // measured defect green reads as a ruling nobody made, and refusing it would
+  // be a narrowing of a published face this card was never given.
+
   // ── the ordering carve-out, ruled 2026-09-01 (#14080) ──────────────────
 
   it.each([
@@ -373,6 +510,10 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922
       { close_date: { $between: ['2026-07-01', ''] } },
       { close_date: { $between: ['', '2026-07-31'] } },
       { close_date: { $between: [5, undefined] } },
+      // The 2026-08-11 reference carve-out (#7596): the two-bound prescription
+      // is the longest thing this arm assembles, inside the same bound.
+      { close_date: { $between: [{ $field: 'a' }, 'M'] } },
+      { close_date: { $between: ['A', { $field: 'b' }] } },
       // The 2026-09-01 ordering carve-out (#14080): `$gte` / `$lte` carry the
       // longest spelling lists, so they are the tallest of the four.
       { close_date: { $gte: null } },