fix(driver-memory,driver-mongodb): refuse a non-boolean $exists comparand with INVALID_FILTER / 400, as $null's is refused (#20897)

.changeset/20897-exists-non-boolean-refused.md

@@ -0,0 +1,22 @@
+---
+'@objectstack/driver-memory': minor
+'@objectstack/driver-mongodb': minor
+---
+
+fix(driver-memory, driver-mongodb)!: a non-boolean `$exists` comparand is refused with `INVALID_FILTER` / 400, as `$null`'s is, instead of selecting the rows with no value (#20897)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered filter-query-face-comparands-refused-at-save) this narrows the query faces to the rule that registered entry already records: its reason states that every query face refuses a non-boolean $null / $exists flag, and its replacement is this change's whole migration (a flag is the boolean itself; $exists true is "has a value", false "has no value"). This change makes that statement true on the two drivers that did not yet refuse. No authorable key, spelling, export or published type moves, and no stored row is read or rewritten; a stored filter carrying such a flag is already refused when it is saved, by that entry. -->
+
+**BREAKING**: this narrows what the in-memory driver and the MongoDB driver accept in a filter. A `$exists` comparand that is not a boolean (a string, a number, `null`, `undefined`, an object) is now refused with `INVALID_FILTER` / 400, where these two drivers used to answer it. It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+`FieldOperatorsSchema` declares `$exists` as a boolean, and `driver-sql`, `driver-sqlite-wasm` and both Turso transports already refused any other comparand. The in-memory driver and the MongoDB driver did not: they read `$exists` as `value === true`, so every other value asked for the rows with NO value. `{ stage: { $exists: "yes" } }` and `{ stage: { $exists: 1 } }` returned the rows without a stage, the opposite of what was written. `0`, `null` and the string `"false"` landed on that same side by the same default, not because anything read them. The in-memory driver's analytics face read the same flag by truthiness and answered the valued rows for the same filter, so that driver gave two different answers.
+
+**What an author sees now.** `400 INVALID_FILTER` with `driver-sql`'s message, beginning `Operator "$exists" on field "FIELD" requires a boolean comparand (true or false).` and naming the position (`filter.stage.$exists`). On the in-memory driver the refusal covers `find`, `findOne`, `count`, `aggregate`, `updateMany`, `deleteMany` and the analytics face (`query()` and `generateSql()`). There, an `undefined` or object comparand is refused first by that face's comparand-type check, also `INVALID_FILTER` / 400, in its own words. A refused write changes nothing.
+
+**What to write instead.** Write the boolean itself. `"$exists": true` matches rows whose field has a value, and `"$exists": false` matches rows whose field has none.
+
+**Who is affected.** A caller that sent a non-boolean `$exists` to `InMemoryDriver` or `MongoDBDriver` (a test suite, a local or embedded deployment, a flow or hook calling the engine in-process) and read the answer as a real one. On `SqlDriver` the same filter was already a 400.
+
+**Unchanged.** `$exists: true` and `$exists: false` answer exactly as before. The aggregation `filter` and `having` positions, which the engine evaluates itself after the driver, are not changed by this entry.

packages/client/src/envelope-caller-census.test.ts

@@ -70,7 +70,9 @@
  * `AnalyticsService` in `analytics-automation-json-erasure.test.ts`, where
  * `analytics` is the SERVICE, not the client. That is a producer call and no
  * part of the SDK caller population, so every row carries its receiver and the
- * ledger classifies it `NOT_SDK`.
+ * ledger classifies it `NOT_SDK`. [#20897] driver-memory's refusal suite does
+ * the same on its own cube service (`MemoryAnalyticsService` bound to
+ * `analytics`), so it carries a `NOT_SDK` row too.
  *
  * ## The verdicts
  *
@@ -479,6 +481,12 @@ const LEDGER: readonly LedgerRow[] = [
         method: 'analytics.query', receiver: 'service', count: 5, verdict: 'NOT_SDK',
         why: 'the real AnalyticsService (the cube read), called to compare its answer for the nested-relation filter with the engine\'s',
     },
+    // ── a producer face outside the SDK: driver-memory's cube service ────────
+    {
+        file: 'packages/drivers/driver-memory/src/memory-exists-non-boolean-refusal.test.ts',
+        method: 'analytics.query', receiver: 'service', count: 2, verdict: 'NOT_SDK',
+        why: 'a MemoryAnalyticsService bound to `analytics`, called directly (no HTTP, no dispatcher envelope) to assert the cube face refuses a non-boolean $exists and answers find()\'s rows for true / false',
+    },
     {
         file: 'packages/client/src/analytics-automation-json-erasure.test.ts',
         method: 'analytics.meta', receiver: 'sdk', count: 2, verdict: 'PAYLOAD_DEPENDENT',
@@ -676,11 +684,18 @@ describe('#13079 §2 — positive controls on the matcher itself', () => {
         // method, so a literal-embedded site lands HERE first, as a phantom
         // producer call. That makes this the assertion most likely to break
         // for a reason that has nothing to do with receivers.
-        expect(service.length, literalNote()).toBe(6);
+        // [#20897] Three producer faces call `analytics.query` bare: the real
+        // AnalyticsService behind the SDK, the same service in `@objectstack/rest`'s
+        // nested-relation pin, and driver-memory's cube service
+        // (`MemoryAnalyticsService`) in its own refusal suite. None of the
+        // receivers is the client, and every file is pinned.
+        expect(service.length, literalNote()).toBe(8);
         expect([...new Set(service.map((s) => s.file))].sort()).toEqual([
             'packages/client/src/analytics-automation-json-erasure.test.ts',
+            'packages/drivers/driver-memory/src/memory-exists-non-boolean-refusal.test.ts',
             'packages/rest/src/analytics-nested-relation-filter.test.ts',
         ]);
+        expect(service.every((s) => s.method === 'analytics.query')).toBe(true);
     });
 });
 
@@ -725,10 +740,10 @@ describe('#13079 §3 — every call site is classified', () => {
         expect(production, literalNote()).toEqual([]);
     });
 
-    it('records the split: 18 payload pins, 10 result-insensitive, 6 not-SDK', () => {
+    it('records the split: 18 payload pins, 10 result-insensitive, 8 not-SDK', () => {
         expect(verdictTotal('PAYLOAD_DEPENDENT')).toBe(18);
         expect(verdictTotal('RESULT_INSENSITIVE')).toBe(10);
-        expect(verdictTotal('NOT_SDK')).toBe(6);
+        expect(verdictTotal('NOT_SDK')).toBe(8);
         // The three above are LEDGER sums and cannot move on a census reading;
         // this one is census-derived, so it carries the note. [#13874]
         expect(sdkSites.length, literalNote()).toBe(28);

packages/drivers/driver-memory/src/filter-refusal.ts

@@ -618,6 +618,42 @@ export function nonBooleanNullComparandError(field: string, value: unknown, path
   );
 }
 
+/**
+ * [#20897, applying #5347's ruling A as #5369 did] `$exists` whose comparand
+ * is not a boolean.
+ *
+ * The symmetric twin of {@link nonBooleanNullComparandError}, and a copy of its
+ * disposition rather than a fresh judgement: `FieldOperatorsSchema` declares
+ * `$exists: z.boolean()` exactly as it declares `$null`, and the 2026-08-06
+ * ruling on #5298 applied #5347-A to `$exists` by name — `driver-sql` has
+ * refused a non-boolean here since. This driver did not, and its answer was the
+ * sharpest of the splits: the live path lowered `val === true` to has-a-value
+ * and EVERYTHING else to no-value, so `{ stage: { $exists: 'yes' } }` returned
+ * the rows with NO value — the author's intent inverted — while this package's
+ * own cube face read the flag by truthiness (`Boolean(raw[0])`) and answered
+ * the valued rows for the same filter. One filter, one package, two answers.
+ * Measured on `origin/main` `f6ccca4a` through `engine.find`, `count`,
+ * `aggregate`, `updateMany`, `deleteMany` and the analytics face, before this
+ * refusal.
+ *
+ * The words are `driver-sql`'s `nonBooleanExistsComparandError`, verbatim —
+ * one condition, one wording (#5240) — with its "this driver" clause re-aimed
+ * at the backend it names, the way the `$null` twin above names `driver-sql`.
+ */
+export function nonBooleanExistsComparandError(field: string, value: unknown, path: string): Error {
+  return unsupportedFilterError(
+    `Operator "$exists" on field "${field}" requires a boolean comparand (true or false). ` +
+      `Received ${describeFilterOperand(value)} (${safeShapePreview(value)}) at ${path}. ` +
+      `@objectstack/spec FieldOperatorsSchema declares $exists as a boolean. It is refused rather ` +
+      `than coerced for the same reason $null is: a non-boolean lands on whichever side ` +
+      `the backend's two-branch conditional happens to default to, and those defaults point in ` +
+      `OPPOSITE directions — driver-sql's \`=== false\` test compiles IS NOT NULL for anything ` +
+      `but false, this driver's \`=== true\` test compiled IS NULL for anything but true. Note ` +
+      `"false" the STRING is truthy, so it lands on the side opposite the false it was written ` +
+      `to mean.`,
+  );
+}
+
 /**
  * [#20444] A non-boolean `$empty` comparand. The leading sentence is
  * `driver-sql`'s `nonBooleanEmptyComparandError`, verbatim — one condition,
@@ -939,6 +975,14 @@ function assertFieldConstraintShape(
     if (op === '$null' && typeof spec[op] !== 'boolean') {
       throw nonBooleanNullComparandError(field, spec[op], `${path}.$null`);
     }
+    // [#20897] `$exists`' comparand is a boolean by the same declaration, and
+    // the ruling that refused `$null`'s third value refused this one too. On
+    // this walk for the reason `$null` is: the live path's `=== true` arm and
+    // the cube face's truthiness read put a third value on OPPOSITE sides, so
+    // the refusal has to fire before either face lowers anything.
+    if (op === '$exists' && typeof spec[op] !== 'boolean') {
+      throw nonBooleanExistsComparandError(field, spec[op], `${path}.$exists`);
+    }
     // [#20444] `$empty`'s comparand is a boolean by the same declaration
     // (`FieldOperatorsSchema`), refused on this walk for the same reason: both
     // faces of this package evaluate `true` / `false` exhaustively, so a third

packages/drivers/driver-memory/src/memory-exists-non-boolean-refusal.test.ts

@@ -0,0 +1,200 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#20897] `$exists` takes a boolean. A non-boolean is refused on EVERY entry
+ * of this package, in `driver-sql`'s words — the `$null` twin's disposition
+ * (#5347-A), applied to `$exists` by the ruling on #5298 (#5369).
+ *
+ * # What was measured
+ *
+ * `FieldOperatorsSchema` declares `$exists: z.boolean()`, and nothing between an
+ * authored `where` and this driver validated it: `driver-sql`, `driver-sqlite-wasm`
+ * and both Turso transports refused a non-boolean, and this package did not.
+ * Measured on `origin/main` `f6ccca4a`, one row with `stage: 'won'` (id 1) and
+ * one with `stage: null` (id 2):
+ *
+ * | entry | `'yes'` | `1` | `'false'` | `0` / `null` |
+ * |---|---|---|---|---|
+ * | `find` / `findOne` / `count` / `aggregate` / `updateMany` / `deleteMany` | id 2 | id 2 | id 2 | id 2 |
+ * | the analytics (cube) face, `query()` | id 1 | id 1 | id 1 | id 2 |
+ *
+ * Two answers inside one package, and neither is a refusal. The query path's
+ * `val === true` test sent every third value to the NO-value side — `'yes'`
+ * asked for the rows without one, the author's intent inverted. The cube face's
+ * `set` arm read the same flag by truthiness, so it answered the opposite rows
+ * for `'yes'` and `1` — and for the string `'false'`, which is truthy.
+ *
+ * # Why every entry is asserted, and against `find()`
+ *
+ * The refusal lives in ONE function (`assertFilterConditionShape`), which every
+ * entry runs before it lowers anything. What would let a future change re-fork
+ * the answers is an entry reaching its lowering WITHOUT that walk — the cube
+ * face's truthiness arm is still there behind it. So each entry is asserted to
+ * refuse, and, for the two booleans the spec declares, to answer exactly the
+ * rows `find()` answers: the same row set as `find()`, or refused with
+ * `INVALID_FILTER` — never a third, quieter answer.
+ */
+
+import { describe, it, expect, beforeEach } from 'vitest';
+import type { FilterCondition } from '@objectstack/spec/data';
+
+import { InMemoryDriver } from './memory-driver.js';
+import { MemoryAnalyticsService } from './memory-analytics.js';
+
+interface WireBearingError extends Error {
+  code?: string;
+  status?: number;
+}
+
+const ROWS = [
+  { id: '1', stage: 'won', score: 10 },
+  // The null-valued row that separates "has a value" from "has none".
+  { id: '2', stage: null, score: 20 },
+];
+
+/**
+ * The exact leading sentence `driver-sql` produces for this condition, copied
+ * from `sql-driver.ts`'s `nonBooleanExistsComparandError`. A literal rather
+ * than an import: this package does not depend on driver-sql (and must not),
+ * so one condition, one wording (#5240) is held by pinning the other side's
+ * text here.
+ */
+const DRIVER_SQL_LEADING_SENTENCE = (field: string) =>
+  `Operator "$exists" on field "${field}" requires a boolean comparand (true or false).`;
+
+/**
+ * The triage pins (`'yes'`, `1`, the string `'false'`), then the rest of the
+ * battery `driver-sql`'s own `$exists` pins run — so the two backends are held
+ * to the same inputs.
+ */
+const NON_BOOLEAN: Array<[label: string, value: unknown]> = [
+  ["the string 'yes'", 'yes'],
+  ['the number 1', 1],
+  ["the STRING 'false'", 'false'],
+  ['the number 0', 0],
+  ['null', null],
+  ['undefined', undefined],
+  ['an object', {}],
+];
+
+const CUBE = {
+  name: 'deals',
+  title: 'Deals',
+  sql: 'deal',
+  measures: { total: { label: 'Total', type: 'count', sql: 'id' } },
+  dimensions: {
+    id: { label: 'Id', type: 'string', sql: 'id' },
+    stage: { label: 'Stage', type: 'string', sql: 'stage' },
+  },
+  public: true,
+} as never;
+
+describe('[#20897] a non-boolean $exists is refused on every entry of driver-memory', () => {
+  let driver: InMemoryDriver;
+  let analytics: MemoryAnalyticsService;
+
+  beforeEach(async () => {
+    driver = new InMemoryDriver({ persistence: false });
+    await driver.syncSchema('deal', {
+      fields: {
+        id: { type: 'text', name: 'id' },
+        stage: { type: 'text', name: 'stage' },
+        score: { type: 'number', name: 'score' },
+      },
+    } as any);
+    for (const row of ROWS) await driver.create('deal', { ...row });
+    analytics = new MemoryAnalyticsService({ driver, cubes: [CUBE] } as never);
+  });
+
+  const sorted = (rows: unknown): string[] => (rows as Array<Record<string, unknown>>).map((r) => String(r.id)).sort();
+  const q = (where: unknown) => ({ where: where as FilterCondition });
+  const cubeQuery = (where: unknown) =>
+    ({ cube: 'deals', measures: ['total'], dimensions: ['id'], where }) as never;
+
+  /**
+   * Every entry of this package that takes a `where`, by name, with the root its
+   * refusal names and whether the shared comparand-TYPE face runs ahead of the
+   * gate there. The analytics face runs it first (its door, ADR-0053 D-D1), so
+   * a flag that face refuses on TYPE — `undefined`, a plain object — keeps that
+   * face's own sentence, the precedence the analytics `where` door and the
+   * engine seam give it too; the envelope and the position are the same.
+   */
+  const ENTRIES: Array<[name: string, run: (where: unknown) => Promise<unknown>, path: string, typeFaceFirst: boolean]> = [
+    ['find', (w) => driver.find('deal', q(w)), 'filter', false],
+    ['findOne', (w) => driver.findOne('deal', q(w)), 'filter', false],
+    ['count', (w) => driver.count('deal', q(w)), 'filter', false],
+    ['aggregate', (w) => driver.aggregate('deal', { ...q(w), aggregations: [{ function: 'count', alias: 'n' }] } as never), 'filter', false],
+    ['updateMany', (w) => driver.updateMany('deal', q(w), { score: 99 }), 'filter', false],
+    ['deleteMany', (w) => driver.deleteMany('deal', q(w)), 'filter', false],
+    ['the analytics face, query()', (w) => analytics.query(cubeQuery(w)), 'where', true],
+    ['the analytics face, generateSql()', (w) => analytics.generateSql(cubeQuery(w)), 'where', true],
+  ];
+
+  /** The comparands the comparand-TYPE face refuses before any flag rule is asked. */
+  const TYPE_FACE_REFUSED: ReadonlySet<unknown> = new Set<unknown>([undefined]);
+  const isTypeFaceRefused = (value: unknown): boolean =>
+    TYPE_FACE_REFUSED.has(value) || (typeof value === 'object' && value !== null);
+
+  const refusalOf = async (run: () => Promise<unknown>): Promise<WireBearingError> => {
+    try {
+      await run();
+    } catch (e) {
+      return e as WireBearingError;
+    }
+    throw new Error('expected this entry to refuse the filter, but it resolved');
+  };
+
+  for (const [entry, run, root, typeFaceFirst] of ENTRIES) {
+    for (const [label, value] of NON_BOOLEAN) {
+      const words = typeFaceFirst && isTypeFaceRefused(value) ? 'the type face\'s words' : 'driver-sql\'s words';
+      it(`${entry} refuses ${label} with INVALID_FILTER / 400, in ${words}`, async () => {
+        const err = await refusalOf(() => run({ stage: { $exists: value } }));
+        expect(err.code).toBe('INVALID_FILTER');
+        expect(err.status).toBe(400);
+        expect(err.message).toContain(`${root}.stage.$exists`);
+        // The type face's sentence is its own contract, pinned in its own
+        // suite; here only the flag rule's first sentence is load-bearing.
+        if (!(typeFaceFirst && isTypeFaceRefused(value))) {
+          expect(err.message).toContain(DRIVER_SQL_LEADING_SENTENCE('stage'));
+        }
+      });
+    }
+  }
+
+  it('refuses it at every depth, and a satisfiable sibling does not let it through', async () => {
+    // The gate is a WALK, not an evaluation: `{ stage: 'won' }` matches and
+    // `{}` is the TRUE identity, yet neither short-circuits the refusal.
+    for (const [where, at] of [
+      [{ $and: [{ stage: { $exists: 'yes' } }] }, 'filter.$and[0].stage.$exists'],
+      [{ $or: [{ stage: 'won' }, { stage: { $exists: 1 } }] }, 'filter.$or[1].stage.$exists'],
+      [{ $or: [{}, { stage: { $exists: 'false' } }] }, 'filter.$or[1].stage.$exists'],
+      [{ $not: { stage: { $exists: 'yes' } } }, 'filter.$not.stage.$exists'],
+    ] as Array<[unknown, string]>) {
+      const err = await refusalOf(() => driver.find('deal', q(where)));
+      expect(err.code).toBe('INVALID_FILTER');
+      expect(err.status).toBe(400);
+      expect(err.message).toContain(at);
+    }
+  });
+
+  it('a refused write leaves the store untouched', async () => {
+    await refusalOf(() => driver.updateMany('deal', q({ stage: { $exists: 'yes' } }), { score: 99 }));
+    await refusalOf(() => driver.deleteMany('deal', q({ stage: { $exists: 1 } })));
+    const rows = (await driver.find('deal', {} as never)) as Array<Record<string, unknown>>;
+    expect(rows.map((r) => [String(r.id), r.score]).sort()).toEqual([['1', 10], ['2', 20]]);
+  });
+
+  describe('the control: true and false answer find()\'s rows on every read entry', () => {
+    for (const [flag, expected] of [[true, ['1']], [false, ['2']]] as Array<[boolean, string[]]>) {
+      it(`$exists: ${flag} selects ${JSON.stringify(expected)}, and every read entry agrees with find()`, async () => {
+        const where = { stage: { $exists: flag } };
+        const found = sorted(await driver.find('deal', q(where)));
+        expect(found).toEqual(expected);
+        expect(await driver.count('deal', q(where))).toBe(expected.length);
+        expect(String((await driver.findOne('deal', q(where)) as Record<string, unknown>).id)).toBe(expected[0]);
+        const cube = (await analytics.query(cubeQuery(where))).rows as Array<Record<string, unknown>>;
+        expect(cube.map((r) => String(r.id)).sort()).toEqual(found);
+      });
+    }
+  });
+});