feat(drivers,formula,objectql): the engine's filter faces answer the staged $empty operator (#20444)

.changeset/20444-empty-operator-engine-arms.md

@@ -0,0 +1,24 @@
+---
+'@objectstack/driver-sql': minor
+'@objectstack/driver-turso': minor
+'@objectstack/driver-memory': minor
+'@objectstack/driver-mongodb': minor
+'@objectstack/formula': minor
+'@objectstack/objectql': minor
+'@objectstack/spec': minor
+---
+
+feat(drivers,formula,objectql): the engine's filter faces answer the staged `$empty` operator (#20444)
+
+Clause-②: yes (widening)
+
+`$empty: true | false` is declared by `@objectstack/spec` (`FieldOperatorsSchema`) with a per-type meaning: a text-like field is empty when it is null or `''`, a multi-value field (multiselect, checkboxes, tags, or a select / radio / lookup / user / file / image with `multiple: true`) when it is null or `[]`, and every other type only when it is null. `$empty: false` is the exact complement. Until now every face in this list refused it (`INVALID_FILTER` / 400), except `matchesFilterCondition`, which answered `false` for every record. **A driver or evaluator called directly now answers it:**
+
+- **By the field's declared type**, through the spec's one expansion (`expandEmptyOperator`): `driver-sql`'s filter compiler (and so `driver-sqlite-wasm` and `driver-turso`'s local transport, which inherit it), `driver-turso`'s remote transport, `driver-memory`'s query path (`find` / `count` / `update` / `delete`) and `driver-mongodb`'s `translateFilter` (its `find`, its aggregate `$match`). The declaration is the one each driver already receives — `initObjects` / `registerObjectMetadata` / `registerExternalObject` on the SQL family, `syncSchema` on the others. On SQL a multi-value field's empty list is tested as stored JSON per dialect (SQLite `json_array_length` behind a `json_valid` guard, PostgreSQL a `jsonb` comparison, MySQL `JSON_LENGTH`), never as an equality comparand.
+- **By value** — null, a missing value, `''` and `[]` are empty (`isEmptyFilterValue`) — on the faces that read no field declaration: `@objectstack/formula`'s `matchesFilterCondition` (the RLS write-side `check`), `driver-memory`'s reference matcher, and `@objectstack/objectql`'s `having` and per-aggregation `filter`. In `having`, a `count` or `sum` holding `0` is not empty.
+
+**Refused, never guessed** (`INVALID_FILTER` / 400): `$empty` on a field whose declaration the driver does not hold (a table built outside its registration, a builtin column such as `id`, a field with no `type`, or `translateFilter` / `RemoteTransport` used standalone without a declaration), a multi-value field on a SQL dialect the driver does not model, and a flag that is not a boolean. `driver-memory`'s analytics (cube) face refuses `$empty` as an operator it cannot compile, as it does `$null`.
+
+New optional API: `translateFilter(where, temporalKind?, valueShape?)` in `@objectstack/driver-mongodb` takes a declared-value-shape resolver (type `ValueShapeResolver`), and `buildAggregationPipeline` a `valueShape` option; `RemoteTransport.setDeclaredValueShapeResolver` in `@objectstack/driver-turso`, which `TursoDriver` wires. `@objectstack/spec`'s shared `FILTER_LOGIC_CASES` table gains seven `$empty` cases: a backend that runs it answers `$empty` or goes red, and its harness must declare the fixture's columns.
+
+`$empty` stays staged: it is not in `FILTER_OPERATORS`, so the engine's front door still refuses it until the flip card adds it, and the view operators `is_empty` / `is_not_empty` still lower to `$null`.

packages/drivers/driver-memory/src/filter-refusal.ts

@@ -364,6 +364,13 @@ export const SUPPORTED_FIELD_OPERATORS: ReadonlySet<string> = new Set<string>([
   ...FILTER_OPERATORS,
   '$like',
   '$ilike',
+  // [#20444] The staged emptiness flag, admitted BY HAND for the reason the
+  // `$like` paragraph above gives, and under its ordering rule: both arms land
+  // with this entry — the reference matcher judges the stored value
+  // (`isEmptyFilterValue`, the spec's reading for a face holding no field
+  // declaration) and the live query path the field's DECLARED row
+  // (`expandEmptyOperator`, from the declaration `syncSchema` recorded).
+  '$empty',
 ]);
 
 /** The vocabulary as it appears in a refusal message, in declaration order. */
@@ -611,6 +618,42 @@ export function nonBooleanNullComparandError(field: string, value: unknown, path
   );
 }
 
+/**
+ * [#20444] A non-boolean `$empty` comparand. The leading sentence is
+ * `driver-sql`'s `nonBooleanEmptyComparandError`, verbatim — one condition,
+ * one wording (#5240).
+ */
+export function nonBooleanEmptyComparandError(field: string, value: unknown, path: string): Error {
+  return unsupportedFilterError(
+    `Operator "$empty" on field "${field}" requires a boolean comparand (true or false). ` +
+      `Received ${describeFilterOperand(value)} (${safeShapePreview(value)}) at ${path}. ` +
+      `@objectstack/spec FieldOperatorsSchema declares $empty as a boolean: true asks for the ` +
+      `empty rows, false for their exact complement.`,
+  );
+}
+
+/**
+ * [#20444] `$empty` on the live query path, aimed at a field this driver holds
+ * no declaration for — an object never passed through `syncSchema`, a field
+ * its schema does not name, or one declared with no `type`.
+ *
+ * What counts as empty is the field's DECLARED row of the ruled table, and the
+ * live path reads it from the declaration rather than from a value, so without
+ * one there is no answer to give: refused, never guessed. The reference matcher
+ * (`memory-matcher.ts`) is the face that holds NO declarations at all, and it
+ * judges the stored value instead — the spec's reading for such a face.
+ */
+export function undeclaredEmptyOperatorFieldError(field: string, path: string): Error {
+  return unsupportedFilterError(
+    `Operator "$empty" on field "${field}" at ${path} targets a field whose declaration this ` +
+      `driver does not hold (no declared type — the object's schema was never synced, or does not ` +
+      `declare the field). What counts as empty is the field's DECLARED row of the ruled table — ` +
+      `null or '' for a text-like type, null or [] for a multi-value field, null only for every ` +
+      `other type — so the operator is refused rather than guessed. Declare the field, or use ` +
+      `"$null" for "has no value".`,
+  );
+}
+
 /**
  * [#5702] A RETIRED filter operator in a field constraint.
  *
@@ -895,6 +938,13 @@ function assertFieldConstraintShape(
     if (op === '$null' && typeof spec[op] !== 'boolean') {
       throw nonBooleanNullComparandError(field, spec[op], `${path}.$null`);
     }
+    // [#20444] `$empty`'s comparand is a boolean by the same declaration
+    // (`FieldOperatorsSchema`), refused on this walk for the same reason: both
+    // faces of this package evaluate `true` / `false` exhaustively, so a third
+    // value would land on whichever side each arm happens to default to.
+    if (op === '$empty' && typeof spec[op] !== 'boolean') {
+      throw nonBooleanEmptyComparandError(field, spec[op], `${path}.$empty`);
+    }
     // [#16810] An ARRAY comparand on a single-value comparison — the operator
     // spelling of the implicit-equality position refused at the top of this
     // function, and the same cell `@objectstack/spec`'s comparand door leaves

packages/drivers/driver-memory/src/memory-20444-empty-operator.test.ts

@@ -0,0 +1,166 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#20444] The staged `$empty` operator on this package's three filter faces.
+ *
+ * - **The live query path** (`InMemoryDriver.find` → mingo) holds the field
+ *   declarations `syncSchema` recorded, so it answers by the field's DECLARED
+ *   row of the ruled 「is empty」 table (ruling A on #20399, record 5865693155;
+ *   the spec's `expandEmptyOperator`): text-like = null or `''`; multi-value =
+ *   null or `[]`; every other type = null only; `$empty: false` the exact
+ *   complement. A field it holds no declaration for is REFUSED.
+ * - **The reference matcher** (`match`) holds no declarations at all, so it
+ *   takes the reading the spec gives such a face — by value
+ *   (`isEmptyFilterValue`): null, a missing key, `''` and `[]` are empty.
+ * - **The analytics (cube) face** does not lower the flag (nor `$null`), and
+ *   refuses it loudly as a declared operator it cannot compile.
+ *
+ * The two value-level faces agree on every value a field's own type can hold;
+ * the one place they part is a stored state the declaration does not predict,
+ * and that cell is pinned below so the divergence is a measurement, not a
+ * surprise.
+ */
+
+import { beforeAll, describe, expect, it } from 'vitest';
+import type { Cube, FilterCondition } from '@objectstack/spec/data';
+import { InMemoryDriver } from './memory-driver.js';
+import { match } from './memory-matcher.js';
+import { MemoryAnalyticsService } from './memory-analytics.js';
+
+const TABLE = 'os20444_empty';
+
+const FIELDS = {
+  id: { type: 'text' },
+  title: { type: 'text' },
+  tags: { type: 'tags' },
+  owners: { type: 'lookup', reference: TABLE, multiple: true },
+  score: { type: 'number' },
+};
+
+const ROWS: Array<Record<string, unknown>> = [
+  { id: 'r1', title: 'x', tags: ['a'], owners: ['u1'], score: 5 },
+  { id: 'r2', title: '', tags: [], owners: [], score: 0 },
+  { id: 'r3', title: null, tags: null, owners: null, score: null },
+  { id: 'r4', title: '  ', tags: ['a', 'b'], owners: ['u2'], score: -1 },
+  // Every column MISSING — the other reading of "no value".
+  { id: 'r5' },
+];
+
+const CASES: Array<{ where: FilterCondition; expected: string[] }> = [
+  { where: { title: { $empty: true } }, expected: ['r2', 'r3', 'r5'] },
+  { where: { title: { $empty: false } }, expected: ['r1', 'r4'] },
+  { where: { tags: { $empty: true } }, expected: ['r2', 'r3', 'r5'] },
+  { where: { tags: { $empty: false } }, expected: ['r1', 'r4'] },
+  { where: { owners: { $empty: true } }, expected: ['r2', 'r3', 'r5'] },
+  { where: { owners: { $empty: false } }, expected: ['r1', 'r4'] },
+  { where: { score: { $empty: true } }, expected: ['r3', 'r5'] },
+  { where: { score: { $empty: false } }, expected: ['r1', 'r2', 'r4'] },
+  { where: { $not: { title: { $empty: true } } }, expected: ['r1', 'r4'] },
+  { where: { $not: { tags: { $empty: false } } }, expected: ['r2', 'r3', 'r5'] },
+  { where: { $or: [{ score: 5 }, { tags: { $empty: true } }] }, expected: ['r1', 'r2', 'r3', 'r5'] },
+  { where: { $and: [{ title: { $empty: false } }, { owners: { $empty: false } }] }, expected: ['r1', 'r4'] },
+  { where: { title: { $empty: false, $ne: 'x' } }, expected: ['r4'] },
+  { where: { tags: { $empty: true, $ne: null } }, expected: ['r2'] },
+];
+
+function refusal(run: () => unknown): Promise<{ code?: string; status?: number } | 'answered'> {
+  return Promise.resolve()
+    .then(run)
+    .then(
+      () => 'answered' as const,
+      (err) => ({ code: (err as { code?: string }).code, status: (err as { status?: number }).status }),
+    );
+}
+
+describe('[#20444] InMemoryDriver — $empty on the live path, the reference matcher and the analytics face', () => {
+  let driver: InMemoryDriver;
+  const ids = async (where: FilterCondition) =>
+    ((await driver.find(TABLE, { where })) as Array<Record<string, unknown>>).map((r) => String(r.id)).sort();
+  const reference = (where: FilterCondition) =>
+    ROWS.filter((r) => match(r, where)).map((r) => String(r.id)).sort();
+
+  beforeAll(async () => {
+    driver = new InMemoryDriver({ persistence: false });
+    await driver.connect();
+    await driver.syncSchema(TABLE, { fields: FIELDS });
+    for (const row of ROWS) await driver.create(TABLE, { ...row });
+  });
+
+  it('the fixture is the five rows', async () => {
+    expect(await ids({})).toEqual(['r1', 'r2', 'r3', 'r4', 'r5']);
+  });
+
+  for (const c of CASES) {
+    it(`${JSON.stringify(c.where)} → ${JSON.stringify(c.expected)} on the live path AND the reference matcher`, async () => {
+      expect(await ids(c.where), 'live').toEqual(c.expected);
+      expect(reference(c.where), 'reference matcher').toEqual(c.expected);
+    });
+  }
+
+  it('$empty: false partitions every declared field with $empty: true, on both faces', async () => {
+    for (const field of ['title', 'tags', 'owners', 'score']) {
+      const empty = await ids({ [field]: { $empty: true } });
+      const full = await ids({ [field]: { $empty: false } });
+      expect([...empty, ...full].sort(), field).toEqual(['r1', 'r2', 'r3', 'r4', 'r5']);
+      expect(reference({ [field]: { $empty: true } }), field).toEqual(empty);
+    }
+  });
+
+  it('the live path REFUSES a field it holds no declaration for; the matcher judges the value', async () => {
+    expect(await refusal(() => driver.find(TABLE, { where: { nope: { $empty: true } } })))
+      .toEqual({ code: 'INVALID_FILTER', status: 400 });
+    expect(await refusal(() => driver.find('never_synced', { where: { title: { $empty: true } } })))
+      .toEqual({ code: 'INVALID_FILTER', status: 400 });
+    expect(reference({ nope: { $empty: true } })).toEqual(['r1', 'r2', 'r3', 'r4', 'r5']);
+  });
+
+  it('the one cell where the declared row and the by-value reading part: a stored state the type cannot hold', async () => {
+    // A number column holding '' is a write-door defect, never a value the
+    // declaration predicts. The declared null-only row does not count it; the
+    // declaration-free matcher does. Pinned so the divergence is known.
+    const odd = new InMemoryDriver({ persistence: false });
+    await odd.connect();
+    await odd.syncSchema('odd', { fields: { id: { type: 'text' }, score: { type: 'number' } } });
+    const rows = [{ id: 'blank', score: '' }];
+    for (const row of rows) await odd.create('odd', { ...row });
+    const live = ((await odd.find('odd', { where: { score: { $empty: true } } })) as Array<Record<string, unknown>>)
+      .map((r) => r.id);
+    expect(live).toEqual([]);
+    expect(rows.filter((r) => match(r, { score: { $empty: true } })).map((r) => r.id)).toEqual(['blank']);
+  });
+
+  it('a non-boolean flag is refused by both faces, on the shared shape gate', async () => {
+    expect(await refusal(() => driver.find(TABLE, { where: { title: { $empty: 'yes' as never } } })))
+      .toEqual({ code: 'INVALID_FILTER', status: 400 });
+    expect(await refusal(() => match(ROWS[0], { title: { $empty: 1 as never } })))
+      .toEqual({ code: 'INVALID_FILTER', status: 400 });
+    // Even where an identity would settle the node before any arm ran.
+    expect(await refusal(() => match(ROWS[0], { $or: [{}, { title: { $empty: 'no' as never } }] })))
+      .toEqual({ code: 'INVALID_FILTER', status: 400 });
+  });
+
+  it('the analytics (cube) face refuses $empty as a declared operator it cannot compile — never drops it', async () => {
+    const cube: Cube = {
+      name: TABLE,
+      title: 'empty',
+      sql: TABLE,
+      measures: { count: { name: 'count', label: 'Rows', type: 'count', sql: 'id' } },
+      dimensions: {
+        id: { name: 'id', label: 'id', type: 'string', sql: 'id' },
+        title: { name: 'title', label: 'title', type: 'string', sql: 'title' },
+      },
+      public: true,
+    } as Cube;
+    const service = new MemoryAnalyticsService({ driver, cubes: [cube] });
+    expect(
+      await refusal(() =>
+        service.query({
+          cube: TABLE,
+          measures: [`${TABLE}.count`],
+          dimensions: [`${TABLE}.id`],
+          where: { title: { $empty: true } },
+        } as never),
+      ),
+    ).toEqual({ code: 'INVALID_FILTER', status: 400 });
+  });
+});