fix(objectql)!: a per-aggregation filter refuses a scalar comparison on a declared JSON-stored field, in where's words

.changeset/21007-aggregation-filter-json-column-refusal.md

@@ -0,0 +1,27 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/core": minor
+"@objectstack/driver-sql": patch
+---
+
+fix(objectql)!: a per-aggregation `filter` refuses `$in` / `$nin` / `$eq` / `$ne` / an ordering / `$between` / implicit equality on a declared JSON-stored field with `INVALID_FILTER` / 400, in the words `where` refuses them in, instead of counting rows the stored arrays cannot support
+
+Clause-②: yes (widening)
+
+<!-- adr-0087: not-required (no-migration-prescription) a refusal of a QUERY shape at the engine's per-aggregation filter position: the operator x declared-type pairs refused are exactly the pairs driver-sql's where has refused on a JSON-stored column since its column-type gate landed, and the per-aggregation position now answers them the same way. No authorable key, spelling or stored metadata shape moves: FilterConditionSchema, AggregationNodeSchema and every object and dataset definition parse and save as before, and nothing reads or rewrites a stored row. There is nothing for objectstack migrate meta to rewrite, since what changes is which query the engine answers, not what any metadata says; the refusal itself names the spelling to use. The other categories are closed on facts: every bumped package publishes (not unpublished); no ADR-0087 id covers a filter operator on a JSON-stored column and this diff adds none (not registered / already-registered); and the change is runtime behaviour plus ADDITIONS only (three new @objectstack/core exports and one new optional trailing parameter on applyInMemoryAggregation), with no published interface or type narrowed or removed (not runtime-interface-only / type-surface-only). -->
+
+**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts in one position, `aggregations[i].filter`, on every driver and for every caller that reaches the engine: the REST query door (`POST /api/v1/data/:object/query`), a flow or hook, and the analytics strategy that lowers a dataset measure's filter onto `engine.aggregate`. The published `applyInMemoryAggregation(rows, ast, timezone, fields)` narrows the same way when it is handed a field map. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+**What is refused.** On a field the object declares JSON-stored (a structured-JSON type such as `json` or `address`, an inherently multi-value option type such as `tags`, `multiselect` or `checkboxes`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`), a per-aggregation `filter` that compares the field with `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin` or implicit equality (`{ "owners": "u1" }`) is refused with `INVALID_FILTER` / 400, whatever the comparand (`null` and an empty list included), at any depth under `$and` / `$or` / `$not`, and before any driver is asked for a row, so an empty table refuses it too. That is the set `driver-sql`'s `where` refuses on such a column, for the same reason.
+
+**What an author sees now.** The same 400 body the same filter gets as a `where`: the filter WAS NOT APPLIED, the comparison can never equal one member of a stored list, and the spelling to use, `{ "FIELD": { "$contains": "a" } }` for membership, or an `$or` of `$contains` for any-of. The field and the operator are withheld from the message, as they are for `where`, and the full diagnostic, naming both and the aggregation position, goes to the server log.
+
+**Why a refusal.** The engine evaluates a per-aggregation filter itself, and it compared the whole stored array against a scalar. Measured through `POST /api/v1/data/:object/query` on SQLite and PostgreSQL 16 over six rows of a `multiple: true` lookup, two of them holding `u1`: `{ owners: { $in: ['u1', 'u9'] } }` counted 0, `{ owners: { $nin: ['u1', 'u9'] } }` counted all 6, the two rows it was asked to exclude among them, `$gt` / `$lte` / `$between` counted 4 / 1 / 5, and `{ tags: { $eq: 'red' } }` counted the row holding `['red']` by JS loose equality. The same filters in `where` were 400 on both dialects.
+
+**Who is affected.** A dashboard, report, dataset measure or caller whose per-aggregation filter compares a JSON-stored field with one of those operators and read the count as a real answer. Also a host calling `applyInMemoryAggregation` directly with a `fields` map: it now judges each `aggregations[i].filter` against that map before any row (an empty `rows` array included) and throws the same `INVALID_FILTER` / 400. It takes an optional fifth argument, `reportWithheld(diagnostic)`, which receives the withheld field, operator and position; without it the diagnostic is dropped. A call without `fields` judges nothing, as before. Write `$contains` for "holds this member", an `$or` of `$contains` for "holds any of these", and `$not` around either for the exclusion.
+
+**Unchanged.** `$contains` and `$notContains` (membership on such a field), `$exists`, `$null` and `$empty`; every operator on a field that is not JSON-stored; `having`; `where`; and a host whose engine has no declaration for the object, where nothing is judged.
+
+**`@objectstack/core`** (three new root exports): `JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText(field, op, bare)` and its return type `JsonColumnOperatorRefusalText` (`{ message, diagnostic }`). They are the operator set and the two texts (the withheld message and the full diagnostic) of the JSON-column refusal, so `driver-sql`'s `where` and the engine's per-aggregation filter refuse with one set and one sentence.
+
+**`@objectstack/driver-sql`**: no behaviour change. Its JSON-column gate reads the set and the text from `@objectstack/core`; every refusal it prints is byte for byte what it printed before.

packages/core/src/index.ts

@@ -125,6 +125,13 @@ export * from './utils/temporal-comparand.js';
 // do not depend on each other, and each driver used to carry its own copy.
 export * from './utils/temporal-storage-form.js';
 
+// [#21007] …and the refusal a scalar comparison gets on a field stored as a
+// JSON column: the operator set and the words. `driver-sql` refuses it on
+// `where`, and `@objectstack/objectql` on the per-aggregation `filter` it
+// evaluates itself — one set and one sentence, here for the reason the entry
+// above gives.
+export * from './utils/json-column-operator-refusal.js';
+
 // [#12350 / ADR-0126 §4] THE activation-ledger row contract, parameterized by
 // `metadata_type`. Same reason as the two entries above: its consumers —
 // `@objectstack/objectql` (packaged actions) and

packages/core/src/utils/json-column-operator-refusal.test.ts

@@ -0,0 +1,69 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#21007] The JSON-column refusal's operator set and words, moved here from
+ * `driver-sql` so the engine's per-aggregation `filter` refuses with them too.
+ *
+ * Two pins, both against what `driver-sql` answered BEFORE the move:
+ *
+ * - **The set** — the 22 spellings `driver-sql`'s module-private
+ *   `JSON_COLUMN_INCOMPATIBLE_OPERATORS` held, member for member.
+ * - **The words** — the SHA-256 of each text, captured from `driver-sql`'s
+ *   built `jsonColumnOperatorError` at the commit before the move (`8f784959c`)
+ *   through a real `SqlDriver` over SQLite: the withheld message (one text for
+ *   every operator), and the diagnostic for an operator, for `$between`, and
+ *   for the bare equality spelling, on a column named `members`. A hash rather
+ *   than a second literal copy, so this file is not a third place the sentence
+ *   lives; the length beside each hash says how far a failure moved it.
+ *
+ * A deliberate change of wording updates the hashes in the PR that makes it —
+ * and then reaches `where` and the per-aggregation `filter` alike, which is the
+ * point of the move.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { createHash } from 'node:crypto';
+import { JSON_COLUMN_INCOMPATIBLE_OPERATORS, jsonColumnOperatorRefusalText } from './json-column-operator-refusal.js';
+
+const sha256 = (text: string): string => createHash('sha256').update(text, 'utf8').digest('hex');
+
+describe('[#21007] JSON_COLUMN_INCOMPATIBLE_OPERATORS', () => {
+  it('holds exactly the spellings driver-sql refused before the move', () => {
+    expect([...JSON_COLUMN_INCOMPATIBLE_OPERATORS].sort()).toEqual([
+      '!=', '$between', '$eq', '$gt', '$gte', '$in', '$lt', '$lte', '$ne', '$nin',
+      '<', '<=', '<>', '=', '==', '>', '>=',
+      'between', 'in', 'nin', 'not_in', 'notin',
+    ]);
+  });
+
+  it('leaves out the membership spelling, the rest of the text family and the null predicates', () => {
+    for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith', '$icontains', '$null', '$exists', '$empty']) {
+      expect(JSON_COLUMN_INCOMPATIBLE_OPERATORS.has(op), op).toBe(false);
+    }
+  });
+});
+
+describe('[#21007] jsonColumnOperatorRefusalText — byte for byte what driver-sql printed before the move', () => {
+  const MESSAGE = { sha: 'c6103dd665625ab3a779822fecbe650bd7f67fc6ea38d17d5fd57cf6ae9193ff', length: 748 };
+
+  it.each([
+    ['an operator', '$in', false, { sha: '358d5aae39170ab3da198beb39368a3f3476dd057dbd5da5f4295cf244eab67e', length: 648 }],
+    ['$between', '$between', false, { sha: '505c094ac5212ac121cdeb4803787ba036f46177450d542fbccf60a9386924fe', length: 658 }],
+    ['the bare equality spelling', '=', true, { sha: '92bb1728ca012c4cde6deb05be3cb0320e113c8c5ac250ecad5b091f3bb87094', length: 660 }],
+  ] as const)('%s', (_name, op, bare, diagnostic) => {
+    const text = jsonColumnOperatorRefusalText('members', op, bare);
+    expect({ sha: sha256(text.message), length: text.message.length }).toEqual(MESSAGE);
+    expect({ sha: sha256(text.diagnostic), length: text.diagnostic.length }).toEqual(diagnostic);
+  });
+
+  it('the message names neither the field nor the operator, and prescribes $contains and an $or of it', () => {
+    const { message, diagnostic } = jsonColumnOperatorRefusalText('secret_col', '$nin', false);
+    expect(message).not.toContain('secret_col');
+    expect(message).not.toContain('"$nin"');
+    expect(message).toContain('WAS NOT APPLIED');
+    expect(message).toContain('{ "FIELD": { "$contains": "a" } }');
+    expect(message).toContain('{ "$or": [{ "FIELD": { "$contains": "a" } }');
+    expect(diagnostic).toContain('Operator "$nin" on field "secret_col"');
+    expect(diagnostic).toContain('{ "secret_col": { "$contains": "a" } }');
+  });
+});

packages/core/src/utils/json-column-operator-refusal.ts

@@ -0,0 +1,150 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#21007] The refusal a SCALAR comparison operator gets when it is aimed at a
+ * field stored as a JSON column — a `multiple: true` field, an inherently
+ * multi-value option type (`tags`, `multiselect`, `checkboxes`) or a
+ * structured-JSON type (`json`, `address`, …): the operator set and the words.
+ *
+ * ## Two faces, one rule
+ *
+ * `driver-sql` refuses these operators on its `where` (#7398): such a column
+ * holds the serialization `["a","b"]`, so `$in` / `$eq` compare that whole text
+ * against one value and match nothing, while `$nin` / `$ne` return the very rows
+ * they were asked to exclude, and the orderings return a lexicographic verdict
+ * over the serialization. `@objectstack/objectql` evaluates a per-aggregation
+ * `filter` itself, row by row, and gave the same three wrong answers in JS —
+ * `{ owners: { $nin: ['u1'] } }` counted the rows holding `u1`. It now refuses
+ * the same operators on the same declared fields, before any driver is asked.
+ *
+ * The two faces cannot import each other (the engine does not depend on a
+ * driver), and a copy each is how one refusal comes to answer one mistake in
+ * two ways. So the set and the sentence live here, on the floor both already
+ * stand on — beside `temporalStorageForm`, which the same two faces share for
+ * the same reason. Each face keeps its own error CONSTRUCTOR (the driver's
+ * carries the #8220 provenance seam, the engine's the ADR-0112 envelope); what
+ * they share is what the caller reads.
+ *
+ * ## The other half of the JSON column's contract
+ *
+ * `$contains` is the membership spelling on such a column (`FILTER_OPERATORS`'
+ * `$contains` docblock, `@objectstack/spec`), and it is what the refusal
+ * prescribes — `$contains` for one member, an `$or` of `$contains` for any-of.
+ * That is why it is ABSENT from the set below, with the rest of the text family
+ * and the null predicates.
+ */
+
+/**
+ * [#7398] Operators whose SQL lowering compares a column's STORED SCALAR to a
+ * value — every spelling either of `driver-sql`'s two comparison emitters
+ * answers (`applyFilterCondition`'s plain-column switch and
+ * `applyNormalizedComparison`'s normalised arms).
+ *
+ * The bare infix forms are here for the same reason they are in `driver-sql`'s
+ * `SCALAR_COMPARAND_OPERATORS`: `applyNormalizedComparison` really does
+ * answer `in` / `nin` / `not_in` / `notin` / `=` / `<>` / `>` …, so a filter
+ * spelled that way against a normalised column compiles, and a gate that only
+ * knew the `$`-forms would leave the failure alive at a different spelling —
+ * the lesson #5234 already paid for there.
+ *
+ * `$between` is included although the card's minimum set stopped at the four
+ * ordering comparisons: it IS `>= AND <=` (`driver-sql` even decomposes a
+ * calendar-day `$between` into `$gte`/`$lt` ahead of its emitter), so refusing
+ * the halves and compiling the compound would be the same wrong answer at one
+ * more spelling.
+ *
+ * Deliberately ABSENT, and this is the load-bearing half of the set: the `LIKE`
+ * family (`$contains`, `$notContains`, `$startsWith`, `$endsWith`,
+ * `$icontains`) and the null predicates (`$null`, `$exists`). `$contains` is
+ * the ONLY working membership spelling on a JSON-array column and downstream
+ * code depends on it (#7398's own tables), while `IS NULL` asks about the
+ * column's presence, which is a well-formed question whatever the column holds.
+ *
+ * [#21007] Moved here from `driver-sql`, unchanged, so the per-aggregation
+ * `filter` refuses exactly the operators `where` refuses.
+ */
+export const JSON_COLUMN_INCOMPATIBLE_OPERATORS: ReadonlySet<string> = new Set([
+  '$eq', '=', '==',
+  '$ne', '!=', '<>',
+  '$gt', '>', '$gte', '>=', '$lt', '<', '$lte', '<=',
+  '$in', 'in',
+  '$nin', 'nin', 'not_in', 'notin',
+  '$between', 'between',
+]);
+
+/** The two texts of one JSON-column refusal — see {@link jsonColumnOperatorRefusalText}. */
+export interface JsonColumnOperatorRefusalText {
+  /**
+   * What the caller is told. It names neither the field nor the operator: on a
+   * read scope the predicate is an administrator's, so both are withheld
+   * (#7929 / #8197), and the sentence says where they went.
+   */
+  readonly message: string;
+  /** The full diagnostic — the field and the operator named — for the server log. */
+  readonly diagnostic: string;
+}
+
+/**
+ * [#7398] The words of the refusal: a scalar-comparison operator met a field
+ * stored as JSON TEXT, so the comparison can never mean what the caller wrote.
+ *
+ * The mechanism is one line of SQL. A `multiple: true` field is stored as the
+ * serialization `["U1","U2"]`, so `members in ('U1')` is FALSE — the text
+ * genuinely is not equal to that id — and `members not in ('U1')` is TRUE:
+ *
+ * - `$in` / `$eq` / bare equality → **0 rows**, fail-CLOSED. Silent, and a
+ *   `200` with an empty array is byte-identical to a query that legitimately
+ *   matched nothing, so no caller has anything to key on.
+ * - `$nin` / `$ne` → **the row it was asked to exclude**, fail-OPEN. That is
+ *   the dangerous half and the reason this is a refusal rather than a
+ *   documented footgun: an exclusion that silently stops excluding WIDENS a
+ *   result set, the direction #3948 / #4209 / #5347 all ruled outranks a
+ *   narrowing one.
+ * - The ordering comparisons are not even uniformly empty: `$lte` matched,
+ *   because `["usr_…"` sorts below `usr_…` on the leading `[`. A lexicographic
+ *   compare over a serialization is a wrong answer, not a narrow one.
+ *
+ * The message states the filter WAS NOT APPLIED, because "no rows" is a
+ * legitimate answer to a legitimate query, so a caller must be told that this
+ * one was never asked. The prescription is `$contains` (and an `$or` of
+ * `$contains` for any-of). It survives redaction with PLACEHOLDER names — the
+ * SHAPE is the repair, and the shape names nothing.
+ *
+ * `bare` is the implicit-equality spelling `{ field: value }`, whose operator
+ * the diagnostic names as `=`.
+ *
+ * [#21007] Moved here from `driver-sql`'s `jsonColumnOperatorError`, byte for
+ * byte, so `where` and the per-aggregation `filter` print one sentence. The
+ * caller builds the error: this returns only the text.
+ */
+export function jsonColumnOperatorRefusalText(
+  field: string,
+  op: string,
+  bare: boolean,
+): JsonColumnOperatorRefusalText {
+  const spelling = bare
+    ? `The bare equality spelling { "${field}": value }`
+    : `Operator "${op}"`;
+  const on = bare ? '' : ` on field "${field}"`;
+  return {
+    message:
+      `A constraint in this filter WAS NOT APPLIED: it aims a scalar comparison operator at a ` +
+      `field this driver stores as a JSON TEXT column (e.g. ["a","b"]), and such an operator ` +
+      `compares that whole serialized text against a single value — it can never equal one ` +
+      `member. Use "$contains" for membership ({ "FIELD": { "$contains": "a" } }), or an $or of ` +
+      `"$contains" for any-of ({ "$or": [{ "FIELD": { "$contains": "a" } }, ` +
+      `{ "FIELD": { "$contains": "b" } }] }). Refused rather than compiled because the answer ` +
+      `was silently wrong in BOTH directions: $in/$eq matched nothing, while $nin/$ne returned ` +
+      `the very rows they were asked to exclude. The field and the operator this filter ` +
+      `used are withheld from the message; the full diagnostic is in the server log.`,
+    diagnostic:
+      `${spelling}${on} WAS NOT APPLIED: "${field}" is a multi-value (or otherwise JSON-valued) ` +
+      `field, stored by this driver as a JSON TEXT column (e.g. ["a","b"]), and "${op}" compares ` +
+      `that whole serialized text against a single value — it can never equal one member. ` +
+      `Use "$contains" for membership ({ "${field}": { "$contains": "a" } }), or an $or of ` +
+      `"$contains" for any-of ({ "$or": [{ "${field}": { "$contains": "a" } }, ` +
+      `{ "${field}": { "$contains": "b" } }] }). Refused rather than compiled because the answer ` +
+      `was silently wrong in BOTH directions: $in/$eq matched nothing, while $nin/$ne returned ` +
+      `the very rows they were asked to exclude.`,
+  };
+}