fix(core)!: the JSON-column gate refuses $startsWith, $endsWith, $icontains and $like / $ilike as it refuses the equality family

.changeset/21009-json-column-text-operators.md

@@ -0,0 +1,24 @@
+---
+"@objectstack/core": minor
+"@objectstack/objectql": minor
+---
+
+fix(core)!: a filter that aims `$startsWith`, `$endsWith`, `$icontains`, `$like` or `$ilike` at a field stored as a JSON column is refused with `INVALID_FILTER` / 400, as `$eq` / `$in` / `$nin` already are, instead of matching the field's serialized text or failing at query time
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) a refusal of a QUERY shape on a JSON-stored column: the five text operators join the operator set driver-sql's where and the engine's per-aggregation filter already refuse there, through the one shared set in @objectstack/core. No authorable key, spelling or stored metadata shape moves: FilterConditionSchema, ViewFilterRuleSchema and every object, view and dataset definition parse and save as before, and nothing reads or rewrites a stored row. There is nothing for objectstack migrate meta to rewrite, since what changes is which query a driver answers, not what any metadata says, and the refusal itself names the spelling to use. The other categories are closed on facts: @objectstack/core publishes (not unpublished); no ADR-0087 id covers a text operator on a multi-valued field, and filter-text-operator-declared-type-refused covers declared non-text types only, so this diff neither registers nor reuses one (not registered / already-registered); and the change is runtime behaviour only, with no published interface or type narrowed or removed: the exported set keeps its ReadonlySet of string type, and objectql's search expander changes only which operator it emits for a multi-valued field (not runtime-interface-only / type-surface-only). -->
+
+**BREAKING**: this narrows which filters are answered on a field stored as a JSON column, on every face that reads `@objectstack/core`'s `JSON_COLUMN_INCOMPATIBLE_OPERATORS`: `driver-sql`'s `where` (and `driver-sqlite-wasm` and `driver-turso`'s local transport, which inherit it) on every read and write face that lowers a filter, and the engine's per-aggregation `filter`. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+**What is refused.** On a field declared multi-valued (an inherently multi-value option type such as `tags`, `multiselect` or `checkboxes`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`) or structured-JSON (`json`, `address`, …), a filter using `$startsWith`, `$endsWith`, `$icontains`, or the staged pattern pair `$like` / `$ilike`, is refused with `INVALID_FILTER` / 400, at any depth under `$and` / `$or` / `$not`. The per-aggregation `filter` refuses the three declared ones; it already refused `$like` / `$ilike` as operators it does not evaluate. Through the engine, a structured-JSON field was already refused all seven text operators by the text-operator declared-type door, which still answers first there, in its own words; what moves for it is a direct driver call.
+
+**What an author sees.** The body the equality family already gets there, byte for byte: the filter WAS NOT APPLIED, and the spelling to use, `{ "FIELD": { "$contains": "a" } }` for membership or an `$or` of `$contains` for any-of. The field and the operator are withheld from the message and named in the server-log diagnostic; a filter positively marked as the caller's own reads them named.
+
+**Why a refusal.** Such a column stores the serialization `["u1","u2"]`, and none of these five operators has a membership reading. Measured through `POST /api/v1/data/:object/query` on a multi-value lookup and a `tags` field: on SQLite `$startsWith: "["` and `$endsWith: "]"` matched every row with a value, `$startsWith: "u1"` matched none of the rows holding `u1`, and `$icontains: "U1"` also matched the row holding only `u10`; on PostgreSQL 16 every one failed at query time with a `500` `DATABASE_ERROR`, a `json` column having no `LIKE` operator; the per-aggregation `filter` counted 0 for each. No membership reading is invented for a prefix, suffix or case-folded test.
+
+**Who is affected.** A saved filter, list view, dashboard widget, report or caller that aims one of these operators at a multi-valued or JSON-stored field. On SQLite it read rows that matched the stored brackets and quotes; it now gets the 400. On PostgreSQL it already failed, with a 500. Write `$contains` for "holds this member", an `$or` of `$contains` for "holds any of these", and `$not` around either for the exclusion.
+
+**`@objectstack/objectql`: `$search` over a multi-valued field answers by membership.** The search expander (`$search` on `find`, `findOne` and `aggregate`, the REST `search` / `$search` parameter included) used to emit `$in` for a term matching a `select` option label and `$icontains` for any other term, against every field in the resolved search set. On a multi-valued field both are refused by the gate above, so one such field in the set failed the whole search: a label term answered 400 on every dialect, and any other term answered 500 on PostgreSQL and, with this change, 400 on SQLite. The auto-default set includes a `select` declared `multiple: true`, as in `examples/app-todo`'s `todo_task.tags`, and `searchableFields` may name a `tags` field or a multi-valued lookup. Such a field is now matched by membership: a term matching option labels becomes one `$contains` per matched option value, and any other term, or any term on a field with no options, becomes `$contains` of the term. No search answers 400 or 500 for it any more. **The visible cost:** to hit a multi-valued field, a term must now equal one of its members or match one of its option labels; SQLite used to match substrings of the stored array's serialized text as well, so a term like `wood` found a row tagged `redwood`, and it no longer does. Scalar fields are searched exactly as before.
+
+**Unchanged.** `$contains` and `$notContains` (membership on such a field), `$exists`, `$null` and `$empty`; every operator on a field that is not JSON-stored, the scalar text column included; `driver-memory`; and `driver-turso`'s remote transport, which compiles its own filters.

packages/core/src/utils/json-column-operator-refusal.test.ts

@@ -7,7 +7,8 @@
  * Two pins, both against what `driver-sql` answered BEFORE the move:
  *
  * - **The set** — the 22 spellings `driver-sql`'s module-private
- *   `JSON_COLUMN_INCOMPATIBLE_OPERATORS` held, member for member.
+ *   `JSON_COLUMN_INCOMPATIBLE_OPERATORS` held, member for member, and
+ *   [#21009] the five text operators that joined them since.
  * - **The words** — the SHA-256 of each text, captured from `driver-sql`'s
  *   built `jsonColumnOperatorError` at the commit before the move (`8f784959c`)
  *   through a real `SqlDriver` over SQLite: the withheld message (one text for
@@ -28,16 +29,23 @@ import { JSON_COLUMN_INCOMPATIBLE_OPERATORS, jsonColumnOperatorRefusalText } fro
 const sha256 = (text: string): string => createHash('sha256').update(text, 'utf8').digest('hex');
 
 describe('[#21007] JSON_COLUMN_INCOMPATIBLE_OPERATORS', () => {
-  it('holds exactly the spellings driver-sql refused before the move', () => {
+  it('holds exactly the spellings driver-sql refused before the move, and the text family [#21009] added', () => {
     expect([...JSON_COLUMN_INCOMPATIBLE_OPERATORS].sort()).toEqual([
-      '!=', '$between', '$eq', '$gt', '$gte', '$in', '$lt', '$lte', '$ne', '$nin',
+      '!=', '$between', '$endsWith', '$eq', '$gt', '$gte', '$icontains', '$ilike', '$in', '$like',
+      '$lt', '$lte', '$ne', '$nin', '$startsWith',
       '<', '<=', '<>', '=', '==', '>', '>=',
       'between', 'in', 'nin', 'not_in', 'notin',
     ]);
   });
 
-  it('leaves out the membership spelling, the rest of the text family and the null predicates', () => {
-    for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith', '$icontains', '$null', '$exists', '$empty']) {
+  it('[#21009] holds every text operator except the membership pair', () => {
+    for (const op of ['$startsWith', '$endsWith', '$icontains', '$like', '$ilike']) {
+      expect(JSON_COLUMN_INCOMPATIBLE_OPERATORS.has(op), op).toBe(true);
+    }
+  });
+
+  it('leaves out the membership pair and the null predicates', () => {
+    for (const op of ['$contains', '$notContains', '$null', '$exists', '$empty']) {
       expect(JSON_COLUMN_INCOMPATIBLE_OPERATORS.has(op), op).toBe(false);
     }
   });
@@ -56,6 +64,14 @@ describe('[#21007] jsonColumnOperatorRefusalText — byte for byte what driver-s
     expect({ sha: sha256(text.diagnostic), length: text.diagnostic.length }).toEqual(diagnostic);
   });
 
+  it('[#21009] a text operator reads the very message the equality family reads, and its diagnostic names it', () => {
+    for (const op of ['$startsWith', '$endsWith', '$icontains', '$like', '$ilike']) {
+      const text = jsonColumnOperatorRefusalText('members', op, false);
+      expect({ sha: sha256(text.message), length: text.message.length }, op).toEqual(MESSAGE);
+      expect(text.diagnostic, op).toContain(`Operator "${op}" on field "members" WAS NOT APPLIED`);
+    }
+  });
+
   it('the message names neither the field nor the operator, and prescribes $contains and an $or of it', () => {
     const { message, diagnostic } = jsonColumnOperatorRefusalText('secret_col', '$nin', false);
     expect(message).not.toContain('secret_col');

packages/core/src/utils/json-column-operator-refusal.ts

@@ -5,6 +5,7 @@
  * field stored as a JSON column — a `multiple: true` field, an inherently
  * multi-value option type (`tags`, `multiselect`, `checkboxes`) or a
  * structured-JSON type (`json`, `address`, …): the operator set and the words.
+ * [#21009] The text operators other than the membership pair get it too.
  *
  * ## Two faces, one rule
  *
@@ -30,15 +31,18 @@
  * `$contains` is the membership spelling on such a column (`FILTER_OPERATORS`'
  * `$contains` docblock, `@objectstack/spec`), and it is what the refusal
  * prescribes — `$contains` for one member, an `$or` of `$contains` for any-of.
- * That is why it is ABSENT from the set below, with the rest of the text family
- * and the null predicates.
+ * That is why it is ABSENT from the set below, with its complement
+ * `$notContains` and the null predicates. [#21009] The remainder of the text
+ * family is IN the set: it has no membership reading, so it matched the
+ * serialization.
  */
 
 /**
  * [#7398] Operators whose SQL lowering compares a column's STORED SCALAR to a
  * value — every spelling either of `driver-sql`'s two comparison emitters
  * answers (`applyFilterCondition`'s plain-column switch and
- * `applyNormalizedComparison`'s normalised arms).
+ * `applyNormalizedComparison`'s normalised arms). [#21009] Or MATCHES that
+ * stored scalar as text: the text family other than the membership pair.
  *
  * The bare infix forms are here for the same reason they are in `driver-sql`'s
  * `SCALAR_COMPARAND_OPERATORS`: `applyNormalizedComparison` really does
@@ -53,15 +57,40 @@
  * the halves and compiling the compound would be the same wrong answer at one
  * more spelling.
  *
- * Deliberately ABSENT, and this is the load-bearing half of the set: the `LIKE`
- * family (`$contains`, `$notContains`, `$startsWith`, `$endsWith`,
- * `$icontains`) and the null predicates (`$null`, `$exists`). `$contains` is
- * the ONLY working membership spelling on a JSON-array column and downstream
- * code depends on it (#7398's own tables), while `IS NULL` asks about the
- * column's presence, which is a well-formed question whatever the column holds.
+ * Deliberately ABSENT, and this is the load-bearing half of the set: the
+ * membership pair (`$contains`, `$notContains`) and the null predicates
+ * (`$null`, `$exists`, `$empty`). `$contains` is the ONLY working membership
+ * spelling on a JSON-array column and downstream code depends on it (#7398's
+ * own tables) — `driver-sql` compiles it as a real per-dialect membership test,
+ * and `$notContains` as its exact complement — while `IS NULL` asks
+ * about the column's presence, which is a well-formed question whatever the
+ * column holds.
  *
  * [#21007] Moved here from `driver-sql`, unchanged, so the per-aggregation
  * `filter` refuses exactly the operators `where` refuses.
+ *
+ * [#21009] The remainder of the text family joined the set: `$startsWith`,
+ * `$endsWith`, `$icontains`, and the staged pattern pair `$like` / `$ilike`
+ * that `driver-sql` answers ahead of `FILTER_OPERATORS`. None has a membership
+ * reading, so on a JSON column each matched the SERIALIZATION as text, and the
+ * answers were wrong the same three ways the equality family's were. Measured
+ * through `POST /api/v1/data/:object/query` on a multi-value lookup holding
+ * `["u1","u2"]`:
+ *
+ * - SQLite: `$startsWith: '['` and `$endsWith: ']'` matched EVERY row with a
+ *   value, while `$startsWith: 'u1'` matched none; `$icontains: 'U1'` matched
+ *   the row holding only `u10`, and `$icontains: '","'` matched every row with
+ *   two members.
+ * - PostgreSQL: a `json` column has no `LIKE` operator, so all five failed at
+ *   query time — a `500` `DATABASE_ERROR` for a filter the caller can fix.
+ * - The per-aggregation `filter` counted `0` for `$startsWith`, `$endsWith` and
+ *   `$icontains` (it already refuses the staged pair as unsupported).
+ *
+ * Each now gets this set's `400`. ⛔ No membership reading is invented for a
+ * prefix, suffix or case-folded test: the prescription stays `$contains`. The
+ * infix spellings `like` / `ilike` are not members because no emitter answers
+ * them as operators — the normalised arms carry no text family, and the
+ * operator switch refuses them as unsupported.
  */
 export const JSON_COLUMN_INCOMPATIBLE_OPERATORS: ReadonlySet<string> = new Set([
   '$eq', '=', '==',
@@ -70,6 +99,7 @@ export const JSON_COLUMN_INCOMPATIBLE_OPERATORS: ReadonlySet<string> = new Set([
   '$in', 'in',
   '$nin', 'nin', 'not_in', 'notin',
   '$between', 'between',
+  '$startsWith', '$endsWith', '$icontains', '$like', '$ilike',
 ]);
 
 /** The two texts of one JSON-column refusal — see {@link jsonColumnOperatorRefusalText}. */
@@ -116,6 +146,11 @@ export interface JsonColumnOperatorRefusalText {
  * [#21007] Moved here from `driver-sql`'s `jsonColumnOperatorError`, byte for
  * byte, so `where` and the per-aggregation `filter` print one sentence. The
  * caller builds the error: this returns only the text.
+ *
+ * [#21009] The text family that joined the set reads these same words,
+ * unchanged. Its prescription holds as written — membership is `$contains` —
+ * while the "scalar comparison" wording and the two directions the closing
+ * sentence names are the equality family's.
  */
 export function jsonColumnOperatorRefusalText(
   field: string,

packages/drivers/driver-sql/src/sql-driver-17343-multi-valued-boolean-membership.test.ts

@@ -270,13 +270,29 @@ describe('[#17343] the per-dialect construct, compiled — the registerExternalO
       const d = typed(config);
       for (const field of ['picks', 'refs']) {
         for (const op of POSITIVE_OPERATORS) {
+          // [#21009] Only `$contains` — the membership spelling — compiles on a
+          // JSON column now; the rest of the family is REFUSED there (`400`),
+          // ahead of both this card's declared-type gate and the emitter. Either
+          // way the gate this file is about does not fire: a refusal is not the
+          // `1 = 0` constant, and the SHAPE per operator is owned by #17590's
+          // and #21009's own files.
+          if (op !== '$contains') {
+            let refusal: (Error & { code?: string; status?: number }) | undefined;
+            try {
+              d.compileWhere({ [field]: { [op]: 'x' } } as FilterCondition);
+            } catch (e) {
+              refusal = e as Error & { code?: string; status?: number };
+            }
+            expect(refusal?.code, `${op} over ${field}`).toBe('INVALID_FILTER');
+            expect(refusal?.status, `${op} over ${field}`).toBe(400);
+            continue;
+          }
           const sql = d.compileWhere({ [field]: { [op]: 'x' } } as FilterCondition);
           expect(sql, `${op} over ${field}`).not.toMatch(/1 = 0|1 = 1/);
-          // [#17590] `$contains` compiles the MEMBERSHIP construct now and the
-          // rest of the family still compiles a pattern match. What this card
-          // is about is neither shape — it is that the declared-type gate does
-          // not fire — so this row asks for "a real predicate over the column",
-          // and the SHAPE per operator is owned by #17590's own file.
+          // [#17590] `$contains` compiles the MEMBERSHIP construct. What this
+          // card is about is not the shape — it is that the declared-type gate
+          // does not fire — so this row asks for "a real predicate over the
+          // column".
           expect(sql, `${op} over ${field}`).toMatch(REAL_PREDICATE[label]!);
         }
       }

packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts

@@ -306,18 +306,31 @@ describe('[#17590] the per-dialect membership construct, compiled', () => {
     });
 
     /**
-     * The other text operators are NOT membership spellings and this card does
-     * not rule on them — they keep the text emitter's lowering (on SQLite, since
-     * #20024, `instr(` for `$icontains` and `substr(CAST(` for `$endsWith`).
-     * Pinned so a later widening is a deliberate edit here rather than a silent
-     * side effect.
+     * The other text operators are NOT membership spellings and this card did
+     * not rule on them — it pinned them unmoved "so a later widening is a
+     * deliberate edit here rather than a silent side effect". [#21009] is that
+     * edit: on a JSON column they matched the serialization (SQLite) or failed at
+     * query time (PostgreSQL), so they are now REFUSED there, in the equality
+     * family's `400`, before either emitter is reached — and no membership
+     * reading is invented for them. On the scalar string column they keep the
+     * text emitter's lowering (on SQLite, since #20024, `instr(` for
+     * `$icontains` and `substr(CAST(` for `$endsWith`).
      */
-    it(`${label}: the rest of the text family is UNMOVED on a JSON column`, () => {
+    it(`${label}: [#21009] the rest of the text family is REFUSED on a JSON column, and unmoved on a scalar one`, () => {
       const d = new CompilerProbeDriver(config).declare();
       for (const op of ['$startsWith', '$endsWith', '$icontains', '$like', '$ilike']) {
-        const sql = d.compileWhere({ tags_: { [op]: 'red' } } as FilterCondition);
-        expect(sql, `${op} on ${label}`).toMatch(/LIKE|GLOB|instr\(|substr\(CAST\(/);
-        expect(sql, `${op} on ${label}`).not.toMatch(CONSTRUCT[label]!);
+        let refusal: (Error & { code?: string; status?: number }) | undefined;
+        try {
+          d.compileWhere({ tags_: { [op]: 'red' } } as FilterCondition);
+        } catch (e) {
+          refusal = e as Error & { code?: string; status?: number };
+        }
+        expect(refusal?.code, `${op} on ${label}`).toBe('INVALID_FILTER');
+        expect(refusal?.status, `${op} on ${label}`).toBe(400);
+        expect(refusal?.message, `${op} on ${label}`).toContain('WAS NOT APPLIED');
+        const sql = d.compileWhere({ label: { [op]: 'red' } } as FilterCondition);
+        expect(sql, `${op} on the scalar label, ${label}`).toMatch(/LIKE|GLOB|instr\(|substr\(CAST\(/);
+        expect(sql, `${op} on the scalar label, ${label}`).not.toMatch(CONSTRUCT[label]!);
       }
     });
   }