fix(sql): date arithmetic follows the SQL engines; type checks wait for the column types

[fupelaqu]

What

Date arithmetic with + and - follows the SQL engines (PostgreSQL, DuckDB, Oracle, Snowflake, BigQuery). It behaves the same in core and in every place a script runs: row SELECT, WHERE, per-group calculations, HAVING, computed columns (SCRIPT AS), CREATE TABLE … AS SELECT, and the materialized-view calculation channel. Type errors are checked once the column types are known, never at parse.

Expression	Result	Type
DATE - DATE	calendar days between the two dates	BIGINT
TIMESTAMP - TIMESTAMP, or a DATE and a TIMESTAMP	fractional days (1.25 = a day and 6 hours)	DOUBLE
DATE ± n, n a whole number	the date n days later / earlier	DATE
DATE ± n, n fractional / TIMESTAMP ± n	n days later / earlier, the fraction as a time of day	TIMESTAMP
date + date, a date under * / %, n - date	refused by name, as the engines do	—

The engine evidence:

• RUN: PostgreSQL 16, DuckDB 1.5.5, Trino 483, MySQL 8.4 and MariaDB 11.5.
• From the documentation: Oracle, SQL Server, Snowflake and BigQuery.
• No engine returns milliseconds. MySQL / MariaDB subtract YYYYMMDD numbers (Jan 31 → Feb 1 = 70), so they are no precedent.

Also in this PR:

• Type checks wait for the column types. Every type rule moved from parse time to validateResolved(), after the mapping is read. With no type to read, nothing is refused and Elasticsearch answers.
  • One comparison rule for =, <, >, BETWEEN, IN, simple CASE, NULLIF and CASE conditions, wherever they appear.
    • A quoted literal that spells a number or a date is read as one.
    • DATE and TIMESTAMP compare on their declared types.
  • Checked against PostgreSQL itself: of the 1,070 statements the move first refused, PostgreSQL accepts 48 and we accept them with PostgreSQL's values (3 are the follow-up below). The 1,022 that PostgreSQL refuses stay refused.
  • Now accepted, though refused at parse before: WHERE CURRENT_DATE - d > 30, WHERE d + 7 > CURRENT_DATE, WHERE COALESCE(n, 0) = '3'.
• A date field elasticsql did not declare is a TIMESTAMP everywhere, DESCRIBE included. This covers indices created by a bulk load, Logstash or an application, judged per field. The schema is derived from the Elasticsearch mapping, whose date holds an instant, as Elasticsearch SQL also reports.
  • A column declared DATE through elasticsql stays DATE.
  • An ALTER TABLE keeps an undeclared date a TIMESTAMP.
• Casts over arithmetic apply. (n + x)::BIGINT answered 3.25; it is now 3. A NULL operand gives NULL.
• GREATEST / LEAST compare dates, in every venue, skipping NULLs.
• Index patterns and comma lists read their indices' mappings, merged field by field (subqueries included). A field mapped differently stays untyped. DDL refreshes the cached types.
• Now run:
  • the clock per group (CURRENT_DATE - MAX(d));
  • COUNT(d) - 1 per group;
  • DATE_TRUNC(ts, DAY) = '2024-01-31';
  • NULLIF with a date literal of either shape (NULLIF(d, '2025-01-01 10:00:00')), in queries and computed columns;
  • CASE b WHEN '1' / WHEN TRUE over a boolean on ES 6.8, and CASE WHEN <boolean column> on a document without the boolean (the ELSE branch).
• ES 6.8: a date column compared with a computed date now runs. The normalisation is rendered for 6.8 only, through a per-major PainlessTarget in each client module, so the ES 7 / 8 / 9 requests for those shapes are byte-identical to before.
• Scalar expressions answer what PostgreSQL answers. Closes Arithmetic over a CASE expression returns the ELSE branch plus one, always #380 and Closes total / NULLIF(cnt, 0) fails the whole query on exactly the rows it protects #392.
  • A CASE or a NULLIF used as an operand is one value.
    • CASE WHEN n > 0 THEN n ELSE 0 END + 1 adds 1 on every branch. It added 1 to the ELSE branch only.
    • x / NULLIF(n, 0), NULLIF(n, 0) + 1 and a % NULLIF(b, 0) give NULL where the NULLIF nulls, in a query and in a computed column. The query failed, and the computed column was silently absent.
  • GREATEST / LEAST answer their arguments' common type on every row. GREATEST(n, 2) over an INT answers 3, where it answered 3.0. With a DOUBLE argument, every row is a double; main answered an integer on the rows where the DOUBLE was missing.
  • ABS is typed by its argument, as abs(integer) → integer. ABS(n) + 1 runs; it failed to compile. ABS(n) = '1.5' is refused by name, as on main.
  • A simple CASE answers its THEN / ELSE values in their own type: CASE s WHEN 'a' THEN 1 ELSE 0 END gives the integers 1 / 0, where it gave the strings "1" / "0".
  • NULLIF('2024-01-31', ts) is typed TIMESTAMP, the literal's reading, and CREATE TABLE … AS SELECT declares it.
  • A number cast to DATE, such as (d - d2)::DATE, is refused by name, as PostgreSQL refuses it. A number cast to TIMESTAMP keeps its epoch-millisecond conversion.
• A CREATE TABLE … AS SELECT whose SELECT is refused leaves no table. Main created the empty table first.

Measured

Check	Result
Generated population. Operators × operand kinds: columns, aggregates, literals, CASE / COALESCE / NULLIF / GREATEST / LEAST, typed NULLs, numeric strings, undeclared fields	5,120 query expressions + 2,880 computed columns, 0 mismatches; no type refusal at parse
DateArithmeticSpec, with runners for es6 rest/jest, es7, es8 and es9, plus GroupByCompleteness, DateFunctionExecution, GatewayApi, ReplGateway and PredicateFunctionResult	ES 8.18.3: 303/303
ES 6.8.23: DateArithmeticSpec + TemporalLiteral, PredicateFunctionResult, DateFunctionExecution (REST); DateArithmeticSpec (Jest)	79/79 REST; Jest 21/21 before the scalar-expression tests were added (CI runs them). 5 pre-existing esMajor >= 7 cancels, unchanged
PostgreSQL classification of the statements newly refused by the type-check move	PostgreSQL accepts 48: 45 accepted with PG's values, 3 = follow-up. PostgreSQL refuses 1,022: 1,022 refused
Before / after census, ES 8.18.3 and 6.8.23 (229 statements against main, then 322 against each fix round)	0 statements that ran now fail. Every changed value and new refusal is listed below
ES 7 / 8 / 9 bodies for the 6.8 comparison shapes	byte-identical to main (24/24)
Release gate against main: 4,771 statements, ES 8.18.3 and 6.8.23, PostgreSQL 16 as the oracle	All 828 that main answered correctly still do: 793 byte-identical, 34 with the same values in the type PostgreSQL gives (GREATEST(n, 2) was 3.0), and 1 that reads CURRENT_DATE (PostgreSQL's fixture was recorded the day before). Where PostgreSQL refuses the statement: 480 new refusals, and 8 changed answers (d + '1', and a text CASE concatenated on every branch). All from the rulings above
Scalar expressions (CASE / NULLIF operands, GREATEST / LEAST, ABS, simple CASE, NULLIF literal first, number → DATE), ES 8.18.3 and 6.8.23	129 statements in committed tests, with PostgreSQL's values; the 45 over GREATEST / LEAST also check the Java class on every row. They fail on the previous head. No column mixes number classes over 3,592 answered statements (main mixes Integer and Double in 5 GREATEST shapes)
Suites	sql 1821, core 1149, bridges es6 260 / es7 267 / es8 267 / es9 267, macrosTests 30; + compile 2.12.20 / 2.13.16; lint clean

Cost: 5 interleaved rounds against main, with a new committed probe, ResolutionCostProbe (parse → schema attach → validateResolved → emission).

• Up to the second fix round:

  • total per statement ×0.999 [0.961–1.060];
  • type checks +0.2–3.4 µs per statement;
  • request generation +0.3–1.0 µs on about 20 shapes whose request changed for correctness;
  • ES 8 execution of the 6.8-fixed comparison shapes ×0.99–1.00 (identical bodies);
  • reading a pattern's mapping: about 1 ms cold, about 7 µs cached.

• The last fix round reads each comparison literal once, and checks a literal's shape before any date parse. Request generation is back to its cost before the comparison rule (WHERE heavy 41.5 → 45.8 µs, after 60.7 µs at the first commit). The remaining type-check cost, up to +1.8 µs on about 20 statements, is the rule doing its work once. The population total is unchanged (566 vs 571 ms). The requests are byte-identical to the first commit.

• The scalar-expression fixes: population total ×0.96 to ×1.02 over two 5-round runs, every range overlapping; no statement slower beyond noise.

Not run locally, left to CI: ES 7.17 and 9.0 legs, and the Jest leg since the scalar-expression tests.

⚠️ Breaking changes against 0.23.x (release notes)

• DATE - DATE is days, not milliseconds. MAX(d) - MIN(d) answered 2592000000 for 30 days and now answers 30. Remove a division by 86400000; write a HAVING threshold in days. Per group, a number added to a date is days (MAX(d) + 1.5).

• DESCRIBE of an index elasticsql did not create reports its date fields as TIMESTAMP, not DATE. So does CREATE TABLE … AS SELECT from it. Their arithmetic keeps the time of day: ts2 - ts = 0.948, and per group MAX(ts2) - MIN(ts) = 1.385 days (it was 1.197E8 ms). Measured on such an index, WHERE ts = '2024-01-31', > and BETWEEN select the same rows as before.

• New refusals of statements that ran, all answering wrong or meaningless before:

  • MAX(d) / 1000, MAX(d) + MIN(d), *, /, % (epoch-millisecond arithmetic);
  • d + 'x' (string concatenation);
  • n + s (INT + KEYWORD, concatenation 3a);
  • WHERE id = n (KEYWORD = INT, no rows);
  • a quoted number that is not whole, beside an integer: COALESCE(n, 0) = '1.5', n + '1.5', NULLIF(n, '1.0'), as PostgreSQL refuses them;
  • 26 CASE conditions comparing incompatible types (PostgreSQL refuses them all; on main 9 answered 0 for every row and 17 failed in Elasticsearch).

  Remedies: CONCAT(CAST(n AS VARCHAR), s), WHERE id = CAST(n AS VARCHAR).

• Values that change because they were wrong:

  • an operator after a CASE applies to every branch (CASE WHEN n > 0 THEN n ELSE 0 END + 1 for n = 3: 3 → 4);
  • GREATEST / LEAST over whole numbers answer whole numbers (3.0 → 3), and over a DOUBLE, a double on every row;
  • a simple CASE answers its THEN / ELSE values in their own type ("0" → 0);
  • casts over arithmetic apply ((n + x)::BIGINT 3.25 → 3);
  • d + '1' is the next day (it concatenated). This is elasticsql's own rule: PostgreSQL 16 refuses d + '1';
  • AVG(n) + 1 per group is 1.5 (it was truncated to 1.0);
  • CAST(n AS TIMESTAMP) over a numeric column is an instant (it returned the number);
  • a computed column over a DATE holding a date-time string stores a value (it stored NULL);
  • a temporal compared with a date string was compared as text, with wrong rows.

• API (binary-incompatible, downstream rebuild):

  • PainlessContext gains target: PainlessTarget, and the bridge conversions take an implicit PainlessTarget;
  • each client module generates a BuildInfo object (RestClientBuildInfo, JestClientBuildInfo, JavaClientBuildInfo: elasticVersion, elasticMajor) from its build's Elasticsearch version; the serializers' ElasticsearchMajor (new on JavaClientSearchBodySerializer) and every painlessTarget read it, so no client source writes a major by hand;
  • object ArithmeticExpression no longer extends AbstractFunction4;
  • new public members, including sql.ComparisonRule, SingleSearch.typeErrors, Validation.typeError, convert.typeErrorsOf and Conversion.typeError;
  • Case.baseType is the type of the THEN / ELSE values (it included the CASE <expr> operand); ABS's reportedType is its argument's type; NullIf.reportedType reads a literal first argument;
  • validate() checks syntax and structure only: a caller that executes a parsed statement outside SearchApi must call validateResolved() after attaching the schema.

Follow-ups (not in this PR)

Measured on main and on this PR's head, against PostgreSQL 16. Each is filed:

• Extensions (SOFTNETWORK-APP/softclient4es-extensions#71): views must call validateResolved() after attaching the schema, and store a DATE / TIMESTAMP per-group calculation. A transform writes decimals, which a date column refuses.
• Arrow (SOFTNETWORK-APP/softclient4es-arrow#198): set operations must call the branch type check, because DuckDB casts implicitly.
• A per-group calculation with no GROUP BY (A calculation over aggregates without GROUP BY fails in Elasticsearch: SELECT MAX(n) - MIN(n) FROM t #413; SELECT MAX(n) - MIN(n) FROM t) fails in Elasticsearch on main for every shape. Its own PR, before the release.
• Wrong results with no error (Wrong results with no error: ES 8 answers 0.0 for MIN / MAX / AVG of a column a whole group lacks, CREATE TABLE … AS SELECT … GROUP BY stores one empty document per source document, and values come back in the class Elasticsearch's JSON gives them (a date per group as epoch milliseconds) #414):
  • on ES 8, MIN / MAX / AVG of a column absent from every document of a group answer 0.0;
  • CREATE TABLE … AS SELECT … GROUP BY stores one empty document per source document;
  • CREATE TABLE … AS SELECT DATEDIFF(…) fails at index creation (scaling_factor);
  • values come back in the class Elasticsearch's JSON gives them: per group, a date (MIN(d), MAX(d) + 1) as a number of epoch milliseconds, and a BIGINT that fits 32 bits as an Integer.
• Comparison and type rules (Comparison and type rules disagree with PostgreSQL: quoted fractions beside an integer column run, a DATE column keeps a row for a timestamp literal, mixed DATE/TIMESTAMP BETWEEN and text/boolean GREATEST are refused, text + text concatenates #415):
  • WHERE n = '1.5' over a bare integer column, and its IN / BETWEEN / > forms, go to Elasticsearch as on main, where PostgreSQL refuses them; the integer ranges per type are not checked either ('99999999999' beside an INT);
  • in WHERE, a bare DATE column compared with a timestamp literal compares the instant, where PostgreSQL compares the date part;
  • over a bare DATE / TIMESTAMP column, a BETWEEN with mixed date and timestamp bounds is refused, where PostgreSQL accepts it;
  • GREATEST / LEAST over text or booleans are refused, where PostgreSQL accepts them;
  • text + text answers a concatenation (UPPER(s) + 'a'; + NULL gives the text 'null'), where PostgreSQL refuses it;
  • refusal messages name ABS(n) as NUMERIC in arithmetic and as INT in a comparison;
  • a whole-number literal is typed BIGINT, where PostgreSQL types it integer when it fits.
• Scalar expressions (Scalar expressions: ROUND(x, 1) returns an integer, function results fail to compile under an operator or in GREATEST, a simple CASE matches NULL or fails over an expression, and ORDER BY GREATEST(…) fails #416):
  • ROUND(x, 1) returns a whole number, even in a plain SELECT;
  • ROUND(x, 1) - 1, ABS(x) + 1 over a DOUBLE, GREATEST(ABS(n), 1) (also over SQRT or a CASE) and GREATEST(n, NULL) fail to compile on every document;
  • a BIGINT literal past the INT range fails to compile (n + 3000000000), except in GREATEST / LEAST;
  • ORDER BY GREATEST(…) sorts by the alias as a field and fails;
  • CASE YEAR(d) WHEN 2024 … and CASE CAST(n AS VARCHAR) WHEN '3' … fail in Elasticsearch;
  • a simple CASE x WHEN x matches a NULL operand;
  • a CASE or a function over an aggregate is refused per group;
  • DATE_TRUNC(d, HOUR) over a DATE fails when compared with a DATE.
• ES 6.8 (comment on ES 6.8: a raw date doc-value reaches Painless un-normalised — DATE_FORMAT/DATETIME_FORMAT in every venue, and any temporal expression compared with a date column #371): CAST(TRUE AS INT) renders a constant conditional that 6.8 refuses to compile; DATE_FORMAT / DATETIME_FORMAT still fail.
• Parse gaps (comment on An arithmetic expression is rejected as the operand of a scalar function, a CAST or a CASE branch #391): IN with CAST elements, INTERVAL over an aggregate, CAST(n + x AS DOUBLE).
• Web docs for the new rules (SOFTNETWORK-APP/softclient4es-web#73).

🤖 Generated with Claude Code