fix(sql): set operations take DuckDB's types and answer one class per column; aggregates without GROUP BY answer one row

[fupelaqu]

What

Set operations type their columns as DuckDB does. DuckDB is the engine that runs UNION, INTERSECT, EXCEPT and every UNION ALL the relational engine (arrow) plans, so core's own UNION ALL now answers the same types, the same values and the same refusals.

One rule types every depth alike: a column, each field of an object, each element of a list. Each node of DuckDB's operator tree is typed over all of its branches at once:

• a run of one union kind is one node, in any order, even parenthesised;
• INTERSECT / EXCEPT are binary.

Branches	Type
numbers, BOOLEAN	the widest (TINYINT < … < DOUBLE)
DATE and TIMESTAMP	TIMESTAMP
text with any other type	VARCHAR, each value spelled as DuckDB spells it (250000000000.0, 1e-05, 2024-01-31 10:30:00)
VARBINARY with text only	VARBINARY (DuckDB's BLOB)
NULL literal	the other branches' type
DATE / TIMESTAMP with a number or BOOLEAN, no text and no VARBINARY branch	refused by name (the relational engine fails on it at runtime)
other pairs DuckDB gives no type (VARBINARY or TIME with a number or a date)	VARCHAR
a list, an object or a point beside another family	refused by name
lists	a list of the element type these rules give
objects and points	DuckDB's merged STRUCT: every field in order of first appearance, each typed by these rules; points stay GEO_POINT, normalised to {lat, lon}

One class per position. A position whose branches' types differ converts every branch's value to its type, at every depth. JDBC and Arrow, which read the type from the first value, then report it right.

• A value is first read as its declared type, as Elasticsearch indexed it. A number stored as text ("5") is a number.
• A date is read with its field's mapping format, as the cluster's Elasticsearch major reads it. This was measured letter by letter and name by name on 6.8.23, 7.17.29 and 8.18.3.
• A date whose reading would depend on the client's JDK or time-zone data fails the statement, naming the field and the format, never a wrong date. That covers zone names and region ids, and forms that resolve differently across JDKs.

What answers differently from 0.23.0:

• n UNION ALL x answers a DOUBLE column, d UNION ALL ts a TIMESTAMP column, and id UNION ALL n a VARCHAR column, one class each. 0.23.0 mixed the classes.
• Refused by name: a date beside a number (d UNION ALL n). This now also applies inside objects and lists, where 0.23.0 answered each object as stored. That is a deliberate change: DuckDB fails on it, Unimplemented type for cast (INTEGER -> DATE).
• Refused by name: a list or an object beside another family.

Also in this PR:

• A calculation over aggregates without GROUP BY answers one row (MAX(n) - MIN(n), COUNT(*) * 2, MAX(d) + 1). Elasticsearch refused these searches before.
  • COUNT(*) reads the bucket count.
  • Over no matching document, counts and sums are 0 on 7.17 and later. On 6.8, COUNT(x) stays NULL: 6.8 refuses the keep_values gap policy.
  • HAVING COUNT(*) = 0 keeps its row.
  • Closes A calculation over aggregates without GROUP BY fails in Elasticsearch: SELECT MAX(n) - MIN(n) FROM t #413.
• A DATE / TIMESTAMP cast over a per-group date calculation applies: (MAX(d) + 1)::DATE is the day. It failed in Elasticsearch before.
• For the relational engine: private[elastic] MultiSearch.columnTypes(requests, operators) and branchTypes(requests, operators) give the type and the verdict over the operator tree (arrow will call them).
• Licences: DuckDB's text for doubles and temporals comes from a port of {fmt} 6.1.2 and code translated from DuckDB 1.5.5, each with its MIT notice. The notices are in THIRD-PARTY-NOTICES, in META-INF/THIRD-PARTY-NOTICES of the core jar, and in the CLI bundles.

Measured

• Verdicts against the relational engine (arrow, statement by statement, on Elasticsearch 8.18.3):

  • scalar set: 698 / 702 agree; the other 4 are Wrong results with no error: ES 8 answers 0.0 for MIN / MAX / AVG of a column a whole group lacks, CREATE TABLE … AS SELECT … GROUP BY stores one empty document per source document, and values come back in the class Elasticsearch's JSON gives them (a date per group as epoch milliseconds) #414;
  • nested and binary set: 530 / 530 with arrays typed (482 / 530 while ARRAY<T> reads as ANY, A column declared ARRAY<INT> or ARRAY<KEYWORD> reads back as ANY, so DESCRIBE and the type checks lose its type #419);
  • points beside objects: 110 / 110.

• Against DuckDB 1.5.5: a random 330 statements across 6 depths and every operator all match the rule. Wherever DuckDB answers, the type is DuckDB's, apart from degenerate shapes where a branch has no non-NULL value.

• Against main, on Elasticsearch 8.18.3:

  • set-operation sets (361 routes): 0 answered → refused, 152 refused → answered;
  • nested set (448 routes): answered → refused only for the 96 date-beside-number route-cases above;
  • general set (all statement kinds): 0 answers that were correct are now wrong.

• Dates: corpora of every pattern letter, width and built-in name.

  • Elasticsearch 8.18.3: 32,597 values read exactly, 3,512 fail loudly.
  • 7.17.29: 41,714 read exactly, 5,351 fail loudly.
  • 6.8.23: 45,760 read exactly, 5,755 fail loudly.
  • 0 wrong reads. The outcome is identical on client JDKs 8, 11, 17 and 21.

• Tests on the final tree:

  • sql/test 1838 / 0;
  • core/test 1197 / 0 on JDK 11 and on JDK 17;
  • the set-operation specs on Elasticsearch 8.18.3 and 6.8.23.

  The A calculation over aggregates without GROUP BY fails in Elasticsearch: SELECT MAX(n) - MIN(n) FROM t #413 and cast specs ran on 8.18.3, 7.17.29 and 6.8.23 earlier in this PR. CI runs every major.

API changes (rebuild downstream modules)

• ElasticConversion.jsonToRows takes a new last parameter, converters: binary incompatible with code compiled against 0.23.0.
• LegProjection gains converters (four fields to five).
• New:
  • MultiSearch.columnTypes, branchTypes, SetOperationColumn, SetOperationType, StructField;
  • SingleSearch.wholeTableCalculation;
  • ElasticAggregation.wholeTableCalculationAggregation;
  • an overload of ElasticConversion.rowProjector.

Follow-ups (not in this PR)

• Wrong results with no error: ES 8 answers 0.0 for MIN / MAX / AVG of a column a whole group lacks, CREATE TABLE … AS SELECT … GROUP BY stores one empty document per source document, and values come back in the class Elasticsearch's JSON gives them (a date per group as epoch milliseconds) #414: MIN(d) beside a timestamp (per-group dates as doubles).
• Scalar expressions: ROUND(x, 1) returns an integer, function results fail to compile under an operator or in GREATEST, a simple CASE matches NULL or fails over an expression, and ORDER BY GREATEST(…) fails #416: ROUND(x, 1) loses its scale.
• A computed column comes back as a one-element list, and a projection of computed columns only returns every source field: SELECT 2 AS k FROM t #418: computed columns as one-element arrays. Includes the class mix of computed numeric and date branches in a set operation.
• A column declared ARRAY<INT> or ARRAY<KEYWORD> reads back as ANY, so DESCRIBE and the type checks lose its type #419: ARRAY<T> columns read as ANY; derived-table columns untyped in the set-operation check.
• Binary fields answer their base64 text, not their bytes #421: a binary field answers its base64 text, not its bytes (core, arrow and JDBC together).

🤖 Generated with Claude Code