Skip to content

Native null short-circuit evaluates shared arguments eagerly when subexpression elimination is off #6972

Closed

Description

@dwsmith1983

Describe the bug

QueryPlanSerde.canShortCircuitNulls keeps every argument of an expression eager when a later argument holds a subexpression that the operator's expressions share, so that an error Spark raises from the hoisted subexpression is raised in Comet too. It treats them as shared whatever spark.sql.subexpressionElimination.enabled says. With subexpression elimination off, Spark does not hoist anything, and its own null short-circuit skips the later argument on a row whose first argument is NULL. Comet evaluates it eagerly on that row, so it raises where Spark returns rows.

Found while working on #6970 (Found in #5867, #5867 (comment)).

Steps to reproduce

On main (7d8d9b1), as a SQL file test:

-- Config: spark.sql.ansi.enabled=true
-- Config: spark.sql.subexpressionElimination.enabled=false

statement
CREATE TABLE t(m1 map<int, int>, m2 map<int, int>, i int) USING parquet

statement
INSERT INTO t VALUES
  (CAST(NULL AS MAP<INT, INT>), CAST(NULL AS MAP<INT, INT>), 0),
  (map(1, 5), map(1, 6), 1)

query
SELECT m1[1 div i], m2[1 div i] FROM t

Spark returns NULL, NULL and 5, 6. Comet runs the projection natively and fails with DIVIDE_BY_ZERO.

Expected behavior

The same rows as Spark. canShortCircuitNulls can return true when spark.sql.subexpressionElimination.enabled is off, since Spark's projection and aggregate only hoist shared subexpressions when it is on. Every caller of withNullShortCircuit (the map lookup and the array functions that use it) picks that up.

Additional context

With the setting on, the same function is also broader than Spark in one way: a later argument counts as shared when any piece of it is shared, and the whole argument is then evaluated for every row, while Spark hoists only the shared piece. For example map_from_arrays(CASE WHEN k = 0 THEN NULL ELSE array(1) END, CASE WHEN k = 0 THEN array(1 / k, 2) ELSE array(1 / k) END) shares only k = 0, which cannot fail. Exact handling would need native support for evaluating only the shared pieces, so that part is noted here rather than proposed.

Activity

  1. dwsmith1983 commented on Oct 14, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions