Skip to content

CometIcebergNativeSuite single-split sort-merge join DPP test fails on Spark 3.5 with AQE #6901

Closed

Description

@dwsmith1983

Describe the bug

DPP - single-split sort-merge join (AQE=true, pruned=a) and (AQE=true, pruned=b) in CometIcebergNativeSuite, added in #6814, fail on the Spark 3.5 profile:

List() was empty Expected AQE to replace the pruned join with an empty relation

Found in #5867: https://github.com/apache/datafusion-comet/actions/runs/38276011652/job/114910128309. #6814's own Spark 3.5 [scans] job failed on the same two tests (https://github.com/apache/datafusion-comet/actions/runs/38066634294/job/114798528903), and they fail the same way on main (38f394e). Spark 3.4, 4.0, 4.1 and 4.2 pass. The pull request tier runs only Spark 4.1, so main's CI doesn't run this profile.

The test asserts a CometEmptyRelationExec whenever isSpark35Plus && aqeEnabled && scans.isEmpty. On Spark 3.5, AQEPropagateEmptyRelation replaces the pruned join with an empty LocalRelation, which plans as LocalTableScanExec. The EmptyRelation node, and with it EmptyRelationExec, only arrived in Spark 4.0 (SPARK-48466), and ShimCometEmptyRelation returns None on 3.x. spark.comet.exec.localTableScan.enabled defaults to false, so the empty scan and the aggregates above it stay in Spark:

HashAggregate
+- ShuffleQueryStage
   +- Exchange SinglePartition
      +- HashAggregate
         +- LocalTableScan <empty>

The query result is correct. Only the plan check is wrong for 3.5.

Steps to reproduce

./mvnw test -Pspark-3.5 -Pscala-2.13 -Dtest=none \
  -Dsuites='org.apache.comet.CometIcebergNativeSuite DPP - single-split sort-merge join'

Expected behavior

The test passes on every Spark profile Comet builds.

Additional context

Requiring CometEmptyRelationExec only on Spark 4.0+ and accepting the empty LocalTableScanExec on 3.5 would match what each version plans.

Willingness to contribute

I can contribute a fix for this bug independently

Activity

  1. added a commit that references this issue on Oct 12, 2026
    42db91b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions