You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CometIcebergNativeSuite single-split sort-merge join DPP test fails on Spark 3.5 with AQE #6901
DPP - single-split sort-merge join (AQE=true, pruned=a) and (AQE=true, pruned=b) in CometIcebergNativeSuite, added in #6814, fail on the Spark 3.5 profile:
List() was empty Expected AQE to replace the pruned join with an empty relation
The test asserts a CometEmptyRelationExec whenever isSpark35Plus && aqeEnabled && scans.isEmpty. On Spark 3.5, AQEPropagateEmptyRelation replaces the pruned join with an empty LocalRelation, which plans as LocalTableScanExec. The EmptyRelation node, and with it EmptyRelationExec, only arrived in Spark 4.0 (SPARK-48466), and ShimCometEmptyRelation returns None on 3.x. spark.comet.exec.localTableScan.enabled defaults to false, so the empty scan and the aggregates above it stay in Spark:
Activity
test: expect Spark 3.5's empty LocalTableScan in the single-split DPP…