Nightly CI failed on 2026-09-25 #6214

Activity

1. github-actions

   added

   ci-nightly-failureOpened by ci.yml when the nightly CI run against main failsOpened by ci.yml when the nightly CI run against main fails

   on Sep 25, 2026
2. andygrove commented on Sep 25, 2026

   

   andygrove

   on Sep 25, 2026

   Member

   More actions

   Two unrelated failures in this run.

   The Spark 4.0 sql_core-1 failure is CollationSuite "hash agg is not used for non binary collations" (None was empty (CollationSuite.scala:467)), the assertion that the UTF8_BINARY GROUP BY plan contains a Spark HashAggregateExec. It comes from #6195 and will fail every nightly until the diff changes. Spark 4.0's DatasourceV2SQLBase runs spark.sessionState.conf.clear() after each test, which drops spark.shuffle.manager from the session conf. Before #6195, Comet read the shuffle manager from there, so Comet shuffle was silently off for every test after the first one in those suites, and this query's final aggregate stayed on Spark. #6195 checks the manager that SparkEnv holds instead, so Comet shuffle now plans the exchange and the final aggregate is a CometHashAggregateExec. Spark 4.1's DatasourceV2SQLBase only unsets two catalog configs, so 4.1 always saw this, and the 4.1.3 diff already accepts CometHashAggregateExec in that assertion. #6195 ran only the 4.1 suite. The new behaviour is correct, so #6220 regenerates the 4.0.4 diff to accept CometHashAggregateExec the same way, with run-spark-4.0-tests applied.

   One side effect: every DatasourceV2SQLBase suite on 3.4, 3.5 and 4.0 now runs Comet shuffle after its first test, where it used to run Spark's. The 3.5 suite and the other 4.0 shards passed in this run. The 3.4 suite runs only with its label or a manual dispatch, and it hasn't run since #6195 landed.

   The Iceberg 1.9 shard-2 failure is TestRewriteDataFilesAction.testParallelPartialProgressWithMaxFailedCommitsLargerThanTotalFileGroup at formatVersion = 3 ("1 rewrite commits failed. This is more than the maximum allowed failures of 0"), a known upstream flake, apache/iceberg#12889. Three threads commit ten file groups one at a time against a Hadoop table. In the job's JUnit output, one thread committed eight groups back to back while group 3's commit backed off four times and then lost a fifth attempt on stale metadata, past Iceberg's default of four commit retries. That all happens in Iceberg's commit code, not Comet's. Upstream fixed the test in apache/iceberg#13208 and apache/iceberg#13598. Both are in 1.10.0 but not 1.9.1, which is why the 1.10 shard-2 passed on the same commit. This is the first time it has failed in the last ten nightlies. If it keeps recurring, we could backport those two test edits into the 1.9.1 diff.
3. andygrove

   closed this as completedin #6220on Sep 25, 2026