You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Parquet write of a zero-partition RDD produces no output file #5303
Without it no write task runs at all, so the output directory ends up with no data file — not even the schema-only file Spark produces. IcebergWriteExec added in #4658 carries the same guard.
CometWriteFilesExec.doExecuteWrite has no equivalent: it calls child.executeColumnar() and goes straight to mapPartitionsInternal. With zero partitions, FileFormatWriter.writeAndCommit collects zero WriteTaskResults and commits an empty job. The result is a _SUCCESS marker over an empty directory, where Spark writes one file carrying the schema.
Note this is not a regression from #5293 — the CometNativeWriteExec it replaced had the same hole.
Steps to reproduce
Any write whose input plan yields a zero-partition RDD (for example a scan over an empty relation), with spark.comet.parquet.write.enabled=true.
Expected behavior
Match Spark: run one write task so the output carries the schema.
Harder here than in the Iceberg case because the dummy RDD must be columnar — parallelize(Array.empty[InternalRow], 1) is row-based, and CometWriteFilesExec feeds batches into an Arrow stream. Options are a single-partition empty ColumnarBatch RDD, or bypassing the native writer for this case and writing the schema-only file on the JVM.
Activity