Repository navigation
Accelerate DataSource V2 writes (AppendDataExec, OverwriteByExpressionExec, OverwritePartitionsDynamicExec) #5121
Description
What is the problem the feature request solves?
Comet accelerates DataSource V2 reads (Parquet, Iceberg via iceberg-rust, CSV) but all DataSource V2 writes fall back to Spark: AppendDataExec, OverwriteByExpressionExec, OverwritePartitionsDynamicExec, and WriteToDataSourceV2Exec.
For ETL pipelines this is the missing half of the job. A pipeline that scans, joins, and aggregates natively still pays a columnar-to-row transition into Spark's row-based writer at the end, and for Iceberg specifically we accelerate reads but not writes.
Describe the potential solution
Suggested staging:
- Start with Iceberg
AppendDataExec(plainINSERT INTO/append), reusing the native Parquet write machinery from [EPIC] Add support for native Parquet writes #1625 and coordinating with the iceberg-rust integration. Native Sort-Merge Writer for Iceberg ClusteredWriter Path #3595 covers the sort-merge/clustered writer path and would build on the same foundation. OverwriteByExpressionExecandOverwritePartitionsDynamicExec(dynamic partition overwrite) next, sinceINSERT OVERWRITEis a standard ETL pattern.- Generic
WriteToDataSourceV2Execlast, gated on the target format being one Comet can write natively.
As with the V1 native Parquet write work, this should be opt-in and experimental until commit protocol semantics are proven out.
Additional context
Related: #1625 (EPIC: native Parquet writes), #3595 (native sort-merge writer for Iceberg clustered path), #3756 (Iceberg feature matrix). Row-level MERGE / UPDATE / DELETE plans are tracked separately since they layer on top of this.
Activity
docs: link tracking issues for operator acceleration gaps
Visorgood commentedon Sep 8, 2026
Hello @andygrove !
I'm interested in working on the first stage (Iceberg AppendDataExec). I’ve been looking through the existing native Parquet writer path. Would it make sense to split the Spark-side integration into a separate issue/PR?
Visorgood commentedon Sep 10, 2026
Hey @andygrove! Any thoughts on this?
andygrove commentedon Sep 22, 2026
Hi @Visorgood, sorry for the slow reply, and thanks for offering to help.
This issue is out of date. Since it was filed, stages 1 and 2 have been done for Iceberg:
AppendData,OverwriteByExpression,OverwritePartitionsDynamic, and copy-on-writeReplaceDataall go through Comet's split write/commit plan and the native iceberg-rust writer. That work was #4658, #5298, and #5361. It's experimental and off by default for now (spark.comet.write.iceberg.splitOperator.enabledandspark.comet.iceberg.write.enabled). The docs are atdocs/source/user-guide/latest/iceberg-writes.md.The remaining Iceberg write work is tracked in #5649, split into phases (correctness, failure handling, coverage, enabling by default, performance). If you'd like to help, anything unassigned there is a great place to start. For example, #5318 (native
MergeRowsExec) is the natural next step for row-level MERGE, and #5306 (reconciling the two config namespaces for native writes) and #5643 (a keep-or-lift decision for each remaining eligibility restriction) are more self-contained. Comment on the one you pick so we can assign it to you.Stage 3 (generic V2 writes) doesn't have a clear target right now, since Spark's built-in file sources write through the V1 path (#1625). So I'm going to close this issue in favor of #5649. If someone has a V2 sink that would benefit, we can open a focused issue for it.
andygrove commentedon Sep 22, 2026
Correction to my comment above: #5318 is @unikdahal's open PR for native
MergeRowsExec, not an unclaimed issue, so please don't pick that one up. Reviewing it would be very welcome, though. For something to own, #5306 and #5643 are still open and unassigned, and merge-on-read writes (WriteDeltaExec, tracked in #5122) have no one working on them yet.