You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg write: clean up task-attempt data files on task failure #5618
When the native Iceberg write path (#5361) fails partway through a task, data files already finalized by that task attempt are left in the table's data location. They are invisible to readers (files resolve only through committed manifests) and are reclaimed by remove_orphan_files, but iceberg-java cleans them up synchronously: SparkWrite's writers run inside Utils.tryWithSafeFinallyAndFailureCallbacks, and DataWriter.abort() calls SparkCleanupUtil.deleteTaskFiles(io, ...). On spot-heavy or preemption-prone clusters the orphan cost accumulates silently on the native path.
This gap is documented in the "Failure handling" section of iceberg-writes.md; this issue tracks closing it.
Suggested shape
TaskContext.addTaskFailureListener in CometIcebergWriteExec.doExecute, mirroring what iceberg-java's writer abort does:
For failures after the native writer has returned (manifest decode, JVM metrics rebuild, TaskCommit construction), the JVM already holds the decoded DataFiles — delete their paths through the table FileIO that is already in the task closure.
For failures inside the native write itself, the JVM never learns the written paths. Closing that half needs the native operator to report the paths of finalized files alongside the error (or a JNI call to fetch them from the failed writer), since iceberg-rust's writers have no abort/Drop cleanup of their own.
Cleanup must be best-effort and must not mask the original task failure.
Activity
fix: delete data files left by a failed native Iceberg write task
fix: delete data files left by a failed native Iceberg write task
fix: delete data files left by a failed native Iceberg write task
fix: Delete completed tasks' data files when an Iceberg write job fai…