Comet writer doesn't respect OVERWRITE mode

Describe the bug

Comet fails if the folder exists and cannot overwrite it

Steps to reproduce

Folder output2 exists

spark.read.parquet("/tmp/writertest/input").repartition(1).write.mode("overwrite").parquet("/tmp/writertest/output2")

25/12/23 08:31:40 ERROR CometExecIterator: Native execution for task 17 failed
org.apache.comet.CometNativeException: Failed to create output file '/tmp/writertest/output2/part-00000-00017.parquet': Is a directory (os error 21)

Expected behavior

Should overwrite the folder with new content

Additional context

No response

Taking a look at this

some more details on implementation

The flow is like

    ↓
DataFrameWriter.saveInternal() 
    ↓
InsertIntoHadoopFsRelationCommand.run()
    ↓
[Line 131] deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
    ↓
[Line 238] committer.deleteWithJob(fs, staticPrefixPath, true)
    ↓
[Line 183] fs.delete(path, recursive=true)  ← ACTUAL DELETION HAPPENS HERE

This Spark implementation now

    val doInsertion = if (mode == SaveMode.Append) {
      true
    } else {
      val pathExists = fs.exists(qualifiedOutputPath)
      (mode, pathExists) match {
        case (SaveMode.ErrorIfExists, true) =>
          throw QueryCompilationErrors.outputPathAlreadyExistsError(qualifiedOutputPath)
        case (SaveMode.Overwrite, true) =>
          if (ifPartitionNotExists && matchingPartitions.nonEmpty) {
            false
          } else if (dynamicPartitionOverwrite) {
            // For dynamic partition overwrite, do not delete partition directories ahead.
            true
          } else {
            deleteMatchingPartitions(fs, qualifiedOutputPath, customPartitionLocations, committer)
            true
          }
        case (SaveMode.Overwrite, _) | (SaveMode.ErrorIfExists, false) =>
          true
        case (SaveMode.Ignore, exists) =>
          !exists
        case (s, exists) =>
          throw QueryExecutionErrors.saveModeUnsupportedError(s, exists)
      }
    }

The keypoint here, we need to support dynamicPartitionOverwrite or fallback to Spark in this case

take

Overwrite mode works on main now: #4946 added SaveMode.Overwrite handling to the native writer (CometNativeWriteExec.scala, test "SaveMode.Overwrite replaces existing parquet data" in CometParquetWriterSuite), and on Spark 4.0+ #5763 leaves SaveMode to Spark. I think this can be closed.

@dwsmith1983

Overwrite mode works on main, as @dwsmith1983 found. I checked it at c230bf4: a native Parquet write with mode("overwrite") into a directory that already held an earlier write replaced it, on Spark 3.5.9 (CometNativeWriteExec) and on Spark 4.1.3 (CometWriteFilesExec). The second write succeeded, and the directory read back only its 5 rows.

#4946 added the SaveMode.Overwrite handling on Spark 3.x, which SaveMode.Overwrite replaces existing parquet data in CometParquetWriterSuite covers, and on Spark 4.0+ #5763 leaves the save mode to Spark. Closing.

Navigation Menu

Repository navigation

Comet writer doesn't respect `OVERWRITE` mode #2970

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Comet writer doesn't respect OVERWRITE mode #2970

Description

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Activity

coderfender commented on Dec 24, 2025

comphead commented on Dec 29, 2025

coderfender commented on Feb 6, 2026

dwsmith1983 commented on Oct 4, 2026

andygrove commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Comet writer doesn't respect `OVERWRITE` mode #2970