You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg write fails with "No column with source column id" after an identity partition field is dropped #5693
After an identity partition field is dropped and a column is re-added, the native writer cannot resolve the partition source column and fails the write:
org.apache.comet.CometNativeException: Unexpected => No column with source column id 2 in schema
Schema { struct: StructType { fields: [NestedField { id: 1, name: "id", required: true,
field_type: Primitive(Long), ... } ...
at org.apache.comet.Native.executePlan(Native Method)
at org.apache.spark.sql.comet.CometIcebergWriteExec.drainAvroPayload(CometIcebergWriteExec.scala:276)
at org.apache.spark.sql.comet.CometIcebergWriteExec.$anonfun$doExecute$9(CometIcebergWriteExec.scala:181)
The partition spec still references source column id 2, which no longer exists in the current schema, and the native manifest encoding treats that as fatal. iceberg-java tolerates it.
Steps to reproduce
Spark 4.1.3, Iceberg 1.11.0, spark.comet.iceberg.write.enabled=true plus the Iceberg Spark SQL test setup from dev/diffs/iceberg/1.11.0.diff:
The write succeeds against a table whose partition spec references a source column that is no longer in the current schema, matching iceberg-java, or the operator declines the write at planning time so it falls back to the JVM writer instead of failing the task.
Additional context
Found by turning the two Iceberg write flags on by default in #5677.
Same test class and same encode_data_files_as_manifest phase as #5691, which panics on a related partition-spec-evolution case. One fix to how the writer resolves the spec against the current schema may close both.
Activity
fix: native Iceberg write panics on an evolved partition spec and on …