You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg write panics generating the partition path for a timestamptz partition column #5694
Writing to a table partitioned on a timestamptz column panics in iceberg-rust while generating the data file's partition path:
org.apache.comet.CometNativeException: called `Option::unwrap()` on a `None` value
at core::option::unwrap_failed
at iceberg::spec::values::temporal::timestamptz::microseconds_to_datetimetz
at <iceberg::spec::values::datum::Datum as core::fmt::Display>::fmt
at <iceberg::spec::values::datum::Datum>::to_human_string
at <iceberg::spec::transform::Transform>::to_human_string
at <iceberg::spec::partition::PartitionSpec>::partition_to_path::{closure#0}
at <iceberg::spec::partition::PartitionSpec>::partition_to_path
at <...DefaultLocationGenerator as ...LocationGenerator>::generate_location
at <...RollingFileWriter<ParquetWriterBuilder, DefaultLocationGenerator, ...>>
microseconds_to_datetimetz unwraps a None when converting the partition value to its human-readable form for the directory name, so the task dies with a Rust panic crossing the JNI boundary rather than an error.
Steps to reproduce
Spark 4.1.3, Iceberg 1.11.0, spark.comet.iceberg.write.enabled=true plus the Iceberg Spark SQL test setup from dev/diffs/iceberg/1.11.0.diff:
Both write to a table whose partition spec covers a wide set of column types, including timestamptz.
Expected behavior
The partition path is generated for timestamptz partition values the same way iceberg-java generates it, and the write succeeds. Any value the conversion genuinely cannot represent produces an error rather than a panic.
Additional context
Found by turning the two Iceberg write flags on by default in #5677.
The panic is in iceberg-rust rather than in Comet's own code, so the fix may need to go upstream. Worth checking whether the value reaching microseconds_to_datetimetz is itself wrong (a unit or epoch mismatch on the way in) before concluding the conversion is at fault.
Activity
fix: native Iceberg write panics on an evolved partition spec and on …