You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Account the native Iceberg writer's buffers in Comet's memory pool #5648
The native Iceberg writer (#5361) buffers rows in parquet-rs before flushing row groups, and the clustered and fanout writers hold per-partition writers open, each with its own buffer. It is not clear whether any of that memory is registered with Comet's memory pool (spark.comet.exec.memoryPool / the unified off-heap pool), or whether it is Rust heap allocation that Spark's memory accounting cannot see. A fanout write over many partitions multiplies the unaccounted buffers by the partition count.
The native scan and shuffle went through this audit already (#5212, memory pool and accounting sweep); the writer landed after it.
Describe the potential solution
Audit what the writer allocates per open file (parquet-rs column writers, dictionary pages, the manifest serializer) and whether WriterProperties sizes (row group size, page size, dictionary page size) bound it.
Register the writer's buffers with the task's memory reservation, or document why the amount is bounded enough not to need it, with the bound stated in terms of the vetted write.parquet.* properties and the partition count.
For the fanout writer, decide whether to cap open writers or account for them, since the JVM FanoutDataWriter has the same shape and users size executors for it.
Add a test that a fanout write over many partitions with a small spark.comet.memoryOverhead fails with a Comet out-of-memory error rather than a process-level one.
Additional context
Part of the native Iceberg writes epic, #5649. Related: #5361, #5212.
Activity
feat: charge native write buffers to the task's memory pool (apache#6247