You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Parquet writes report row counts but not row contents to WriteTaskStatsTrackers #5307
WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow) is a per-row callback. Comet's native write path has columnar batches, not InternalRows, so CometWriteFilesExec.recordRows calls it n times with InternalRow.empty rather than materializing every row just to hand it straight back.
That is exactly right for BasicWriteTaskStatsTracker, the only implementation Spark ships, which ignores the row argument and just increments a counter (BasicWriteStatsTracker.scala). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows.
Today this logs a warning per non-BasicWriteTaskStatsTracker instance. A warning is the honest minimum, but it is not a guarantee.
Describe the potential solution
The obstacle is that a plan-time guard is not possible: WriteJobDescription.statsTrackers only exists at execution time, by which point getSupportLevel has already accepted the write and there is no way to fall back.
Options:
Materialize rows only when a non-basic tracker is present — correct, and pays the row-conversion cost solely in the case that needs it.
Fail the write with a clear message instead of warning, so nobody gets wrong statistics silently.
Find a plan-time signal for the tracker set so the write can fall back to Spark gracefully.
Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet.
Additional context
Introduced by #5293, which moved native writes onto Spark's WriteFilesExec seam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression.
Activity
LinSimon-901101 commentedon Sep 29, 2026
take