Audit the PRs in 1.1.0 for regressions since 1.0.0 #6399

Activity

1. andygrove

   self-assigned this

   on Sep 29, 2026
2. github-actions

   added

   requires-triage

   on Sep 29, 2026
3. andygrove commented on Sep 29, 2026

   

   andygrove

   on Sep 29, 2026

   MemberAuthor

   More actions

   Phase 1 is done. Sonnet agents checked all 83 fix → origin links from the Phase 0 blame against the 1.0.0 tag, and I spot-checked the verdicts that mattered. That is 86 links minus 3 that were a main commit and its branch-1.1 backport.

   Seven are regressions, and all seven were fixed before 1.1.0-rc1. They're listed in #6402 with the introducing PR, the fix, and why review missed each one. None of the seven introducing PRs was backported to branch-1.0, and all seven fixes are in rc1. The five fixes that merged to main after the cut without reaching branch-1.1 (#6261, #5880, #5403, #5846 and #5169) all fix bugs that were already in 1.0.0, so none of them needs a backport as a regression fix.

   Of the other 76, 45 fixed bugs that were already in 1.0.0. There the blame link was noise, because the fix touched lines that a PR in the window had only moved or edited. 28 were bugs in functionality that is new in 1.1.0 and didn't break anything that worked in 1.0.0. Most of those are in the native Iceberg writer, the native Celeborn shuffle and the native in-memory cache, which are all off by default. The last 3 weren't fixes for a defect. I overrode two regression verdicts:

   • fix: bound shuffle schema cache retention and preserve eviction order #6098: the shuffle schema cache that perf: decode shuffle blocks against a cached schema instead of re-parsing per block #5809 added had a wrong eviction order and no byte cap, but 1.0.0 had no cache and nothing that worked there broke.
   • fix: skip the executor memory overhead warning when a factor is set or in local mode #6198: a new memory-overhead warning from fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin #6054 fired when it shouldn't. That's noise, not a regression.

   The reasons review missed the seven fall into three groups:

   • CI coverage. The pull request CI runs the Comet suites on the default Spark profile only and skips the Iceberg suites. Iceberg's TestForwardCompatibility caught deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 #5262's break after merge, and the nightly Spark 3.4 job caught fix: let decimal SUM recover from an intermediate overflow like Spark #6041's. No CI profile could have caught feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775's, because CI builds only the newest patch of each Spark line and the behavior it got wrong changed in earlier patches.
   • New native paths on by default. feat: support WindowGroupLimitExec #4870 documented its signed-zero rank mismatch in the compatibility guide but still enabled WindowGroupLimitExec by default, instead of gating it behind allowIncompatible. feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775 has the same shape.
   • Tests that covered the targeted case but not its neighbor:
     • feat: support unicode case sensitive field names for reading parquet #5602 tested case-insensitive duplicate fields but not byte-identical ones.
     • feat: support S3 compliant filesystems #5314 tested the default scheme list but not one naming only s3.
     • perf: serialize Python input directly from Comet Arrow vectors #5368 tested incompatible schemas but not two operators labelling UTC differently.

   The seven introducing PRs (#5602, #5262, #5314, #5368, #6041, #4870 and #4775) are the known answers for checking the Phase 2 triage.
4. andygrove commented on Sep 29, 2026

   

   andygrove

   on Sep 29, 2026

   MemberAuthor

   More actions

   Phase 2 is done. Sonnet agents triaged all 292 PRs that touch production code, in 25 batches. Each PR got HIGH, MEDIUM or LOW for how plausibly its diff regressed something that worked in 1.0.0. The result is 47 HIGH, 119 MEDIUM and 126 LOW. A HIGH is a reason to look closer, not a finding, and I expect most of them to turn out fine in Phase 3.

   The method changed in two ways from the plan above. First, the packets left out later fixes to each PR's lines, because including them would have given away the seven known answers from Phase 1. Second, one agent looked up my local notes, which describe some of those answers, so the later batches were told to judge each PR only from its own diff and the code.

   For calibration, five of the seven PRs behind the Phase 1 regressions were rated HIGH. Two of those five aren't clean hits: #5368 shared a batch with its own fix, and the agent that rated #5602 had read my notes. Of the five blind checks, #4775, #4870 and #5262 were rated HIGH, and #6041 and #5314 were rated MEDIUM. Both misses only showed up under a non-default configuration:

   • spark.sql.codegen.factoryMode=NO_CODEGEN on Spark 3.4
   • a fs.comet.libhdfs.schemes list naming s3 but not s3a

   Both were at least flagged as touching a default path. That's within the plan's tolerance of one or two misses, so I'm not rerunning the triage.

   The 47 HIGH PRs go to Phase 3, grouped by area:

   • Expressions and casts (12): feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775 feat: support listagg / string_agg aggregate (Spark 4.0+) #4816 fix: normalize floating-point values in native collect_set #5166 refactor: delegate ANSI integer arithmetic to arrow checked kernels #5280 feat: run rlike natively by default for Java-equivalent literal patterns #5415 feat: support Map for CreateArray literal #5452 fix: normalize noncanonical NaN literals in comparisons #5472 feat: add native spark_sequence kernel for integral element types #5614 feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638 fix: align string to timestamp parsing with Spark's segment rules #5682 feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher #5692 fix: dispatch Iceberg system functions wrapped as ApplyFunctionExpression #5773
   • Planner, operators and shuffle (13): feat: support WindowGroupLimitExec #4870 fix: enable FIRST/LAST partial merge #5041 fix: make CometExplodeExec respect batch size #5362 fix: revert unsafe partial aggregates after final fallback #5421 perf: bulk copy fixed-width columns in ArrowWriter #5442 feat: add raw Celeborn native shuffle reader (7/n) #5531 perf: cache parsed plan data across a stage tasks #5615 perf: slice the child instead of gathering it when unnesting #5667 fix: recover native Celeborn shuffle from oversized rows #5668 feat: Enable adaptive partial aggregation for eligible native shuffle plans #5723 feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ #5763 perf: give collect_list and collect_set a native GroupsAccumulator #5803 perf: spill every shuffle partition of a task into one file #5916
   • Scans, meaning Parquet, Iceberg and Delta (15): fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values #5177 feat: support _metadata constant columns in native Parquet scan #5237 deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 #5262 feat: Support native scans with unprojected Spark 4 VARIANT columns #5377 fix: isolate object-store registration by backend and configuration #5503 feat: support unicode case sensitive field names for reading parquet #5602 fix: match Spark's duplicate field and field id semantics in parquet field lookup #5654 fix: apply Spark's Parquet conversion rules to nested struct/list/map fields #5681 feat: normalize marked Variant arrays at the native Parquet boundary #5715 fix: keep Iceberg complex null checks on native scans #5732 fix: reject duplicate Parquet field names before decoding #5786 feat(iceberg): Iceberg table format V3: apply deletion vector on reads #5853 fix: reject a file without field ids at any depth whether or not id matching is on #6116 fix(iceberg): don't push transform residuals as their source column, fail on residual errors #6154 fix: park the native scan loop instead of busy-polling while waiting on native I/O #6219
   • Memory, FFI, config and shims (7): perf: serialize Python input directly from Comet Arrow vectors #5368 fix: make CometDiskBlockWriter spill registry per-task instead of executor-global #5493 fix: prevent memory leak after failed Arrow vector import #5539 fix: preserve Arrow Field metadata across C Data exports #5552 fix: Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6025 perf: make one thread-local access per tracked allocation #6166 fix: read shuffle write buffer, spill limit and off-heap sizes in bytes #6191

   The dependency-upgrade review in Phase 3 starts from #5262, #5324, #5865 and #6094.

   The common reasons for a HIGH were:

   • An expression or operator that became native by default, with a Spark difference that the tests don't pin down (NaN and -0.0, regex dialects, patch-version behavior).
   • A rewrite or "no behavior change" refactor of a default-path kernel with no new correctness tests.
   • A default-path change bundled into a PR for an opt-in feature. For example, fix: recover native Celeborn shuffle from oversized rows #5668 made partitions compute eagerly on every native shuffle read, and feat: add raw Celeborn native shuffle reader (7/n) #5531 added a new hard error on the local shuffle read path.
   • A logged fallback turned into a thrown error, as in fix(iceberg): don't push transform residuals as their source column, fail on residual errors #6154.

   One open question goes first in Phase 3 because it could affect the release. #6261 fixed a hang that needs a single Tokio worker, and it isn't on branch-1.1. A standalone executor without spark.executor.cores gets exactly one worker. Phase 1 traced the memory-release bug in #6261 to before 1.0.0, but nobody has run the hang reproducer on 1.0.0. So it isn't known yet whether the hang is new in 1.1.0.
5. andygrove commented on Sep 29, 2026

   

   andygrove

   on Sep 29, 2026

   MemberAuthor

   More actions

   Phase 3, expressions and casts: all 12 PRs have been reviewed against 1.0.0 and against Spark. Every reproducer was then run on 1.0.0 and 1.1.0-rc1 builds, which covers Phase 4 for this area. The #6261 check is done too: that hang was already in 1.0.0.

   Five regressions that ship in 1.1.0 are confirmed, and #6402 has the details:

   • Native regr_* aggregates return wrong results for a constant variable merged from several partitions #6423, from feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775: regr_slope, regr_intercept and regr_r2 return large wrong values instead of NULL for a constant variable merged across partitions.
   • Typed Dataset.filter returns wrong rows over a native aggregate #6424, from feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher #5692: a typed Dataset.filter over a native aggregate returns wrong rows. fix: zero sliced boolean offsets at every level before exporting to the JVM #6339 fixes it on main and needs a backport.
   • Decimal-returning DSv2 functions lose their scale through the codegen dispatcher #6425, from feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher #5692: decimal-returning DSv2 functions lose their scale in the codegen dispatcher, e.g. 0.03 instead of 3.00.
   • Native Iceberg years/months/days/hours differ from Iceberg for some pre-1970 timestamps #6426, from feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638 and fix: dispatch Iceberg system functions wrapped as ApplyFunctionExpression #5773: the native Iceberg years, months, days and hours differ from Iceberg for rare pre-1970 timestamps.
   • Native IF fails with "column types must match schema types" when its branches differ in nested nullability #6334, from feat: support Map for CreateArray literal #5452: a native IF with a map or struct literal branch throws when every row of a batch takes the same branch.

   Each one has a draft fix PR open or in progress, linked from its issue.

   No regression was found in #5472, #5166, #5415, #5614, #5280 or #5682. Several of them fix wrong answers that 1.0.0 gave. #4816 changes the order of values in listagg only where Spark leaves that order unspecified, so it's not a regression.

   Why review missed these:

   • A path became native by default without a test for the case where it diverges from Spark: a constant merged from several partitions in feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775, pre-1970 unit boundaries in feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638, and a batch where every row takes one branch in feat: support Map for CreateArray literal #5452.
   • feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher #5692's catch-all sent new shapes through the codegen dispatcher that the dispatcher's tests never covered: sliced boolean inputs and decimal results whose scale differs from the declared type.
6. andygrove commented on Sep 30, 2026

   

   andygrove

   on Sep 30, 2026

   MemberAuthor

   More actions

   Phase 3, planner, operators and shuffle: all 13 PRs have been reviewed against 1.0.0 and against Spark. Every reproducer was then run on 1.0.0 and 1.1.0-rc1 builds, which covers Phase 4 for this area.

   Three more regressions that ship in 1.1.0 are confirmed, and #6402 has the details:

   • From perf: slice the child instead of gathering it when unnesting #5667 and fix: make CometExplodeExec respect batch size #5362: explode, posexplode and explode_outer of an array of structs with a boolean field return wrong booleans and NULLs after the first spark.comet.batchSize output rows of each input batch, with default configs. 3,000 rows of 12 elements gave wrong booleans in 22,824 of the 36,000 output rows. The new chunking slices the output, and Arrow Java ignores the bit offset of the sliced boolean children, which is Sliced booleans nested in structs, and sliced inputs to the JVM UDF bridge, reach the JVM misaligned and return wrong results #6288. fix: [branch-1.1] zero sliced boolean offsets at every level before exporting to the JVM (#6339) #6449, the branch-1.1 backport of fix: zero sliced boolean offsets at every level before exporting to the JVM #6339, fixes every case on rc1, so the same backport covers this and Typed Dataset.filter returns wrong rows over a native aggregate #6424.
   • Match Spark ordering and rank semantics for floating values nested in arrays and structs #5507, from feat: support WindowGroupLimitExec #4870: the native WindowGroupLimit treats -0.0 and 0.0 inside an array or struct sort key as different, so a rank <= k filter drops tied rows. 1.0.0 ran WindowGroupLimit in Spark.
   • From feat: Enable adaptive partial aggregation for eligible native shuffle plans #5723: adaptive partial aggregation decides once per task, after 100,000 rows, and never goes back. A task whose keys repeat after a mostly distinct start sends every later row through the shuffle, 10x the rows in the reproducer. The results are correct, and the only off switch is the testing config that the tuning guide documents.

   No regression was found in #5041, #5442, #5531, #5615, #5668, #5763 or #5803. #5421 and #5916 are deliberate trade-offs. #5421 moves partial AVG, stddev, first, last and a few others back to Spark when the final aggregate runs in Spark, mostly for anyone running without the Comet shuffle manager, because 1.0.0 could return NULL averages there. That's worth a line in the release notes. #5916 puts all of a map task's spill in one local directory, which #6010 tracks.

   One memory concern isn't verified. Since #5803, a native partial collect_list directly over UNION ALL or coalesce keeps every input batch it has read, including the columns it doesn't collect, and those aren't counted against the memory pool until the aggregate emits. I couldn't make that observable in a test on both builds.

   Why review missed these:

   • The explode tests exploded int arrays or pulled struct fields out into top-level columns, so a boolean below the top level never crossed back to the JVM from a sliced batch. Sliced booleans nested in structs, and sliced inputs to the JVM UDF bridge, reach the JVM misaligned and return wrong results #6288 names LIMIT and aggregates as triggers, not explode.
   • fix: normalize scalar float sort and window rank keys #5469 fixed the scalar case of the WindowGroupLimit bug and moved the nested case to Match Spark ordering and rank semantics for floating values nested in arrays and structs #5507 as a documented incompatibility, instead of falling back for it by default.
   • feat: Enable adaptive partial aggregation for eligible native shuffle plans #5723's tests covered uniformly high and uniformly low cardinality, but not a distinct prefix followed by repeats. Review asked for a supported off switch, and the PR documented the testing route instead.
7. andygrove commented on Oct 1, 2026

   

   andygrove

   on Oct 1, 2026

   MemberAuthor

   More actions

   Phase 3, scans: all 15 Parquet, Iceberg and Delta PRs have been reviewed against 1.0.0 and against Spark. The reproducers were then run on 1.0.0 and 1.1.0-rc1 builds, which covers Phase 4 for this area.

   Three more regressions that ship in 1.1.0 are confirmed, and #6402 has the details:

   • Native Iceberg scan fails on files written before a nested field was added, and since 1.1.0 null checks and explode hit it too #6504, from fix: keep Iceberg complex null checks on native scans #5732: on an Iceberg table where a struct column, or an array's element struct, gained a nested field, IS NULL and IS NOT NULL on the column, non-outer explode of the array, and joins on the struct now fail. 1.0.0 fell back to Spark for these. The root cause is older (Type casting error when reading files persisted with old schema for complex type iceberg-rust#2617): a plain read of such a column fails natively in 1.0.0 too.
   • Native Parquet scan returns wrong _metadata.file_block_start when Spark splits a file #6505, from feat: support _metadata constant columns in native Parquet scan #5237: _metadata.file_block_start and file_block_length come out wrong when Spark splits a Parquet file into several partitions, because DataFusion and parquet-mr assign row groups to splits differently. 1.0.0 fell back to Spark for metadata columns.
   • Native Parquet scan fails on an empty file whose nested field type differs from the read schema #6506, from fix: apply Spark's Parquet conversion rules to nested struct/list/map fields #5681: an empty or fully pruned Parquet file with a nested field that the read schema can't convert now fails the query when the file is opened. Spark and 1.0.0 return the other files' rows. For such files that do have rows, the same change fixes a wrong answer.

   All three have a workaround in the draft release notes (#6469), checked against the reproducers.

   No regression was found in #5262 (its scan side), #5377, #5503, #5602, #5654, #5715, #5853, #6116, #6154 or #6219. #6154 fixes wrong rows that 1.0.0 returned for some Iceberg transform residuals.

   Two findings aren't counted:

   • fix: reject duplicate Parquet field names before decoding #5786 now rejects Parquet files with byte-identical duplicate field names in a few shapes that 1.0.0 read correctly. That's deliberate and documented, and only writers other than Spark produce such names.
   • fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values #5177's TIMESTAMP_MILLIS overflow check covers a whole 8192-row batch, so a LIMIT can fail on a value Spark never reads. That needs timestamps past year 294,000, which Spark can't write.

   Why review missed these:

   • fix: keep Iceberg complex null checks on native scans #5732 was argued from residual pushdown and Iceberg's post-scan filter, which do hold, but nobody asked what else the removed fallback was shielding, and no fixture evolves a nested schema.
   • feat: support _metadata constant columns in native Parquet scan #5237 reasoned that the metadata columns are constant per split. That's true, but which split reads a row group is a reader decision, and DataFusion and parquet-mr make it differently. Its tests used files too small to split.
   • fix: apply Spark's Parquet conversion rules to nested struct/list/map fields #5681 moved a check that Spark makes per value to file open, and its tests always had rows to decode.
   • fix: reject duplicate Parquet field names before decoding #5786's reviews measured it against main, which already had feat: support unicode case sensitive field names for reading parquet #5602's behavior, rather than against 1.0.0.
8. andygrove commented on Oct 1, 2026

   

   andygrove

   on Oct 1, 2026

   MemberAuthor

   More actions

   Phase 3, memory, FFI, config and shims: all 7 PRs have been reviewed against 1.0.0. None of them has a regression to count, so there was nothing to run on the 1.0.0 and rc1 builds for this area.

   Four findings aren't counted, but are worth knowing:

   • fix: Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6025's IRSA credential provider is on by default and doesn't fall back. 1.0.0 fell back to the node role or Pod Identity when the web-identity call failed, so a cluster with broken IRSA that quietly read S3 as the node role now fails native Iceberg reads with "failed to load signing credential". Not falling back is the point of the PR, and it fixes Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm #6024's 403 storm, but the 1.1.0 section of the upgrade guide doesn't mention it. The documented opt-out is per catalog, so path-loaded Iceberg tables can only opt out with spark.comet.scan.icebergNative.enabled=false.
   • fix: read shuffle write buffer, spill limit and off-heap sizes in bytes #6191 now rejects an invalid spark.comet.debug.memory value such as yes instead of ignoring it, which fails every native plan. It's a testing setting, but one sentence in the upgrade guide would cover it.
   • fix: preserve Arrow Field metadata across C Data exports #5552 sends a broadcast UNION ALL uncoalesced when its branches carry different top-level field metadata. The results are correct, and the PR accepted that cost on purpose.
   • feat: always count native allocations and log executor native memory usage #6162, which isn't in this group, made the allocation-tracking wrapper always on, and it costs about 2% on TPC-H Q21 after perf: make one thread-local access per tracked allocation #6166.

   No regression was found in #5368, #5493, #5539, #6166 or #6191's intended changes, which the upgrade guide documents. #5368's code only runs with spark.comet.exec.pyarrowUDF.enabled, and #5493's riskiest code was test-only and removed by #6066 before rc1.
9. andygrove commented on Oct 1, 2026

   

   andygrove

   on Oct 1, 2026

   MemberAuthor

   More actions

   Phase 3, dependency upgrades: I reviewed the upstream changes between the versions in 1.0.0 and 1.1.0-rc1, starting from the code Comet calls on default paths. That covers DataFusion and datafusion-spark 54.1 to 55.1, arrow and parquet 58.4 to 59.3, opendal 0.57 to 0.58.2, and the 154 iceberg-rust commits from 3d84c81 to bb1e4a4. The reproducers were run on 1.0.0 and rc1 builds. This completes Phase 3.

   Two regressions that ship in 1.1.0 are confirmed, both from DataFusion 55, and #6402 has the details:

   • array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701: array_distinct and array_union now treat -0.0 and 0.0 as one value and return 0.0. Spark versions without SPARK-54918 keep them apart: 3.4, 3.5, 4.0 before 4.0.5 and 4.1 before 4.1.4. 1.0.0 matched Spark.
   • A native final aggregate that has spilled can fail the task during its replay #6254: a native final hash aggregate that has spilled can't spill again while it reads the spill back, so a task near its memory limit fails with Failed to acquire N bytes. At 96m of off-heap memory, rc1 failed every run and 1.0.0 passed every run.

   Both were already filed, so I added the regression label and the evidence to each. Their workarounds are in the draft release notes (#6469).

   No regression was found in arrow, parquet, opendal or iceberg-rust. A few upstream changes improve on 1.0.0. Comparisons now treat -0.0 as equal to 0.0, as Spark does. map_from_entries raises Spark's DUPLICATED_MAP_KEY error. A Date32 to TIMESTAMP_NTZ overflow now errors as it does in Spark. The DataFusion review wasn't exhaustive: it didn't cover the sort-merge join and sort spill changes, row-based group values, or several string kernels.

   One gap is worth checking before rc2. Native Iceberg reads from S3, GCS or Azure now depend on opendal 0.58 installing its HTTP transport when the native library loads. That works in local runs, but no CI job reads from a real object store, so a quick S3 smoke test on the rc2 build would cover it.

   Why review missed these:

   • deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 #5262 switched the array_distinct and array_union signed-zero tests to ignore to get the upgrade through, so a behavior change from 1.0.0 went in as a skipped test rather than a fallback.
   • A native final aggregate that has spilled can fail the task during its replay #6254 needs a final aggregate that spills under a tight budget, and no CI test runs one.
10. andygrove commented on Oct 1, 2026

    

    andygrove

    on Oct 1, 2026

    MemberAuthor

    More actions

    Phase 5: what to do with each regression, and why review missed them.

    What to do with each regression

    rc2 comes from branch-1.1, where #6449 already fixes #6424 and #6464. For the other 11:

    Regression	Effect	Fix	Recommendation
    #6423 regr_*	wrong results	#6451 (approved), backport #6489	rc2
    #6426 Iceberg temporal functions	wrong results for rare pre-1970 inputs	#6456 (approved), backport #6486	rc2
    #6334 native IF	error	#6458 (approved), backport #6491	rc2
    #5507 WindowGroupLimit	wrong results for nested float keys	#6468 (merged), backport #6487	rc2
    #6466 adaptive partial aggregation	slower	#6474 (merged), backport #6488	rc2
    #6505 _metadata block columns	wrong results	#6510 (approved), backport #6511	rc2
    #6425 DSv2 decimal functions	wrong results	#6455 (needs review), backport #6490	rc2 if it's reviewed in time, otherwise release note
    #5701 array_distinct/array_union	wrong results with -0.0	#5750 (changes requested)	rc2 if a narrow fallback is ready, otherwise release note
    #6506 empty-file conversion error	error, rare	#6515 (needs review)	release note, fix in 1.1.1
    #6254 spilled aggregate replay	error under memory pressure	upstream apache/datafusion#25814	release note, fix in 1.1.1
    #6504 Iceberg nested-field evolution	error	none yet	release note, fix in 1.1.1

    The fixes I'd take in rc2 are small, and each either falls back to Spark or restores 1.0.0's behavior. All of them except #6425 are approved or merged. The draft release notes in #6469 have a checked workaround for every entry, so anything that misses rc2 is covered there. #6515 changes when scan errors are raised, so I'd rather it had some time on main than went in at the last minute.

    Why review missed them

    Across all 20 regressions, the 7 fixed before rc1 and the 13 that shipped in it, the same few things come up:

    • Tests used the inputs the author had in mind, not the ones where the new path differs from Spark. This is by far the biggest group: -0.0 and NaN (fix: normalize scalar float sort and window rank keys #5469, Match Spark ordering and rank semantics for floating values nested in arrays and structs #5507, array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701), a constant like 0.1 merged across partitions (Native regr_* aggregates return wrong results for a constant variable merged from several partitions #6423), pre-1970 unit boundaries (Native Iceberg years/months/days/hours differ from Iceberg for some pre-1970 timestamps #6426), a batch where every row takes one branch (Native IF fails with "column types must match schema types" when its branches differ in nested nullability #6334), output sliced past the first batch (Native explode returns wrong booleans for arrays of structs after the first batch of output #6464), files Spark splits or that are empty (Native Parquet scan returns wrong _metadata.file_block_start when Spark splits a file #6505, Native Parquet scan fails on an empty file whose nested field type differs from the read schema #6506), data that defeats a heuristic (Adaptive partial aggregation can shuffle 10x more rows than 1.0.0, with no supported way to turn it off #6466), and an asymmetric config (fix: decide libhdfs routing from the scheme as written #5825).
    • A change removed or widened a guard without asking what else it was shielding: the Iceberg null-check fallback (Native Iceberg scan fails on files written before a nested field was added, and since 1.1.0 null checks and explode hit it too #6504), the codegen dispatcher catch-all (Typed Dataset.filter returns wrong rows over a native aggregate #6424, Decimal-returning DSv2 functions lose their scale through the codegen dispatcher #6425), and native map literals (Native IF fails with "column types must match schema types" when its branches differ in nested nullability #6334).
    • A known divergence shipped on a default path. feat: support WindowGroupLimitExec #4870 documented its signed-zero gap and still enabled the operator (fix: normalize scalar float sort and window rank keys #5469), and deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 #5262 switched the array_distinct and array_union tests to ignore instead of falling back (array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701).
    • Tests compared against something other than Spark: Comet's own accumulator for regr_* (Native regr_* aggregates return wrong results for a constant variable merged from several partitions #6423), and iceberg-rust instead of Iceberg Java for the temporal functions (Native Iceberg years/months/days/hours differ from Iceberg for some pre-1970 timestamps #6426).
    • CI couldn't see it. The pull request tier runs only the default Spark profile (fix: correct two nightly test failures on Spark 3.4 and 4.2 #6156), CI builds only the newest patch of each Spark line (fix: gate the regr_r2 degenerate-case swap on the Spark patch release #6042), and nothing runs Iceberg's forward-compatibility tables (fix: read Iceberg tables partitioned by an unknown transform #5759) or a spill under a tight memory pool (A native final aggregate that has spilled can fail the task during its replay #6254).
    • A dependency upgrade was reviewed as an API migration rather than as a set of behavior changes (array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701, A native final aggregate that has spilled can fail the task during its replay #6254, fix: read Iceberg tables partitioned by an unknown transform #5759).
    • One review asked for a supported off switch, and the PR merged without it (Adaptive partial aggregation can shuffle 10x more rows than 1.0.0, with no supported way to turn it off #6466).

    Ten of the 18 PRs that introduced these had a single approver.

    #6516 turns these lessons into checks in the review-comet-pr, review-comet-expression-pr and review-comet-memory-pr skills.
11. andygrove

    added a commit that references this issue on Oct 1, 2026

    docs: check behavior against the latest release and add the 1.1.0 aud…

    ...

    9124348
12. 0lai0

    added a commit that references this issue on Oct 3, 2026

    docs: check behavior against the latest release and add the 1.1.0 aud…

    ...

    9c0fde0