Repository navigation

feat: Joom 1.1-1 — Delta scan, correctness fixes, wide-row performance, cost-based engine choice - #2

Merged

msafonov merged 72 commits into

Oct 3, 2026

Commits on Sep 26, 2026

fix: retain Comet execution for warehouse workaround regressions

notimesea
authored and
msafonov
committed
65be8b7
fix: preserve zero partitions when native scan files are pruned

notimesea
authored and
msafonov
committed
45fad9e

Commits on Sep 27, 2026

revert: drop the sort merge reservation cap

msafonov
and
claude
committed
d153b3a
feat: spill all non-streaming native window shapes

msafonov
and
claude
committed
e90fcec
test: compare spilled window shapes with Spark

msafonov
and
claude
committed
e435466
test: reproduce native sort failing to spill after the Spark share shrinks

msafonov
and
claude
committed
943936f
build: vendor datafusion-physical-plan 55.1.0 unchanged

msafonov
and
claude
committed
6eaf12e
fix: keep native sort spill memory when the Spark share shrinks

msafonov
and
claude
committed
880575b
fix: keep the spill merge's headroom across all merge passes

msafonov
and
claude
committed
b3cd74a
fix: reserve the encoded rows a merge keeps for reuse

msafonov
and
claude
committed
d7d47f3
fix: count a single-column merge's sort key once

msafonov
and
claude
committed
d82e3a3
fix: leave as much memory again for the final spill merge's consumers

msafonov
and
claude
committed
69e6389
fix: let a sort's spill merge split a run batch wider than its budget

msafonov
and
claude
committed
a4dc994
fix: keep a sort spill's batches reserved until they are written

msafonov
and
claude
committed
426ca8e
fix: count buffers a sort's chunks share once

msafonov
and
claude
committed
1b8d480
fix: reserve a buffer a sort's buffered batches share once

msafonov
and
claude
committed
6ae0323
fix: keep the merge headroom for a sort's final in-memory merge

msafonov
and
claude
committed
5b75a38

Commits on Sep 28, 2026

fix: reserve what a sort's spill merge holds for each run

msafonov
and
claude
committed
399e6c9
fix: restore sort merge reservation cap and native test contract

msafonov
and
codex
committed
4ff3708
fix: allow small Comet batches and cap JVM shuffle batches at use time

msafonov
and
codex
committed
ea38aad
debug: measure wide batches at shuffle decode and FFI output

msafonov
and
codex
committed
ca6b963
test: bound unreserved sort output at the JVM handoff

msafonov
and
codex
committed
5f6cbb8
test: measure wide binary payload copies in sort kernels

msafonov
and
codex
committed
39f49a0
fix: avoid repeated copies of wide binary payloads in native sort

msafonov
and
codex
committed
be99cc1
test: measure dictionary expansion before wide binary sort

msafonov
and
codex
committed
5da7791
fix: select direct shuffle reads during initial planning

msafonov
and
codex
committed
4956d83
test: benchmark binary columnar-to-row allocation overhead

msafonov
and
codex
committed
c0b0448
perf: avoid intermediate binary copies in row materialization

msafonov
and
codex
committed
3d99265
test: cover default Kryo broadcast payloads

msafonov
and
codex
committed
a52c168
perf: reuse unique dictionary values instead of casting in JVM shuffle row conversion

msafonov
and
claude
committed
d9ea392
feat: revert isolated native operators feeding Spark operators

msafonov
and
claude
committed
2eda535

Commits on Sep 29, 2026

feat: run each query stage in one engine and pick shuffle formats per boundary

msafonov
and
claude
committed
edaf6ca
perf: evaluate small window partitions together and coalesce window output

msafonov
and
claude
committed
95dc9ae
fix: do not fail an aggregate whose first reservation was refused when it spills

msafonov
and
claude
committed
85bef53
fix: let the task's other memory consumers make the JVM shuffle writer spill

msafonov
and
claude
committed
b65fdbe
revert: remove UnifyStageEngines

msafonov
and
claude
committed
0a92f71
revert: remove RevertIsolatedNativeOperators

msafonov
and
claude
committed
4f92985
feat: pick shuffle and broadcast formats from the engines on both sides

msafonov
and
claude
committed
01df400
feat: choose each operator's engine by a cost over the whole plan

msafonov
and
claude
committed
fd190a6
feat: native Delta scan (port of apache/datafusion-comet#5365)

msafonov
and
claude
committed
c69bd9c
fix(delta): claim GCS scans whose Hadoop auth keys only select ADC

msafonov
and
claude
committed
a435236
fix: do not hash decimals above precision 18 in native shuffle (port of apache/datafusion-comet#6005)

msafonov
and
claude
committed
88cfb29
feat: run sorts of wide rows with narrow keys in Spark when Spark reads them

msafonov
and
claude
committed
2d3cc89
feat: move sorts with variable-width payloads to Spark on the initial plan

msafonov
and
claude
committed
909335e
test: time native sorts of wide binary payloads end to end

msafonov
and
claude
committed
3b37b4a
perf: sort the keys of wide rows and gather each payload once in native sort

msafonov
and
claude
committed
d796f09

Commits on Sep 30, 2026

fix: gather a native sort's output from the batches it references

msafonov
and
claude
committed
f66011d
refactor: keep wide-row sorts in Spark across AQE re-plans without per-thread state

msafonov
and
claude
committed
9ff3736
fix: count the encoded rows a merge keeps instead of summing them per batch

msafonov
and
claude
committed
b2771dd
perf: coalesce small input batches of a native sort before buffering them

msafonov
and
claude
committed
f7a13eb

Commits on Oct 1, 2026

fix(delta): do not resolve subqueries when planning reads the scan's partitioning

msafonov
and
claude
committed
0e17ce3
fix: rebase the offsets of sliced build batches when coalescing a broadcast

msafonov
and
claude
committed
d913126
fix: keep the streamed order of a sort-merge join with a join filter

msafonov
and
claude
committed
f761b31
perf: coalesce small blocks when reading a Comet shuffle

msafonov
and
claude
committed
2d30872
feat: keep shuffles of wide rows in Spark

msafonov
and
claude
committed
c2562fb
perf: reuse generated columnar-to-row projections across tasks

msafonov
and
claude
committed
9560d4a
feat: decide wide-row sorts and shuffles by leaf columns

msafonov
and
claude
committed
b83e917

Commits on Oct 2, 2026

fix: spill a large in-memory sort before producing its output

msafonov
and
claude
committed
568955d
perf: gather wide shuffle rows from the batches each partition references

msafonov
and
claude
committed
94abcc0
feat: price the cost-based engine choice by rows and leaf columns

msafonov
and
claude
committed
d1148b2
feat: price the cost-based engine choice per row and decide it on every plan

msafonov
and
claude
committed
cea0ae5
feat: enable the cost-based engine choice by default

msafonov
and
claude
committed
f9793a7

Commits on Oct 3, 2026

feat: price the cost-based engine choice with the calib29 measurements

msafonov
and
claude
committed
1f7b89b
fix: clone the shared sort source with Arc::clone in the handoff test

msafonov
and
claude
committed
6b11f2e
feat: keep the cost-based engine choice disabled by default

msafonov
and
claude
committed
71a7c07
test: give the child sort spill metrics test less memory so the sort spills

msafonov
and
claude
committed
2b28fed
feat: keep PartitionAggregateWindowExec disabled by default

msafonov
and
claude
committed
bc64dfc
fix: pool C2R projections only after every reader of their rows is done

msafonov
and
claude
committed
8e06527
ci: list the missing suites so check-suites.py passes

msafonov
and
claude
committed
bb53574
style: drop a needless string interpolator flagged by scalafix

msafonov
and
claude
committed
c8b47b4
build: compile with Scala 2.13 and Rust 1.99

msafonov
and
claude
committed
1a52c8c
style: drop an unused pattern binding flagged by scalafix on Scala 2.13

msafonov
and
claude
committed
4f150e1