Skip to content

fix: [branch-1.0] zero sliced boolean offsets at every level before exporting to the JVM (#6339) #6579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andygrove merged 2 commits into from
Oct 4, 2026
Merged

fix: [branch-1.0] zero sliced boolean offsets at every level before exporting to the JVM (#6339) #6579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9fe07d
fix: zero sliced boolean offsets at every level before exporting to t…
andygrove Sep 29, 2026
6647d0f
fix: support Rust 1.99 lints
andygrove Oct 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/source/contributor-guide/ffi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,18 @@ pub extern "system" fn Java_..._exportVector(
}
```

### Array Offsets

Arrow Java's C Data import ignores `ArrowArray.offset` at every level
([apache/arrow-java#88](https://github.com/apache/arrow-java/issues/88)) and reads each buffer from its start. arrow-rs
folds a slice into the buffers for almost every type, but a sliced `BooleanArray` keeps its bit offset, and a struct
exports offset 0 even when its children are sliced. So every array native exports to the JVM first goes through
`zero_offsets` (in `native/common/src/ffi_offsets.rs`), which re-slices boolean bitmaps to start at bit 0 at every
level and shares every other buffer. `move_to_spark` applies it to executed batches and decoded shuffle blocks, and
`JvmScalarUdfExpr` applies it to the inputs of the JVM UDF bridge. A new native to JVM export path has to call it too,
or sliced booleans reach the JVM misaligned
([#6288](https://github.com/apache/datafusion-comet/issues/6288)).

### Wrapper Object Lifecycle (Native → JVM)

```
Expand Down
34 changes: 17 additions & 17 deletions native/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading