Conversation
Add immutable ArrowSchema and ArrowArray ABI structs plus C Data import for null and primitive arrays. Validate pointer layout, child counts, buffer counts, offsets, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Trust declared null counts like Arrow C++ and arrow-rs, require a validity bitmap for unknown null counts like nanoarrow, and ignore reserved flag bits for forward compatibility. Move the base ArrowSchema and ArrowArray structures into Julia owned storage at import and mark the sources released without calling their callbacks, following the C Data Interface move semantics used by arrow-rs from_raw and nanoarrow ArrowArrayMove. Callers may free or reuse the passed structures once from_c_data returns; release_c_data or finalization releases the moved copies through the producer callbacks. Keep imported buffers rooted behind a shared release owner. Copy misaligned fixed width buffers into aligned Julia storage before typed access, and materialize Julia owned data from copy, collect, deepcopy, and serialize so no read path bypasses the release liveness gate. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #607 +/- ##
==========================================
+ Coverage 87.43% 88.16% +0.72%
==========================================
Files 26 28 +2
Lines 3288 3666 +378
==========================================
+ Hits 2875 3232 +357
- Misses 413 434 +21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| const ARROW_FLAG_NULLABLE = Int64(2) | ||
| const ARROW_FLAG_MAP_KEYS_SORTED = Int64(4) | ||
|
|
||
| const _CDATA_MAX_FORMAT_BYTES = 4096 |
There was a problem hiding this comment.
I don't object this for safety but is 4096 reasonable limit?
There was a problem hiding this comment.
I believe so, but I may be wrong. Documented the choice and added boundary tests in ca63747.
…tics Address Kou's review of the primitive importer. Report actual and expected buffer and child counts, name the schema child count explicitly, and describe dictionary NULL consistency accurately. Document host-dependent ABI padding, the format scan policy, finalizer re-registration, indexed access, and Julia deepcopy after the C header move. Add a C compiler ABI probe and regression coverage for scan boundaries, bit-offset null counts, release field preservation, finalizer retry, detached copies, and validation messages. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Generated-by: OpenAI Codex
|
Resolved and restacked. Thanks for your patience @kou |
|
@samtalki thanks for all the work on this stack, including splitting it up for review and working through the ownership, release, and validation details with @kou. You mentioned needing this for PowerIO.jl. If you’re interested, I’d welcome your feedback on whether the new C Data API covers that use case. There’s no expectation that you rework this whole stack against the new internals. My goal w/ the rewrite though was to make the C data model the core of everything since it's the more foundational layer, instead of building it on top of the, IMO, now dated architecture of the original Arrow.jl. It allows us to easily build the C data, stream, IPC, file formats, and normal Julia interop (Tables.jl, etc.) as small layers/adapters on top. |
Field metadata now crosses the C boundary in both directions, per the C data interface encoding (int32 pair count, then length-prefixed key and value bytes, native endian): _cmetadata! encodes on export (NULL when absent), _import_cmetadata parses on import — recursively through children and dictionary value fields. Counts and lengths are producer-declared like every other C Data pointer, but negative values refuse (they would wrap the walk). This closes the functional gap the C-data PR research recorded (schema metadata neither imported nor exported). Trim note: a Union-typed keyword makes the kwcall tuple imprecise — the import branches on the metadata's presence. C-string imports are now bounded: a missing NUL terminator refuses cleanly at 1 MiB instead of scanning memory unbounded — adopted from Samuel Talkington's C-data hardening in apache/arrow-julia#607. Pins: battery metadata round-trips (nested children, dictionary fields, absent-metadata NULL, hostile negative-length blobs) and a metadata round-trip in the trim workload (zero verifier errors, zero warnings). Co-authored-by: Samuel Talkington <talkington.samuel@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary
First PR of the C Data Interface stack for GH-184, extracting null and primitive array import from #603 as requested there.
ArrowSchemaandArrowArrayC layout structs with layout assertions.Arrow.from_c_data(schema_ptr, array_ptr; convert=true)for null and primitive arrays.null_count == -1requires the validity bitmap it is resolved from, like nanoarrow; reserved flag bits are ignored.ArrowSchemaandArrowArraystructures into Julia owned storage and marks the sources released without calling their callbacks; callers may free or reuse the passed structures oncefrom_c_datareturns.copy,collect,deepcopy, andserializematerialize Julia owned data through the release liveness gate.Struct, boolean, string, binary, list, and export support land in the follow up PRs.
Stack
Base is
main.Follow up PRs:
Size
Against
main: 6 files changed, 1246 insertions, 1 deletion.Validation
git diff --check main...HEAD: passed.ArrowSchemaandArrowArrayare isbits structs with expected C layout sizes.AI Assistance
Co-authored-by: Robert Buessow robert.buessow@relational.ai
Co-authored-by: Olle Martensson olle.martensson@gmail.com
Generated-by: OpenAI Codex