Conversation
This was referenced Jul 6, 2026
samtalki
force-pushed
the
agent/cdata-import-breadth-v2
branch
2 times, most recently
from
July 8, 2026 17:46
e306a2b to
fd30d90
Compare
samtalki
force-pushed
the
agent/cdata-import-breadth-v2
branch
from
July 8, 2026 19:53
fd30d90 to
e84c39d
Compare
Add immutable ArrowSchema and ArrowArray ABI structs plus C Data import for null and primitive arrays. Validate pointer layout, child counts, buffer counts, offsets, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Trust declared null counts like Arrow C++ and arrow-rs, require a validity bitmap for unknown null counts like nanoarrow, and ignore reserved flag bits for forward compatibility. Move the base ArrowSchema and ArrowArray structures into Julia owned storage at import and mark the sources released without calling their callbacks, following the C Data Interface move semantics used by arrow-rs from_raw and nanoarrow ArrowArrayMove. Callers may free or reuse the passed structures once from_c_data returns; release_c_data or finalization releases the moved copies through the producer callbacks. Keep imported buffers rooted behind a shared release owner. Copy misaligned fixed width buffers into aligned Julia storage before typed access, and materialize Julia owned data from copy, collect, deepcopy, and serialize so no read path bypasses the release liveness gate. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
samtalki
force-pushed
the
agent/cdata-import-breadth-v2
branch
from
July 12, 2026 20:27
e84c39d to
d12b4e7
Compare
…tics Address Kou's review of the primitive importer. Report actual and expected buffer and child counts, name the schema child count explicitly, and describe dictionary NULL consistency accurately. Document host-dependent ABI padding, the format scan policy, finalizer re-registration, indexed access, and Julia deepcopy after the C header move. Add a C compiler ABI probe and regression coverage for scan boundaries, bit-offset null counts, release field preservation, finalizer retry, detached copies, and validation messages. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Generated-by: OpenAI Codex
Add C Data import for struct arrays on top of the null and primitive array importer. Root structs are exposed as Tables.jl column tables with DataAPI metadata; nested structs import as NamedTuple element vectors with parent validity masking and parent offsets applied to the children. Parse field names and schema metadata from the C schema for the imported table. Validate child pointer layout, child counts against the schema, struct child lengths, name and metadata bounds, and nesting depth, bound validation with a total node budget so aliased or cyclic child pointers cannot make it explode combinatorially, and reject duplicate struct field names, which NamedTuple rows and column lookup cannot represent. Adjust dictionary encoding offsets for non 1-based reference pools so categorical pools with missing values keep correct dictionary indices. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add import support for boolean, string, binary, list, fixed size, temporal, timestamp, duration, interval, and decimal C Data formats. Extend malformed input, offset, null count, release, copy, collect, metadata, and GC rooting coverage for the broader importer. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
samtalki
force-pushed
the
agent/cdata-import-breadth-v2
branch
from
September 8, 2026 14:59
d12b4e7 to
8e9e579
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR builds on #603 and expands
Arrow.from_c_dataimport coverage.Stack
Base remains
mainbecause these cross fork PRs are not represented as true stacked bases on GitHub.Depends on #603, currently
1b94281.Incremental review link:
samtalki/arrow-julia@agent/cdata-import-foundation...agent/cdata-import-breadth-v2
Size
Incremental over #603: 2 files changed, 905 insertions, 19 deletions.
Total against
main: 5 files changed, 2661 insertions, 4 deletions.Validation
git diff --check agent/cdata-import-foundation...HEAD: passed.AI Assistance
Co-authored-by: Robert Buessow robert.buessow@relational.ai
Co-authored-by: Olle Martensson olle.martensson@gmail.com
Generated-by: OpenAI Codex