Rewrite Arrow.jl internals for 3.0

[quinnj]

Summary

• preserve per-batch dictionary snapshots, accept undeclared stream replacements, and read dictionary deltas in streams, files, and range scans; resume Arrow.append from the final dictionary state even when an update follows the last record batch; fixes Arrow.Table uses last dictionary for all batches in streaming decode, resulting in incorrect reads #610
• replace the Arrow 2.x internals with a validated core, new IPC reader and writer, C Data and C Stream support, and a materialized public table facade
• compile each Tables.Scan once, keep storage and public-domain plans separate, preserve stable empty-result schemas, and support sparse byte-range reads with footer statistics
• deepen column construction around one recursive adapter for native, retained-schema, dictionary, Union, and ArrowTypes-backed columns
• retain automatic ArrowTypes custom serialization, public-domain lifting, extension metadata, and nested custom-type round trips
• use registered DataStrings 1.0 as the StringView representation package and keep Arrow.jl's export surface small
• use DataDecimals 1.0 and Durations 1.4 for supported top-level decimal, interval, and timestamp columns; every zone-naive timestamp reads as Durations.Timestamp{P} at the column unit (a written DateTime column reads back as an instant-equal Timestamp{Millisecond}) and every timezone-declared timestamp as Durations.ZonedTimestamp{P,Z} (no TimeZones.jl requirement), both writing back zero-conversion with every comparison operator pushing down; preserve wire descriptors on rewrite
• make the public value domain uniform at every depth: temporal, decimal, and interval leaves materialize as the same public values inside struct, list, map, and union rows and under run-end-encoded and dictionary wrappers as at the top level; the writer accepts them back at every depth (fresh and retained), scan literal lowering is transparent through run-end encoding, and the reader allocation budget covers the dynamic conversion work at every seam (docs/dev/DESIGN-nested-public-values.md is the design contract)
• discard inherited batch statistics when rewriting edited tables so filters cannot skip changed rows
• reject invalid writer options before opening destination files, release owned files after failures, and preserve the original error
• save only referenced Utf8View/BinaryView bytes, retain shared ranges, and avoid sorting and per-entry copies when referenced bytes already cover each copied buffer prefix
• remove JSON3 and StructTypes integration and support claims
• deepen deterministic fuzzing, Apache Arrow corpus coverage, PyArrow/nanoarrow oracles, C Data checks, allocation limits, trim checks, documentation, and package administration

Current validation

Latest head: 0ffdf1436664228629cbe619249011a415d216f7.

Appending to a valid IPC stream with a trailing dictionary replacement or delta previously either wrote incorrect values or rejected a matching dictionary. The reader now retains the final dictionary map; append resumes that map while earlier record batches keep their original snapshots. Public file and IO regressions cover replacements, deltas, and uncompressed/LZ4/Zstd streams.

• Full package suites pass on Linux Julia 1.10.12 and 1.13.0 with four threads and bounds checks, including CloudStore integration, all four acceptance batteries, and Aqua.
• All 624 focused dictionary and incremental-writer assertions pass on Julia 1.10.12 and 1.13.1. These include 90 assertions covering trailing updates and refusal of a mismatching pool without altering the destination.
• Six independently generated PyArrow 25.0.1 streams reproduce the append defect on the prior head; the candidate passes all 30 public reproduction controls. PyArrow full validation confirms exact values and row counts in all six output streams after two public appends each.
• Pinned Apache Arrow corpus: 388 pass, 0 fail, 63 declared skips. PyArrow 25.0.1/nanoarrow 0.9.0 oracle: 315 pass, 0 fail, 20 declared skips. C Data/C Stream oracle: 147 pass, 0 fail, 8 declared skips. These ran on Linux Julia 1.12.6.
• Strict Documenter 1.19.0 and the exact CI JuliaFormatter 2.12.4 check pass.
• An independent private review covered the complete runtime/test diff and coverage workflow changes.

All 55 hosted checks pass at 0ffdf1436664228629cbe619249011a415d216f7: both CI runs and the conformance run completed successfully. All 34 native coverage uploads were accepted for apache/arrow-julia at that exact commit, and all 34 reports finished processing, with aggregate coverage at 91.93%. Prior green job logs show coverage uploads were rejected with Token required because branch is protected. Uploading jobs now use scoped OIDC for repository-owned runs and fail on upload errors; public forks retain tokenless upload support. Coverage percentage thresholds and processing paths are unchanged. The notification count now matches one 17-report CI run, so main commits and branches without a duplicate PR run can report coverage; all 34 uploads from this PR's push and PR runs were independently verified. The scheduled nightly upload path receives the same authentication repair; scheduled execution is not part of the local validation above.

Dependency

JuliaData/Tables.jl#380 is merged and released as Tables.jl 1.14.0. Commit dd8c272 removes the temporary source and CI pins, raises the Tables.jl compatibility floor to 1.14, and resolves Tables from the General registry everywhere (package, CI workflows, conformance image, docs).

Breaking changes

This is the Arrow.jl 3.0 rewrite. Important changes include:

• Julia 1.10 is now required.
• Arrow.Table and Arrow.Stream return materialized Julia vectors instead of lazy Arrow vector wrappers.
• Arrow.write is eager and whole-buffer. Arrow.Writer and IPC-stream append are reimplemented with fixed-schema semantics; the curried write and tobuffer compatibility forms remain. Obsolete writer tuning keywords are ignored with warnings.
• Arrow.write(io, table) writes file format by default; use file=false for stream format.
• ArrowTypes-based custom serialization remains automatic. ArrowTypes stays exported only as an Arrow 2.x compatibility exception. Packages that define mappings should depend on and import ArrowTypes directly. Arrow.ArrowTypes remains as a qualified compatibility binding.
• most APIs are no longer exported and must be called through Arrow.

Fresh declared heterogeneous Julia Union columns are written as canonical dense Arrow Unions. A retained Union rewrite fails closed when materialization has lost its original route. Nested retained dictionary pools follow the same fail-closed rule.

See CHANGELOG.md and docs/src/migration.md for the complete list and migration guidance.

Release preparation

• DataStrings, DataDecimals, and Durations 1.4 (Timestamp/ZonedTimestamp) resolve from General; the bundled ArrowStrings package and its release steps are removed
• the Apache release-candidate template and CI now enforce a vote window of at least 72 hours
• Arrow.jl remains at 3.0.0-DEV pending the rewrite merge and Apache release process

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.

🤖 Generated with Claude Code

[davidanthoff]

Very cool!

One question: could ArrowCore.jl be a real standalone, dependency-free package, similar to how ArrowStrings.jl is split out already? From reading the code, it looks like ArrowCore.jl could contain what it has at the moment in this branch, plus cdata.jl, and the uncompressed IPC read/write path (with the vendored FlatBuffers/Meta layer). And the Tables.jl facade, ArrowTypes lowering, and the compression codecs could then stay in Arrow.jl on top of it.

As far as I can tell this split already exists at the code level: ArrowCore only imports Mmap, cdata.jl only uses ArrowCore, and neither ipc_read.jl nor ipc_write.jl references Tables at all — the codecs only appear at the buffer (de)compression step. So ArrowCore.jl would depend on Base and stdlibs only: no binary deps (Lz4_jll/Zstd_jll), no Tables.jl. The two places where the package boundary doesn't quite match the code today seem small: EnumX (but Meta is generated, so the generator could emit self-contained enum modules instead), and compression (which could be a plain dispatch seam — ArrowCore.jl defines compress/decompress generics and errors helpfully on compressed buffers, Arrow.jl keeps its hard codec deps and implements them; so using Arrow behaves exactly as it does now.

The reason I'm asking: we could use this in the Julia VS Code extension. The REPL process could use ArrowCore.jl and ArrowStrings.jl to hand large tables to the extension's interactive table viewer — write an uncompressed IPC file, and the TypeScript side reads it with Arrow JS, with the file-format footer giving the viewer random access to just the visible row range. That would be really nice! But the way we ship code that loads into the user's REPL process rules out anything with a binary dependency. And we also couldn't use anything that pulls in Tables.jl, because that would pin the Tables.jl version users see in their REPL to whatever we ship. So this could only work if there is a core package without those deps.

(I'd also have some uses for this in Queryverse, but the VS Code one probably the more interesting one)

And, as all ideas these days, heavily Claude helped ;)

[quinnj]

I think it's close to that, but currently we still have the new ArrowStrings.jl package which provides the native offset + string view representation. We want that as a stand-alone package because I want to be able to use it as a dep for CSV.jl to do direct parse into that representation, I.e. we can parse a csv file directly into a zero-copy arrow representation.

So not quite zero-dependency.

I'm also not sure just ArrowCore.jl would be quite enough to have arrow JS be able to read easily. Currently we have the CData & IPC layers on top of the core buffers to actually fulfill those interfaces.

I do have another project/data format I've been working on that would be a much cleaner/simpler binary table format with cross-language implementations. I've been slowly iterating on it, but it could potentially be a really good fit for this.

[davidanthoff]

Yeah, I also think it is super close.

I think the ArrowStrings.jl would not be a problem, i.e. if ArrowCore.jl were to depend on ArrowStrings.jl that would work (we use a trick for that kind of situation a lot in the extension). The main thing is really that the compression stuff would have to not be in there, because we can't handle the binary dependencies that are pulled in.

And yes, the cdata and ipc layers would have to move into ArrowCore.jl as well. Claude thinks that would be easy, but who knows ;)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.66667% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.93%. Comparing base (3712291) to head (0ffdf14).
⚠️ Report is 55 commits behind head on main.

Files with missing lines	Patch %	Lines
ext/ArrowTimeZonesExt.jl	86.95%	3 Missing ⚠️
src/ArrowTypes/test/tests.jl	0.00%	3 Missing ⚠️
src/ArrowTypes/src/ArrowTypes.jl	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #609      +/-   ##
==========================================
+ Coverage   87.43%   91.93%   +4.49%     
==========================================
  Files          26       27       +1     
  Lines        3288    12367    +9079     
==========================================
+ Hits         2875    11369    +8494     
- Misses        413      998     +585     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[quinnj]

The trailing-dictionary append repair and coverage authentication changes are fully validated at 0ffdf1436664228629cbe619249011a415d216f7. Both hosted CI runs and conformance passed all 55 jobs. Every one of the 34 native coverage uploads was accepted for this repository and commit; all 34 reports finished processing (91.93% aggregate coverage). Local suites, independent PyArrow round trips, and conformance results are recorded in the PR description.

The rewrite remains draft and unmerged. The scheduled nightly authentication change has not been exercised by a fresh scheduled run.

AI disclosure: This work was prepared with assistance from OpenAI Codex.