Skip to content

feat(wasm)!: support large-file downloads and disk streaming - #214

Open
mickvandijke wants to merge 2 commits into
mainfrom
feat/wasm-large-file-downloads
Open

mickvandijke wants to merge 2 commits into
mainfrom
feat/wasm-large-file-downloads

Conversation

@mickvandijke

@mickvandijke mickvandijke commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Files larger than 1 GB are currently rejected even when a browser reads them in bounded ranges. Removing that check alone would still leave 32-bit file offsets on wasm32. This change uses 64-bit file positions and adds BrowserFileReader.pipeTo() for public/private files, with bounded writes, backpressure, cancellation and writer cleanup.

In-memory downloads fill a JavaScript buffer through bounded WASM reads, accept an optional output-memory budget, and report allocation failures with the streaming alternative. Manifest validation accepts larger files and more than 1,024 chunks. The companion SDK and upload limits are outside this PR.

Linear issue

Closes V2-1398

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Proposed for reviewer confirmation: client download buffering, offset validation and destination handling; no wire, node, storage-format or payment changes.

Compatibility

  • Wire: none; existing WebRTC requests and protocol verification are retained.
  • Storage: none; native DataMaps, ciphertext, per-chunk KDF and BLAKE3 verification are retained.
  • API: Rust browser descriptor PublicFileDescriptor.size changes from usize to u64; affected Rust WASM-export signatures change. JavaScript file sizes and offsets remain numbers, now validated as safe integers. pipeTo, the optional download memory budget, and native data_download_range_u64 are additive. Native data_download_range retains its signature. Progress text for complete WASM downloads now reports byte counts.

Semver impact

  • breaking
  • feature
  • fix

Breaking is proposed for Rust browser API consumers; existing JavaScript call arities remain supported.

Test evidence

  • Generated WASM on the rebased branch: 190 tests passed, including a complete 4,303,347,835-byte native-encrypted file streamed with a matching BLAKE3 hash and writes no larger than 4 MiB.
  • Large-file cases cover public/private seeks across 4 GiB, actual disk output, write backpressure, cancellation, writer errors, allocation failures, memory budgets and invalid offsets.
  • Native shared engine: 130 tests passed; browser manifest validation: 3 tests passed.
  • Rust formatting, WASM lint and ADR governance passed. Native library/example lint passed before rebasing onto current main.
  • WASM regressions run in Node with a mocked WebRTC host and real Rust protocol/crypto processing. A production compatibility smoke and real-browser integration have not been run for this change.
  • Test fixture and reproducible native generator: ant-core/wasm-tests/large-download.test.mjs and ant-core/examples/generate-browser-large-file.rs.

New dependency

None.

ADR

ADR-0004: large-file download destinations. The existing Proposed ADR is amended and remains Proposed.

Mitigation / rollback

Revert this PR and regenerate consuming WASM artifacts. Applications can keep using bounded readRange and set an explicit output-memory budget. Root-map/index memory, destination capacity and JavaScript's safe-integer range (just under 8 PiB) remain practical limits; disk output requires a caller-provided writable destination.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant