feat(wasm)!: support large-file downloads and disk streaming - #214
Open
mickvandijke wants to merge 2 commits into
Open
mickvandijke wants to merge 2 commits into
mickvandijke wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Files larger than 1 GB are currently rejected even when a browser reads them in bounded ranges. Removing that check alone would still leave 32-bit file offsets on wasm32. This change uses 64-bit file positions and adds
BrowserFileReader.pipeTo()for public/private files, with bounded writes, backpressure, cancellation and writer cleanup.In-memory downloads fill a JavaScript buffer through bounded WASM reads, accept an optional output-memory budget, and report allocation failures with the streaming alternative. Manifest validation accepts larger files and more than 1,024 chunks. The companion SDK and upload limits are outside this PR.
Linear issue
Closes V2-1398
Risk tier
Proposed for reviewer confirmation: client download buffering, offset validation and destination handling; no wire, node, storage-format or payment changes.
Compatibility
PublicFileDescriptor.sizechanges fromusizetou64; affected Rust WASM-export signatures change. JavaScript file sizes and offsets remain numbers, now validated as safe integers.pipeTo, the optional download memory budget, and nativedata_download_range_u64are additive. Nativedata_download_rangeretains its signature. Progress text for complete WASM downloads now reports byte counts.Semver impact
Breaking is proposed for Rust browser API consumers; existing JavaScript call arities remain supported.
Test evidence
ant-core/wasm-tests/large-download.test.mjsandant-core/examples/generate-browser-large-file.rs.New dependency
None.
ADR
ADR-0004: large-file download destinations. The existing Proposed ADR is amended and remains Proposed.
Mitigation / rollback
Revert this PR and regenerate consuming WASM artifacts. Applications can keep using bounded
readRangeand set an explicit output-memory budget. Root-map/index memory, destination capacity and JavaScript's safe-integer range (just under 8 PiB) remain practical limits; disk output requires a caller-provided writable destination.