Conversation
Resolve missing CloudStore metadata once and require a strong ETag before reading ranges. Check that the refreshed size agrees with the supplied handle, so same-size replacement fails with HTTP 412 instead of mixing versions. Generated-by: OpenAI Codex
Retain provider responses and reject mismatched ranges, object versions, content encodings, and byte counts before Arrow interprets the bytes. Keep the released CloudStore request paths and disable HTTP decompression so returned bytes retain their stored offsets. Cover equal-length wrong-column responses and protocol faults through both provider clients, and exercise the signed S3 and Azure Table and Stream paths. Generated-by: OpenAI Codex
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arrow cloud reads could combine different versions of an object or accept bytes from the wrong range. In a reproduced scan, selecting column
a = 1:30000returned columnb's-7values: the server returned an equally sized slice, and the extension discarded theContent-Rangethat identified it.Pin a strong ETag before reading, refreshing missing metadata once and checking the known size. Retain each range response and require HTTP 206, exact range bounds and total size, the pinned ETag, and the requested byte count. Request the identity representation, disable HTTP decompression, and reject encoded responses. Whole-object ranges follow the same checks; empty reads send no request. Complete object handles need no extra metadata request, and overwritten objects still surface HTTP 412.
This targets
core-rewriteand depends on #609. The request path is verified against released CloudStore 1.6 and 1.8, with no runtime dependency or compatibility changes. Validation follows normal HTTP body buffering; it does not add a hard network receive limit.Validation:
git diff --checkalso pass.The research, implementation, and tests were AI-driven with OpenAI Codex, including an independent agent review. The broader Arrow rewrite in #609 still needs upstream review.
Co-authored by Codex