DDIR inspect: format each record before writing to stderr - #910
Merged
Merged
Conversation
Reuse a record buffer per operator so nested Debug formatting does not issue many unbuffered writes under the global stderr lock. Preserve record formatting and synchronous output. Co-Authored-By: Astra (OpenAI Codex) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
inspecton both backends wrote each record witheprintln!while formatting it. For nestedDebugvalues that is many small unbuffered writes to stderr, each taking stderr's global lock, and with several workers they queue behind one another. This formats each record into aStringthe operator reuses, then writes it with oneeprint!. Output is unchanged: the same record syntax, still synchronous, and Rust test capture still works. The vec backend also stops cloning the value just to print it. Two files, +15/−2.Measured on the stock DDIR tour, which inspects heavily (52,814 records). Three paired runs each, order reversed in the middle run, M4 release build. In every pair the inspected records are identical byte for byte (compared as sorted multisets), on both backends at 1 and 4 workers, and the final query outputs match the existing oracle.
Memory cost. Peak footprint goes up. For vec at 1 worker the rise is consistent: 35.11–35.19 → 41.00–41.13 MiB. Corgi at 1 worker is about 0.7 MiB higher. Each operator's buffer keeps the capacity of its largest record, but this measurement does not show that the buffer accounts for the whole increase. The 4-worker peaks vary between runs.
Other workloads. SCC initial and churn times are within 0.7%. The AoC runner has the same median elapsed time (2.13 s at 1 worker, 2.24 s at 4 workers).
How this was found. Removing
inspectfrom the tour takes its first tick from about 1.5 s to 5–15 ms. Most of the tour's measured time was stderr formatting, not dataflow work.This makes
inspectcheaper. It does not make it the right way to get data out: we'd like a columnar path for results, as a separate piece of work.Tests.
cargo test --release -p interactive: 104 passed, 5 ignored (all ignored before this change too). All 33 AoC answers match. The supported sessions match on both backends at 1 and 4 workers.Measurements and the change are by Astra (OpenAI Codex). I re-ran the tests on master-next 9cfc909.
🤖 Generated with Claude Code