Repository navigation
[python] Use Native scan statistics in explain - #10609
Merged
JingsongLi merged 8 commits intoOct 17, 2026
Merged
Conversation
JingsongLi
marked this pull request as ready for review
October 17, 2026 00:51
sundapeng
added a commit
to sundapeng/paimon
that referenced
this pull request
Oct 17, 2026
Native CI builds pypaimon-rust from paimon-rust main, and Python Tests skip native planning tests, so no job forces native planning through a released binding. That is how apache#10609 broke 0.4.0 without notice: every native plan failed on Plan.stats() and silently fell back to the Python scanner. Native CI / pypaimon-rust 0.4.0 installs the release from PyPI and runs the regular suite with native planning forced. Tests marked native_plan need Rust main, so they stay in the main job. On the previous commit this job fails test_native_plan_retains_independent_payload_column_files and plans natively only twice.
sundapeng
added a commit
to sundapeng/paimon
that referenced
this pull request
Oct 17, 2026
…ts() apache#10609 calls rust_plan.stats(), which pypaimon-rust 0.4.0, the release the Python test jobs install, does not have. The AttributeError sends every native plan back to the Python scanner, so ContiguousWindowPlanTest.test_native_plan_retains_independent_payload_column_files fails on master. Read the counters only when the binding has them, as snapshot_id already does; such plans report no native statistics.
sundapeng
added a commit
to sundapeng/paimon
that referenced
this pull request
Oct 17, 2026
Native CI builds pypaimon-rust from paimon-rust main, and Python Tests skip native planning tests, so no job forces native planning through a released binding. That is how apache#10609 broke 0.4.0 without notice: every native plan failed on Plan.stats() and silently fell back to the Python scanner. Native CI / pypaimon-rust 0.4.0 installs the release from PyPI and runs the regular suite with native planning forced. Tests marked native_plan need Rust main, so they stay in the main job. On the previous commit this job fails test_native_plan_retains_independent_payload_column_files and plans natively only twice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Use Native planning counters in PyPaimon's existing explain flow and exercise Java-compatible fixed-bucket pruning through Native scans.
Depends on apache/paimon-rust#1140. This PR remains draft until the API is available on Rust main, which Native CI builds.
Changes
Rust owns core counter collection, schema statistics, Data Evolution projection and split planning. Python converts and displays the result, reconciling final totals after its existing post-processing and deferred LIMIT. No new format support.
Verification