Skip to content

[python] Use Native scan statistics in explain - #10609

Merged
JingsongLi merged 8 commits into
apache:masterfrom
JingsongLi:codex/native-scan-statistics
Oct 17, 2026
Merged

JingsongLi merged 8 commits into
apache:masterfrom
JingsongLi:codex/native-scan-statistics

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Use Native planning counters in PyPaimon's existing explain flow and exercise Java-compatible fixed-bucket pruning through Native scans.

Depends on apache/paimon-rust#1140. This PR remains draft until the API is available on Rust main, which Native CI builds.

Changes

  • Exercise Native same-key OR, repeated IN intersections, correlated partition predicates and composite bucket keys through the existing scan/read API.
  • Align Python's selector with Java's conservative treatment of pending buckets and predicates which become AlwaysFalse after partition substitution. Partition filtering rejects nonmatching partitions.
  • Native scans always collect lightweight counters. Use scan.plan() and plan.stats(); remove the collect_stats bridge flag and the Native with_stats call. Preserve early manifest/sidecar pruning and LIMIT stopping.
  • Keep snapshot metadata on Plan. Preserve counters through pinned snapshots, incremental windows, streaming conversion, post-processing and captured authorization without rerunning the Python scanner.
  • Keep a small Native stats handle and defer conversion to Python counters until Plan.stats() is called. Cache the converted result. Authorization wrapping preserves lazy conversion, and pickle materializes Python scalars instead of serializing the process-local stats handle.
  • Reconcile final split/file/byte totals and deferred LIMIT metadata with the returned Python Plan when statistics are first requested, retaining Native decode/prune counters. Empty incremental windows use the same complete zero-counter shape as an empty core Plan.
  • Preserve scan_with_stats() for both planners. Render actual Native work through ExplainResult.planning_counters. Full before/after funnels and Python identity sets are unavailable for Native because skipped metadata is not decoded.

Rust owns core counter collection, schema statistics, Data Evolution projection and split planning. Python converts and displays the result, reconciling final totals after its existing post-processing and deferred LIMIT. No new format support.

Verification

  • After the review fixes, expanded Native regression: 516 passed, 13 skipped, 122 subtests passed. Its remaining authorization mock was updated to use the complete Native counter factory; both affected unit suites then passed all 119 cases (46 subtests), including that authorization case.
  • Added 12 REST regression cases covering shard/slice with deferred LIMIT and empty timestamp windows: all 12 failed before the fix and passed afterward. Added a unit case preserving work counters, saturating final bytes, excluding unknown sizes and retaining zero LIMIT. Statistics remain lazy, cached and serializable.
  • Native regressions verify counters on ordinary plans, lazy conversion and reuse, pickle/JSON serialization, LIMIT stopping, historical/empty snapshots, cross-schema netting, Data Evolution projection, captured grants, and no data-file reads during explain.
  • Changed Python files passed repository Flake8 configuration and Python 3.6 syntax parsing.
  • Companion Rust verification: 2105 table-layer tests, 168 binding read/streaming tests and 6 DataFusion statistics tests passed; Rust formatting and CI's Rust 1.98.0 workspace Clippy passed.

@JingsongLi JingsongLi changed the title [python] Show complete pruning statistics in Native explain [python] Complete Native scan pruning and explain statistics Oct 16, 2026
@JingsongLi JingsongLi changed the title [python] Complete Native scan pruning and explain statistics [python] Use Native scan statistics in explain Oct 16, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review October 17, 2026 00:51
@JingsongLi
JingsongLi merged commit 0bc71fb into apache:master Oct 17, 2026
11 of 15 checks passed
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
Native CI builds pypaimon-rust from paimon-rust main, and Python Tests
skip native planning tests, so no job forces native planning through a
released binding. That is how apache#10609 broke 0.4.0 without notice: every
native plan failed on Plan.stats() and silently fell back to the Python
scanner.

Native CI / pypaimon-rust 0.4.0 installs the release from PyPI and runs
the regular suite with native planning forced. Tests marked native_plan
need Rust main, so they stay in the main job. On the previous commit
this job fails test_native_plan_retains_independent_payload_column_files
and plans natively only twice.
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
…ts()

apache#10609 calls rust_plan.stats(), which pypaimon-rust 0.4.0, the release
the Python test jobs install, does not have. The AttributeError sends
every native plan back to the Python scanner, so
ContiguousWindowPlanTest.test_native_plan_retains_independent_payload_column_files
fails on master. Read the counters only when the binding has them, as
snapshot_id already does; such plans report no native statistics.
sundapeng added a commit to sundapeng/paimon that referenced this pull request Oct 17, 2026
Native CI builds pypaimon-rust from paimon-rust main, and Python Tests
skip native planning tests, so no job forces native planning through a
released binding. That is how apache#10609 broke 0.4.0 without notice: every
native plan failed on Plan.stats() and silently fell back to the Python
scanner.

Native CI / pypaimon-rust 0.4.0 installs the release from PyPI and runs
the regular suite with native planning forced. Tests marked native_plan
need Rust main, so they stay in the main job. On the previous commit
this job fails test_native_plan_retains_independent_payload_column_files
and plans natively only twice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant