Skip to content

[core] Improve fixed-bucket pruning and expose scan statistics - #1140

Open
JingsongLi wants to merge 6 commits into
apache:mainfrom
JingsongLi:codex/native-scan-statistics
Open

JingsongLi wants to merge 6 commits into
apache:mainfrom
JingsongLi:codex/native-scan-statistics

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already supports scan explanations and fixed-bucket pruning. Complete Native bucket selection using Java semantics and expose counters from ordinary planning through Plan.

Companion PyPaimon integration: apache/paimon#10609 (draft until this API reaches Rust main).

Changes

  • Complete fixed-bucket pruning using Java's BucketSelector: same-field OR clauses, intersections of repeated Equal/IN constraints, per-partition predicate specialization, and the 1000-value combination bound. Keep pending and dynamic bucket assignments conservative.
  • Preserve early manifest/sidecar filtering. Disable unsafe raw partition specialization under masking while retaining independent safe key conjuncts. Cache key-only selectors across partitions and use ordered lookup for large intersections; unordered literals fail open.
  • Unify counters as ScanStats, requested through with_stats().plan() and retrieved through Plan.stats(). Remove tuple-returning plan_with_trace/plan_with_stats APIs. DataFusion uses the same API and statistics terminology.
  • Collect lightweight statistics only. Remove ScanStatsMode, the complete flag, full-funnel fields, extra schema-statistics reads and deferred LIMIT handling. Counters describe actual planning work while retaining manifest, bucket, row-range and sidecar pruning plus LIMIT stopping.
  • Keep snapshot metadata only on Plan.snapshot_id(). Preserve statistics and snapshot metadata through score attachment and audit split regrouping. Audit counters describe the final regrouped plan.
  • Expose the core Plan and read-only ScanStats through thin Python bindings, including combined Delta/Changelog incremental scans. Statistics collection does not open data files.

No new dependencies or public bucket-selector API. No formats or read capabilities beyond PyPaimon's existing support.

Verification

  • cargo test --locked -p paimon --lib table::: 2104 passed, 3 ignored, including 135 table-scan cases.
  • DataFusion scan statistics integration: 6 passed (unrelated NaN/signed-zero case excluded).
  • Python binding tests/test_read.py: 152 passed.
  • Expanded PyPaimon Native regression: 501 passed, 13 skipped and 122 subtests passed; two remaining full-statistics expectations were adapted, then all 20 focused Native explain/integration cases passed.
  • cargo +1.98.0 fmt --all -- --check and CI's cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings passed.

Coverage includes ordinary/statistics plan equality; zero/positive LIMIT; sparse manifest/Avro sidecar decoding; pending buckets; masking and authorization; correlated partition predicates; historical/empty snapshots; cross-schema ADD/DELETE netting; Data Evolution projection; and combined Delta/Changelog windows.

@JingsongLi JingsongLi changed the title [core] Add complete statistics for Native scan planning [core] Complete scan pruning and Native explain statistics Oct 16, 2026
@JingsongLi JingsongLi changed the title [core] Complete scan pruning and Native explain statistics [core] Improve fixed-bucket pruning and expose scan statistics Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant