Skip to content

[core] Improve fixed-bucket pruning and expose scan statistics - #1140

Merged
JingsongLi merged 7 commits into
apache:mainfrom
JingsongLi:codex/native-scan-statistics
Oct 17, 2026
Merged

JingsongLi merged 7 commits into
apache:mainfrom
JingsongLi:codex/native-scan-statistics

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already supports scan explanations and fixed-bucket pruning. Complete Native bucket selection using Java semantics and expose lightweight counters from ordinary planning through Plan.

Companion PyPaimon integration: apache/paimon#10609 (draft until this API reaches Rust main).

Changes

  • Complete fixed-bucket pruning using Java's BucketSelector: same-field OR clauses, intersections of repeated Equal/IN constraints, per-partition predicate specialization, and the 1000-value combination bound. Keep pending and dynamic bucket assignments conservative.
  • Preserve early manifest/sidecar filtering. Disable unsafe raw partition specialization under masking while retaining independent safe key conjuncts. Cache key-only selectors across partitions and use ordered lookup for large intersections; unordered literals fail open.
  • Every scan collects lightweight ScanStats by default. Call scan.plan() and read plan.stats() directly; the getter always returns statistics. Remove with_stats, collection flags/modes, full-funnel fields, tuple-returning plan_with_trace/plan_with_stats APIs, and extra collection-only reads or delayed LIMIT handling.
  • Store statistics inline in Plan. Summarize final file counts and bytes together, cache them, and refresh final totals after split transformations while retaining the work already performed. DataFusion uses the same statistics API.
  • Keep snapshot metadata only on Plan.snapshot_id(). Preserve counters and snapshot metadata through audit regrouping, index-score attachment, authorized plans, combined incremental windows and stateful streaming All/Delta/Changelog planning. Diff keeps independent statistics for each snapshot state.
  • Thin Python bindings expose core Plan and read-only ScanStats. Collection uses local counters without locks or atomics and retains normal manifest, bucket, row-range, sidecar and LIMIT optimizations; no extra data-file reads.

No new dependencies or public bucket-selector API. No formats or read capabilities beyond PyPaimon's existing support.

Verification

  • Rust table layer: 2105 passed, 3 ignored, including sidecar, LIMIT, snapshot, incremental, audit and score coverage.
  • Python binding read and stateful-streaming suites: 168 passed; initial and follow-up stream plans assert exact default manifest/file counts.
  • DataFusion scan-statistics integration: 6 passed (unrelated NaN/signed-zero case excluded).
  • Expanded PyPaimon Native planning/reading regression: 503 passed, 13 skipped, 122 subtests passed. Authorization unit suite including the new lazy-wrapper regression: 52 passed.
  • New Plan transformation regression checks final count/byte cache refresh, empty transformations, retained snapshot/pruning counters/LIMIT state, and overflow/unknown-size semantics.
  • CI's Rust 1.98.0 workspace Clippy with fulltext,vortex and -D warnings passed; Rust formatting, Python repository Flake8 and Python 3.6 syntax checks passed.

@JingsongLi JingsongLi changed the title [core] Add complete statistics for Native scan planning [core] Complete scan pruning and Native explain statistics Oct 16, 2026
@JingsongLi JingsongLi changed the title [core] Complete scan pruning and Native explain statistics [core] Improve fixed-bucket pruning and expose scan statistics Oct 16, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 5af64a3 into apache:main Oct 17, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants