Repository navigation

feat: let the Parquet scan skip optional filters that are not worth their cost - #25682

Draft

adriangb wants to merge 16 commits into

apache:mainfrom

pydantic:optional-filter-parquet

Draft

feat: let the Parquet scan skip optional filters that are not worth their cost#25682
adriangb wants to merge 16 commits into
apache:mainfrom
pydantic:optional-filter-parquet

Commits on Sep 24, 2026
- feat: add OptionalFilterPhysicalExpr to mark filters not needed for correctness
  
  adriangb
  and
  claude
  committed
  c839c76
- feat: add filter_stats (FilterCost, Clock) for filter measurements
  
  adriangb
  and
  claude
  committed
  0337936
- feat: add OptionalFilterGate to pause optional filters that cost more than they save
  
  adriangb
  and
  claude
  committed
  62a11a7
Commits on Sep 25, 2026
- feat(gate): add a measured overhead for each evaluated row to the cost
  
  adriangb
  and
  claude
  committed
  3b8994a
- feat(gate): share pauses between the gates of one plan site
  
  adriangb
  and
  claude
  committed
  9755fd1
- feat(gate): decide only on windows of at least MIN_OBSERVED_ROWS rows
  
  adriangb
  and
  claude
  committed
  3ccb85e
- feat(gate): use the measured work of the filter producer as the saving
  
  adriangb
  and
  claude
  committed
  0c024cc
- feat: add optional_filter_mode config
  
  adriangb
  and
  claude
  committed
  b8fb779
- feat: let the Parquet scan skip optional filters that are not worth their cost
  
  adriangb
  and
  claude
  committed
  5becb4e
Commits on Sep 26, 2026
- fix(parquet): count only skippable rows in the saving of an optional row filter
  
  adriangb
  and
  claude
  committed
  9fdef55
- feat(parquet): share the pauses of optional filter gates in a scan
  
  adriangb
  and
  claude
  committed
  ee93043
- test(parquet): gate windows of MIN_OBSERVED_ROWS rows in row filter tests
  
  adriangb
  and
  claude
  committed
  7caf968
- feat(parquet): order row filter predicates by measured rows removed per nanosecond
  
  adriangb
  and
  claude
  committed
  9f36c9d
- test(parquet): size the optional filter integration tests for gate windows
  
  adriangb
  and
  claude
  committed
  f68a421
Commits on Sep 28, 2026
- Merge apache/main into optional-filter-parquet
  
  adriangb
  and
  claude
  committed
  214446d
- fix(parquet): no scan equivalences from optional conjuncts
  
  adriangb
  and
  claude
  committed
  38cf3e7

Commits

Commits on Sep 24, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Commits on Sep 28, 2026