Skip to content

[core] Support composite BTree prefix and range queries - #10346

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/composite-btree-ranges
Oct 3, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/composite-btree-ranges

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Extend composite BTree global index queries for Data Evolution tables with equality left prefixes and a range on the first non-equality key column. This follows #10345.

  • Build one typed tuple interval for >, >=, <, <= and BETWEEN, including combined bounds and contradictory constraints.
  • Seek directly to virtual prefix/range boundaries in SST blocks and prune files by typed key metadata. Preserve the persisted file format and full-key point/Bloom paths.
  • Prefer more bounded columns, then more equality columns, then fewer total key columns. For a leading-column-only condition, prefer a dedicated single-column index when its coverage includes the entire composite definition.
  • Apply the existing selected-file scan budget to every group of a definition and decline the whole definition if any group exceeds it. Preserve consistent eager/reader fallback and partial coverage behavior.
  • Keep conditions on later key columns as row filters and support the existing AND/OR planning paths.
  • Document the selection and scan rules. Python and dedicated composite IN/IS NULL optimization remain deferred.

Tests

All scoped final tests pass without the fast-build profile:

Scope Runtime/profile Result
Composite, scalar BTree and SST regression tests JDK 8 377 passed
Query planning and Data Evolution table regression tests JDK 8 78 passed
Flink composite index SQL/lifecycle test Flink 1 / JDK 8 1 passed
Flink composite index SQL/lifecycle test Flink 2 / JDK 11 1 passed
Spark composite index procedure suite Spark 3 / JDK 8 / Scala 2.12 2 passed
Spark composite index procedure suite Spark 4 / JDK 17 / Scala 2.13 2 passed

Coverage includes both index file versions, compression and cross-block seeks, open/closed boundaries, NULL components, conflicting constraints, local row-ID ranges, independent single-column indexes, partial coverage, whole-definition budget fallback and eager/reader equivalence.

Checkstyle, Spotless, Maven Enforcer, Apache RAT and git diff --check also pass. Two independent review iterations completed, with the final review reporting no actionable findings.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1934e40. No blocking issues found.

Checked the virtual tuple boundaries and SST seeks, open/closed endpoints and NULL exclusion, contradictory constraints, metadata pruning, composite/scalar selection, whole-definition budget fallback, residual predicates, and eager/reader coverage behavior. The persisted index format and full-key point/Bloom path remain unchanged.

Local validation on JDK 8, without fast-build:

  • 408 Common regression tests covering composite/scalar BTree and SST readers passed.
  • 167 Core regression tests covering query planning, index refresh, Data Evolution tables and related paths passed.
  • The Flink 1 composite SQL/lifecycle test passed.
  • Both Spark 3 / Scala 2.12 composite procedure tests passed.
  • Two additional temporary randomized tests passed: 2,400 interval evaluations across BTree v1/v2, with and without local row-ID clipping, compared against row-level evaluation of the covered predicates. These exercise reordered and conflicting equality/range constraints, NULLs, and boundary suffixes.

Full GitHub CI still has pending jobs at review time. Flink 2 and Spark 4 were not rerun locally.

@JingsongLi
JingsongLi merged commit 289e926 into apache:master Oct 3, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants