Skip to content

[core] Support composite BTree IN and NULL queries - #10347

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/composite-btree-in-null
Oct 3, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/composite-btree-in-null

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Completes the remaining Java composite BTree query support after #10346. Python support remains deferred.

  • Support IN, IS NULL and IS NOT NULL in composite key plans, including discrete prefixes followed by a range and full-key multi-point lookups.
  • Intersect predicates on each key column and deduplicate typed values before expanding at most 256 intervals. Larger expansions use the existing fallback.
  • Scan the selected file union once for budgeting, retain Bloom point lookups, and keep later key predicates as data filters.
  • Use the leading column's scalar coverage when choosing between composite and single-column indexes, including definitions registered under secondary columns.
  • Preserve NULL values during Spark literal conversion so an IN list containing NULL is pushed down.
  • Update the BTree documentation and cover both eager and reader queries across FAST, FULL and DETAIL modes.

The existing index file format and serialization are preserved.

Tests

Scope Profile / JDK Tests
Composite BTree storage and predicate planning JDK 8 23
Global index planning and data evolution table queries JDK 8 80
Flink composite BTree SQL Flink 1 / JDK 8 1
Flink composite BTree SQL Flink 2 / JDK 11 1
Spark composite BTree procedure and SQL Spark 3 / JDK 8 2
Spark composite BTree procedure and SQL Spark 4 / JDK 17 2

109 focused test executions. Final runs include Checkstyle, Spotless and Maven Enforcer; Apache RAT and git diff --check also pass.

Regressions cover both BTree file versions, compressed block boundaries, typed domain intersection and deduplication, the 256-interval boundary, NULL semantics, shared-file budgeting, deleted scalar postings, reader/eager parity, scan modes and unindexed tails.

Engine regressions delete scalar posting files while preserving their manifests. During predicate verification, each nonempty SQL query must then fail with a missing composite index file when the composite files are temporarily deleted, and Spark asserts the eager/reader split types. Files are restored before the build lifecycle assertions continue. Disabling index queries makes the strengthened test fail, and the tests exposed and verify the fix for Spark string NULL literal conversion.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3cd65fb. No blocking issues found.

Checked discrete-domain intersection and typed deduplication, NULL semantics, the 256-interval expansion limit and fallback, multi-point/Bloom lookups, shared-file budgeting, leading-column scalar coverage, and residual predicates in eager/reader plans. The persisted index format is unchanged.

Local validation on JDK 8, without fast-build:

  • 412 Common BTree/SST regression tests passed.
  • 169 Core query-planning, index-refresh and Data Evolution regression tests passed.
  • The Flink 1 composite SQL/lifecycle test passed.
  • Both Spark 3 / Scala 2.12 composite procedure tests passed.
  • Two additional temporary randomized tests passed: 4,000 integer-key IN/NULL/range predicate combinations across BTree v1/v2 and with/without local row-ID clipping, checked against row-level evaluation of the covered predicates. Both small-IN OR forms and native IN predicates were exercised, including duplicate/NULL literals and conflicting constraints.

Full GitHub CI still has pending jobs at review time. Flink 2 and Spark 4 were not rerun locally.

@JingsongLi
JingsongLi merged commit 71e3c92 into apache:master Oct 3, 2026
4 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants