Repository navigation
[core] Support composite BTree IN and NULL queries - #10347
Merged
JingsongLi merged 2 commits intoOct 3, 2026
Merged
Conversation
leaves12138
approved these changes
Oct 3, 2026
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Reviewed 3cd65fb. No blocking issues found.
Checked discrete-domain intersection and typed deduplication, NULL semantics, the 256-interval expansion limit and fallback, multi-point/Bloom lookups, shared-file budgeting, leading-column scalar coverage, and residual predicates in eager/reader plans. The persisted index format is unchanged.
Local validation on JDK 8, without fast-build:
- 412 Common BTree/SST regression tests passed.
- 169 Core query-planning, index-refresh and Data Evolution regression tests passed.
- The Flink 1 composite SQL/lifecycle test passed.
- Both Spark 3 / Scala 2.12 composite procedure tests passed.
- Two additional temporary randomized tests passed: 4,000 integer-key IN/NULL/range predicate combinations across BTree v1/v2 and with/without local row-ID clipping, checked against row-level evaluation of the covered predicates. Both small-IN OR forms and native IN predicates were exercised, including duplicate/NULL literals and conflicting constraints.
Full GitHub CI still has pending jobs at review time. Flink 2 and Spark 4 were not rerun locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Completes the remaining Java composite BTree query support after #10346. Python support remains deferred.
IN,IS NULLandIS NOT NULLin composite key plans, including discrete prefixes followed by a range and full-key multi-point lookups.The existing index file format and serialization are preserved.
Tests
109 focused test executions. Final runs include Checkstyle, Spotless and Maven Enforcer; Apache RAT and
git diff --checkalso pass.Regressions cover both BTree file versions, compressed block boundaries, typed domain intersection and deduplication, the 256-interval boundary, NULL semantics, shared-file budgeting, deleted scalar postings, reader/eager parity, scan modes and unindexed tails.
Engine regressions delete scalar posting files while preserving their manifests. During predicate verification, each nonempty SQL query must then fail with a missing composite index file when the composite files are temporarily deleted, and Spark asserts the eager/reader split types. Files are restored before the build lifecycle assertions continue. Disabling index queries makes the strengthened test fail, and the tests exposed and verify the fix for Spark string NULL literal conversion.