Skip to content

[vortex] Do not push unrepresentable timestamp predicate literals - #9675

Merged
JingsongLi merged 1 commit into
apache:masterfrom
jackylee-ch:vortex-unrepresentable-ts
Sep 11, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
jackylee-ch:vortex-unrepresentable-ts

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Purpose

toTimestampLiteral takes the precision from the column type but the value from the predicate literal, and nothing reconciles them — Spark builds the literal from its own type with Timestamp.fromMicros, so a TIMESTAMP(0) column can receive a millisecond literal. Dividing by 1000 (or discarding the sub-millisecond part on TIMESTAMP(3)) then yields a bound that answers some operators wrongly: f_ts < 1500ms on a seconds column pushed < 1s and dropped the 1000ms row, which no post-scan filter can restore.

Math.floorDiv is not enough. For a residual literal, col < L needs the bound rounded down but col > L needs it rounded up, and = is unsatisfiable while != is a tautology — one direction cannot serve all six operators. So a literal that is not exactly representable at the column's grain is no longer pushed, and the existing null handling drops the leaf. ParquetFilters.normalizeDecimal uses RoundingMode.UNNECESSARY for the same reason.

Tests

Three roundTrip cases in VortexPredicateConverterTest: millisecond literal on TIMESTAMP(0) with < and !=, a pre-epoch literal, and a sub-millisecond literal on TIMESTAMP(3). All three return 0 rows on master.

Written with Claude Code; verification is mine.

@jackylee-ch
jackylee-ch force-pushed the vortex-unrepresentable-ts branch from 5516d0c to 18885b5 Compare September 7, 2026 11:21
toTimestampLiteral takes the precision from the column type but the value
from the predicate literal, and nothing reconciles them. Spark builds the
literal from its own type with Timestamp.fromMicros, so a TIMESTAMP(0)
column can receive a millisecond literal. Dividing by 1000, or discarding
the sub-millisecond part on TIMESTAMP(3), then yields a bound that answers
some operators wrongly: "f_ts < 1500ms" on a seconds column pushed "< 1s"
and dropped the 1000ms row, which no post scan filter can restore.

Math.floorDiv is not enough, because "col < L" needs the residual rounded
down while "col > L" needs it rounded up. Refuse to push a literal that is
not exactly representable at the column's grain and let the existing null
handling drop the leaf, as ParquetFilters.normalizeDecimal does with
RoundingMode.UNNECESSARY.
@jackylee-ch
jackylee-ch force-pushed the vortex-unrepresentable-ts branch from 18885b5 to 2408119 Compare September 7, 2026 12:56

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prevents real false-negative pruning: a lower-precision Vortex timestamp column can receive a higher-precision Spark literal, and truncating that literal changes strict comparisons. I traced null conversion through the leaf and compound visitors: unsupported AND leaves are omitted conservatively, while an OR with an unsupported child is not pushed. That preserves matching rows for residual evaluation.

No blocking issue found in the changed guards, including pre-epoch values and sub-millisecond/microsecond remainders. The added real-format round-trip tests exercise the lost-row cases. I have not rerun the native Vortex suite here.

@JingsongLi
JingsongLi merged commit 56a5493 into apache:master Sep 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants