Skip to content

Align temporal string schema evolution with Java - #1115

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-temporal-string-evolution
Oct 12, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-temporal-string-evolution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Align the Native reader's existing temporal-to-character schema evolution with Java:

  • Format TIME with DateTimeUtils' precision loop, including .0 for whole seconds and negative-millisecond normalization.
  • Format TIMESTAMP with a space separator and its declared minimum fraction width, retaining significant nanoseconds.
  • Format TIMESTAMP_LTZ in the process local time zone without an ISO zone suffix.
  • Apply Java CHAR padding and VARCHAR truncation to scalar temporal casts.
  • Reuse the scalar conversion inside ARRAY/MAP/ROW string rendering, preserving NULL containers and nested NULLs.
  • Share temporal formatting with non-legacy partition paths and fix TIME's whole-second partition text.
  • Preserve Java's signed epoch-day narrowing, four-digit year rendering, and millisecond wrapping for ordinary timestamps outside datetime ranges.

The behavior lives in Rust core; the Python bindings need no new API or conversion logic. Legacy partition formatting is unchanged. No dependencies are added.

Evidence

An existing-capability probe exercised 85 Python-supported atomic schema changes through REST tables. It identified TIME/TIMESTAMP/LTZ string differences. Re-running the same probe after these changes found no mismatches.

Java references: TimestampToStringCastRule, TimeToStringCastRule, DateTimeUtils, BinaryStringUtils, and Timestamp.toLocalDateTime. Actual Java CastExecutors were also exercised for precision, bounded strings, time zones, pre-epoch timestamps and wide years.

Verification

  • 73 focused Rust tests: evolution_string, nested_evolution and partition_utils.
  • cargo +1.98.0 fmt --all -- --check.
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings.
  • Rebuilt Python extension and ran 195 REST schema-evolution tests with Native planning/reading/writing/committing enabled: 447 Native reads, 261 writes and commits.
  • LTZ REST cases also passed in America/New_York and Asia/Shanghai, including winter/summer instants.

Companion Python PR: apache/paimon#10519, with Java-aligned pure-Python casts and cross-path regressions. Its Native CI will require this Rust change to be merged first.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the implementation and focused tests. I found no blocking issues.

@JingsongLi
JingsongLi merged commit 6aa60c5 into apache:main Oct 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants