Skip to content

[python] Align temporal string schema evolution with Java and Native - #10519

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-temporal-string-evolution
Oct 12, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-temporal-string-evolution

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

What changes

Make existing temporal-to-character schema evolution produce Java's text in both Python and Native reads:

  • TIME precision follows DateTimeUtils' digit loop, including .0 at whole seconds.
  • TIMESTAMP uses a space separator, declared minimum fractional width, and exact nanosecond digits.
  • TIMESTAMP_LTZ uses the local time zone without a trailing Z or offset.
  • Scalar CHAR/VARCHAR casts apply Java padding/truncation.
  • ARRAY/MAP/ROW string rendering uses the same temporal leaf casts and preserves NULL containers/elements.
  • Share formatting with existing row-to-string rendering.
  • Use integer Gregorian dates and Java's signed millisecond/epoch-day rules rather than restricting Arrow data to Python datetime's year range.

No new Python capability or public API is introduced. These casts and schema changes already existed; this aligns their results with Java and Rust core.

Rust dependency

Depends on apache/paimon-rust#1115. This PR is a draft until that change is merged. The Native CI continues to build paimon-rust from upstream main, so the Rust PR must merge first.

Verification

  • 195 REST schema-evolution tests passed with Native planning, reading, writing and committing enabled: 447 reads, 261 writes and commits.
  • New tests cover Append, primary-key and Data Evolution tables; nested ARRAY/MAP/ROW values; predicate reads; CHAR/VARCHAR lengths; pre-epoch values, nanoseconds and wide dates.
  • LTZ REST tests passed in UTC, America/New_York and Asia/Shanghai, including winter/summer instants.
  • 179 related Python tests and 32 subtests passed: existing schema evolution, nested/PK reads, schema manager, row rendering and casting.
  • Changed files passed Flake8 and production code passed Python 3.6 grammar validation.
  • Re-ran an 85-combination REST cast probe: no remaining differences among the exercised existing Python casts.
  • Checked actual Java CastExecutors for temporal formatting, wide years, signed epoch-day narrowing, seconds wrapping and local-zone overflow.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the temporal formatting and schema-evolution paths, including the focused Python regression tests. I found no blocking issues.

@JingsongLi
JingsongLi marked this pull request as ready for review October 12, 2026 23:21
@JingsongLi
JingsongLi merged commit c9f93bd into apache:master Oct 12, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants