[core] Keep all sibling tags of a snapshot as Iceberg refs - #10248
Open
LuciferYang wants to merge 4 commits into
Open
LuciferYang wants to merge 4 commits into
LuciferYang wants to merge 4 commits into
Conversation
The ref rebuild kept only the first tag name of each snapshot, so with several tags on one snapshot the siblings silently disappeared from Iceberg on the next commit — VERSION AS OF for them failed, and tags that notifyCreation had just added vanished again. Flatten every tag name of a snapshot into its own ref. Assisted-by: GLM-5.3
LuciferYang
marked this pull request as draft
October 2, 2026 04:03
LuciferYang
marked this pull request as ready for review
October 2, 2026 04:36
LuciferYang
marked this pull request as draft
October 2, 2026 06:34
LuciferYang
marked this pull request as ready for review
October 2, 2026 19:51
Contributor
Author
|
@JingsongLi Could you take a look? When one snapshot has several tags, the next commit kept only one of them as an Iceberg ref, so reading the others by tag through Iceberg failed with "Cannot find ref". The Flink Iceberg ITCase now reads |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Paimon allows several tags on one snapshot. With Iceberg compatibility enabled, every commit rebuilds the Iceberg
refsmap fromTagManager.tags(), butIcebergCommitCallbackkept only the first tag name of each snapshot (entry.getValue().get(0)). The other tags on that snapshot dropped out of the Iceberg metadata on the next commit, so reading them by tag through Iceberg failed with "Cannot find ref". Which tag survived depended on the order of the tag directory listing.Every tag name of a retained snapshot now gets its own ref. Snapshots with a single tag are unaffected.
This closes #10246.
Tests
FlinkIcebergITCaseBase#testCreateTagsnow also putstag3on snapshot 4, which already hastag2, and inserts one more row so the commit rebuilds the refs. ReadingT$refsthrough the Iceberg catalog must still returntag1,tag2andtag3. On master one of the two tags on snapshot 4 is missing.IcebergCompatibilityTest.testSiblingTagsOnOneSnapshotAllBecomeRefsputs two tags on one snapshot and then commits again to trigger the rebuild. It asserts that the refs are exactlyfirstandsecond, both pointing at that snapshot. The tags are created throughTagManagerwith no tag callbacks, so the previous metadata has no refs and the rebuild from the tag list is the only thing that can produce them. Without the fix, only one of the two tags comes back.API and Format
No.
Documentation
No.