Skip to content

[core] Keep all sibling tags of a snapshot as Iceberg refs - #10248

Open
LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:m/core-076-iceberg-refs
Open

LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:m/core-076-iceberg-refs

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Paimon allows several tags on one snapshot. With Iceberg compatibility enabled, every commit rebuilds the Iceberg refs map from TagManager.tags(), but IcebergCommitCallback kept only the first tag name of each snapshot (entry.getValue().get(0)). The other tags on that snapshot dropped out of the Iceberg metadata on the next commit, so reading them by tag through Iceberg failed with "Cannot find ref". Which tag survived depended on the order of the tag directory listing.

Every tag name of a retained snapshot now gets its own ref. Snapshots with a single tag are unaffected.

This closes #10246.

Tests

  • FlinkIcebergITCaseBase#testCreateTags now also puts tag3 on snapshot 4, which already has tag2, and inserts one more row so the commit rebuilds the refs. Reading T$refs through the Iceberg catalog must still return tag1, tag2 and tag3. On master one of the two tags on snapshot 4 is missing.
  • IcebergCompatibilityTest.testSiblingTagsOnOneSnapshotAllBecomeRefs puts two tags on one snapshot and then commits again to trigger the rebuild. It asserts that the refs are exactly first and second, both pointing at that snapshot. The tags are created through TagManager with no tag callbacks, so the previous metadata has no refs and the rebuild from the tag list is the only thing that can produce them. Without the fix, only one of the two tags comes back.

API and Format

No.

Documentation

No.

The ref rebuild kept only the first tag name of each snapshot, so with
several tags on one snapshot the siblings silently disappeared from
Iceberg on the next commit — VERSION AS OF for them failed, and tags
that notifyCreation had just added vanished again.

Flatten every tag name of a snapshot into its own ref.

Assisted-by: GLM-5.3
@LuciferYang
LuciferYang marked this pull request as draft October 2, 2026 04:03
@LuciferYang
LuciferYang marked this pull request as ready for review October 2, 2026 04:36
@LuciferYang
LuciferYang marked this pull request as draft October 2, 2026 06:34
@LuciferYang
LuciferYang marked this pull request as ready for review October 2, 2026 19:51
@LuciferYang

Copy link
Copy Markdown
Contributor Author

@JingsongLi Could you take a look? When one snapshot has several tags, the next commit kept only one of them as an Iceberg ref, so reading the others by tag through Iceberg failed with "Cannot find ref". The Flink Iceberg ITCase now reads T$refs through the Iceberg catalog and fails on master with one of the two tags on snapshot 4 missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Iceberg export keeps only one tag per snapshot, dropping sibling tags

1 participant