[server] Update Tablet Server to use per-table/partition remote data directory by LiebingYu · Pull Request #2782 · apache/fluss

LiebingYu · 2026-03-03T07:18:57Z

Purpose

Linked issue: close #2756

Brief change log

Tests

API and Format

Documentation

LiebingYu · 2026-05-26T03:01:32Z

@wuchong ready for review

…directory

platinumhamburg

Thanks @LiebingYu, I left some comments.

platinumhamburg · 2026-10-10T09:18:15Z

            generator.writeNumberField(MAX_TIMESTAMP_FIELD, remoteLogSegment.maxTimestamp());
            generator.writeNumberField(
                    SEGMENT_SIZE_IN_BYTES_FIELD, remoteLogSegment.segmentSizeInBytes());
+            if (remoteLogSegment.remoteLogDir() != null) {


Do we need to persist remote_log_dir in the manifest JSON? The table or partition registration already records the assigned directory, and registerReplica() has that value when it loads the manifest. Writing it on both the manifest and every segment creates several copies that can disagree. The loader only fills it in when the manifest-level field is absent, so a missing or different segment-level value would go unnoticed until a file operation. Could we attach the directory when loading and keep it out of the persisted format? If manifests need to be usable without the registration metadata, I'd store one value and validate it against the assigned directory.

Thanks for the insightful comments. I remove the remoteLogDir in RemoteLogManifest and RemoteLogSegment. And only keep it in RemoteLogTablet, this is a runtime info. Besides I modified the RemoteLogStorage methods to carry the remote path information.

PTAL!

platinumhamburg

The per-replica directory is now used for remote log I/O and KV snapshot upload and recovery. I found a few remaining issues to address.

The Coordinator's table and partition cleanup still builds KV and log paths from the cluster-level remote.data.dir, so it can miss data assigned to another directory. Configuration validation also allows a cluster to set only remote.data.dirs, while ServerBase.start() still constructs an FsPath from remote.data.dir; that configuration fails at startup. The new test fixture always supplies both settings, so it does not catch this case.

Could we update cleanup to use the registered table or partition directory, and add tests for startup with only remote.data.dirs and for remote cleanup after dropping entities placed in non-default directories? The changed follower writer-snapshot recovery path should also be tested on a non-default directory.

Could you also clarify whether remote.data.dirs supports different filesystem schemes? The filesystem-token RPC currently returns a token for only the filesystem initialized from remote.data.dir. If mixed schemes are supported, this path needs to handle them; otherwise, the restriction should be validated and documented.

platinumhamburg · 2026-10-12T03:07:42Z

            if (deleteRemote && replicaToDelete.isLeader()) {
                kvManager.deleteRemoteKvSnapshot(
-                        replicaToDelete.getPhysicalTablePath(), replicaToDelete.getTableBucket());
+                        replicaToDelete.getRemoteDataDir(),


Passing the assigned remoteDataDir to the Tablet Server's bucket cleanup is right, but the Coordinator's table/partition cleanup still derives its KV and log paths from the cluster-level remote.data.dir. For a table or partition assigned to another directory, RemoteStorageCleaner sweeps the wrong location. This also leaves no effective fallback if the per-bucket remote deletion did not run or failed. Could we pass the registered table/partition directory into this cleanup path and obtain the FileSystem from each target path?

Agreed, this is a remaining gap. Coordinator cleanup is explicitly tracked in #2785 under #2292. That follow-up will use the registered table/partition directory, resolve the filesystem from each target path, and add non-default-directory cleanup tests. Could we keep that work in #2785?

platinumhamburg · 2026-10-12T03:10:12Z

+            FsPath remoteKvTabletDir =
+                    FlussPaths.remoteKvTabletDir(
+                            remoteKvDir, PhysicalTablePath.of(tablePaths.get(t)), tb);
+            assertThat(remoteKvTabletDir.getFileSystem().exists(remoteKvTabletDir)).isTrue();


These assertions establish that the snapshot was written to the assigned directory, but the test never drops an entity placed outside the default directory. Could we drop one non-partitioned KV table and one partition in a non-default directory, then wait until their remote KV and log directories are gone? That would cover the Coordinator cleanup path as well as the per-bucket cleanup path.

Coordinator cleanup and the corresponding drop tests are tracked separately in #2785 as part of #2292. Could we address this there, keeping this PR focused on the Tablet Server’s per-table/partition remote I/O and recovery paths?

platinumhamburg · 2026-10-12T03:11:00Z

            // 2. download writer id snapshots from remote storage.
            File snapshotFile = FlussPaths.writerSnapshotFile(log.getLogDir(), nextFetchOffset);
-            buildWriterIdSnapshotFile(snapshotFile, remoteLogSegmentWithMaxStartOffset, rlm);
+            buildWriterIdSnapshotFile(


This changes the path used to fetch the writer snapshot during follower recovery. The existing follower recovery test exercises the default remote directory, while the new multi-directory tests verify readback without forcing this recovery path. Could we run the writer-ID follower recovery scenario on a table assigned to a non-default directory and assert that the recovered writer state is intact?

Added a follower recovery test for a table assigned to dir2, verifying the recovered writer IDs, sequences, and offsets. Also added unit coverage for partitioned and non-partitioned tables.

LiebingYu · 2026-10-12T09:42:40Z

@platinumhamburg Thanks for the review. Fixed the startup issue and added tests for configuring either remote.data.dir or remote.data.dirs alone. Also added follower writer-snapshot recovery coverage for a non-default directory, verifying the recovered writer state.

Coordinator cleanup and the corresponding non-default-directory drop tests are tracked separately in #2785 under #2292. Could we keep those changes in that follow-up?

Filesystem resolution uses each target path’s scheme, but token distribution currently covers only the default directory’s filesystem. Mixed schemes requiring separate tokens are therefore not fully supported. Independent authentication is tracked in #2518; this PR does not add per-path token distribution.

The following issues are listed here: #2292.

platinumhamburg

LGTM.

gyang94 reviewed Mar 8, 2026

View reviewed changes

Comment thread fluss-common/src/main/java/org/apache/fluss/remote/RemoteLogSegment.java Outdated

LiebingYu force-pushed the 4-update-ts-remote branch from 9e54aff to 2610b9e Compare March 11, 2026 07:46

LiebingYu force-pushed the 4-update-ts-remote branch 3 times, most recently from c3e0d21 to 6009c88 Compare May 26, 2026 01:58

LiebingYu force-pushed the 4-update-ts-remote branch 3 times, most recently from 4ebfe61 to 2e6a85e Compare July 21, 2026 02:58

LiebingYu force-pushed the 4-update-ts-remote branch from 2e6a85e to 8154d48 Compare July 25, 2026 07:24

LiebingYu force-pushed the 4-update-ts-remote branch 2 times, most recently from 235608f to 76bf1c2 Compare October 10, 2026 02:54

LiebingYu and others added 4 commits October 10, 2026 16:36

[server] Update Tablet Server to use per-table/partition remote data …

d54b18d

…directory

fix

5982423

fix

083a9f1

fix

6836eca

LiebingYu force-pushed the 4-update-ts-remote branch from 76bf1c2 to 6836eca Compare October 10, 2026 08:37

platinumhamburg requested changes Oct 10, 2026

View reviewed changes

LiebingYu added 2 commits October 11, 2026 12:07

address baiye's comments

a39f421

fix

6456dc2

platinumhamburg approved these changes Oct 12, 2026 •

edited

Loading

View reviewed changes

platinumhamburg requested changes Oct 12, 2026

View reviewed changes

fix only set remote.data.dirs

97fd7f6

LiebingYu added 2 commits October 12, 2026 17:43

fix

e0e46f9

fix

5ed1c9a

platinumhamburg approved these changes Oct 13, 2026

View reviewed changes

LiebingYu merged commit 70384ea into apache:main Oct 13, 2026
30 of 31 checks passed

Conversation

LiebingYu commented Mar 3, 2026

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Uh oh!

LiebingYu commented May 26, 2026

Uh oh!

platinumhamburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

platinumhamburg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiebingYu commented Oct 12, 2026

Uh oh!

platinumhamburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants