HDDS-16739. S3 ListObjectVersions should return the null version of each object by smengcl · Pull Request #11426 · apache/ozone

smengcl · 2026-10-06T23:02:56Z

Generated-by: Claude Code (Fable 5.1, Opus 5.5)

What changes were proposed in this pull request?

Since HDDS-16591, GET /bucket?versions (ListObjectVersions) returns 501 NotImplemented. Before that, the request fell through to ListObjects and clients parsed the response as an empty version list. Bucket cleanup tools commonly call ListObjectVersions and then delete by the returned version IDs. The ceph/s3-tests cleanup fixture does so before and after each test, so the nightly S3 compatibility report dropped from 37.0% to 0.4% for s3-tests.

Ozone does not support bucket versioning yet, so every bucket behaves like an AWS S3 bucket that never had versioning enabled. This PR makes S3 Gateway respond the way AWS S3 does for such a bucket:

ListObjectVersions lists each object as a single Version with VersionId null and IsLatest true, with no DeleteMarker entries. It reuses the existing key listing in BucketEndpoint and paginates with key-marker, NextKeyMarker and NextVersionIdMarker. PUT and DELETE on ?versions are still rejected.
null is accepted as the version ID of the current object in DeleteObjects, DeleteObject, PutObjectTagging and DeleteObjectTagging. Any other version ID is still rejected with NotImplemented.
DeleteObjects returns VersionId in Deleted and Error entries when the request specified it.
The S3 API doc is updated separately in HDDS-16739. Document ListObjectVersions in the S3 API page ozone-site#566.

Not changed: the request is still audited and authorized as a bucket listing (GET_BUCKET), GetBucketVersioning still returns NotImplemented, and GetObject and HeadObject still ignore versionId.

This is the unversioned subset of the ListObjectVersions work planned in HDDS-15728, which can extend it with the merged listing for versioned buckets.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16739

How was this patch tested?

New unit tests in TestBucketList, TestObjectMultiDelete, TestObjectNotImplemented, TestObjectTaggingPut and TestObjectTaggingDelete. TestBucketNotImplemented now expects only PUT and DELETE on ?versions to be rejected.
New AWS SDK v2 integration test testDeleteObjectsListedByListObjectVersions, which paginates ListObjectVersions, deletes the returned versions with DeleteObjects and checks that the bucket is empty.
Ran the s3gateway unit tests and TestS3SDK locally.

🤖 Generated with Claude Code

…ach object Versioning is not implemented, so every bucket behaves like an AWS S3 bucket that never had versioning enabled. ListObjectVersions now lists each object with VersionId "null" instead of returning NotImplemented, and "null" is accepted as the version ID of the current object in DeleteObjects, DeleteObject, PutObjectTagging and DeleteObjectTagging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

smengcl · 2026-10-06T23:05:06Z

Before we have proper versioning implementation, we could have this. Thoughts, @fmorg-git ?

Copilot

Copilot review overview

🟡 Changes recommended

ListObjectVersions must reject version-id-marker when key-marker is absent.

Review effort: Balanced
Findings: 1

Open (1)

Validate key-marker when version-id-marker is supplied · New

What changed in this PR

Adds unversioned ListObjectVersions compatibility to S3 Gateway and permits operations targeting the current object via version ID null.

Changes:

Returns each object as its latest null version with pagination support.
Accepts null versions for deletion and tagging while rejecting unsupported versions.
Adds unit, SDK integration, and interface documentation coverage.

File	Description
`TestObjectTaggingPut.java`	Tests tagging the null version.
`TestObjectTaggingDelete.java`	Tests removing null-version tags.
`TestObjectNotImplemented.java`	Tests null-version object deletion.
`TestObjectMultiDelete.java`	Tests null-version bulk deletion responses.
`TestBucketNotImplemented.java`	Allows GET on `?versions`.
`TestBucketList.java`	Tests version listing and pagination.
`S3Consts.java`	Defines version constants and parameters.
`ObjectTaggingHandler.java`	Accepts null-version tagging requests.
`ObjectOperationNotImplementedHandler.java`	Allows null-version deletion.
`MultiDeleteResponse.java`	Serializes response version IDs.
`ListVersionsResult.java`	Models ListObjectVersions responses.
`EndpointBase.java`	Adds version-ID validation helper.
`BucketOperationNotImplementedHandler.java`	Routes version listings to the endpoint.
`BucketEndpoint.java`	Implements listing and null-version bulk deletion.
`KeyMetadata.java`	Adds version metadata fields.
`AbstractS3SDKV2Tests.java`	Adds end-to-end SDK coverage.
`S3.md`	Documents ListObjectVersions support.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

fmorg-git · 2026-10-06T23:10:21Z

hi @smengcl, yeah I was actually working on something very similar, but you got the PR up before me. FYI @Gargi-jais11

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fmorg-git · 2026-10-06T23:23:54Z

hi @smengcl - fyi the change is bigger than this because the actions for STS differ between ListObjectVersions and ListObjects. Maybe I can handle the STS portions after this PR goes in?

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

smengcl · 2026-10-07T18:02:03Z

hi @smengcl - fyi the change is bigger than this because the actions for STS differ between ListObjectVersions and ListObjects. Maybe I can handle the STS portions after this PR goes in?

Sounds good! Thx

fmorg-git · 2026-10-13T01:04:22Z

+
+    StringWriter writer = new StringWriter();
+    JAXB.marshal(result, writer);
+    assertThat(writer.toString())


curious - for such a small set of files, would it be worth it to have an assertion on the actual expected output instead of a few contains and doesNotContain checks?

…ull version-id-marker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Gargi-jais11

Thanks @smengcl for the patch. Please find the inlined comments.

Gargi-jais11 · 2026-10-13T04:07:06Z

    int listType = queryParams().getInt(QueryParams.LIST_TYPE, 1);
    boolean fetchOwner = queryParams().getBoolean(QueryParams.FETCH_OWNER, false);
-    return listType != 2 || fetchOwner;
+    return isListObjectVersions() || listType != 2 || fetchOwner;


here we always includes Owner on ListObjectVersions whenever isListObjectVersions() is true, even when fetch-owner=false. On AWS, owner is omitted by default and only included when fetch-owner=true (ListObjectVersions).
use the same rule as ListObjectsV2 here (fetchOwner only)

Gargi-jais11 · 2026-10-13T04:10:33Z

+  static boolean isUnsupportedVersionId(@Nullable String versionId) {
+    return versionId != null && !NULL_VERSION_ID.equals(versionId);
+  }


versionId="" is treated as unsupported not the same as absent or "null", so DeleteObject / tagging etc. with an empty versionId returns NotImplemented.
we should be treating blank the same as absent and only reject non-null values.

Gargi-jais11 · 2026-10-13T04:12:41Z

+   * @param listing result of listing the bucket with {@code key-marker} as the marker
+   * @param versionIdMarker {@code version-id-marker} of the request
+   */
+  static ListVersionsResult of(ListObjectResponse listing, String versionIdMarker) {


AWS ListObjectVersions responses include KeyCount, but ListVersionsResult never maps it, so the XML omits KeyCount.

smengcl and others added 2 commits October 6, 2026 12:23

HDDS-16739. Return VersionId in DeleteObjects errors, tidy tests

0d1880e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI balanced review requested due to automatic review settings October 6, 2026 23:02

Copilot started reviewing on behalf of smengcl October 6, 2026 23:03 View session

smengcl requested a review from jojochuang October 6, 2026 23:04

smengcl added enhancement New feature or request AI-gen labels Oct 6, 2026

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java Outdated

HDDS-16739. Reject version-id-marker without key-marker

c674dd8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added documentation Improvements or additions to documentation s3 S3 Gateway labels Oct 6, 2026

fmorg-git reviewed Oct 6, 2026

View reviewed changes

Comment thread hadoop-hdds/docs/content/interface/S3.md Outdated

peterxcli requested review from Gargi-jais11 and peterxcli October 7, 2026 00:37

HDDS-16739. Drop S3.md change, the doc moves to ozone-site

376ca6c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

smengcl mentioned this pull request Oct 7, 2026

HDDS-16739. Document ListObjectVersions in the S3 API page apache/ozone-site#566

Draft

Merge branch 'master' into HDDS-16739

dc193a1

smengcl marked this pull request as ready for review October 9, 2026 01:15

smengcl requested a review from fmorg-git October 11, 2026 21:56

fmorg-git reviewed Oct 13, 2026

View reviewed changes

Comment thread hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java Outdated

Comment thread hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java

smengcl marked this pull request as draft October 13, 2026 02:05

HDDS-16739. Address review: ignore ListObjectsV2 params, reject non-n…

ea6bafe

…ull version-id-marker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Gargi-jais11 reviewed Oct 13, 2026

View reviewed changes

Conversation

smengcl commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

smengcl commented Oct 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

fmorg-git commented Oct 6, 2026

Uh oh!

Uh oh!

fmorg-git commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smengcl commented Oct 7, 2026

Uh oh!

Uh oh!

fmorg-git Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smengcl commented Oct 6, 2026 •

edited

Loading

fmorg-git commented Oct 6, 2026 •

edited

Loading