Repository navigation
HDDS-16467. Run S3 smoketests with AWS CLI v1 and v2 (parallel CI) - #11569
Closed
jojochuang wants to merge 3 commits into
Closed
jojochuang wants to merge 3 commits into
jojochuang wants to merge 3 commits into
Conversation
Split acceptance matrix for suites that run the full s3 smoketest tree (HA-unsecure, HA-secure, secure, EC) into separate jobs for aws and aws2 via OZONE_AWS_CLI, instead of running both sequentially in one job. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I322d41797b57cc2c807ec574f42a133a36c3914f
Contributor
|
Please make sure to cancel CI run when changing an open pull request to draft. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
CLI argument handling can select unintended test coverage, and prefixed bucket names remove existing routing regression coverage.
1 open finding
What changed in this PR
Extends Ozone’s S3 smoketests to AWS CLI v1 and v2, with parallel CI jobs intended to reduce runtime.
Changes:
- Splits four acceptance suites into separate CLI-version jobs.
- Adds a shared runner supporting single-CLI or sequential dual-CLI execution.
- Updates Robot tests for CLI selection, response differences, and test-data isolation.
| File | Description |
|---|---|
| hadoop-ozone/dist/src/main/smoketest/s3/rclone.robot | Uses the selected CLI for credentials. |
| hadoop-ozone/dist/src/main/smoketest/s3/presignedurl.robot | Uses the selected CLI for credentials. |
| hadoop-ozone/dist/src/main/smoketest/s3/objectlist.robot | Uses the selected CLI for uploads. |
| hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot | Adapts credentials and timestamp parsing. |
| hadoop-ozone/dist/src/main/smoketest/s3/freon.robot | Updates credential handling and test prefixes. |
| hadoop-ozone/dist/src/main/smoketest/s3/conditionalput.robot | Prefixes keys and adjusts argument quoting. |
| hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot | Adds shared CLI compatibility helpers. |
| hadoop-ozone/dist/src/main/smoketest/s3/bucketlist.robot | Uses the selected CLI for credential changes. |
| hadoop-ozone/dist/src/main/smoketest/s3/bucketcreate.robot | Prefixes endpoint-named test buckets. |
| hadoop-ozone/dist/src/main/compose/testlib.sh | Adds the shared S3 test runner. |
| hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh | Adopts the shared runner. |
| hadoop-ozone/dist/src/main/compose/ozonesecure/test-vault.sh | Adopts the shared runner. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test.sh | Adopts the shared runner. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-haproxy-s3g.sh | Adopts the shared runner. |
| hadoop-ozone/dist/src/main/compose/ozone-ha/test.sh | Adopts the shared runner across layouts. |
| hadoop-ozone/dist/src/main/compose/ozone-ha/test-haproxy-s3g.sh | Adopts the shared runner. |
| hadoop-ozone/dist/src/main/compose/common/ec-test.sh | Adopts the shared runner for EC. |
| hadoop-ozone/dev-support/checks/acceptance.sh | Passes CLI selection to acceptance tests. |
| dev-support/ci/acceptance_suites.sh | Generates CLI-specific matrix entries. |
| .github/workflows/ci.yml | Runs the expanded acceptance matrix. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| : ${OZONE_OLD_VERSIONS:="${2:-}"} | ||
| export OZONE_OLD_VERSIONS | ||
| else | ||
| export OZONE_AWS_CLI="${2:-}" |
adoroszlai
reviewed
Oct 16, 2026
Contributor
There was a problem hiding this comment.
Summary from #11509 (comment):
- We should implement the parallel version in the first commit, not as follow-up.
- We should not run a full matrix. Adding parallel split may restore original wall-clock CI time, overall time would still increase significantly, affecting all other Apache projects (and our other repos), with whom we share the queue of runners. Simply running v1 and v2 in distinct environments (e.g. non-HA and HA, respectively) would also make all the plumbing changes unnecessary.
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I05021b5967786d35795aba3ba2e86e2cee92030d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
smoketest/s3/with AWS CLI v1 and v2 (from HDDS-16467. Run S3 smoketests with AWS CLI v1 and v2 #11509 base).OZONE_AWS_CLI=aws|aws2) instead of running both CLIs sequentially in one job.Motivation
Sequential v1+v2 in
execute_s3_smoketestsadded ~30m wall clock on HA-unsecure acceptance. Parallel matrix legs should restore ~single-pass duration at the cost of extra runners.Test plan
Generated-by: Cursor (Auto)
Made with Cursor