Repository navigation
CLDSRV-1012: Fix CI development/7.10 - #6321
Conversation
Fail the check (both on startup and /ready route) only if all backends are failing: so that the working backends can still be used. Issue: CLDSRV-110 (cherry picked from commit f567903)
Now fail if ALL backends of only one client fails. Previously, the deep healthcheck would fail if ALL backends/locations failed globally across all clients (data, metadata, vault, kms). This change modifies the logic to fail if ANY client has ALL its backends/locations failing. This ensures: 1. For data backend with multiple sproxyd location constraints: - Returns HTTP 200 if at least ONE location is healthy - Returns HTTP 500 only if ALL locations fail 2. Each client (data, metadata, vault, kms) is evaluated independently - If ALL locations of the data client fail, overall check fails - If ALL locations of metadata fail, overall check fails - etc. The new logic uses: - `results.some()` to check across clients - `keys.every()` within each client to check all its locations (cherry picked from commit ae94080)
(cherry picked from commit 6dd43a5)
The eslint version used on 7.10 cannot parse logical assignment operators and enforces consistent quoting of object keys. Co-Authored-By: Claude <noreply@anthropic.com>
They had been disabled for a long while and cannot be reenabled yet because they don't pass, so keep on skipping them for now. Tickets have been created to take care of them: - CLDSRV-440 - CLDSRV-441 - CLDSRV-442 - CLDSRV-443 (cherry picked from commit df646e4) Backport to 7.10: only the skip of the 'Azure container creation' tests (CLDSRV-441) is selected from this commit. All other changes are ignored.
Hello bourgoismickael,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
|
/create_integration_branches |
ConflictA conflict has been raised during the creation of I have not created the integration branch. Here are the steps to resolve this conflict: git fetch
git checkout -B w/8.8/improvement/CLDSRV-1012-fix-7.10-ci origin/development/8.8
git merge origin/w/7.70/improvement/CLDSRV-1012-fix-7.10-ci
# <intense conflict resolution>
git commit
git push -u origin w/8.8/improvement/CLDSRV-1012-fix-7.10-ciThe following options are set: create_integration_branches |
ConflictA conflict has been raised during the creation of I have not created the integration branch. Here are the steps to resolve this conflict: git fetch
git checkout -B w/9.2/improvement/CLDSRV-1012-fix-7.10-ci origin/development/9.2
git merge origin/w/9.1/improvement/CLDSRV-1012-fix-7.10-ci
# <intense conflict resolution>
git commit
git push -u origin w/9.2/improvement/CLDSRV-1012-fix-7.10-ciThe following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
The following options are set: create_integration_branches |
francoisferrand
left a comment
francoisferrand
left a comment
There was a problem hiding this comment.
LGTM, but I don't see how this relates to fixing CI due multiple backend and GCP failing...
multiple-backend fails on development/7.X since after 2026-02-05.
Because of location constraints with GCP buckets (zenko-gcp-*) that no longer exists (since after Feb 2026).
Cloudserver doesn't start in the CI because of the initial healthcheck that fails on 1 GCP backend and blocks startup. If healthcheck is relaxed, cloudserver does start. |
|
The whole healthcheck log line that blocks startup {
"name": "S3",
"error": {
"InternalError": true
},
"healthStatus": {
"us-east-1": {
"code": 200,
"message": "OK"
},
"scality-internal-file": {
"code": 200,
"message": "OK"
},
"dataFile": {
"code": 200,
"message": "OK"
},
"scality-internal-mem": {
"code": 200,
"message": "OK"
},
"scality-us-west-1": {
"code": 200,
"message": "OK"
},
"awsbackendencryption": {
"message": "Congrats! You own the bucket"
},
"awsbackend": {
"message": "Congrats! You own the bucket"
},
"awsbackendmismatch": {
"message": "Congrats! You own the bucket"
},
"awsbackend2": {
"message": "Congrats! You own the bucket"
},
"azurebackend": {
"message": "Congrats! You can access the Azure storage account"
},
"azurebackend2": {
"message": "Congrats! You can access the Azure storage account"
},
"azurebackendmismatch": {
"message": "Congrats! You can access the Azure storage account"
},
"azurenonexistcontainer": {
"message": "Congrats! You can access the Azure storage account"
},
"gcpbackend": {
"error": true,
"external": true,
"mpu": {
"gcpBucket": "zenko-gcp-mpu",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.374Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
},
"main": {
"gcpBucket": "zenko-gcp-bucket",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.643Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
}
},
"gcpbackend2": {
"error": true,
"external": true,
"main": {
"gcpBucket": "zenko-gcp-bucket-2",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.431Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
},
"mpu": {
"gcpBucket": "zenko-gcp-mpu-2",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.523Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
}
},
"gcpbackendmismatch": {
"error": true,
"external": true,
"mpu": {
"gcpBucket": "zenko-gcp-mpu",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.374Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
},
"main": {
"gcpBucket": "zenko-gcp-bucket",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.442Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
}
},
"gcpbackendproxy": {
"error": true,
"external": true,
"main": {
"gcpBucket": "zenko-gcp-bucket",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.433Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
},
"mpu": {
"gcpBucket": "zenko-gcp-mpu",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.491Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
}
},
"gcpbackendnoproxy": {
"error": true,
"external": true,
"mpu": {
"gcpBucket": "zenko-gcp-mpu",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.375Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
},
"main": {
"gcpBucket": "zenko-gcp-bucket",
"error": {
"message": null,
"code": 404,
"time": "2026-10-02T14:46:04.376Z",
"requestId": "...",
"statusCode": 404,
"retryable": false
}
}
},
"memorybucket": {
"code": 200,
"message": "OK"
},
"vaultMem": {
"code": 200,
"message": "OK"
},
"memoryKms": {
"code": 200,
"message": "OK"
}
},
"time": 1790952364643,
"req_id": "c59948bfaf4ea0b554de",
"level": "info",
"message": "initial health check failed, delaying startup",
"hostname": "runnervm8df0l",
"pid": 130
} |
|
/approve |
Build failedThe build for commit did not succeed in branch w/8.8/improvement/CLDSRV-1012-fix-7.10-ci The following options are set: approve, create_integration_branches |
ConflictA conflict has been raised during the update of Please resolve the conflict on the integration branch ( Here are the steps to resolve this conflict: git fetch
git checkout w/9.0/improvement/CLDSRV-1012-fix-7.10-ci
git pull # or "git reset --hard origin/w/9.0/improvement/CLDSRV-1012-fix-7.10-ci"
git merge origin/development/9.0
# <intense conflict resolution>
git commit
git merge origin/w/8.8/improvement/CLDSRV-1012-fix-7.10-ci
# <intense conflict resolution>
git commit
git push -u origin w/9.0/improvement/CLDSRV-1012-fix-7.10-ciThe following options are set: approve, create_integration_branches |
|
Added a commit in branch 8.8 d73b60b to fix ceph in the CI using a wrong network interface |
ConflictA conflict has been raised during the update of Please resolve the conflict on the integration branch ( Here are the steps to resolve this conflict: git fetch
git checkout w/9.2/improvement/CLDSRV-1012-fix-7.10-ci
git pull # or "git reset --hard origin/w/9.2/improvement/CLDSRV-1012-fix-7.10-ci"
git merge origin/development/9.2
# <intense conflict resolution>
git commit
git merge origin/w/9.1/improvement/CLDSRV-1012-fix-7.10-ci
# <intense conflict resolution>
git commit
git push -u origin w/9.2/improvement/CLDSRV-1012-fix-7.10-ciThe following options are set: approve, create_integration_branches |
ConflictA conflict has been raised during the update of Please resolve the conflict on the integration branch ( Here are the steps to resolve this conflict: git fetch
git checkout w/9.3/improvement/CLDSRV-1012-fix-7.10-ci
git pull # or "git reset --hard origin/w/9.3/improvement/CLDSRV-1012-fix-7.10-ci"
git merge origin/development/9.3
# <intense conflict resolution>
git commit
git merge origin/w/9.2/improvement/CLDSRV-1012-fix-7.10-ci
# <intense conflict resolution>
git commit
git push -u origin w/9.3/improvement/CLDSRV-1012-fix-7.10-ciThe following options are set: approve, create_integration_branches |
|
I have successfully merged the changeset of this pull request
The following branches have NOT changed:
This pull request did not target the following hotfix branch(es) so they
Please check the status of the associated issue CLDSRV-1012. Goodbye bourgoismickael. The following options are set: approve, create_integration_branches |
4 participants
Need to fix CI to backport a hotfix for RD-2378.
multiple-backendfails ondevelopment/7.Xsince after 2026-02-05.Because of location constraints with GCP buckets (
zenko-gcp-*) that no longer exists (since after Feb 2026).On 7.x, any failing location blocks startup. Fixed in 8.x and 9.x with relaxed health check.
Changes
||=,quote-props)In 9.5, the test config still points GCP locations at
zenko-gcp-*, but the tests using them are skipped. The raw-node GCP tests that run use their owncldsrvci-*buckets and bypass cloudserver.Should we recreate those buckets for 7.X hotfixes instead of this PR ?