[shiftstack] Bound soft-transient oc exec retries during pod polling - #4192
tusharjadhav3302 wants to merge 1 commit into
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Hi @tusharjadhav3302. Thanks for your PR. I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
84b7935 to
7139f5d
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 56s |
|
recheck |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 07s |
|
/agentic_review |
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR |
7139f5d to
1fb42bd
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 18m 25s |
|
recheck |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 49m 18s |
michburk
left a comment
michburk
left a comment
There was a problem hiding this comment.
One concern, otherwise looks much cleaner. Thanks!
| bash -c 'if cat /tmp/cifmw_cmd_rc 2>/dev/null; then exit 0; | ||
| elif [ -f /tmp/cifmw_cmd_pid ] && kill -0 $(cat /tmp/cifmw_cmd_pid) 2>/dev/null; then exit 10; | ||
| else exit 2; fi' | ||
| cmd: "{{ role_path }}/files/shiftstack_cmd_poll.sh" |
There was a problem hiding this comment.
Just one slight concern over using ansible.builtin.command and role_path here.
Have you tested this and does this work as expected? Situations like this can be a little tricky, as role_path refers to the role path on the controlling host, but the command executes on the remote host, so the file might not be exactly where you expect.
Maybe ansible.builtin.script would be safer?
There was a problem hiding this comment.
Good catch — switched the wait task to ansible.builtin.script with cmd: files/shiftstack_cmd_poll.sh (same pattern as other roles here). That copies the poller from the controller role files/ onto the remote host before running it, so we are not relying on role_path existing on the target.
1fb42bd to
a2bc174
Compare
a2bc174 to
92c0704
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 30s |
michburk
left a comment
michburk
left a comment
There was a problem hiding this comment.
One last concern about the default value. I just want to double check that the patterns you list here match the possible error patterns you expect.
Ride through brief API VIP blips (connection refused / i/o timeout) with a consecutive failure budget instead of failing on the first poll or spinning for the full 8h window when the API is truly unreachable. Hard-transient patterns from openstack-k8s-operators#4150 keep the full poll timeout. Move the poller into roles/shiftstack/files/shiftstack_cmd_poll.sh and invoke it with ansible.builtin.script so it is copied from the controller. Match kubectl's "connection ... was refused" via connection.*refused. Signed-off-by: tusharjadhav3302 <tjadhav@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
8fba0ac to
47ed24d
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 39m 10s |
|
recheck |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 5m 13s |
|
recheck |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 5m 18s |
This branch has not been deployed
4 participants
Summary
oc execerrors (connection refused,i/o timeout) while polling long-running shiftstack tests inshiftstackclient.cifmw-dnsmasqrestarted mid-test) without spinning for ~8h when the API is truly unreachable.Why
On
shiftstack-rhel9-rhoso18.0-ocp4.23(serval70, 2026-09-17) the outer poller failed immediately on:The background
osp_verification_4.23_nightlyrun continued and completed afterward. Re-addingconnection refusedto the unlimited hard-transient list was deliberately rejected in #4150 review; this change uses a bounded soft budget instead (default 12 × 5s ≈ 60s).Test plan
connection refusedduring polling no longer fails the testconfigMade with Cursor