[shiftstack] Bound soft-transient oc exec retries during pod polling - #4192
tusharjadhav3302 wants to merge 1 commit into
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @tusharjadhav3302. Thanks for your PR. I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
84b7935 to
7139f5d
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 56s |
|
recheck |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 07s |
|
/agentic_review |
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR |
| vars: | ||
| _hard_transient_re: "{{ cifmw_shiftstack_exec_transient_patterns | join('|') }}" | ||
| _soft_transient_re: "{{ cifmw_shiftstack_exec_soft_transient_patterns | join('|') }}" | ||
| ansible.builtin.shell: | |
There was a problem hiding this comment.
Not blocking: Logic itself lgtm but I don't love having a big block of shell script in the middle of this ansible file, do you think it would make sense to pull this script out into a dedicated file? It could live somewhere like roles/shiftstack/files/shiftstack_cmd_poll.sh
There was a problem hiding this comment.
Good call — extracted the poller to roles/shiftstack/files/shiftstack_cmd_poll.sh and invoke it from the wait task via role_path. Soft/hard transient handling and exit codes are unchanged; the Ansible task just sets the poll env vars and normalizes the attempts metadata from stderr.
Ride through brief API VIP blips (connection refused / i/o timeout) with a consecutive failure budget instead of failing on the first poll or spinning for the full 8h window when the API is truly unreachable. Hard-transient patterns from openstack-k8s-operators#4150 keep the full poll timeout. Move the poller into roles/shiftstack/files/shiftstack_cmd_poll.sh. Signed-off-by: tusharjadhav3302 <tjadhav@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
7139f5d to
1fb42bd
Compare
This branch has not been deployed
4 participants
Summary
oc execerrors (connection refused,i/o timeout) while polling long-running shiftstack tests inshiftstackclient.cifmw-dnsmasqrestarted mid-test) without spinning for ~8h when the API is truly unreachable.Why
On
shiftstack-rhel9-rhoso18.0-ocp4.23(serval70, 2026-09-17) the outer poller failed immediately on:The background
osp_verification_4.23_nightlyrun continued and completed afterward. Re-addingconnection refusedto the unlimited hard-transient list was deliberately rejected in #4150 review; this change uses a bounded soft budget instead (default 12 × 5s ≈ 60s).Test plan
connection refusedduring polling no longer fails the testconfigMade with Cursor