Skip to content

Retry transient oc exec errors during shiftstack pod polling - #4150

Merged
brjackma merged 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry
Sep 17, 2026
Merged

brjackma merged 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry

Conversation

@tusharjadhav3302

Copy link
Copy Markdown
Contributor

Summary

  • Treat known transient oc exec stderr patterns as retryable while polling long-running shiftstack tests in shiftstackclient
  • Prevents false post-deployment failures when the API/admission webhook returns transient errors (for example SCCExecRestrictions timeouts) while the background command is still running
  • Improves the unreachable-pod failure message with the polling attempt limit

Context

shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 failed in post-deployment after ~32 minutes even though:

  • RHOSO deploy and openstack-edpm were healthy
  • The nested OCP 4.22 install inside shiftstackclient continued and completed successfully afterward

Failure signature:

admission plugin "security.openshift.io/SCCExecRestrictions" failed to complete validation in 13s

Related Jira: https://redhat.atlassian.net/browse/OSPRH-36061

Test plan

  • Re-run shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 and confirm post-deployment survives transient oc exec polling errors during osp_verification_4.22_nightly.yaml
  • Confirm non-transient failures (for example pod NotFound) still fail promptly

Made with Cursor

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b4f2059d92234a2898c1b7c94a2ac513

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 33m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 31m 09s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 51m 37s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 20m 17s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 17m 08s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 41m 41s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 03s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 19s
cifmw-molecule-shiftstack FAILURE in 21m 20s

@eshulman2

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@bshewale

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from e75f403 to fb296a0 Compare August 31, 2026 07:32
@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

2 similar comments
@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from fb296a0 to 6df275c Compare August 31, 2026 12:31
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/2260089efffe45f2ac95d8615f6248ac

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 11m 00s
podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000186871 failed in 0s
cifmw-crc-podified-edpm-baremetal FAILURE in 49m 00s
cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 50m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 16s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 22m 25s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 44m 37s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 22s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 16s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 6df275c to b18c45e Compare September 1, 2026 06:52
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from b18c45e to 2857eaa Compare September 1, 2026 19:56
Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 2857eaa to 5aedad7 Compare September 2, 2026 09:55
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from a06e258 to 4ac0715 Compare September 2, 2026 10:13
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/f6ebbb51f37a4d78a54474aca81d3978

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 23s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 37m 53s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 2h 08m 38s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 12m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 02s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 43m 49s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 2h 02m 57s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 45s
✔️ cifmw-pod-pre-commit SUCCESS in 10m 30s
cifmw-molecule-shiftstack FAILURE in 19m 05s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 4ac0715 to a418566 Compare September 3, 2026 05:46
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/fb9c14f1c3ae4ac6bed2b13a4a3cc21a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 34m 41s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 33m 51s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 57m 40s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 17m 51s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 57s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 50m 19s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 2h 03m 29s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 51s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
cifmw-molecule-shiftstack FAILURE in 19m 35s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from a418566 to ecb6282 Compare September 7, 2026 06:42
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d95dc30870044ae0ab7dea2df1043216

✔️ openstack-k8s-operators-content-provider SUCCESS in 14m 30s
podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000193806 failed in 0s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000193807 failed in 0s
cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000193808 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 47s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 38m 26s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000193827 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 27s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 27s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 41s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ff3e53fff38b47d1a48763482d1c6894

openstack-k8s-operators-content-provider FAILURE in 7m 23s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 24s
openstack-k8s-operators-content-provider-bootc FAILURE in 28m 35s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 30s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 07s
✔️ cifmw-molecule-shiftstack SUCCESS in 30m 23s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from bc9331b to af5a7b1 Compare September 15, 2026 11:26
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/effed30064374bc5be7f234de9b46bf3

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 24m 55s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 22s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 30s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 59m 49s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 12s
openstack-k8s-operators-content-provider-bootc FAILURE in 10m 08s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 08s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 47s
cifmw-molecule-shiftstack FAILURE in 22m 35s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from af5a7b1 to 7151ea5 Compare September 15, 2026 14:09
@brjackma

Copy link
Copy Markdown
Contributor

/lgtm

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c631f59d4b23433a953d75ae267f2b6a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 13s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 15s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 38m 41s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 00m 16s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 34s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 06m 41s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000203956 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 15s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 45s
cifmw-molecule-shiftstack FAILURE in 19m 10s

@brjackma

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/dee38fc1a3a74f4a8bb57e1c39b44d1a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 16s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 55s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 27m 10s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 56s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 22s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 08m 12s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 32m 07s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 47s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 18s
cifmw-molecule-shiftstack FAILURE in 23m 54s

@brjackma

Copy link
Copy Markdown
Contributor

/retest-required

@brjackma

Copy link
Copy Markdown
Contributor

recheck-required

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ad387c3d1a7e4a9ba2eeac86c9285a98

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 32s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 25m 39s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 39s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 59m 33s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 00s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 16m 22s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 28m 02s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 58s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 25s
cifmw-molecule-shiftstack TIMED_OUT in 31m 08s

@brjackma

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/bf8b061248304b098be41e3bda978011

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 25m 57s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 15s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 08s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 07m 53s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 55s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 24m 25s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000205054 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 26s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
cifmw-molecule-shiftstack FAILURE in 18m 44s

@Valkyrie00

Copy link
Copy Markdown
Contributor

Recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/19351ee2fcb14a4ba80756a65af26b71

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 30m 02s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 39s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000205281 failed in 0s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 51m 44s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 45s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 58m 01s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 24m 18s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 38s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 26s
cifmw-molecule-shiftstack FAILURE in 21m 07s

…ling

Long-running osp_verification tests poll shiftstackclient via oc exec.
Transient API/admission failures (for example SCCExecRestrictions webhook
timeouts) were treated as pod loss and aborted post-deployment early.

Keep polling on configurable transient stderr patterns so the test can
finish when the background command is still running. Patterns live in
defaults; failure messages report actual attempt count.

Related: OSPRH-36061
Signed-off-by: Tushar Jadhav <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@Valkyrie00
Valkyrie00 force-pushed the fix/shiftstack-exec-transient-retry branch from 7151ea5 to 8f9a744 Compare September 17, 2026 08:52
@openshift-ci openshift-ci Bot removed the lgtm label Sep 17, 2026

@Valkyrie00 Valkyrie00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/2775dc4423ec49d2a5440fb7569a1a91

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 21m 14s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 04s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 28m 32s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 05m 59s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 11s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 07m 45s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 31m 21s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 06s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 23s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 25s

@brjackma
brjackma merged commit 756b525 into openstack-k8s-operators:main Sep 17, 2026
10 checks passed
tusharjadhav3302 added a commit to tusharjadhav3302/ci-framework that referenced this pull request Sep 18, 2026
Ride through brief API VIP blips (connection refused / i/o timeout)
with a consecutive failure budget instead of failing on the first poll
or spinning for the full 8h window when the API is truly unreachable.
Hard-transient patterns from openstack-k8s-operators#4150 keep the full poll timeout. Enable
pipefail in the poller shell for ansible-lint.

Signed-off-by: tusharjadhav3302 <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants