Skip to content

[kustomize_deploy] Bootstrap SKMO leaf admin-two before leaf OSCP apply - #4182

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
Deydra71:fix/skmo-leaf-admin-two
Sep 23, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
Deydra71:fix/skmo-leaf-admin-two

Conversation

@Deydra71

@Deydra71 Deydra71 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

After osp-secret password randomization, create admin-two in central Keystone using AdminPassword from the leaf kustomize manifest, before oc apply.

Remove the stale prepare-leaf bootstrap that reads the static osp-secrets.env template instead of the deployed leaf secret.

Add _leaf_oscp.resources is defined and _keystone_lb_svc.resources is defined. Without this check, accessing resources on the error dict raises an AttributeError, which Ansible treats as a fatal exception (and doesn't retry).

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@Deydra71 Deydra71 changed the title [SKMO] Bootstrap SKMO leaf admin-two before leaf OSCP apply [kustomize_deploy] Bootstrap SKMO leaf admin-two before leaf OSCP apply Sep 15, 2026
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/46884b0f90354feda7ccf697e3de1d8a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 51m 47s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 26s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 53s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 00m 33s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 34s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 19m 51s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 24m 09s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 24s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 46s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 44s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 5m 40s

@vakwetu vakwetu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/299d14a9022c41f898f828fd94331cb7

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 20m 50s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 28s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 32m 02s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 00m 27s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 27s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 16m 09s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 26m 47s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 16s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 15s
✔️ cifmw-architecture-validate-hci SUCCESS in 6m 04s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 6m 59s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8edd09ce66c34ae3b914a95333b41c58

✔️ openstack-k8s-operators-content-provider SUCCESS in 27m 12s
❌ podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000206077 failed in 0s
❌ cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000206078 failed in 0s
❌ cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000206079 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 20s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 36m 28s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 25m 33s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 18s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 55s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 37s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 6m 07s

@Deydra71

Copy link
Copy Markdown
Contributor Author

The test project with dependencies on this PR openstack-k8s-operators/architecture#827 and openstack-k8s-operators/octavia-operator#688 passed

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/6171fcc9fa36414099d5bc8c701cffec

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 53m 01s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 29s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 56s
❌ cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000207056 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 33s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 47m 30s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 29m 12s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 21s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 08s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 39s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 6m 01s

@Deydra71

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/7139967809044b97960278d46f8ff542

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 07m 24s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 45s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 32m 47s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 53m 05s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 39s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 05m 18s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 30m 19s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 26s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 50s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 50s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 5m 53s

@vakwetu vakwetu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Sep 21, 2026
@Deydra71

Copy link
Copy Markdown
Contributor Author

Hello @Valkyrie00 @evallesp ! Can I please get a review on this PR? Thanks!

@Valkyrie00

Copy link
Copy Markdown
Contributor

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Dry runs modify central Keystone 🐞 Bug ≡ Correctness
Description
The new bootstrap lacks a cifmw_kustomize_deploy_generate_crs_only guard and performs live
project, user, password, and role changes even when deployment is configured only to generate
resources. A matching SKMO stage therefore reaches central Keystone despite the later manifest apply
being explicitly skipped in generate-only mode.
Code

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[R8-11]

+  when:
+    - stage.path is defined
+    - "'multi-namespace-skmo/control-plane2' in stage.path"
+    - _osp_secret_manifest_check.rc == 0
Evidence
The include runs before the guarded apply, while the included task's conditions do not exclude
generate-only mode and its shell tasks mutate Keystone.

roles/kustomize_deploy/tasks/execute_step.yml[302-306]
roles/kustomize_deploy/tasks/execute_step.yml[334-351]
roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[94-121]
roles/kustomize_deploy/README.md[31-32]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The SKMO bootstrap performs live central Keystone mutations when `cifmw_kustomize_deploy_generate_crs_only` is enabled, violating generate-only behavior.

## Fix Focus Areas
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[8-11]
- roles/kustomize_deploy/tasks/execute_step.yml[302-305]

## Recommended Fix
Add `not cifmw_kustomize_deploy_generate_crs_only | bool` to the bootstrap guard so no Kubernetes queries or Keystone mutations occur during generate-only runs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Bootstrap targets the wrong cluster 🐞 Bug ≡ Correctness
Description
The new k8s_info calls and oc commands omit cifmw_openshift_kubeconfig, token, and context
settings even though the surrounding deployment uses them. When the configured cluster is not the
process default, readiness checks and Keystone mutations fail or run against another cluster before
the manifest is applied to the intended one.
Code

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[R64-67]

+          kubernetes.core.k8s_info:
+            api_version: keystone.openstack.org/v1beta1
+            kind: KeystoneAPI
+            namespace: "{{ _central_namespace }}"
Evidence
The new cluster operations provide no explicit connection configuration, while both the existing
secret lookup and subsequent apply consistently use the role's configured kubeconfig.

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[63-83]
roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[94-121]
roles/kustomize_deploy/tasks/inject_osp_secret_keys.yml[63-73]
roles/kustomize_deploy/tasks/execute_step.yml[345-359]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The SKMO bootstrap does not use the OpenShift connection configured for the deployment and may query or modify a different cluster.

## Fix Focus Areas
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[63-83]
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[94-121]

## Recommended Fix
Pass `kubeconfig`, optional API token, and optional context to both `k8s_info` tasks, and set `KUBECONFIG` plus the configured `PATH` for both shell tasks.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Transient API failures abort bootstrap 🐞 Bug ☼ Reliability
Description
Both new readiness loops dereference .resources without first checking that the registered
k8s_info result defines it. A transient authentication, connection, or API error returns an
error-shaped result, so evaluating the first until expression can fail instead of consuming the
configured retries.
Code

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[R71-73]

+          until:
+            - _keystoneapi_info.resources | length > 0
+            - _keystoneapi_info.resources[0].status.conditions is defined
Evidence
The new loops use .resources as their first predicate, whereas this PR adds the exact
missing-field guard to equivalent existing waits to prevent Ansible from aborting retries.

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[63-76]
roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[78-92]
hooks/playbooks/skmo/configure-leaf-keystone-internal.yaml[87-94]
hooks/playbooks/skmo/configure-leaf-keystone-internal.yaml[131-137]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new Kubernetes readiness loops can abort on an error result because they access `.resources` before confirming that field exists.

## Fix Focus Areas
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[71-76]
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[87-92]

## Recommended Fix
Add `_keystoneapi_info.resources is defined` and `_osc_pod_info.resources is defined` as the first predicates in their respective `until` lists before any length or index access.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View high (1)
4. Config values execute shell commands 🐞 Bug ⛨ Security
Description
The bootstrap embeds project and user values unquoted and embeds the manifest password inside unsafe
double quotes in Bash source. Shell metacharacters from skmo-values.yaml or a preserved existing
secret can therefore alter the command and execute on the Ansible host during project or user
configuration.
Code

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[R107-110]

+            if oc -n {{ _central_namespace }} rsh openstackclient \
+              openstack user show {{ _leaf_admin_user }} >/dev/null 2>&1; then
+              oc -n {{ _central_namespace }} rsh openstackclient \
+                openstack user set --password "{{ _leaf_admin_password }}" {{ _leaf_admin_user }}
Evidence
The values are loaded from architecture YAML or decoded from the generated secret and then inserted
directly into Bash; existing cluster secret values are preserved unchanged rather than restricted to
the random generator's safe alphabet.

roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[35-60]
roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[94-118]
roles/kustomize_deploy/tasks/inject_osp_secret_keys.yml[193-229]
roles/kustomize_deploy/files/osp_secret_manifest.py[76-104]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Repository configuration and existing secret values are interpolated directly into Bash source, permitting command injection and argument corruption.

## Fix Focus Areas
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[94-100]
- roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml[104-118]

## Recommended Fix
Apply Ansible's `quote` filter to every interpolated namespace, project, user, and password value, removing the manual password double quotes; preferably replace shell construction with argument-vector commands where control flow permits.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
Review mode: ⚖️ Balanced: This changes deployment sequencing and credential/bootstrap behavior across Ansible tasks, Kubernetes readiness handling, and Keystone authorization, creating meaningful operational and security risk but not enough independent logic density to warrant extended review.

Grey Divider

Tip of the day
💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml
Comment thread roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml
Comment thread roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml
Comment thread roles/kustomize_deploy/tasks/configure_skmo_leaf_admin_user.yml Outdated
@Valkyrie00

Valkyrie00 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Hey @Deydra71 👋, I just left a comment on a specific point, I think it's important to take a look at it, and if you could take a look at what qodo found, but overall LGTM.

After osp-secret password randomization, create admin-two in central
Keystone using AdminPassword from the leaf kustomize manifest, before
oc apply. Remove the stale prepare-leaf bootstrap that read the static
osp-secrets.env template instead of the deployed leaf secret.

Add `_leaf_oscp.resources is defined` and `_keystone_lb_svc.resources is defined`
Without this check, accessing .resources on the error dict raises an AttributeError,
which Ansible treats as a fatal exception (and doesn't retry).

Signed-off-by: Veronika Fisarova <vfisarov@redhat.com>
@Deydra71
Deydra71 force-pushed the fix/skmo-leaf-admin-two branch from cc62440 to 6a1a8a0 Compare September 22, 2026 09:34
@openshift-ci openshift-ci Bot removed the lgtm label Sep 22, 2026
@Deydra71

Copy link
Copy Markdown
Contributor Author

@Valkyrie00 Thanks for the reviews&assistance! I addressed findings #3 and #4.

Findings #1 and #2 are not valid. SKMO never runs in geenrate only mode, it's always as full deployment and in one OCP cluster

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/851ca651720948149f98d57d8c170352

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 46m 10s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 11s
❌ cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000210029 failed in 0s
❌ cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000210030 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 26s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 18m 27s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 28m 53s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 07s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 21s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 37s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 6m 10s

@Deydra71

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/22be58a0d4dd4af2a8a7108f82566d27

✔️ openstack-k8s-operators-content-provider SUCCESS in 6h 10m 23s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 52s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 36m 18s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 55s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 32s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 4h 03m 29s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 28m 36s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 21s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 59s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 57s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 6m 13s

@Valkyrie00 Valkyrie00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@brjackma

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brjackma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit f6464b6 into openstack-k8s-operators:main Sep 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants