Skip to content

(multiple) Allow OCP cluster reuse when switching architectures - #4145

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
abays:fix/devscripts-ocp-reuse-across-architectures
Aug 31, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
abays:fix/devscripts-ocp-reuse-across-architectures

Conversation

@abays

@abays abays commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

The devscripts compliance check in 110_check_ocp.yml required an exact match of extra disk volume count to consider an existing OCP deployment reusable. When switching architectures after a non-deepscrub clean (e.g. va-multi with 3 extra disks to va-hci with 2), the volume count mismatch forced a full make clean followed by make all, triggering an hour-plus OCP rebuild that is unnecessary — the OCP boot disk images are architecture-agnostic and extra disks are blank storage volumes that deploy_layout.yml recreates with the correct count.

Remove _needed_volumes from the compliance condition so that the boot disk image count and cluster directory are sufficient to determine reusability. Clean up stale volume attachment XML files during non-deepscrub cleanup to prevent a decrease in extra_disks_num from leaving orphan XMLs that would cause create_vms.yml to attach the wrong number of disks. Add a safety-net block in ocp_layout.yml to restore libvirt networks destroyed by make clean for cases where a genuine full rebuild is still required (e.g. master count change).

The devscripts compliance check in `110_check_ocp.yml` required an
exact match of extra disk volume count to consider an existing OCP
deployment reusable. When switching architectures after a non-deepscrub
clean (e.g. va-multi with 3 extra disks to va-hci with 2), the volume
count mismatch forced a full `make clean` followed by `make all`,
triggering an hour-plus OCP rebuild that is unnecessary — the OCP boot
disk images are architecture-agnostic and extra disks are blank storage
volumes that `deploy_layout.yml` recreates with the correct count.

Remove `_needed_volumes` from the compliance condition so that the boot
disk image count and cluster directory are sufficient to determine
reusability. Clean up stale volume attachment XML files during
non-deepscrub cleanup to prevent a decrease in `extra_disks_num` from
leaving orphan XMLs that would cause `create_vms.yml` to attach the
wrong number of disks. Add a safety-net block in `ocp_layout.yml` to
restore libvirt networks destroyed by `make clean` for cases where a
genuine full rebuild is still required (e.g. master count change).

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Andrew Bays <abays@redhat.com>
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Allow OCP cluster reuse across architecture switches

🐞 Bug fix 🕐 20-40 Minutes

Grey Divider

AI Description

• Reuse architecture-agnostic OCP clusters despite changes to blank extra-disk counts.
• Remove stale disk attachment XML during non-deepscrub cleanup.
• Restore required libvirt networks before full OCP rebuilds.
Diagram

graph TD
  A["Non-deepscrub cleanup"] --> B["Remove stale XML"] --> C["Check OCP artifacts"] --> D{"Cluster reusable?"}
  D -->|Yes| E["Reuse OCP cluster"]
  D -->|No| F["Devscripts cleanup"] --> G["Restore networks"] --> H["Bootstrap OCP"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Persist a compatibility manifest
  • ➕ Explicitly records which durable cluster properties permit reuse
  • ➕ Can accommodate future architecture-sensitive boot artifacts
  • ➖ Introduces manifest versioning and migration complexity
  • ➖ Requires synchronizing metadata with every deployment and cleanup path
2. Reconcile volumes before compliance
  • ➕ Retains strict volume-count validation
  • ➕ Makes the on-disk pool match the requested layout before reuse
  • ➖ Couples cluster compatibility to disposable blank storage
  • ➖ Requires more libvirt mutations and failure handling during preflight

Recommendation: The PR's artifact-based compliance check is the best fit because boot images and cluster state determine reuse, while blank disks are recreated by layout deployment. Removing stale XML prevents incorrect attachments, and restoring networks safely covers the genuine rebuild path; a compatibility manifest is only warranted if durable artifacts later become architecture-dependent.

Files changed (3) +75 / -1

Bug fix (3) +75 / -1
110_check_ocp.ymlIgnore disposable volume counts during OCP compliance +0/-1

Ignore disposable volume counts during OCP compliance

• Removes the extra-volume count from the reuse decision. Existing clusters now comply when their cluster directory and expected boot images are present, allowing reuse across layouts with different blank-disk counts.

roles/devscripts/tasks/110_check_ocp.yml

clean_layout.ymlDelete stale OCP volume attachment XML +14/-0

Delete stale OCP volume attachment XML

• Finds and removes generated volume attachment XML during cleanup. This prevents obsolete definitions from attaching too many disks after reducing the requested extra-disk count.

roles/libvirt_manager/tasks/clean_layout.yml

ocp_layout.ymlRestore fixed libvirt networks before OCP rebuild +61/-0

Restore fixed libvirt networks before OCP rebuild

• After a noncompliant cluster triggers devscripts cleanup, detects missing fixed networks and defines, starts, and enables them. This preserves VBMC connectivity required for bare-metal OCP bootstrap.

roles/reproducer/tasks/ocp_layout.yml

@qodo-code-review

qodo-code-review Bot commented Aug 26, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Missing disks accepted ✗ Dismissed 🐞 Bug ≡ Correctness
Description
Removing _needed_volumes makes 110_check_ocp.yml mark a cluster compliant even when it has fewer
extra volumes than the requested layout. On a switch that increases extra_disks_num, the compliant
path skips OCP VM/volume creation, so the reused masters never receive the newly required disks.
Code

roles/devscripts/tasks/110_check_ocp.yml[92]

-    - _needed_volumes | bool
Evidence
The removed fact was computed by comparing discovered master-0 volume files with the requested
extra_disks_num; without it, compliance depends only on cluster/base-image existence. Compliance
then bypasses the reproducer block that calls libvirt_manager's OCP layout, and that skipped path
is where create_vms.yml creates the requested extra disks. Repository scenarios include layouts
with three and four OCP disks, proving that an increase is a supported configuration transition and
can expose the undersized reuse path.

roles/devscripts/tasks/110_check_ocp.yml[70-93]
roles/reproducer/tasks/ocp_layout.yml[212-238]
roles/libvirt_manager/tasks/ocp_layout.yml[90-104]
roles/libvirt_manager/tasks/create_vms.yml[91-117]
scenarios/reproducers/va-multi.yml[80-88]
scenarios/reproducers/va-multi-skmo.yml[80-88]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The compliance check ignores extra-volume count entirely, so a cluster with fewer disks than requested is reused and the missing disks are never created.

## Issue Context
Reuse should tolerate surplus blank volumes when moving to a layout with fewer disks, but it must still reject an existing cluster whose volume count is below `extra_disks_num`. A lower-bound comparison preserves the intended reuse case without accepting an undersized cluster.

## Fix Focus Areas
- roles/devscripts/tasks/110_check_ocp.yml[70-93]
- roles/reproducer/tasks/ocp_layout.yml[212-238]
- roles/libvirt_manager/tasks/create_vms.yml[91-117]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
Review mode: ⚖️ Balanced

Grey Divider

Tip of the day
💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread roles/devscripts/tasks/110_check_ocp.yml
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/716d38251b5f4f3cb407c252cc1e1a98

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 24m 31s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 35m 00s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 48m 48s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 23m 05s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 52s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 50m 57s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 49m 31s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 00s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 00s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 16s
cifmw-molecule-libvirt_manager FAILURE in 43m 08s
✔️ cifmw-molecule-reproducer SUCCESS in 15m 13s

@abays

abays commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/716d38251b5f4f3cb407c252cc1e1a98

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 24m 31s ✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 35m 00s ✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 48m 48s ✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 23m 05s ✔️ cifmw-pod-zuul-files SUCCESS in 4m 52s ✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 50m 57s ✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 49m 31s ✔️ noop SUCCESS in 0s ✔️ cifmw-pod-ansible-test SUCCESS in 10m 00s ✔️ cifmw-pod-pre-commit SUCCESS in 9m 00s ✔️ cifmw-molecule-devscripts SUCCESS in 10m 16s ❌ cifmw-molecule-libvirt_manager FAILURE in 43m 08s ✔️ cifmw-molecule-reproducer SUCCESS in 15m 13s

2026-08-26 13:46:48.654462 | controller | TASK [libvirt_manager : Deactivate default] ************************************
2026-08-26 13:46:48.654466 | controller | Wednesday 26 August 2026  13:46:48 +0000 (0:00:00.131)       0:02:07.796 ******
2026-08-26 13:46:48.888979 | controller | fatal: [instance]: FAILED! => changed=false
2026-08-26 13:46:48.889004 | controller |   msg: network default not found

Could be related. Let's retry and see what happens.

@abays

abays commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@evallesp evallesp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@nemarjan nemarjan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@openshift-ci

openshift-ci Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nemarjan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit 04593d8 into openstack-k8s-operators:main Aug 31, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants