problem
We are experiencing intermittent/random VM deployment failures in CloudStack 4.22.1 with XCP-ng 8.3.
During the investigation, we observed that the template copy/base volume associated with the deployment is no longer present in the primary storage. However, the corresponding database entry still exists in the template_spool_ref table.
When CloudStack subsequently attempts to deploy a VM from the affected template, it appears to retrieve the stale storage reference from the database and attempts to access a VDI that no longer exists.
The deployment then fails with an XCP-ng API error indicating that the VDI UUID is invalid.
versions
Apache CloudStack: 4.22.1
Hypervisor: XCP-ng 8.3
Storage: Presetup [iSCSI]
The steps to reproduce the bug
VM Launch is failing below is observed in logs:
2026-09-08 15:02:51,295 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
2026-09-08 15:02:51,295 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
What to do about it?
Should the corresponding template_spool_ref entry be removed or marked appropriately when the template copy is deleted?
problem
We are experiencing intermittent/random VM deployment failures in CloudStack 4.22.1 with XCP-ng 8.3.
During the investigation, we observed that the template copy/base volume associated with the deployment is no longer present in the primary storage. However, the corresponding database entry still exists in the template_spool_ref table.
When CloudStack subsequently attempts to deploy a VM from the affected template, it appears to retrieve the stale storage reference from the database and attempts to access a VDI that no longer exists.
The deployment then fails with an XCP-ng API error indicating that the VDI UUID is invalid.
versions
Apache CloudStack: 4.22.1
Hypervisor: XCP-ng 8.3
Storage: Presetup [iSCSI]
The steps to reproduce the bug
VM Launch is failing below is observed in logs:
What to do about it?
Should the corresponding template_spool_ref entry be removed or marked appropriately when the template copy is deleted?