Nasbackup quiesce fixes

[MitchDrage]

Description

Fixes #14295, one commit per issue:

1. Failed scheduled backup shown as MANUAL. When cleanup fails, the NAS provider keeps the backup row in Error state, but the schedule id was only written on success. The provider now returns that row and BackupManagerImpl records the schedule id, name and description on it.
2. Guest agent timeout not configurable. New zone-scoped setting nas.backup.quiesce.agent.timeout (default 30 seconds, 0 keeps libvirt's default of 5), passed to nasbackup.sh as --quiesce-timeout and on to virsh --timeout for freeze and thaw. The per-VM override from the issue is left out.
3. cleanup() removing the destination under a running job. cleanup() now runs virsh domjobabort and waits up to 60 seconds for the job to end. If it is still running, the files and mount are left in place and the backup goes to Error state. Also fixed the Failed branch of the job polling loop, which called cleanup without exiting.
4. Failed freeze never thawed. The script now always thaws after a freeze attempt and logs the freeze error. A failed thaw only fails the backup if the freeze succeeded, so VMs without a guest agent still get an unquiesced backup.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

I've listed this as major as I'm having backups continuously fail. I've applied the timeout to nasbackup.sh by hand as a workaround and I'm waiting on the next scheduled run to confirm it. I'll add a comment tomorrow as to the result, or add some changes to fix it.

How Has This Been Tested?

• Unit tests: new cases in BackupManagerTest, NASBackupProviderTest and a new LibvirtTakeBackupCommandWrapperTest. These and the NAS plugin suite pass.
• Script: ran nasbackup.sh against stub virsh/mount commands for each failure path, including an abort that never completes.
• Real host (4.22.1.1, libvirt 11.10.0, QEMU 10.1.0, NFS repository): a quiesced backup with --timeout 30 on freeze and thaw completed. Separately, aborting a running push backup ended the job about 1.4 seconds after domjobabort returned, after which the destination could be removed and unmounted.

How did you try to break this feature and the system with this change?

• Simulated: Guest agent not running, with quiesce on: freeze and thaw both fail, and the backup still completes unquiesced.
• Simulated: Thaw failing after a successful freeze: the job is aborted before the destination is removed and unmounted.
• Simulated: An abort that never completes: the files and mount are left alone and the backup goes to Error state.
• Simulated: libvirt reporting the job as Failed: the script now exits. The original kept polling.
• Real host: Aborting a backup mid-copy: the VM was unaffected and QEMU released the files within about 1.4 seconds.

[abh1sar]

@MitchDrage thanks for the PR. I'll review/test it in the next few days.

[Copilot]

Copilot review overview

🟡 Changes recommended

Ambiguous freeze/thaw and job-status failures can still leave guests frozen or delete destinations under active jobs, while failed-row retention remains incomplete.

Review effort: Balanced
Findings: 2 · 1

Open (3)

• Avoid reporting success when guest freeze state is ambiguous · New
• Do not treat domjobinfo failures as completed backup jobs · New
• Apply retention cleanup to failed backup records · New

What changed in this PR

Fixes NAS backup quiescing, cleanup, and failed scheduled-backup handling.

Changes:

• Adds configurable guest-agent freeze/thaw timeouts.
• Aborts active libvirt jobs before cleanup.
• Preserves metadata for failed scheduled backups and expands tests.

File	Description
BackupManagerTest.java	Tests failed scheduled-backup linkage.
BackupManagerImpl.java	Persists metadata on failed backups.
nasbackup.sh	Handles quiescing, aborts, and cleanup.
LibvirtTakeBackupCommandWrapperTest.java	Tests timeout argument forwarding.
LibvirtTakeBackupCommandWrapper.java	Passes timeout to the script.
NASBackupProviderTest.java	Tests failure and cleanup behavior.
NASBackupProvider.java	Adds timeout configuration and returns Error backups.
TakeBackupCommand.java	Carries the quiesce timeout.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 84.21053% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.03%. Comparing base (2e63c60) to head (90eee16).
⚠️ Report is 11 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...rg/apache/cloudstack/backup/TakeBackupCommand.java	50.00%	3 Missing ⚠️
...ource/wrapper/LibvirtTakeBackupCommandWrapper.java	66.66%	0 Missing and 2 partials ⚠️
...rg/apache/cloudstack/backup/BackupManagerImpl.java	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #14305      +/-   ##
============================================
+ Coverage     18.00%   18.03%   +0.03%     
- Complexity    16219    16261      +42     
============================================
  Files          5936     5936              
  Lines        535716   535849     +133     
  Branches      65596    65614      +18     
============================================
+ Hits          96459    96665     +206     
+ Misses       428268   428166     -102     
- Partials      10989    11018      +29     

Flag	Coverage Δ
uitests	4.04% <ø> (+0.02%)	⬆️
unittests	19.11% <84.21%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[MitchDrage]

@vladimirpetrov @sureshanaparti could this go into the 4.22.2 milestone? It's a bug fix against 4.22.

[abh1sar]

LGTM. Reviewed and tested.

1. Failed scheduled backup shown as MANUAL - Verified
2. Guest agent timeout not configurable - Verified that the timeout value is transmitted and honoured
3. cleanup() removing the destination under a running job. - Verified that cleanup aborted the job
4. Failed freeze never thawed - Verified that the guest was thawed in this case.

Also did regression testing around common backup operations and incremental backups. All passed.

[abh1sar]

@blueorangutan package

[blueorangutan]

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19472

[Damans227]

thanks @MitchDrage, fixes look right. LGTM

[abh1sar]

@blueorangutan test

[blueorangutan]

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-17107)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53685 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14305-t17107-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[abh1sar]

Can we mere this @vladimirpetrov @sureshanaparti ?