CKS/CSI: fix PVC deletion script to avoid '(standard input)' and use kubectl -o name/jsonpath by Pearl1594 · Pull Request #11968 · apache/cloudstack

Pearl1594 · 2025-11-03T18:09:25Z

Description

This PR fixes PVC deletion script to avoid "(standard input)" and use kubectl -o name/jsonpath

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-11-03T18:17:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 17.86%. Comparing base (a9b2f33) to head (459ae94).
⚠️ Report is 1 commits behind head on 4.22.

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #11968      +/-   ##
============================================
- Coverage     17.86%   17.86%   -0.01%     
+ Complexity    16037    16036       -1     
============================================
  Files          5928     5928              
  Lines        534479   534479              
  Branches      65410    65410              
============================================
- Hits          95468    95465       -3     
- Misses       428173   428175       +2     
- Partials      10838    10839       +1

Flag	Coverage Δ
uitests	`4.02% <ø> (ø)`
unittests	`18.92% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sonarqubecloud · 2025-11-03T19:10:40Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Pearl1594 · 2025-11-04T00:44:10Z

@blueorangutan package

blueorangutan · 2025-11-04T00:46:04Z

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-11-04T02:03:18Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15622

github-actions · 2026-03-04T02:15:19Z

This PR is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.

DaanHoogland · 2026-03-04T15:49:13Z

@Pearl1594 , anything to do here?

weizhouapache

code lgtm

have not tested it

Pearl1594 · 2026-06-29T14:16:33Z

@Pearl1594 , anything to do here?

This was more of a cleanup related PR - nothing more to do @DaanHoogland - ready for review / merge..

DaanHoogland · 2026-06-29T15:32:08Z

@blueorangutan package

blueorangutan · 2026-06-29T15:34:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-06-29T16:28:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18411

DaanHoogland · 2026-06-30T05:15:45Z

@blueorangutan test

blueorangutan · 2026-06-30T05:18:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-06-30T19:51:44Z

[SF] Trillian test result (tid-16451)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49543 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11968-t16451-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

clgtm, but I see more similarities than differences in the four changed blocks. Will it make sense to modularize a bit?

DaanHoogland · 2026-07-02T14:05:28Z

also, this code is touched by the kubernetes integration tests, is it? (i.e. smoke tests suffice?)

DaanHoogland · 2026-08-31T11:00:45Z

also, this code is touched by the kubernetes integration tests, is it? (i.e. smoke tests suffice?)

is it @Pearl1594 i.e. can we merge?

Pearl1594 · 2026-08-31T13:36:35Z

I don't believe we don't deploy the CSI driver as part of CKS test suite - so no, it isn't covered by tests @DaanHoogland

Pearl1594 · 2026-08-31T13:52:45Z

I believe it's ready for merge. I tested it almost a year ago, I think it's worthwhile testing again.

sudo87 · 2026-09-09T07:07:00Z

Tested this on a live CKS cluster (KVM, CSI enabled). Deployed a workload with a PVC (reclaimPolicy: Delete), then deleted the cluster to trigger this script.

With the patched script, the log correctly showed:

Processing PVC pr11968-test-pvc in namespace default (PV: pvc-... has reclaimPolicy=Delete)
  Deleting Deployment: pr11968-test-app
Deleting PVC pr11968-test-pvc in namespace default
Script completed successfully!

Deployment was identified and deleted by its actual name — no (standard input) errors. Confirms the fix works as intended.

weizhouapache · 2026-09-09T07:14:05Z

Tested this on a live CKS cluster (KVM, CSI enabled). Deployed a workload with a PVC (reclaimPolicy: Delete), then deleted the cluster to trigger this script.

With the patched script, the log correctly showed:
Processing PVC pr11968-test-pvc in namespace default (PV: pvc-... has reclaimPolicy=Delete)
  Deleting Deployment: pr11968-test-app
Deleting PVC pr11968-test-pvc in namespace default
Script completed successfully!
Deployment was identified and deleted by its actual name — no (standard input) errors. Confirms the fix works as intended.

cool, thanks @sudo87
let's merge

weizhouapache · 2026-09-09T07:15:07Z

oh, wait
@Pearl1594
is this applicable for 4.22 too ?

…kubectl -o name/jsonpath

Pearl1594 · 2026-09-09T14:20:17Z

oh, wait @Pearl1594 is this applicable for 4.22 too ?

Yes @weizhouapache - I've rebased it. Thanks

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Updates the PVC cleanup script to avoid grep -l on piped JSON (which emits "(standard input)") by enumerating workload resources by name and inspecting their PVC references via kubectl ... -o jsonpath.

Changes:

Switch workload discovery from kubectl get ... -o json | grep -l to kubectl get ... -o name loops.
Add per-resource kubectl get <kind> <name> -o jsonpath=... checks for PVC references across multiple workload types.
Extend StatefulSet checks to include both pod template volumes and volumeClaimTemplates.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    /opt/bin/kubectl get statefulsets -n "$namespace" -o name 2>/dev/null | \
+    while IFS= read -r resource; do
+        if [ -z "$resource" ]; then
+            continue
+        fi
+        name=${resource#*/}
+        # Check both template volumes and volumeClaimTemplates
+        sts_volumes=$(/opt/bin/kubectl get statefulset "$name" -n "$namespace" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName} {.spec.volumeClaimTemplates[*].metadata.name}' 2>/dev/null || echo "")


+    /opt/bin/kubectl get deployments -n "$namespace" -o name 2>/dev/null | \
+    while IFS= read -r resource; do


+        dep_volumes=$(/opt/bin/kubectl get deployment "$name" -n "$namespace" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' 2>/dev/null || echo "")
+        if [ -n "$dep_volumes" ] && echo "$dep_volumes" | grep -F -q "$pvc_name"; then


+    /opt/bin/kubectl get deployments -n "$namespace" -o name 2>/dev/null | \
+    while IFS= read -r resource; do
+        if [ -z "$resource" ]; then
+            continue
+        fi
+        name=${resource#*/}
+        dep_volumes=$(/opt/bin/kubectl get deployment "$name" -n "$namespace" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' 2>/dev/null || echo "")
+        if [ -n "$dep_volumes" ] && echo "$dep_volumes" | grep -F -q "$pvc_name"; then
+            echo "$(timestamp) -   Deleting Deployment: $name"
+            /opt/bin/kubectl delete deployment "$name" -n "$namespace" --ignore-not-found=true


+            /opt/bin/kubectl delete deployment "$name" -n "$namespace" --ignore-not-found=true
            deleted_count=$((deleted_count + 1))
        fi
    done


weizhouapache · 2026-09-09T14:37:47Z

oh, wait @Pearl1594 is this applicable for 4.22 too ?

Yes @weizhouapache - I've rebased it. Thanks

thanks @Pearl1594

let's merge when smoke tests pass.

@blueorangutan package

blueorangutan · 2026-09-09T14:40:05Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-09-09T16:14:20Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19182

boring-cyborg Bot added the component:kubernetes label Nov 3, 2025

github-actions Bot added the no-pr-activity label Mar 4, 2026

DaanHoogland added this to the 4.23.0 milestone Mar 4, 2026

github-actions Bot removed the no-pr-activity label Mar 5, 2026

winterhazel added needs-testing needs-review labels Mar 17, 2026

weizhouapache approved these changes Jun 29, 2026

View reviewed changes

weizhouapache modified the milestones: 4.23.0, 4.22.2 Jun 29, 2026

DaanHoogland approved these changes Jul 2, 2026

View reviewed changes

Pearl1594 mentioned this pull request Jul 6, 2026

[CKS] KubernetesCluster destroy hangs indefinitely (no timeout) when SSH-based PV cleanup blocks on a stuck PVC #13544

Open

DaanHoogland added this to CloudStack Testing Aug 10, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 10, 2026

DaanHoogland assigned sudo87 Aug 10, 2026

DaanHoogland removed the needs-review label Aug 31, 2026

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Aug 31, 2026

DaanHoogland unassigned sudo87 Sep 1, 2026

sudo87 self-assigned this Sep 9, 2026

sudo87 moved this from Ready to In review in CloudStack Testing Sep 9, 2026

CKS/CSI: fix PVC deletion script to avoid '(standard input)' and use …

459ae94

…kubectl -o name/jsonpath

Copilot AI lite review requested due to automatic review settings September 9, 2026 14:19

Pearl1594 force-pushed the delete-resources-tied-pvc branch from fdd4ce4 to 459ae94 Compare September 9, 2026 14:19

boring-cyborg Bot added component:compute component:kvm component:orchestration component:saml component:storage component:UI labels Sep 9, 2026

Pearl1594 changed the base branch from main to 4.22 September 9, 2026 14:19

Copilot AI reviewed Sep 9, 2026

View reviewed changes

weizhouapache added status:tested and removed needs-testing labels Sep 9, 2026

Copilot started reviewing on behalf of Pearl1594 September 9, 2026 18:55 View session

		/opt/bin/kubectl get deployments -n "$namespace" -o name 2>/dev/null \| \
		while IFS= read -r resource; do

		dep_volumes=$(/opt/bin/kubectl get deployment "$name" -n "$namespace" -o jsonpath='{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' 2>/dev/null \|\| echo "")
		if [ -n "$dep_volumes" ] && echo "$dep_volumes" \| grep -F -q "$pvc_name"; then

Uh oh!

Conversation

Pearl1594 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonarqubecloud Bot commented Nov 3, 2025

Quality Gate passed

Uh oh!

Pearl1594 commented Nov 4, 2025

Uh oh!

blueorangutan commented Nov 4, 2025

Uh oh!

blueorangutan commented Nov 4, 2025

Uh oh!

github-actions Bot commented Mar 4, 2026

Uh oh!

DaanHoogland commented Mar 4, 2026

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Pearl1594 commented Jun 29, 2026

Uh oh!

DaanHoogland commented Jun 29, 2026

Uh oh!

blueorangutan commented Jun 29, 2026

Uh oh!

blueorangutan commented Jun 29, 2026

Uh oh!

DaanHoogland commented Jun 30, 2026

Uh oh!

blueorangutan commented Jun 30, 2026

Uh oh!

blueorangutan commented Jun 30, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jul 2, 2026

Uh oh!

DaanHoogland commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pearl1594 commented Aug 31, 2026

Uh oh!

Pearl1594 commented Aug 31, 2026

Uh oh!

sudo87 commented Sep 9, 2026

Uh oh!

weizhouapache commented Sep 9, 2026

Uh oh!

weizhouapache commented Sep 9, 2026

Uh oh!

Pearl1594 commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

weizhouapache commented Sep 9, 2026

Uh oh!

blueorangutan commented Sep 9, 2026

Uh oh!

blueorangutan commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Pearl1594 commented Nov 3, 2025 •

edited

Loading

codecov Bot commented Nov 3, 2025 •

edited

Loading

DaanHoogland commented Aug 31, 2026 •

edited

Loading