Detect and quarantine wedged vGPU VFs

[yummybomb]

Summary

Killing the process holding a vendor VFIO VF during guest driver init (~8s window) silently corrupts the VF: every later boot on it fails while sysfs, NVML, and the vGPU plugin all look healthy. #321 removed hypeman's own kill triggers; this layer detects the wedges external kills (OOM, operator, deploy restarts) still cause, and takes the VF out of placement.

Detection — guest agent report, no lifecycle change. On a wedged VF the guest driver is present, trying, and failing: its kernel loops NVRM: ... RmInitAdapter failed! in the guest kernel log. The guest agent watches /dev/kmsg for that line and reports it as a HYPEMAN-GPU-INIT-FAILED marker — the same guest→host channel as the existing HYPEMAN-* markers, landing in the instance's logs/app.log. The agent also probes driver init at boot with nvidia-smi -L when the binary is present: the device open runs RmInitAdapter, so on a wedged VF the probe itself triggers the failure line without waiting for the workload to touch the GPU, and on success it emits a HYPEMAN-GPU-INIT-OK marker. A new VGPUSentinelController tails app.log for every instance holding a vendor VFIO assignment and records failures on the marker; the full marker shape is required since a bare token could appear in echoed exec command lines. The agent only watches when an NVIDIA PCI function is present, throttles re-emission to one per 30s, and matches the full kernel-line shape, never the driver-build-specific (0x22:0x65:884) tuple. A no-driver image never produces the kernel line and can never trigger a failure record. An image that skips the guest agent does not report — v1 accepts that coverage bound; the raw kernel line still reaches app.log for manual diagnosis. The controller exits on hosts without a vGPU framework instead of re-probing every 5s.

Quarantine. Failure reports persist to <data-dir>/gpu/vf-health.json (versioned envelope {"version": 1, "records": [...]}, keyed by VF address, survives restarts) under the placement lock, tallied per instance assignment. A VF is quarantined once failures accumulate from gpu.vf_quarantine_threshold distinct assignments (default 2), so a one-off transient does not remove a healthy VF while a genuinely wedged VF — which fails every boot — is out after its second victim. A HYPEMAN-GPU-INIT-OK marker clears the VF's failure tally; it never clears a quarantine, which stays a manual recovery. One wedge produces one failure record per assignment no matter how many times the marker repeats or the controller restarts. The store fails closed on an unreadable state file: mutations are refused, and vGPU placement and advertised availability are disabled (with load retried on each attempt) rather than treating the empty in-memory set as healthy and returning quarantined VFs to rotation. A report whose persist fails is rolled back from memory so the next scan retries it. Start archives the previous boot's serial log before persisting a new assignment (fatal on failure for GPU instances), so a scan can never replay the prior boot's report against a freshly assigned VF. The threshold softens but does not prevent host-wide quarantine from a systemic non-wedge init failure (e.g. a driver-mismatch rollout emitting the same line on every VF): such changes are validated on a test host first, and the init-failures counter is the early alerting signal if one gets through.

Placement.

• Quarantined VFs are excluded from selection and from advertised profile availability. Healthy siblings on the same card stay available.
• Cards are ordered by quarantined-VF count before load: a card with a wedged VF becomes overflow-only, so it drains toward the SR-IOV recovery cycle instead of staying warm.
• The final VF pick within the chosen card is randomized: the previous lowest-address tiebreak routed every first create on an idle host to the same VF, making one wedged VF look like a total GPU outage.

Resources API. GET /resources now reports allocatable_slots (the free-and-healthy count admission control uses; 0 while VF health state is unavailable) and quarantined_slots alongside total_slots/used_slots, so reported capacity matches admission.

Observability. Below-threshold failures log at warn and count in hypeman_instances_vgpu_sentinel_init_failures_total; quarantines log at error and count in hypeman_instances_vgpu_sentinel_quarantines_total; the hypeman_instances_vgpu_quarantined_vfs gauge and hypeman_instances_vgpu_vf_health_store_unavailable (1 while the persisted state file cannot be loaded and placement is failing closed) round it out.

GPU.md documents the new behavior and the manual recovery flow (DCGM quiesce → SR-IOV cycle → restart services → verification boot → clear the VF's entry in vf-health.json → restart hypeman immediately).

Out of scope (follow-up layer)

An operator force-cycle endpoint (DCGM quiesce → sriov-manage -d/-e → verification boot → clear quarantine). Until then recovery stays the documented manual runbook.

Testing

• Unit tests: kmsg record parsing/matching in the guest agent; marker patterns for both failure and OK markers (incl. echoed-command and raw-kernel-line negative cases); the nvidia-smi init probe (emit on success, skip without the binary); tail behavior across partial lines, oversized lines, log archival, and new assignments; failure-report retry on persist failure; threshold crossing, per-assignment dedup, configured thresholds, and success reset (incl. that success never clears a quarantine); versioned-envelope validation and load-failure refusal; already-quarantined rescans after controller restarts; kernel-facility-only kmsg matching (userspace /dev/kmsg writes cannot forge the report); repeated marker emission so a printk write-split cannot lose a report; placement exclusion / card bias / tiebreak; availability accounting incl. the new resources fields.
• go test -race ./lib/devices ./lib/paths ./lib/providers ./lib/resources ./lib/system/guest_agent ./cmd/api/config and the vGPU-related lib/instances tests pass locally; go vet clean. The full lib/instances suite fails only on environment-dependent VM/network tests, identically on the unmodified base.
• The detection signal (kernel-line timing, serial-console delivery into app.log) and the recovery sequence were validated on real L40S hardware by deliberately wedging VFs; the guest-agent watcher and the controller have not yet run against a live wedge end-to-end — that run should happen on the dev GPU host before this merges, and should now also exercise the threshold path (two victim boots) and the nvidia-smi OK probe.

---

Note

High Risk
Changes vGPU placement, admission control, and persistent VF health state on GPU hosts; misconfiguration or systemic false positives could quarantine capacity or block all vGPU placement until state is repaired.

Overview
Adds automatic detection and quarantine for vendor VFIO vGPU VFs that look healthy on the host but fail guest driver init (RmInitAdapter), so one wedged VF no longer absorbs every placement.

The guest agent watches /dev/kmsg and (when nvidia-smi exists) probes at boot, writing strict HYPEMAN-GPU-INIT-FAILED / HYPEMAN-GPU-INIT-OK lines to app.log. A new VGPUSentinelController tails those logs for vendor VFIO instances, persists per-VF failure tallies in vf-health.json, and quarantines after gpu.vf_quarantine_threshold distinct instance assignments (default 2). OK markers clear below-threshold tallies but never lift quarantine.

Placement skips quarantined VFs, deprioritizes parent GPUs that have quarantined siblings, and randomizes tie-breaks among free VFs (replacing lowest-address bias). Corrupt or unreadable VF health state fails closed: placement and advertised availability are disabled until the file is fixed. GPU instance start archives the prior boot’s app.log before assigning a VF so stale markers cannot count against a new assignment.

GET /resources exposes allocatable_slots and quarantined_slots; admission/reservation now keys off allocatable slots (not total_slots - used_slots), with explicit errors when placement is disabled. Metrics cover init failures, quarantines, quarantined VF count, and store-unavailable state. GPU.md and config examples document recovery (DCGM quiesce → SR-IOV cycle → clear vf-health.json + restart).

^{Reviewed by Cursor Bugbot for commit 1ae633a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sjmiller609]

Overall looks good, a few small feedbacks and improvements we should consider.

• Versioning: use a versioned envelope for future counters and recovery state:

{"version": 1, "records": [...]}

• Failure threshold: currently 1 detection quarantines a VF. Should this be configurable? e.g. quarantine after failures from N distinct VM assignments, default 2. Also we could have a sucess reset: successful GPU-initialization sentinel that clears the VF’s failure count. I think we should do this if there's any case where this error happens even when the VF is not wedged.

• Resources API: expose allocatable and quarantined slots so reported capacity matches admission:

 "total_slots": 64,
 "used_slots": 5,
+"allocatable_slots": 57,
+"quarantined_slots": 2

 "profiles": [{
   "name": "NVIDIA L40S-12Q",
-  "available": 59
+  "available": 57
 }]

• GPU-less hosts: exit the controller when no GPU framework exists; continuous five-second discovery is unnecessary.

• Terminology: replace “convict/convictions” terminology, that seems confusing.

• Cleanup follow-up: acknowledged that automated GPU reset and cleanup are intentionally deferred; this PR handles detection and quarantine only.

[yummybomb]

Addressed the review feedback (commits be50050, 7b24855):

• Versioning: vf-health.json is now {"version": 1, "records": [...]}; unknown versions fail closed like any other invalid state.
• Failure threshold: quarantine now requires failures from gpu.vf_quarantine_threshold distinct instance assignments, default 2, deduped per assignment so rescans/restarts can't double-count. Success reset is in too: the guest agent probes driver init at boot with nvidia-smi -L (the device open runs RmInitAdapter, so on a wedged VF the probe also forces detection instead of waiting for the workload) and emits HYPEMAN-GPU-INIT-OK on success, which clears the VF's failure tally. It never clears a quarantine — that stays manual.
• Resources API: allocatable_slots and quarantined_slots are now in the /resources GPU status; profile available already excluded quarantined VFs.
• GPU-less hosts: the controller exits when discovery finds no vGPU framework instead of re-probing every 5s (still retries on discovery errors so a transient sysfs failure doesn't kill it).
• Terminology: convict/conviction is gone; the metric split into vgpu_sentinel_init_failures_total (below-threshold reports, the early signal for systemic non-wedge failures) and vgpu_sentinel_quarantines_total.
• Cleanup follow-up: unchanged, force-cycle endpoint stays a follow-up.

Still pending before merge: the live end-to-end run on the dev GPU host, which should now cover the threshold path (two victim boots) and the OK probe.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7b24855. Configure here.}

[github-actions]

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at 6492c75 (pushed)

go get github.com/kernel/hypeman-go-staging@6492c755cfb8b5186fc44fd326b40eedc6e0f406

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

230 files generated at 632e29f (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at 642ac17 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	bd_76CwLJwY-curved-alder
Timestamp	2026-08-25T19:09:20.367Z
stlc	8413509
Spec hash	ad58e37697bf
Config hash	55e15f6f4434

---

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-25 19:09:45 UTC