Skip to content

Abstract vGPU devices behind a framework dispatch - #366

Open
yummybomb wants to merge 22 commits into
mainfrom
hypeship/vgpu-framework-abstraction
Open

Abstract vGPU devices behind a framework dispatch#366
yummybomb wants to merge 22 commits into
mainfrom
hypeship/vgpu-framework-abstraction

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Bottom layer of the vendor VFIO vGPU stack (this#322#363#364#321). Behavior-preserving refactor only — no lifecycle semantics change in this layer.

Kernel 6.8 hosts assign NVIDIA vGPUs through a vendor-specific VFIO interface instead of mdev, so the mdev-shaped seams get generalized before the new backend lands above:

  • VGPUDevice / VGPUFramework abstraction — mdev moves behind a framework dispatch (CreateVGPU / DestroyVGPU / DiscoverVGPU), with assignments described by a framework + device path instead of a bare mdev UUID.
  • Hypervisor wiring — QEMU receives the generalized vGPU path through VGPUDevicePath, while Cloud Hypervisor continues to receive the existing mdev path as a passthrough device.
  • Metadata — VF allocation tracked with a single field; instance metadata carries GPUFramework / GPUDevicePath alongside the mdev UUID.
  • QEMU args — vGPU attaches via sysfsdev, and the dead mdev branch is dropped from PCI passthrough args.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/hypervisor/... and targeted lib/instances suites pass (TestSocketCacheKeyChangesWhenSocketIsRecreated and the network/image-dependent instances tests fail identically on the unmodified stack head in this environment)

Note

Medium Risk
Changes GPU attach/teardown and VM config paths across instance lifecycle and QEMU/CH, though Linux behavior still delegates to mdev; metadata migration relies on legacy UUID fallback.

Overview
Introduces a framework-agnostic vGPU layer (VGPUDevice, VGPUFramework, CreateVGPU / DestroyVGPU) so mdev is one backend today and future VFIO paths can plug in without threading mdev UUIDs through the stack.

Instance and hypervisor wiring now treat vGPU separately from whole-GPU PCI passthrough: metadata gains GPUFramework and GPUDevicePath (with legacy fallback from GPUMdevUUID), lifecycle create/start/stop/delete goes through releaseStoredVGPU, and hypervisor.VMConfig uses dedicated VGPUDevicePath for QEMU vfio-pci,sysfsdev=… and Cloud Hypervisor device passthrough. Preflight validation counts vGPU on its own so microvm slot limits stay correct.

Naming cleanup: SR-IOV VF occupancy is Allocated instead of HasMdev; resource reporting follows the same field.

Reviewed by Cursor Bugbot for commit 45c4460. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb yummybomb changed the title hypeship/vgpu framework abstraction Abstract vGPU devices behind a framework dispatch Aug 6, 2026
@yummybomb
yummybomb marked this pull request as ready for review August 6, 2026 19:20
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch 2 times, most recently from c3d6a2f to f677357 Compare August 6, 2026 19:40
Comment thread lib/instances/start.go Outdated
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from a903554 to 4499016 Compare August 7, 2026 20:52
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from 12bd9d9 to 0abb18e Compare August 10, 2026 18:26
Comment thread lib/instances/vgpu.go Outdated
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from 0abb18e to 5f9e4fd Compare August 10, 2026 18:58
Comment thread lib/instances/stop.go
Comment thread lib/instances/create.go
@yummybomb
yummybomb requested a review from sjmiller609 August 11, 2026 21:35
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from fe97bda to d86b86b Compare August 12, 2026 15:20
Comment thread lib/hypervisor/qemu/config.go
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch 2 times, most recently from ec4e189 to 40e61b2 Compare August 13, 2026 18:28

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • consider a struct param for DestroyVGPU(ctx, framework, devicePath, mdevUUID) — three positional strings are easy to swap silently as backends multiply
  • error strings still say "create vGPU mdev for profile %s" — reword in whichever layer introduces non-mdev backends
  • stop/delete log "uuid", stored.GPUMdevUUID — will be empty for path-only assignments; log the device path instead

@yummybomb

Copy link
Copy Markdown
Contributor Author

addressed the review in 113d9e6 + 9396bc0:

  • DestroyVGPU now takes a devices.VGPUAssignment struct instead of three positional strings — pulled the exact shape down from the next PR in the stack, so its diff collapses on restack
  • stop/delete now log device_path (always populated) instead of the possibly-empty mdev uuid
  • the "create vGPU mdev" error strings stay as-is in this layer since mdev is the only backend here; the layer that introduces the vendor backend already rewords them to "create vGPU for profile"

@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch 4 times, most recently from 27b2b18 to 199c405 Compare August 21, 2026 20:47
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from 199c405 to 28b9531 Compare August 24, 2026 19:04

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 28b9531. Configure here.

Comment thread lib/instances/create.go
Since vGPU devices attach through VGPUDevicePath, PCIDevices only carries
whole-device passthrough paths; the mdev special case was unreachable.
Stop and delete keep best-effort vGPU release (log and continue,
metadata always cleared) and start no longer releases a stale stored
assignment. Those behavior changes belong to the lifecycle-hardening
layer, not this behavior-preserving refactor.
vGPU attachment moved from PCIDevices to VGPUDevicePath, which let a
microvm config carrying only a vGPU pass the profile validation that
previously rejected the mdev path via the PCIDevices check.
@yummybomb
yummybomb force-pushed the hypeship/vgpu-framework-abstraction branch from f0d868d to 45c4460 Compare August 24, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants