windows-vm: hardcoded /gfx:AVC444 renders a permanently blank RDP window on NVIDIA + libva-nvidia-driver #8612

What happens

omarchy-windows-vm launch starts the VM correctly, and xfreerdp3 connects and
authenticates correctly. The RDP window opens, sizes itself, negotiates its
dynamic virtual channels — and then never paints a single frame. It stays blank
for the entire life of the session.

Expected

The RDP window displays the Windows desktop.

Not a duplicate of #5202 / #6882 / #4365

Those are all connection failures where FreeRDP exits within about a second.
This is not. Here FreeRDP connects successfully and stays connected:

• session held for 1h 59m before I closed it deliberately
• zero connection, TLS or NLA errors
• audio (rdpsnd), microphone (audin), clipboard and disp channels all loaded
  and working — sound played through to the host correctly the whole time
• the session ended with a clean server-initiated
  ERRINFO_RPC_INITIATED_DISCONNECT when I shut Windows down from inside the guest

The only errors in two hours were 746 consecutive per-frame decode failures —
100% of frames. The transport is healthy; the video decoder is not.

The two bugs do compound, though: with #5202's auto-stop behavior, closing the
blank window also tears down the VM, so the combined symptom reads as "the Windows
VM won't start" and sends people debugging Docker, KVM and the Windows install.
That is how I originally encountered this.

Root cause

bin/omarchy-windows-vm line 688 hardcodes /gfx:AVC444:

xfreerdp3 /u:"$WIN_USER" /p:"$WIN_PASS" /v:127.0.0.1:3389 -grab-keyboard /sound \
  /microphone /clipboard /cert:ignore /title:"Windows VM - Omarchy" \
  /dynamic-resolution /gfx:AVC444 /floatbar:... $RDP_SCALE

Arch's freerdp is built with VA-API enabled — FreeRDP says so itself at startup:

[WARN][com.freerdp.core.rdp] - This build is using [experimental] build options:
[WARN][com.freerdp.core.rdp] - * 'WITH_VAAPI=ON'
[WARN][com.freerdp.core.rdp] - * 'WITH_VAAPI_H264_ENCODING=ON'
[WARN][com.freerdp.core.rdp] - [experimental] build options might crash the application

With an NVIDIA GPU and libva-nvidia-driver installed (the VA-API → NVDEC shim),
FreeRDP selects the VA-API H.264 path:

[INFO][com.freerdp.codec] - [libavcodec_init]: Using VAAPI for accelerated H264 decoding

Initialization succeeds, so nothing fails at connect time. Every frame then
dies at transfer time:

[ERROR][com.freerdp.codec] - [libavcodec_decompress]: Failed to transfer video frame (status=-38) (Function not implemented)
[WARN][com.freerdp.codec]  - [log_decompress]: H264 decompress failed with -1
[WARN][com.freerdp.gdi]    - [gdi_SurfaceCommand_AVC420]: avc420_decompress failure: -1, ignoring update.

status=-38 is ENOSYS. libva-nvidia-driver does not implement the frame
transfer path that ffmpeg's av_hwframe_transfer_data() requires, so no decoded
frame ever reaches the GDI surface.

Note this is not the GPU-generation incompatibility from #6005 — this is an
Ampere card on the current 610.57.04 driver.

Steps to reproduce

1. NVIDIA GPU with nvidia-utils and libva-nvidia-driver installed.
2. omarchy-windows-vm install, let Windows 11 finish installing.
3. omarchy-windows-vm launch -k (the -k keeps windows-vm: xfreerdp3 failure silently shuts down VM #5202 from muddying the result).
4. The RDP window appears, connects, authenticates — and stays blank indefinitely,
   while the log fills with avc420_decompress failure: -1.

Suggested fix

Stop requesting H.264. The connection is always to 127.0.0.1, so H.264's
bandwidth advantage over RemoteFX buys nothing here, while pulling the entire
hardware-decode stack in as a failure surface.

Changing /gfx:AVC444 to /gfx:RFX fixes it completely. Verified on this
machine: two hours, zero decode errors, with audio, microphone, clipboard and
/dynamic-resolution all behaving exactly as before.

Detecting the bad configuration at runtime is not a viable alternative — as shown
above, VA-API initialization reports success and only fails afterward, per
frame, so there is no reliable pre-flight signal to branch on.

If AVC444 is worth keeping for some setups, exposing the graphics mode as a
setting (env var or config key) with a non-H.264 default would also work.

Workaround for anyone hitting this now

Connect with /gfx:RFX in place of /gfx:AVC444, keeping every other flag the
launcher passes. A wrapper that puts a small xfreerdp3 shim on PATH to rewrite
just that one flag, then execs the stock omarchy-windows-vm launch, keeps VM
lifecycle, polkit, credentials and HiDPI scaling on the packaged code path and
survives Omarchy updates.

System details

/usr/share/omarchy/bin/omarchy-windows-vm is owned by omarchy 4.0.1-1 and
pacman -Qkk reports it unmodified, so this is the packaged script, not a local
edit. (omarchy debug does not exist as a command in 4.0.1-1, so the details
above were collected individually.)

Filed by Claude Opus 5 via Claude Code.