Skip to content

Fix Cocoa startup crash when the VM has no console - #312

Open
charles-hood wants to merge 1 commit into
omacom:mainfrom
charles-hood:fix/cocoa-no-console-crash
Open

charles-hood wants to merge 1 commit into
omacom:mainfrom
charles-hood:fix/cocoa-no-console-crash

Conversation

@charles-hood

@charles-hood charles-hood commented Oct 4, 2026 •

Copy link
Copy Markdown

What changed and why

The runtime's QEMU segfaults at startup when the Cocoa display runs a VM that has no console at all:

macos/.build/qemu-gpu-runtime/bin/qemu-system-aarch64 -machine virt -nodefaults -S -display cocoa

It stops with EXC_BAD_ACCESS at address 0x38 in qemu_console_surface, called from cocoa_display_init. This happens with gl=off and gl=on.

The cause is in qemu-texture-borrowing-11.1.patch. cocoa_display_init calls qemu_console_surface(dcl.con) straight after qemu_console_lookup_default(), and that lookup returns NULL when there is no console. Pristine QEMU never reads the surface at that point. Its listener registration handles a NULL console by showing the "This VM has no graphic display device." placeholder.

The fix is one statement in that patch:

surface = dcl.con ? qemu_console_surface(dcl.con) : NULL;

Nothing reads surface before qemu_console_register_listener() replaces it. I checked this with breakpoints on every reader in ui/cocoa.m during startup, with a GPU device in both GL modes. So with no console, registration installs QEMU's placeholder as upstream does, and with a console the statement does what it did before. Since nothing reads it there, the line could also be removed outright; I kept it, guarded, so the console case stays exactly as it was. Happy to switch to removing it if you prefer. I edited the patch that introduced the bug rather than adding a new one. Because the replacement keeps the same line count, every later patch applies at the same lines, with no fuzz or offsets. texture_patch_sha256 is now b8e107ecd82813854bac7f6bb110722f52937d64b6f98611c4dd02842977fc2f. The ui/cocoa.m hunks were regenerated with GNU diff 3.12 from a tree with the patch applied (the same tool reproduces the previous section byte for byte), and the section headers are unchanged.

The app's own configuration always adds a virtio-gpu device and was never affected. The crash hits anyone who runs the runtime by hand without a display device. It needs a VM with no console of any kind. Without -nodefaults, the default serial port and monitor are vc text consoles, so the crash needs -nodefaults, or both defaults replaced (for example -serial stdio -qmp ...).

New test: macos/Tests/test-cocoa-no-console.py, with cocoa-no-console-harness.c, added to make test-contracts. It compiles the patched startup statements against stubs, with the console lookup returning NULL and then a real console. It also checks that the patch's only surface read is still among those statements and that the builder pins the patch's hash. It fails on the previous patch (abort at the NULL console) and passes now.

Testing

MacBook Pro, Apple M4 Pro (Mac16,8), macOS 27.0.1 (26A434):

  • make runtime on the base commit and on this branch. On both, the patch stack applies with no fuzz, offsets or rejects.
  • python3 macos/Tests/test-cocoa-no-console.py passes. With the previous patch it fails.
  • make test-contracts and make test pass.
  • Before the fix, the command above crashes in qemu_console_surface with -display cocoa and -display cocoa,gl=on. After the fix, both open a window with the placeholder (screenshot below), handle mouse movement, and exit with status 0 from the menu's Quit.
  • A GPU configuration (-machine virt,accel=hvf,gic-version=3 -cpu host -m 1024 -nodefaults -S -device virtio-gpu-gl-pci,romfile= -display cocoa,gl=on) starts and shows "Display output is not active." before and after the change.
  • Not tested: make app and booting the guest (no Docker on this machine).
  • Typing into the placeholder window crashes in vt100_keysym. Unpatched QEMU 11.1.1 (Homebrew) does the same, so that is upstream behaviour and this PR leaves it alone.
placeholder-gl-on

The texture-borrowing patch reads the default console's surface at the top
of cocoa_display_init. When QEMU has no console at all (for example
-nodefaults with no display device), qemu_console_lookup_default() returns
NULL and qemu_console_surface() dereferences it, so QEMU segfaults before
the window opens. Pristine QEMU shows its "This VM has no graphic display
device." placeholder in that case.

Leave surface NULL when there is no console. Listener registration then
installs QEMU's placeholder, as it does upstream. With a console the
statement is unchanged. Add a contract test that compiles the patched
startup statements against a missing console.
@charles-hood
charles-hood marked this pull request as ready for review October 4, 2026 00:54

@themartiano themartiano left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be merged after v0.5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants