Skip to content

fix(menu): contain the keybindings Lua bind scan - #12916

Open
ansonboby wants to merge 2 commits into
omacom:quattrofrom
ansonboby:fix/menu-keybindings-lua-runaway
Open

ansonboby wants to merge 2 commits into
omacom:quattrofrom
ansonboby:fix/menu-keybindings-lua-runaway

Conversation

@ansonboby

Copy link
Copy Markdown

Summary

SUPER + K could OOM-kill the session and take Hyprland down (issue #12852). Root cause: the keybindings menu's Lua bind scan dofiles the user's config under a fake hl environment, and two growth paths there were unbounded:

  • Dispatcher expressions serialize their own arguments back into the expression, so a nest of hl.dsp.*(...) calls doubles in size every level — ~35 levels is ~16GB, matching the reported 18GB.
  • lua_literal() recursed through tables with no cycle detection, so a cyclic config argument recursed without bound.

The uncapped lua child then exhausted RAM+swap (kernel OOM killer fired on the lua process), Hyprland segfaulted in its NVIDIA EGL teardown, and the parent bash parked in anon_pipe_read forever — the reported hang with 0-byte keybindings.* temp files.

Containment in layers — a runaway now fails the scan instead of the session

  1. lua_literal caps: cycle detection, max nesting depth 10, max serialized size 64KiB — the scan aborts cleanly on any of them.
  2. Instruction watchdog (debug.sethook) around the config run: a config stuck in an infinite loop burns its budget and the pcall catches it instead of spinning until the OOM killer arrives.
  3. Shell seatbelt: the lua child runs under timeout 20 with a 1GiB address-space ulimit, so anything that slips past the Lua caps still cannot take the session's memory.
  4. Scratch hygiene: killed refreshes leave keybindings.XXXXXX mktemp files the records-glob prune never matched; those now get cleaned.

A failed scan is the safe outcome: the menu renders binds without their dispatch, a shape every downstream path already handles (unresolved __lua binds render as plain rows today).

Test plan

  • 3 new regressions in test/shell.d/keybindings-menu-test.sh: cyclic bind argument, 40-level nested dispatcher bomb, infinite config loop — each must fail the scan and still render the menu row. All 22 assertions in the suite pass, plus menu-guards-test.sh and bin-style-test.sh.
  • Real-config sanity: an actual user hyprland.lua scans in 0.23s with all 225 entries rendered — the watchdog does not false-positive on legitimate configs.

Closes #12852.

@llstrk

llstrk commented Sep 23, 2026

Copy link
Copy Markdown

Verified: No production defect found in the scoped review of 3201694f. The real --print path preserved an ordinary Lua bind's recovered dispatch and rendered fallback rows for the tested scan failures:

Synthetic config Observed scan result Menu result
Ordinary exec bind Dispatch recovered Row rendered
Cyclic argument Cycle error Unresolved row rendered
40 nested dispatchers Size-cap error Unresolved row rendered
Infinite Lua loop Instruction-budget error Unresolved row rendered

Optional test improvement: The new cyclic-case row assertion also passes on the unpatched baseline with Lua 5.5.1: the existing pcall catches a stack-overflow error. Keep the fallback assertion, but capture DEBUG=1 stderr and assert the cycle-specific diagnostic so this case detects removal of the new cycle check.


Review information

Test scope: Synthetic configurations through the actual menu script, with stubbed hyprctl and xkbcli. Full shipped/user configurations, the full shell suite, live dispatch, and the reported compositor/OOM crash were not tested.

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Automated AI review: Astra Medium initial inspection, independent Opus 5.5 High and GPT 6 Sol Xhigh technical reviews, followed by Astra synthesis against source and retained test evidence and a fresh Astra Medium editorial check.

ansonboby added a commit to ansonboby/omarchy that referenced this pull request Sep 23, 2026
The cyclic-case row assertion also passes on an unpatched baseline: the
existing pcall catches Lua's stack overflow and still yields a fallback
row, so the test could not detect removal of the cycle check. Run the
case with DEBUG=1 and assert the cycle-specific diagnostic, per the
omacom#12916 review.
@omarchybot omarchybot added the bug Something isn't working label Sep 27, 2026
The Lua bind scan dofile's the user's config under a fake environment and
serializes bind arguments back into dispatcher expressions, whose size
doubles every nesting level. A cyclic table, a nested dispatcher chain or
a runaway loop grew without bound: the scan ballooned to ~18GB, the OOM
killer fired, and Hyprland segfaulted tearing down its NVIDIA context.
The uncapped child also left bash parked in anon_pipe_read forever.

Containment in layers, so a runaway fails the scan instead of the
session:

- lua_literal detects cycles, caps nesting at 10 and total serialized
  size at 64KiB, aborting the scan cleanly
- the config run carries an instruction watchdog, so an infinite loop
  burns its budget and pcall catches it
- the lua child runs under timeout 20 with a 1GiB address-space ulimit:
  whatever escapes the caps above cannot take the session's memory
- stale keybindings.XXXXXX mktemp scratch from killed refreshes is
  pruned; the records glob never matched those names

A failed scan is the safe outcome: the menu renders binds without their
dispatch, which every downstream path already handles.

Regression tests cover all three runaways (cyclic argument, 40-level
nested dispatcher, infinite config loop) plus the real-config path,
which still completes in 0.23s.
The cyclic-case row assertion also passes on an unpatched baseline: the
existing pcall catches Lua's stack overflow and still yields a fallback
row, so the test could not detect removal of the cycle check. Run the
case with DEBUG=1 and assert the cycle-specific diagnostic, per the
omacom#12916 review.
@ansonboby
ansonboby force-pushed the fix/menu-keybindings-lua-runaway branch from 5331744 to cd56498 Compare September 29, 2026 14:23
@ansonboby

Copy link
Copy Markdown
Author

Rebased onto current quattro at cd564986 — the branch had gone conflicting.

The conflict was upstream's f8146395 (the terminal launcher taking the focused window's pid from Lua) landing in the same dispatcher chain this PR touches, plus its v13 → v14 cache-key bump. Resolved by keeping both sides: upstream's terminal-bind test and all three runaway cases (cyclic argument, 40-level nested dispatcher, infinite config loop) — they are independent assertions rather than alternatives.

Checked after the rebase:

  • scope unchanged — still just bin/omarchy-menu-keybindings and test/shell.d/keybindings-menu-test.sh;
  • upstream's new branch takes arg straight from o.bind_commands[...], so it never enters the serializer and is outside the caps by construction; the caps still cover every path that does, through call_expression;
  • keybindings-menu-test.sh (19 checks), manifest-entrypoints-test and bin-style-test pass.

@omarchybot

Copy link
Copy Markdown
Collaborator

Looked at this at cd564986. Classifying it needs two models to agree: Claude Opus 5.5 reads it as a bug fix, but the second model, Codex Medium, didn't run because the review budget for today is used up. Without that agreement the classification isn't settled, so the fix hasn't been run on a test machine or reviewed line by line yet. Nothing in this comment is a finding about the code.

What I did check: the branch sits on current quattro and still touches only bin/omarchy-menu-keybindings and test/shell.d/keybindings-menu-test.sh. The stock Omarchy config never passes one dispatcher into another (o.bind in default/hypr/helpers.lua wraps only plain strings with hl.dsp.exec_cmd). So the doubling path only matters for a user config that nests dispatchers, which is also all #12852 can establish, since the reporter couldn't pin down the trigger.

Next: a later run, when the second model is available, will classify it, reproduce the runaway on a worker against quattro and this head, and review it. That review will include whether the timeout/ulimit wrapper alone would already contain it. Nothing is needed from you in the meantime.

@jandrusk

jandrusk commented Oct 8, 2026

Copy link
Copy Markdown

For the maintainer's comparison: the instruction-budget hook and the timeout 20 / ulimit -v wrapper here should also contain the hang in #13979 and #12052 (a config that iterates hl.get_workspaces() or another hyprlua getter). #14128 is a narrower fix for that case, with its own debug.sethook guard. If this lands first, #14128 could probably shrink to its mock-getter change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

omarchy-menu-keybindings: embedded lua runs the Hyprland config and balloons to ~18GB, OOM-kills, and takes Hyprland down (NVIDIA teardown SIGSEGV)

4 participants