Agents panel freezes permanently when inotify quota is exhausted: FileView watches die on the next atomic record rewrite and never re-arm #9974

What happened

The Agents panel on one of our machines displayed week-old usage data (day
bars, weekly limit gauges — everything) for three days, until the shell was
restarted. The usage records on disk were perfectly fresh the whole time:
omarchy-agent-usage-update kept rewriting
~/.local/state/omarchy/agents/usage/*.json every 15 minutes with correct
data, but the panel never re-read them.

The journal shows the trigger. The user's inotify quota was momentarily
exhausted (heavy parallel agent workloads; possibly compounded by #7150,
where the shell itself leaks inotifywait watchers), and every re-arm attempt
failed:

Aug 30 01:28:41 gt omarchy-shell[278245]:   WARN : inotify_add_watch(/home/ll/.local/state/omarchy/agents/usage) failed: (No space left on device)
Aug 30 01:28:41 gt omarchy-shell[278245]:   WARN : inotify_add_watch(/home/ll/.local/state/omarchy/agents/usage/fireworks.json) failed: (No space left on device)

The failure mode is sticky by construction:

1. plugins/agents/Agent.qml reads each record through
   FileView { watchChanges: true; onFileChanged: reload() }.
2. omarchy-agent-usage-update replaces each record atomically
   (mktemp + mv), i.e. a new inode every 15 minutes, so the watch must be
   re-armed after every rewrite.
3. When one re-arm fails with ENOSPC, the FileView keeps its last parsed
   content and there is no retry path: once the quota frees up again (it did —
   the machine sat at ~3.4k watches out of 524k when we debugged this), the
   watch stays dead. The panel is frozen until the shell process is restarted.

printErrors: false on that FileView makes it fully silent apart from the
low-level WARN above, so to the user this looks like "my usage limits never
reset".

What was expected

A missed watch re-arm should degrade to eventual consistency, not to a
permanent freeze of the panel.

Steps to reproduce

1. On a running shell, exhaust the user inotify watch quota (e.g.
   sysctl fs.inotify.max_user_watches=<current usage> as root, or spawn
   enough inotifywait instances).
2. Wait for one omarchy-agent-usage-update cycle (≤15 min) so the records
   are atomically replaced while re-arm fails — the journal shows the
   inotify_add_watch … (No space left on device) WARNs.
3. Restore the quota.
4. The panel keeps showing the pre-exhaustion data forever; new
   omarchy-agent-usage-update --force runs update the files but not the
   panel. omarchy-restart-shell fixes it.

Possible fix

Either re-arm with retry when inotify_add_watch fails, or add a cheap
low-frequency fallback reload so a lost watch only ever costs bounded
staleness. We are running the second option locally (a 2-minute
Timer { onTriggered: view.reload() } next to the FileView in Agent.qml)
and it rides out quota exhaustion cleanly.

System

• Omarchy 4.0.0-1 (Arch), quickshell-git 0.3.0.r20.g28771c7-2
• Multi-monitor desktop, heavy multi-agent usage (many watcher-holding
  processes)

Filed by Claude Fable 5 via Claude Code.