omarchy-agent-usage-codex: 4s RPC timeout intermittently drops Codex usage/limits

[matthewjaybarr-png]

What happens

omarchy-agent-usage-codex spawns codex app-server and makes two JSON-RPC
calls to it — account/read then account/rateLimits/read — each bounded by
a 4-second timeout (rpc_request(..., timeout=4)).

A successful run typically completes in ~2s, but response time varies enough
that the call intermittently exceeds 4s. When it does, the script's exception
handler overwrites the entire result for that refresh:

except Exception as exc:
    result["usageStatusText"] = "Codex limits unavailable"
    result["authHelpText"] = str(exc)   # e.g. "account/read"

result["limits"] stays [] (its initial default), so any usage record
consumer reading the cached JSON — the bar widget and any app that reads
~/.local/state/omarchy/agents/usage/codex.json — loses the rate-limit
display for that whole refresh cycle, even though the previous cycle had
valid data.

Reproduction

Ran the collector 8 times back-to-back on an idle-ish machine:

for i in $(seq 8); do
  /usr/bin/omarchy-agent-usage-codex --force
done

3 of 8 runs failed with the exact signature above (usageStatusText: "Codex limits unavailable", authHelpText: "account/read"), each taking ~5s
end-to-end vs ~2s for a success — consistent with hitting the 4s timeout on
account/read and returning shortly after.

Fix

The 4s timeout margin against codex app-server's actual response time is
too tight. Bumping both RPC timeouts (account/read,
account/rateLimits/read) from 4s to something like 10s resolves it in local
testing (0 failures in 6 subsequent runs). Patch is a one-line-per-call
change in fetch_codex_rpc().

Environment

• omarchy package: 4.0.4-1
• codex-cli: 0.157.1

[matthewjaybarr-png]

Update: applied a local patch bumping both RPC timeouts (account/read,
account/rateLimits/read) from 4s to 10s and re-tested more carefully.

That patch did not reliably fix it — a batch of 12 forced runs after the
patch still failed 8/12, worse than the original ~3/8 baseline. Digging
further with a hand-rolled RPC client against codex app-server directly:

• account/read returned in ~0.15s.
• account/rateLimits/read took ~5.2s and came back as a JSON-RPC error
  response (not a timeout): "failed to fetch codex rate limits: error sending request for url (https://chatgpt.com/backend-api/wham/usage)".

So the underlying issue looks like an intermittently slow/erroring network
call from codex app-server out to the ChatGPT usage backend
(chatgpt.com/backend-api/wham/usage), not just a too-tight client timeout.
Note fetch_codex_rpc()'s current error handling also silently swallows
this case in a different way: an {"error": ...} JSON-RPC response still
matches the request id, so rpc_request() returns it normally, and
limit_window()'s (limits_msg.get("result") or {}) quietly turns the
error into an empty limits dict with no usageStatusText set at all — a
second, distinct failure mode from the timeout one originally reported here.

My repeated testing (dozens of forced collector runs in a short window) may
itself have aggravated this against the real backend, so take the exact
failure rates with a grain of salt — but the error-message evidence for a
backend-side reliability issue (independent of the 4s timeout) seems solid.
A fix here probably needs to also treat an "error" field in the RPC
response as a failure (set usageStatusText) in addition to any timeout
tuning.

[matthewjaybarr-png]

Update 2: retracting the "OpenAI backend flakiness" framing from my last
comment — it was premature.

The failures are intermittent enough that I can't currently pin them to one
cause. Some data points:

• One batch of 12 forced collector runs failed 8/12 while a Surfshark
  WireGuard VPN connection on my machine was up and had recently
  reconnected (NetworkManager auto-reactivated it after a drop); ufw logs
  showed inbound return packets on that tunnel being classified INVALID and
  dropped around the same window, which looked like a plausible culprit
  (stale conntrack state after the reconnect breaking in-flight streams).
• But a later batch failed just as badly (4/5) with that VPN fully
  disconnected, and several direct hand-rolled RPC tests against
  codex app-server in between succeeded cleanly and quickly (all <1.1s,
  including a real account/rateLimits/read response). So the VPN isn't a
  sufficient explanation either.

What does look solid: this isn't specific to the usage-collector script.
The same machine independently hit this directly in an interactive Codex
CLI session tonight: Warning: Falling back from WebSockets to HTTPS transport. stream disconnected before completion: failed to lookup address information: Try again — a plain OS-level DNS resolution failure
(EAI_AGAIN) mid-connection, after a transport fallback. That's consistent
in shape with what's hitting omarchy-agent-usage-codex's RPC calls, and
the collector script currently can't show it (stderr=subprocess.DEVNULL
on the codex app-server subprocess swallows whatever Codex itself would
have logged).

So: real, intermittent, currently unattributed root cause (network stack on
this machine, DNS resolution, or something upstream — not clear yet). Two
concrete, independent fixes still worth doing regardless of root cause:

1. Don't discard the codex app-server subprocess's stderr in
   fetch_codex_rpc() — capture it and include a snippet in
   authHelpText on failure, so the actual error (timeout vs. DNS vs. RPC
   error-response) is visible instead of a generic "Codex limits
   unavailable" / "account/read".
2. Treat a JSON-RPC response containing an "error" key as a failure (set
   usageStatusText) rather than silently defaulting to {} via
   (limits_msg.get("result") or {}) — this is the second bug noted in my
   previous comment, distinct from any timeout/network issue.

[omarchybot]

#14049 has merged into quattro, removing the account/read dependency, fixing buffered reads, reporting rate-limit RPC errors, and capturing app-server stderr for exit diagnostics. The latest comments describe an unresolved intermittent failure with uncertain cause. There is still no general last-good-limits fallback; keeping this open pending confirmation.

Scope checked by GPT-6 in Codex and Codex Medium against the reports and landed source. The second opinion agreed with this scope; independence is not guaranteed. No runtime tests were rerun for this reconciliation.