Fix Codex limits timing out on buffered app-server replies

[tossbaws]

Problem

The agents panel shows Codex limits unavailable even when codex login is valid.

rpc_request() in bin/omarchy-agent-usage-codex calls select() on proc.stdout and then readline() on the text wrapper. The app-server sends notifications (remoteControl/status/changed, account/updated) around the same time as replies. When a notification and a reply end up in the same read, readline() pulls both into Python's buffer and returns only the first line. select() can't see lines that are already buffered, so the reply sits there until the deadline and the collector raises TimeoutError.

Relationship to #13733

#13733 asks for account/rateLimits/read first and makes account/read a fallback, which avoids the most common failure. It keeps the buffered reader, though, so the same race can still cost the limits. With a notification in the same write as the rate-limits reply, quattro fails with account/rateLimits/read.

This may also be why account/read looked like it never answered. With the reader below, a live probe here (codex-cli 0.157.1) got the account/read reply in about 0.46s.

This PR is merged up to date with quattro and keeps #13733's request order.

Fix

• Read the raw fd with os.read and split lines ourselves. Leftover bytes are kept on the process object, so data that arrives with one reply carries into the next request.
• Use time.monotonic() for the request deadline so wall-clock changes can't stretch or shorten it (suggested in review).

Testing

• test/shell.d/agent-usage-codex-scanner-test.sh: a new case uses a stand-in app-server that sends a notification in the same write as the initialize reply and another in the same write as the account/rateLimits/read reply. It fails on quattro with account/rateLimits/read and passes here. Read Codex limits without waiting on account/read #13733's test and all the existing tests in the file still pass.
• Live runs against a real Codex Plus login return the plan and both limit windows.

🤖 Generated with Claude Code

[ibarani]

Confirmed the same buffering failure independently on Omarchy 4.0.4-1. Our local fix uses the same raw-pipe read and retained-buffer approach as this PR.

I tested this PR's collector at 71423eb069d489745d0ee31ef9976ce8737327af against a pipe-based regression harness:

• Original reader times out when a notification and matching response arrive in one write.
• This PR consumes the coalesced response successfully.
• Unread responses survive across successive requests.
• Fragmented JSON, invalid JSON, and non-object JSON are handled.
• An incomplete line respects the timeout instead of blocking in readline().

All five checks passed (the first check asserts the original failure). These tests run without credentials, network access, or a Codex process. Separately, a live protocol probe with codex-cli 0.159.0 successfully returned account and rate-limit responses; this was not an end-to-end test of this PR's collector.

One small improvement from our local patch: use time.monotonic() for both the deadline and remaining-time calculation, so wall-clock adjustments cannot change the timeout duration.

Reproduction and regression harness

Save this as verify_rpc.py, then run python verify_rpc.py /path/to/original-collector /path/to/pr-collector.

import io, os, runpy, sys, threading, time
from types import SimpleNamespace
from contextlib import contextmanager
old = runpy.run_path(sys.argv[1])['rpc_request']
new = runpy.run_path(sys.argv[2])['rpc_request']
@contextmanager
def pipe(data=b''):
    r,w=os.pipe()
    p=SimpleNamespace(stdin=io.StringIO(),stdout=os.fdopen(r,'r'))
    if data:os.write(w,data)
    try:yield p,w
    finally:p.stdout.close();os.close(w)
data=b'{"method":"account/updated"}\n{"id":2,"result":{"ok":true}}\n'
with pipe(data) as (p,w):
    try:old(p,2,'account/read',timeout=.1)
    except TimeoutError:print('PASS: original fails with notification and response in one write')
    else:raise AssertionError('original did not reproduce failure')
with pipe(data) as (p,w):
    assert new(p,2,'account/read',timeout=.1)['result']['ok']
    print('PASS: patched reader consumes coalesced response')
with pipe(b'{"id":1}\n{"id":2}\n') as (p,w):
    assert new(p,1,'initialize',timeout=.1)['id']==1
    assert new(p,2,'account/read',timeout=.1)['id']==2
    print('PASS: unread responses persist across requests')
with pipe(b'bad json\n[]\n{"id":') as (p,w):
    t=threading.Thread(target=lambda:(time.sleep(.03),os.write(w,b'2}\n')))
    t.start()
    assert new(p,2,'account/read',timeout=.2)['id']==2
    t.join()
    print('PASS: fragmented response and malformed notifications')
with pipe(b'{"id":') as (p,w):
    start=time.monotonic()
    try:new(p,2,'account/read',timeout=.05)
    except TimeoutError:pass
    else:raise AssertionError('expected timeout')
    assert time.monotonic()-start<.2
    print('PASS: incomplete line respects deadline')

[greptile-apps]

Confidence Score: 5/5

[High risk] Changes how the agent reads replies from a subprocess.

The PR appears safe to merge; no outstanding findings remain.

Summary

The PR replaces buffered Codex app-server reads with raw file-descriptor reads, preserves bytes between requests, and adds a regression test for notifications batched with replies.

_{Reviews (4) · Last reviewed commit: "Merge quattro into fix-codex-rpc-buffere..."}

[tossbaws]

Confirmed the same buffering failure independently on Omarchy 4.0.4-1. Our local fix uses the same raw-pipe read and retained-buffer approach as this PR.

I tested this PR's collector at 71423eb069d489745d0ee31ef9976ce8737327af against a pipe-based regression harness:

• Original reader times out when a notification and matching response arrive in one write.
• This PR consumes the coalesced response successfully.
• Unread responses survive across successive requests.
• Fragmented JSON, invalid JSON, and non-object JSON are handled.
• An incomplete line respects the timeout instead of blocking in readline().

All five checks passed (the first check asserts the original failure). These tests run without credentials, network access, or a Codex process. Separately, a live protocol probe with codex-cli 0.159.0 successfully returned account and rate-limit responses; this was not an end-to-end test of this PR's collector.

One small improvement from our local patch: use time.monotonic() for both the deadline and remaining-time calculation, so wall-clock adjustments cannot change the timeout duration.

Reproduction and regression harness
Save this as verify_rpc.py, then run python verify_rpc.py /path/to/original-collector /path/to/pr-collector.

import io, os, runpy, sys, threading, time
from types import SimpleNamespace
from contextlib import contextmanager
old = runpy.run_path(sys.argv[1])['rpc_request']
new = runpy.run_path(sys.argv[2])['rpc_request']
@contextmanager
def pipe(data=b''):
    r,w=os.pipe()
    p=SimpleNamespace(stdin=io.StringIO(),stdout=os.fdopen(r,'r'))
    if data:os.write(w,data)
    try:yield p,w
    finally:p.stdout.close();os.close(w)
data=b'{"method":"account/updated"}\n{"id":2,"result":{"ok":true}}\n'
with pipe(data) as (p,w):
    try:old(p,2,'account/read',timeout=.1)
    except TimeoutError:print('PASS: original fails with notification and response in one write')
    else:raise AssertionError('original did not reproduce failure')
with pipe(data) as (p,w):
    assert new(p,2,'account/read',timeout=.1)['result']['ok']
    print('PASS: patched reader consumes coalesced response')
with pipe(b'{"id":1}\n{"id":2}\n') as (p,w):
    assert new(p,1,'initialize',timeout=.1)['id']==1
    assert new(p,2,'account/read',timeout=.1)['id']==2
    print('PASS: unread responses persist across requests')
with pipe(b'bad json\n[]\n{"id":') as (p,w):
    t=threading.Thread(target=lambda:(time.sleep(.03),os.write(w,b'2}\n')))
    t.start()
    assert new(p,2,'account/read',timeout=.2)['id']==2
    t.join()
    print('PASS: fragmented response and malformed notifications')
with pipe(b'{"id":') as (p,w):
    start=time.monotonic()
    try:new(p,2,'account/read',timeout=.05)
    except TimeoutError:pass
    else:raise AssertionError('expected timeout')
    assert time.monotonic()-start<.2
    print('PASS: incomplete line respects deadline')

Appreciate the suggestion. I went ahead and added that to the latest; good catch.

[jelenv]

Independently confirmed the same issue on Omarchy 4.0.4 with codex-cli 0.160.0, using the stock collector through my custom agents widget.

• The stock RPC probe reproduced Codex limits unavailable / account/read in 2 of 5 live runs.
• In a subsequent captured failure, after the unchanged 4-second deadline, select() reported the OS pipe was not readable, yet draining Python’s text buffer returned the successful account/read response (id 2, no RPC error).
• A synthetic notification + response sent in one write reproduced the same timeout; raw-read live probes received account responses in about 0.3 seconds.

This confirms the buffered-reader race rather than a login failure. We have not tested this PR’s patch or applied a local workaround; looking forward to the upstream fix. Thanks for working on it!

[dhh]

Thank you! Brought in with your authorship in #14049.