Skip to content

fix: detect early Codex app-server exits - #8073

Closed
dcalliari wants to merge 4 commits into
omacom:quattrofrom
dcalliari:fix/codex-collector-approval-policy
Closed

dcalliari wants to merge 4 commits into
omacom:quattrofrom
dcalliari:fix/codex-collector-approval-policy

Conversation

@dcalliari

@dcalliari dcalliari commented Aug 24, 2026 •

Copy link
Copy Markdown

Summary

  • Detect Codex app-server termination during requests and notifications immediately.
  • Return an operation-specific error instead of waiting for the RPC timeout.
  • Add regression coverage for startup EOF and failure while sending the initialized notification.

The approval-policy change previously included in this PR was removed because upstream commit 4cd8a08, merged as #7649, already resolved it with -a on-request. Issue #8032 was closed by that PR, so this PR does not close it.

Testing

  • bash test/shell.d/agent-usage-codex-scanner-test.sh
  • git diff --check

The full ./test/shell suite was also started; it reported an unrelated missing omarchy-pkgs checkout in config-test.sh and did not complete within the task timeout.

Copilot AI balanced review requested due to automatic review settings August 24, 2026 16:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

When select() times out and poll() then finds the process gone, the child may have written its final response and exited in between. Raising at that point discarded a complete reply that base would have read on the next pass. Once the process has exited everything it wrote is already in the pipe, so a zero-timeout select tells the two cases apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex Medium <noreply@openai.com>
@omarchybot omarchybot added the verified Omarchy Triage has verified that this issue is ready for final review label Sep 28, 2026
@omarchybot

Copy link
Copy Markdown
Collaborator

Reviewed and tested on a fresh Omarchy worker VM. The problem is real, and the change is proportionate to it. On quattro, when the Codex app-server dies at startup, the AI Agents panel shows the bare initialize from #8032. When it closes stdin before the initialized notification, the panel shows [Errno 32] Broken pipe and the collector prints a traceback to stderr. With this change, both say Codex app-server exited before <step> and stderr stays clean. When a child process of the server keeps stdout open, quattro waits the full 8 seconds, and this change reports the exit in about 0.3 s.

What ran, on the worker:

  • agent-usage-codex-scanner-test.sh passes at the head. With the collector reverted to quattro, the two new cases fail with exactly those messages.
  • Checked by hand: a fake app-server that answers every request and then exits still gives the panel its plan and both limit windows.
  • ./test/cli passes.
  • ./test/shell has 3 failures ("Hi is 18 columns wide", "a logo of plain block art animates", "omarchy-theme-install refuses a URL it cannot check"). All three fail the same way on quattro without this change, and none of them touches the Codex collector.

I pushed one commit to your branch, 6d70929. In rpc_request, if the app-server wrote its last reply and exited between the select() timeout and proc.poll(), the new exit check raised and threw away that complete reply. Now, once the process has exited, the collector checks the pipe one more time and reads anything already there before it reports the exit.

The second opinion was Codex Medium. It found that race, and it reviewed the fix in a second round and found nothing else open. It can read my notes, so its agreement on the rest isn't guaranteed to be independent.

This is waiting on the maintainer. Nothing is needed from you.

@omarchybot omarchybot removed the verified Omarchy Triage has verified that this issue is ready for final review label Sep 29, 2026
omarchybot and others added 2 commits October 1, 2026 13:59
quattro now asks for the limits before account/read, so the initialized notification goes through rpc_send ahead of the reordered requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
quattro's account/read fallback swallows only TimeoutError so a missing plan never costs the limits already read. With app-server exits now raised as RuntimeError, an exit during that fallback escaped to the outer handler and reported the limits as unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@omarchybot

Copy link
Copy Markdown
Collaborator

This branch had stopped merging cleanly: #13733 on quattro reordered the requests so the limits are read first and account/read is only a fallback. I merged quattro into the branch (e324b16) and kept your rpc_send for the initialized notification ahead of the new order.

The merge needed one follow-up (37e9c87). quattro's fallback only catches TimeoutError, so a missing plan can never cost limits that were already read. This branch now raises RuntimeError when the app-server exits, so an exit during that fallback got past the except and reported the limits as unavailable. The fallback now catches both, and test/shell.d/agent-usage-codex-scanner-test.sh has a case for an app-server that exits on account/read.

Tested on a disposable Omarchy worker at 37e9c87:

  • quattro with these tests fails at startup with the bare initialize error from AI Agents panel: Codex tab stuck showing "initialize" (invalid --ask-for-approval value 'untrusted') #8032. The branch reports which step the app-server died at, and the codex scanner test passes 27 of 27.
  • The merge without the follow-up fails the new case: it loses the limits with "Codex app-server exited before account/read".
  • ./test/cli and ./test/shell fail in the same places on quattro, so this branch adds no failures: 1 CLI test (vscode theme reference) and 8 shell files (ascii, branding-about-animation, kernel headers and kernel migration, launch-browser, captive portal, passwordless grant, video background).

Reviewed by Claude Opus 5.5 only. There was no second opinion this time: Codex Medium is configured, but the bot's daily cap on reviews was used up, so nobody else has checked the merge or the follow-up. For the same reason the class check could not be repeated, so the verified label stays off for now. The next run asks again, and until then this waits on that second review.

@omarchybot omarchybot removed the bug Something isn't working label Oct 1, 2026
@greptile-apps

greptile-apps Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Improves error handling for a subprocess communication flow.

The PR appears safe to merge, with a non-blocking gap in coverage of the poll-based exit path.

Findings

  1. P2 Poll exit path remains untested ▶

Summary

The PR translates app-server EOF and send failures into operation-specific errors and preserves fetched limits when the server exits during the optional account read.

  • Adds regression tests for startup EOF, initialized-notification send failure, and account/read exit.
  • The new poll-based exit path remains untested.

Reviews (1) · Last reviewed commit: "Keep the limits when the app-server exit..."

Comment on lines +83 to +84
exec 1>&-
sleep 2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Poll exit path remains untested The startup fixture closes stdout while the child is still alive. That makes select report EOF, so the test exercises the readline() error path rather than the new proc.poll() check. The account/read fixture also reaches EOF. A regression in poll-based exit detection could therefore pass all the new tests; a fixture that keeps the stdout pipe open briefly after the direct child exits would cover that path.

@dhh

dhh commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Thanks! The same fix is coming in through #14049 (from #8977, which detects early app-server exits). Closing in favor of that.

@dhh dhh closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants