Codex auth is structurally broken: single-use refresh token stored as a snapshot secret, no successful run since 2026-07-03

[rssrn]

Summary

The codex agent has not completed a run since 2026-07-03. Every invocation since
2026-07-06 has failed on OAuth refresh, four for four. The agent-fallback chain hides
this: the run is recorded under whichever agent eventually answered, so the dashboard shows
a healthy-looking outcome and nothing surfaces the fact that the configured primary agent
is dead.

Evidence

runs rows attributed to codex:

agent	outcome	n	last
codex	success	3	2026-07-03T06:00:04Z

Every codex invocation after that appears only inside another run's fallback_attempts:

started_at	project	fell back to	outcome
2026-07-06T06:00:04Z	labro-proactive	claude-code	partial
2026-07-27T06:00:03Z	labro-proactive	claude-code	partial
2026-08-12T06:00:03Z	newschart-proactive	claude-code	partial
2026-08-28T06:25:05Z	labro-proactive	claude-code	partial

Stderr from the 08-28 run:

ERROR codex_login::auth::manager: Failed to refresh token: Your access token could not be
refreshed because your refresh token was already used. Please log out and sign in again.

The reason recorded in fallback_attempts is the generic turn.failed — the 401 is only
visible in the container log, not in the DB, so this was invisible to the dashboard for
eight weeks.

Root cause

Credentials are supplied as CODEX_AUTH_JSON_BASE64 — a base64 snapshot of auth.json,
stored as a repo secret in labro-rssrn and rendered into deploy/.env by
upgrade-image.yml / labro-restart.yml.

OAuth refresh tokens are single-use. The first refresh inside the container consumes the
stored token and issues a new one, which lives only in the container's ephemeral
~/.codex/auth.json. The next container recreate restores the original snapshot — a token
that has already been spent — and the refresh 401s. From then on there is no path back:
the stored secret can never again be valid.

So this is not "the secret went stale and needs rotating". Re-authenticating and re-storing
auth.json buys exactly one refresh cycle before the same failure recurs. The mechanism is
structurally incompatible with a snapshot-in-a-secret.

Options

1. Persist auth.json outside the container. Mount it from /opt/labro/data/ so the
   refreshed token survives recreates, and seed the file once by hand. Cheapest fix; keeps
   the ChatGPT-subscription auth path. Needs the file to be writable by the container and
   excluded from any rsync that would clobber it.
2. Switch codex to an API key (CODEX_API_KEY, already a declared secret). API keys do
   not rotate, so the snapshot problem disappears entirely — at the cost of usage-based
   billing rather than the subscription.
3. Drop codex from the model lists. It is the primary for all three *-proactive
   projects and nothing else. If neither of the above is worth the effort, removing it makes
   the configuration honest and saves the wasted first attempt on every proactive run.

Related

Whatever is chosen, the masking is a separate problem: a configured agent can fail on every
single invocation for two months without anything reporting it. That belongs with #60, which
already covers skipped hiding crashes and total source failures — same class of defect, in
the agent layer rather than the task-source layer. Worth cross-referencing rather than
solving here.

[rssrn]

Deprioritised — codex is no longer in any model chain, so this is no longer live breakage.

As of f7c8969 in labro-rssrn, every lane runs claude-code first with two free OpenRouter tiers behind it. Codex was the primary only for the three *-proactive projects and appears nowhere now, so nothing invokes it and nothing fails on it.

Nothing in the analysis above is retracted. The root cause stands: CODEX_AUTH_JSON_BASE64 stores a snapshot of auth.json, OAuth refresh tokens are single-use, and every container recreate restores an already-spent token — so re-authenticating buys exactly one refresh cycle before the same 401 returns. That remains true and remains unfixed; it is simply no longer costing anything, because the failing first attempt on every proactive run is gone.

Dropping to P3 rather than closing. This becomes actionable again the moment codex is wanted back in a chain, and the options in the body (persist auth.json outside the container, or switch to CODEX_API_KEY) are the decision to make at that point.

Two related notes for whoever picks this up:

• CODEX_API_KEY and CODEX_AUTH_JSON_BASE64 are now unused secrets in labro-rssrn. Deliberately left in place — they are harmless, and re-adding them would be friction if codex returns.
• The masking that let this run for eight weeks unnoticed is the more valuable half of this issue, and it is tracked separately in skipped masks crashes and total source failures — a broken run is indistinguishable from an idle one #60. A configured agent failed on every single invocation for two months and nothing reported it, because the fallback chain records the run under whichever agent eventually answered. Fixing codex would not fix that.