fix(reports): count errors as misses and stop losing cost on error paths

[bai-uipath]

Cost went missing on the paths where a task did not finish, and the pass rate paid a bonus for erroring. Ordered below by how much each part changes what a run reports.

Scope: the recovery fix below is the only part that touches the run path (the agent's cancel branch and the orchestrator's timeout handler). Everything else changes the rate card and what gets reported, not what gets executed. git diff main -- evalboard/ is empty, so the dashboard is untouched.

A hard-killed task recorded nothing at all

A task that runs to the wall spends the most getting there, and it was recording none of it. Not just the cost: no turns, no tokens, no commands, nothing.

A task-level timeout is a different code path from a turn-level one, and that is where it went:

• A turn-level timeout raises TurnTimeoutError, which reaches _on_attempt_failure, which drains agent.pending_turn into result.iterations. The partial keeps its tokens and gets priced. (What happened to that price is the section below.)
• A task-level timeout comes from the ThreadedWatchdog, which SIGKILLs the agent by PID and cancels the asyncio task awaiting the turn. It never reaches _on_attempt_failure, and nothing had parked a record to drain anyway.

Reproduced live on a 30s wall (task_timeout has a ge=30 floor, so 30s is as tight as it goes), against a task written to outlast it:

status = TIMEOUT   duration = 30.6s   total_turns = 0
iterations = []    total_tokens = None   total_cost_usd = None
cost_complete = False    run tasks_unpriced = 1

❌ That is byte-for-byte the shape of the production rows this was chased from: TIMEOUT, zero turns, no tokens, no cost. Across seven recent runs there are 26 such rows totalling 377 minutes of wall clock that recorded nothing.

✅ That spend is now recovered rather than declared missing. The kill does not destroy the telemetry. The agent's finalize still reduces everything the event stream delivered into a complete record and prices it from the buckets. What lost it was the last line of that method, which parks the record on pending_turn only for a turn flagged as crashed: the Claude agent flagged an external cancel as COMPLETED, so the record was assembled and then dropped. It now finalizes as a crash, which is what Codex and Antigravity already did, and their copy of that fragment moves to a shared kernel on the base class.

Reading the slot was the other half, and nothing did on this path: the cancel arrives as a BaseException, so it never reaches the retry executor's per-attempt hook. _drain_killed_turn reads it in the TaskTimeoutError handler, ahead of teardown (whose agent.stop() clears the slot) and ahead of finalization, so a recovered turn feeds token aggregation, command statistics and model resolution exactly like a turn that ended on its own.

Measured on the VM against the same task, same box, with only these two seams toggled:

Harness	Model	Seams off	Seams on
Claude Code	claude-sonnet-4-6	❌ 0 turns, 0 commands, no tokens, no cost	✅ 9 commands, 235,312 tokens, $0.1830
Antigravity	gemini-3.1-pro-preview	❌ 0 turns, 0 commands, no tokens, no cost	✅ 12 commands, 93,478 tokens, $0.1209
Codex	gpt-5.6-terra	✅ 20 commands, 94,398 tokens, $0.0739	✅ 10 commands, 91,556 tokens, $0.0640

Two things that table settles. Codex never lost this money: its turn runs on a worker thread that finishes the record and returns it before the cancel lands, so the row arrives priced through the ordinary path (crashed=false on both runs). It still gains the drain for the timings where the cancel wins the race. And Antigravity's ghost proves the orchestrator half carries its own weight: that agent already parked its partial before this PR, so the row was blank purely because nothing read the slot.

⚠️ One field stays blank on a recovered Claude row: total_turns is the SDK's num_turns, which only arrives on the terminal result message a hard kill never delivers. visible_turns and actual_commands (9 and 9 above) do reflect the work, and the tokens and cost are whole.

⚠️ The row still reports cost_complete: false. Recovery captures what the stream delivered, which is most of it, but the generation the agent was waiting on when it was killed was never delivered by anyone. The total stays a floor and the row says so. _cost_complete used to call these rows fully priced, because it asked "does every turn that burned tokens carry a cost?" and a row with no turns answers yes vacuously. It now returns False for any TIMEOUT, keyed on the status rather than on elapsed time, so a setup failure stays free (it genuinely spent nothing) while a hard kill never does.

Keying on the status rather than on emptiness matters, and live testing is what showed it. A first attempt flagged only TIMEOUT rows with no preserved turn. But a task killed mid-dialog keeps the turns that already finished, and those carry costs, so it still reported as complete while the in-flight turn was just as lost. Re-projecting that captured run through the corrected rule:

final_status    = TIMEOUT
preserved turns = 3   (only 2 carry token_usage; the third is the killed turn, usage None)
total_cost_usd  = 0.0995112
cost_complete   = False     # was True under the emptiness rule

A killed turn's tokens were captured but never priced

The turns that were recorded had a second, unrelated problem, and it is the one visible on the evalboard: 26 turns, 30m03s, 1,201,607 tokens captured, cost None.

status         = ERROR       crash_reason = Agent turn timed out after 1800s
total_tokens   = 1,201,607   total_cost_usd = None      model = claude-opus-5

It is a two-step failure, and the second step is the one that surprises:

1. On a clean turn the Claude Code SDK reports its own dollar cost. The rate card is never consulted, which is why the other 972 rows in that run priced fine despite claude-opus-5 being absent from the rate card entirely.
2. A timeout kill means no terminal result message, so no SDK cost. The rate card is the only fallback left, and with no entry for the model the fallback is a no-op (test_backfill_noop_for_unknown_model pins that). Full token counts, no money.

The fix is the missing rate, verified by replaying that row's real buckets through the corrected card:

that row (32 uncached, 18,519 out, 99,959 cache write, 1,083,097 cache read)
         None  ->  $1.6294
all 9 killed rows in that run
         $0    ->  $40.37

This is also what prices the turn the section above recovers: a hard kill delivers no result message either, so the recovered turn is only worth dollars if the card can price it.

The rate card was wrong in five other places

Audited every entry against the published tables. What was wrong, and by how much:

Entry	Was	Is	Effect
claude-opus-5	(absent)	5 / 25 / 6.25 / 0.50	the case above
claude-opus-4-8, -4-7, -4-6, -4-6-20250514, -4-5-20251101	15 / 75 / 18.75 / 1.50	5 / 25 / 6.25 / 0.50	3x overcharge on every fallback-priced turn
claude-haiku-4-5-20251001	0.80 / 4 / 1 / 0.08	1 / 5 / 1.25 / 0.10	25% undercharge (those were Haiku 3.5's rates)
gpt-5.6-sol, -terra, -luna	cache write at 1.25x input	cache write == input	OpenAI bills no cache-write fee; matches every other OpenAI entry

Opus 4.5 and later dropped to $5/$25 while Opus 4.1 and Opus 4 stayed at $15/$75, so the version boundary is the price boundary and a newer Opus is not the more expensive one. That is what the old entries got backwards.

Also added the aliases that silently went unpriced because only the dated id was keyed: claude-opus-4-5, claude-opus-4-1, claude-opus-4, claude-sonnet-4-5, claude-haiku-4-5, claude-haiku-3-5, plus claude-fable-5 and claude-mythos-5.

Verified correct and left alone: every Sonnet 4.x entry, the Claude 3.x entries, all nine remaining OpenAI entries, and the Gemini entries (including the documented >200K-tier caveat). virtuoso-* correctly stays out: the coder_eval_uipath plugin registers it through register_pricing.

One deliberate non-correction: claude-sonnet-5 stays at the standard $3/$15 rather than the $2/$10 introductory rate in effect through 2026-08-31. A static table cannot express a promo window, and of the two available errors this one overstates cost for a few weeks instead of understating it indefinitely afterwards. The comment says so.

So the next missing rate cannot hide

The rate card is a static table baked into the installed version, so this recurs the next time a model ships. Three guards, none of which block a run:

• check_pricing_coverage warns at dispatch for any pinned agent.model the card cannot price. A warning, not a refusal, so a new model stays evaluable the day it ships: cost degrades, the evaluation does not. Its limit, worth knowing: it only sees pinned agent.model, so a model reached through BEDROCK_MODEL gets no dispatch warning.
• tasks_unpriced / cost_complete count what a run actually lost, per row and per run. The report prints (floor — N task(s) have spend missing from this total), or unavailable when nothing could be priced. The wording is cause-agnostic because both failures above land here and the report cannot always tell which applied. One predicate, row_cost_incomplete, is shared by the report and RunSummary so the two cannot disagree about which rows lost money. One edge it does not cover: a run where every row is a ghost renders no token section at all, so the caveat lives only in run.json. A run with any priced row shows it.
• A CI test walks every committed experiment and fails if one names an unpriced model.

Judge and simulator spend is reported at all

Per-row judge_cost_usd and simulator_cost_usd, run-level eval_overhead_cost_usd. This was captured per criterion and rolled up nowhere, so it was absent from every total.

llm_judge prices its own call from criterion.model. The simulator prices from the route model rather than the subject's, because it resolves model=None through to BEDROCK_MODEL; pricing it off the subject would mis-bill every task that pins agent.model.

Kept beside the agent bill rather than folded into it: judge cost is a property of the suite's criteria and identical across harnesses, so folding it in would make two harnesses look closer than they are. **Total Cost** on the report always means the two added together.

Errors count as misses

RunSummary.pass_rate is tasks_succeeded / tasks_run. The old rate divided by tasks_run - tasks_error, so the more a harness fell over, the smaller its denominator got.

How much this actually moves: on a healthy run, very little. Measured across the last six runs:

run	tasks	errors	old	new	Δ
07-28 sonnet-4-6	1005	4 (0.4%)	93.4%	93.0%	0.4
07-27 sonnet-4-6	1002	6 (0.6%)	93.1%	92.5%	0.6
07-26 sonnet-4-6	986	9 (0.9%)	88.9%	88.1%	0.8
07-25 opus-5	984	11 (1.1%)	89.1%	88.1%	1.0
07-25 virtuoso-1-5	984	78 (7.9%)	84.5%	77.8%	6.7
07-24 gpt-5.6-luna	984	0	84.0%	84.0%	0.0

So this is not a headline-number change for the Claude nightly: it costs well under a point. It matters exactly when a harness errors a lot, which is also exactly when the old formula flattered it most. The virtuoso-1-5 row is a live 6.7-point example, and the degenerate case is real: a run that errored on 854 of 861 rows reported 100.0% because 7 of its 7 evaluable rows passed.

pass_rate is a pydantic computed_field, so it serializes into run.json and cannot drift from the counts it derives from. VariantAggregate gets the same property so an A/B whose variants error at different rates compares like with like. error_share ships beside it, diagnostic only, never adjusting the rate: a drop at a high error share is an infrastructure night, the same drop at a normal share is the model.

Minor points

One denominator across every surface. Four surfaces derived their own rate; each now reads the published field (reports.py, reports_experiment.py, reports_html.py). The dashboard was never part of this: evalboard/lib/trends.ts already counted every row in its denominator, so its numbers were correct before this PR and are unchanged by it.

Error diagnostics on the row. error_message (truncated to 400 chars) and error_category, projected from the existing error_details. Nothing new is captured, only surfaced. Without them an errored run is untriageable without fetching per-task artifacts one at a time.

A crash that burned nothing is free, and reported free. An error before the agent ran genuinely cost zero. On a live errored run the zero-token row came back cost_complete: true with tasks_unpriced: 0, which is what keeps the flag a signal rather than something that fires on every error.

Budget kills (TOKEN_BUDGET_EXCEEDED / COST_BUDGET_EXCEEDED) are category failed, not error: they carry their spend on the row like any other task and sit in the denominator as misses.

⚠️ simulator_cost_usd is a floor. UserSimulator records only uncached_input_tokens and drops both cache buckets, so a cached prefix is largely absent from the count. Widening that changes what the run captures rather than what this reports, so it is not done here. Same for a kill that lands inside the simulator between agent turns: the agent's turns are safe, that one simulator call is not.

How it was verified

✅ Live hard kill on three harnesses, on the VM, at this commit: Claude Code, Codex and Antigravity, each against a task built to outlast the 30s floor, each run twice with only the two recovery seams toggled. That is the table above. Claude and Antigravity go from a blank row to a priced one; Codex was already whole. A mid-dialog kill from an earlier session, re-projected through the rule, is what caught the emptiness version of _cost_complete under-flagging.

✅ Live Bedrock run for the judge and simulator paths, which had no live evidence: a dialog task with a simulated user and an llm_judge, subject and judge both sonnet-4-6.

run.agent_cost_usd          = 0.0995112
run.eval_overhead_cost_usd  = 0.008076     # = 0.007518 + 0.000558, exactly
row.judge_cost_usd          = 0.007518
row.simulator_cost_usd      = 0.000558

The route-versus-subject branch was then exercised against that run's real telemetry: repinning the subject to haiku left the simulator cost unchanged, proving it prices at the route; removing the route dropped it to the haiku rate, proving the fallback. A run with no judge reports eval_overhead_cost_usd: null rather than $0.00.

✅ Real-run replay for the cost and rate work: the timed-out row and the nine killed rows above, pulled from blob and repriced through the corrected card. Historical replay for the denominator, plus the six-run table above.

✅ Recovery is pinned by a test that fails without it. An agent-level test streams a turn carrying real usage, cancels it from outside exactly as the task watchdog does, and asserts the record lands on pending_turn with its tokens and a cost backfilled from the buckets. With the one-line finalize change reverted, that assertion reports partial = None: the production symptom. An orchestrator-level test drives a real task timeout and asserts the parked turn reaches both result.iterations and total_token_usage, and a companion asserts an empty slot still lands a clean TIMEOUT row, since the drain runs on the way to a saved row and must not raise.

✅ Every remaining spend path traced rather than assumed. Subject-agent turns and their internal sub-agents fold into the turn total. llm_judge makes exactly one call per criterion with no retry loop and no text-channel fallback, and its criterion.model defaults to a priced model, so it cannot silently go unpriced. agent_judge runs a single communicate(), so its TurnRecord carries the whole sub-agent loop with the SDK's cost. Variant reports feed the same token section, so an A/B gets the same cost lines. Two things specifically looked for and not found: a retry around the judge invoke that would keep only the last attempt's usage, and any double-count between the agent bill and the overhead.

✅ Unit, snapshot and CI. Tests pin the pass-rate reductio, the empty-run None, the error-message truncation, the pricing pre-flight's silent / warning / unpinned paths, the floor and unavailable renderings, and a legacy row reading as cost-complete. Two pre-existing tests that hardcoded the old Opus and Haiku rates were updated to the corrected ones, which is the check that the card change took effect. Report snapshots regenerated: the only diffs are the Success Rate → Pass Rate label, with no rate value moving.

Local: 3840 passed / 8 skipped, make check / typecheck / lint clean. Three failures in test_reports_stats_nonfinite.py are unrelated and pre-existing (cohens_d raises on non-finite input; reproduces with this branch's changes stashed).

🤖 Generated with Claude Code

[github-actions]

Claude finished @bai-uipath's task in 1m 52s —— View job

---

📋 Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards and process
• Read CLAUDE.md for project conventions
• Get full diff with git diff origin/main...HEAD
• Review each changed file with full context:
  • Core pricing and cost accounting changes
  • Model and results updates
  • Report generation changes
  • New test coverage
• Perform cross-file consistency checks
• Analyze "what's missing" from the changeset
• Design-level scrutiny
• Post comprehensive review feedback