fix(reports): count errors as misses and stop losing cost on error paths

docs/REPORT_SCHEMA.md

@@ -51,21 +51,62 @@ run-level summary; full per-replicate detail lives in each `task.json`.
 | `framework_version` | `str` | Coder Eval version chip. |
 | `environment_info` | `dict` | Version/dependency info (may nest, e.g. `tool_plugins`). |
 
+These are **computed**, not stored — derived from the counts and rows above on every
+serialization, so they cannot drift from what they summarize. Read them rather than
+re-deriving your own; independent re-derivations are how two consumers end up
+publishing different numbers for the same run.
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `pass_rate` | `float \| None` | `tasks_succeeded / tasks_run` — errors are in the denominator, counted as misses. `None` on an empty run (0/0 is unknown, not 0%). |
+| `error_share` | `float \| None` | `tasks_error / tasks_run`. Diagnostic only; never adjusts the rate. |
+| `total_cost_usd` | `float \| None` | **The bill**: agent + judge + simulator, summed over the rows. `None` when nothing could be priced. |
+| `agent_cost_usd` | `float \| None` | Subject-agent spend alone. The harness-vs-harness comparison figure — judge spend is a property of the suite's criteria and identical across harnesses, so leaving it in would make two harnesses look closer than they are. |
+| `eval_overhead_cost_usd` | `float \| None` | Judge + simulator spend. The other half of `total_cost_usd`. |
+| `tasks_cost_incomplete` | `int` | Rows whose recorded spend is missing money (unpriced model, or a hard kill that lost an in-flight turn). |
+| `cost_complete` | `bool` | `tasks_cost_incomplete == 0`. When false, every cost figure above is a **floor**, not the bill. A run is never failed for this — see [Missing cost is never fatal](#missing-cost-is-never-fatal). |
+
 ### `task_results[]` — the flat per-task row
 
 Each entry is an **untyped dict** (a denormalization, not a Pydantic model) with keys
 including: `task_id`, `replicate_index`, `variant_id`, `status`
 ([`FinalStatus`](#finalstatus)), `weighted_score`, `duration`, `iteration_count`,
 `tags`, `task_path`, `model_used`, `reference_similarity`, the token buckets
 (`input_tokens` = uncached input, `output_tokens`, `cache_creation_input_tokens`,
-`cache_read_input_tokens`, `total_tokens`), `total_cost_usd`, `expected_commands`,
+`cache_read_input_tokens`, `total_tokens`), the cost fields
+(`total_cost_usd` = agent + judge + simulator, plus the `agent_cost_usd` /
+`judge_cost_usd` / `simulator_cost_usd` slices and the `cost_complete` flag),
+`expected_commands`,
 `actual_commands`, `commands_efficiency`, `agent_config`, `sdk_options`,
 `installed_tools`, turn accounting (`total_turns`, `visible_turns`, `expected_turns`,
 `max_turns_exhausted`, `has_final_reply`), and early-stop fields (`stopped_early`,
 `early_stop_reason`, `turns_remaining_at_stop`). `iterations` here is a **reduced**
 turn digest (`{iteration, duration_seconds, command_count, assistant_turn_count,
 crashed, crash_reason}`) — the full transcript is in `task.json`.
 
+### Missing cost is never fatal
+
+Pricing degrades; the evaluation does not. A model absent from the rate card, a turn
+the backend never priced, a hard-killed task that lost its in-flight spend: each one
+lowers a total and sets `cost_complete: false`. None of them raises, none of them
+books a zero, and none of them changes a run's exit code.
+
+The reasoning is that the two failure modes are not symmetric. A missing cost is
+recoverable after the fact — the token counts are on the record, so a corrected rate
+card reprices the run from its artifacts. A failed run is not: the tokens are already
+spent and the only way back is to run it again. So the framework warns loudly and
+keeps going.
+
+The warning fires up front. `check_pricing_coverage` walks every model the run pins
+(subject agents and judge criteria) before the first task dispatches, and logs the
+ones the card cannot price — early enough to fix the card and restart while it is
+still cheap. After that the run is on its own: totals become floors, and
+`tasks_cost_incomplete` says how many rows are behind that floor.
+
+Consumers should treat any cost field as a lower bound whenever `cost_complete` is
+false, and must not read `None` as `0.0` — "nothing could be priced" and "it was
+free" are different facts.
+
 ---
 
 ## `task.json` — `EvaluationResult`
@@ -247,6 +288,12 @@ respectively), checked after each completed agent turn — see
 - `TokenUsage.total_tokens` is not serialized; sum the buckets (or use the computed
   `input_tokens` + `output_tokens` + cache buckets).
 - `EarlyStopInfo` presence is itself the "stopped early" signal.
+- `total_cost_usd` is the whole bill (agent + judge + simulator) at both row and run
+  level; `agent_cost_usd` is the agent-only slice. `TokenUsage.total_cost_usd` is a
+  different thing: the cost of those tokens, so always agent-only. `run_limits.max_usd`
+  gates on that one, since judge and simulator spend is not known mid-run.
+- A cost of `None` means unpriced, not free, and any total is a floor while
+  `cost_complete` is false — see [Missing cost is never fatal](#missing-cost-is-never-fatal).
 
 ## See also
 

src/coder_eval/agent.py

@@ -147,6 +147,17 @@ def _finalize_and_raise_crash(
             raise AgentCrashError(message) from cause
         raise AgentCrashError(message)
 
+    def _finalize_external_cancel(self, finalize: _FinalizeFn) -> None:
+        """Finalize a turn cancelled from outside (the task watchdog) as a crash. Does NOT raise.
+
+        Only the ``crashed`` branch parks the record on ``pending_turn``; finalizing
+        as ``COMPLETED`` drops it, and the unwinding frame takes the return value
+        with it, so a killed turn's telemetry survives only via this path. The caller
+        re-raises the ``CancelledError`` afterwards.
+        """
+        self._state = AgentState.ERROR
+        finalize(AgentEndStatus.CRASHED, crashed=True, crash_reason="turn cancelled")
+
     def _capture_partial_turn(self, collector: EventCollector) -> None:
         """Build the crashed partial ``TurnRecord`` into ``pending_turn`` (best-effort).
 

src/coder_eval/agents/antigravity_agent.py

@@ -514,8 +514,7 @@ def _on_turn_timeout() -> None:
             raise
         except asyncio.CancelledError:
             if not state.finalized:
-                self._state = AgentState.ERROR
-                state.finalize(AgentEndStatus.CRASHED, crashed=True, crash_reason="turn cancelled")
+                self._finalize_external_cancel(state.finalize)
             raise
         except Exception as e:
             if state.stopped_early_hit and not state.timeout_hit:

src/coder_eval/agents/claude_code_agent.py

@@ -980,6 +980,11 @@ def _on_turn_timeout() -> None:
             if self._timed_out(state.timeout_hit, deadline):
                 assert timeout is not None
                 self._finalize_and_raise_timeout(state.finalize, timeout)
+            # Cancelled from outside this turn: park the telemetry on `pending_turn`
+            # for the caller to drain. Otherwise the `finally` below finalizes as
+            # COMPLETED, which keeps no record.
+            if not state.finalized:
+                self._finalize_external_cancel(state.finalize)
             raise
         except ProcessError as e:
             # When the watchdog SIGKILLs the subprocess, the SDK surfaces it as a

src/coder_eval/agents/codex_agent.py

@@ -864,8 +864,7 @@ def _on_turn_timeout() -> None:
             # stays balanced and the pending-turn contract holds. finalize is
             # idempotent, so the timeout case is a no-op here.
             if not state.finalized:
-                self._state = AgentState.ERROR
-                state.finalize(AgentEndStatus.CRASHED, crashed=True, crash_reason="turn cancelled")
+                self._finalize_external_cancel(state.finalize)
             raise
         except Exception as e:
             # Catches failures OUTSIDE the inner turn block — notably thread_start

src/coder_eval/criteria/llm_judge.py

@@ -201,7 +201,7 @@ async def _invoke_tool_channel(
                 tool_spec=SUBMIT_VERDICT_ANTHROPIC_TOOL,
             )
             verdict, err = extract_verdict_from_anthropic_response(response)
-            response_usage = token_usage_from_anthropic_dict(response)
+            response_usage = token_usage_from_anthropic_dict(response, model=criterion.model)
         case DirectRoute():
             anthropic_response = await invoke_anthropic_judge_async(
                 model=criterion.model,
@@ -212,7 +212,7 @@ async def _invoke_tool_channel(
                 tool_spec=SUBMIT_VERDICT_ANTHROPIC_TOOL,
             )
             verdict, err = extract_verdict_from_anthropic_response(anthropic_response)
-            response_usage = token_usage_from_anthropic_dict(anthropic_response)
+            response_usage = token_usage_from_anthropic_dict(anthropic_response, model=criterion.model)
         case LiteLLMRoute():
             # Defensive: the evaluation route is pinned to Bedrock/Direct by
             # resolve_evaluation_route, so a LiteLLM route should never reach the

src/coder_eval/evaluation/judge_usage.py

@@ -15,6 +15,7 @@
 from typing import Any
 
 from coder_eval.models import TokenUsage
+from coder_eval.pricing import calculate_cost
 
 
 def _coerce_int(value: Any) -> int:
@@ -31,12 +32,17 @@ def _coerce_int(value: Any) -> int:
         return 0
 
 
-def token_usage_from_anthropic_dict(resp: dict[str, Any]) -> TokenUsage | None:
+def token_usage_from_anthropic_dict(resp: dict[str, Any], *, model: str | None = None) -> TokenUsage | None:
     """Extract usage from an Anthropic / Bedrock-invoke Messages response dict.
 
     Both ``invoke_anthropic_judge_async`` (``response.model_dump()``) and
     ``invoke_bedrock_judge_async`` (parsed ``/invoke`` JSON) carry an Anthropic-shaped
     ``usage`` block. Returns ``None`` when usage is missing or carries no tokens.
+
+    ``model`` prices the call from the rate card. Neither judge backend returns a
+    cost, so without it the judge's spend is invisible in every rollup. Left
+    unpriced (``total_cost_usd=None``) when the model is absent from the card,
+    which ``RunSummary.tasks_cost_incomplete`` then surfaces.
     """
     u = resp.get("usage")
     if not isinstance(u, dict):
@@ -47,4 +53,14 @@ def token_usage_from_anthropic_dict(resp: dict[str, Any]) -> TokenUsage | None:
         cache_creation_input_tokens=_coerce_int(u.get("cache_creation_input_tokens")),
         cache_read_input_tokens=_coerce_int(u.get("cache_read_input_tokens")),
     )
-    return None if tu.is_empty() else tu
+    if tu.is_empty():
+        return None
+    if model:
+        tu.total_cost_usd = calculate_cost(
+            model,
+            uncached_input_tokens=tu.uncached_input_tokens,
+            output_tokens=tu.output_tokens,
+            cache_creation_tokens=tu.cache_creation_input_tokens,
+            cache_read_tokens=tu.cache_read_input_tokens,
+        )
+    return tu

src/coder_eval/models/__init__.py

@@ -127,6 +127,12 @@
     TaskConfigRecord,
     ThresholdCheck,
     TurnRecord,
+    eval_overhead_cost,
+    eval_result_total_cost,
+    judge_cost_usd,
+    row_cost_incomplete,
+    simulator_cost_usd,
+    sum_costs,
 )
 
 # Routing
@@ -291,6 +297,14 @@
     "TaskConfigRecord",
     "RunSummary",
     "SkippedTask",
+    # Cost helpers, shared by RunSummary's computed fields and the reports so
+    # every surface agrees on what a total costs and which rows lost money.
+    "row_cost_incomplete",
+    "eval_overhead_cost",
+    "sum_costs",
+    "eval_result_total_cost",
+    "judge_cost_usd",
+    "simulator_cost_usd",
     # Judge defaults
     "DEFAULT_JUDGE_MODEL",
     # Judge

src/coder_eval/models/experiment.py

@@ -6,7 +6,7 @@
 from pathlib import Path
 from typing import Any, Literal, Self
 
-from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+from pydantic import BaseModel, ConfigDict, Field, computed_field, field_validator, model_validator
 
 from coder_eval.models.enums import FinalStatus
 from coder_eval.models.limits import RunLimits
@@ -194,7 +194,12 @@ class VariantResult(BaseModel):  # noqa: CE009 -- persisted result model; round-
 
 
 class VariantAggregate(BaseModel):  # noqa: CE009 -- persisted result model; round-trip leniency like models/results.py
-    """Aggregated statistics for a single variant across all tasks."""
+    """Aggregated statistics for a single variant across all tasks.
+
+    ``pass_rate`` uses the same denominator as ``RunSummary.pass_rate``: every task
+    the variant ran, errors included as misses. Otherwise an A/B whose variants
+    error at different rates compares two different denominators.
+    """
 
     variant_id: str
     tasks_run: int
@@ -228,6 +233,12 @@ def _check_task_count_invariant(self) -> VariantAggregate:
             raise ValueError(f"Task count invariant violated: {total} != {self.tasks_run}")
         return self
 
+    @computed_field  # type: ignore[prop-decorator]
+    @property
+    def pass_rate(self) -> float | None:
+        """``tasks_succeeded / tasks_run`` as a 0-1 fraction. ``None`` on an empty variant."""
+        return self.tasks_succeeded / self.tasks_run if self.tasks_run else None
+
 
 class TaskExperimentSummary(BaseModel):  # noqa: CE009 -- persisted result model; round-trip leniency like models/results.py
     """Cross-variant summary for a single task."""