Add the UK national calibration step over ledger-backed target references

[MaxGhenis]

Summary

The national calibrate step scoped in #623, stacked on #727 (base branch; the first commit here is tree-identical to its head). Part of #623 and #701 — the post-calibration aggregate-scaler adjudication (2026-08-14 comment) stays open and out of scope: no incumbent scalers are ported.

• uk_runtime/national_calibration.py: resolves activated references from uk/target_references.json against the pinned chronicle facts; an activated reference that cannot compile aborts the build (the policyengine-uk-data#456 silent-drop class is impossible by construction). Builds the benefit-unit-grain target matrix per the reference declarations and calibrates household weights via microcosm.calibrate.calibrate() (public API only).
• Activation: dwp.uc.households (caseload, sum of benefit units with universal_credit.amount > 0) and obr.universal_credit_in_cap (expenditure), both period 2025 under the resolver's per-reference resolve-at-or-before semantics. Facts: chronicle#184 (fixture-backed here; live resolution needs the released chronicle bundle carrying Add proximal L1 solver path #184 — stated dependency).
• Diagnostics: per-target estimate/target/rel-error plus loss/ESS/weight-ratio in the step manifest, following the US build's calibration-diagnostics pattern.
• Gate: release-blocking activated-vs-entered count check, per the UK gate adjudication: reference registry, spec-armed thresholds, weight-ratio re-baseline (#630) #706 registry pattern (uk/gates.json).
• Driver: tools/build_uk_national_dataset.py grows an opt-in calibration flag + diagnostics JSON output; the staging seam's behaviour without the flag is unchanged.

No dataset was built; running this on the real staging H5 is a later, gated act. Refs: #707, #727, chronicle#184, policyengine-uk-data#452.

🤖 Generated with Claude Code

[juaristi22]

Adversarial review of the rebased branch: findings and dispositions

A Codex adversarial review ran against this branch after the rebase onto post-#727 main (all findings independently reproduced before being accepted). Disposition of each:

Fixed in this PR (6c8a95a)

3. Checkpoint resume dropped calibration evidence. UKNationalCalibrationStage exposed checkpoint_metadata() but no resume_from_checkpoint, so a build resumed past a completed calibration left manifest = None; _stage_calibration_evidence handed the coverage gate {}, and the release-blocking uk_calibration_reference_coverage entry failed closed on a swallowed KeyError (the driver sidecar wrote {"targets": [], "calibration": null}). Fixed following the retained-leaves/SPI-restoration precedents: the checkpoint record now carries the output frame's content identity, and resume rehydrates manifest/diagnostics with fail-closed validation of the three coverage counts plus a drifted-record refusal. Tests: JSON round-trip + refusals; resumed evidence through the real evaluator; end-to-end crash-resume build asserting the gate passes without re-executing calibration.

Arriving with the stacked #622-completion PR

2. Benunit-grain references cannot compile against household weights. matrix.py's only cross-entity path is person-grain collapse (_entity_row, no entity check): on any frame where benunits ≠ households the stage aborts (array is not broadcastable to correct shape → No targets compiled into the constraint system; repro: 6-person/4-benunit/3-household frame). Nastier: when n(benunit) == n(person) coincidentally — the shape of this PR's fixtures — the scatter silently produces a garbage row that happens to be numerically right, so the existing tests pass by accident. Disposition (adjudicated): fixed entity-general in microcosm-calibrate by the #622-completion PR, whose regenerated surface carries 112 benunit-grain references and a non-1:1:1 regression frame (which must also cover the silent-coincidence case). Until it lands, armed builds (--ledger-facts) abort fail-closed; no silent path exists.

1. The 2025 UC caseload activation binds the December stock, not the intended basis. With chronicle#184's monthly facts now landed, dwp.uc.households at period: 2025 under identity-only operations resolves latest-at-or-before → December 2025 (7.17m) instead of the calendar-year-2025 average (~6.76m) named in the #701 thread — about 6% high, and the resolved/declared count checks cannot catch it. Disposition: the #622 PR ships the calendar_year_average (and latest_plateau) value operations and the regenerated declaration carrying the basis. Until then the 2025 activation resolves December by default — recorded here as a known interim state, not a decision.

Discovered while implementing the resume fix (needs an owner)

4. Prepared scratch columns crash the real staging writer. The stage's prepared measure columns are slash-named (dwp/uc/households) and persist on the returned frame; the staging writer is pandas-HDFStore-based with data_columns=True, which refuses / in column names — so a real armed build would crash at the staging write after a successful calibration. The new e2e test works around it with an explicit cleanup stage (marked as such). Candidate fixes: the stage restores pristine entity tables after the solve (keeping only calibrated weights + mass record), or the shared #622 materializer owns scratch-column lifecycle. Flagging for adjudication rather than fixing here.

Rebase housekeeping already on the branch

• 9dcc83b4: the new gate joins Port the LCFS consumption, ETB VAT, and ETB services/NHS layers as declarative source stages #730's closed unarmed-battery enumeration (TestUnevidencedArms).
• b5c926e7: the three UK gate-battery vintage digests + entry-gates mirrors re-pinned for the new gates.json entry (data-contract module and its schema-3-style test copies).

🤖 Generated with Claude Code