Integrate fixture CTS workflow with verified OTT solver contract

.github/workflows/ci.yml

@@ -8,9 +8,11 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
+    timeout-minutes: 20
     steps:
       - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
       - uses: astral-sh/setup-uv@v3
         with:
           enable-cache: true
@@ -24,6 +26,29 @@ jobs:
         run: uv run ruff format --check src tests scripts
       - name: Pyright
         run: uv run pyright
-        continue-on-error: true
       - name: Test
         run: uv run pytest -q -m "not slow"
+      - name: Execute real local CTS optimizer and replay diagnostics
+        env:
+          JAX_PLATFORMS: cpu
+          XLA_PYTHON_CLIENT_PREALLOCATE: 'false'
+        run: |
+          uv run python -m cts.train.main \
+            method=cts task=fixtures_math backend=local_nano \
+            trainer.max_steps=2 trainer.warmup_steps=0 trainer.batch_size=2 \
+            trainer.log_every=1 method_cfg.group_size=3 \
+            method_cfg.max_new_tokens=8 method_cfg.max_len=48 \
+            method_cfg.alignment.num_iters=5 method_cfg.alignment.use_ott=false \
+            out_dir=artifacts/ci/cts-fixture
+          uv run python -m cts.eval.run_eval \
+            --replay src/cts/data/fixtures/gsm8k_tiny.jsonl --split test \
+            --domain math --audited-labels --out artifacts/ci/replay-diagnostic.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: local-cts-execution-${{ github.sha }}
+          path: artifacts/ci/
+          if-no-files-found: error
+      - name: Install media verifier
+        run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends ffmpeg
+      - name: Verify published fixture demo
+        run: uv run python scripts/verify_fixture_demo.py

AGENTS.md

@@ -2,7 +2,7 @@
 
 ## Rules
 
-- **Layout**: source under `src/cts/`, tests under `tests/`, all generated artifacts under `artifacts/` (git-ignored).
+- **Layout**: source under `src/cts/`, tests under `tests/`, all runtime experiment outputs under `artifacts/` (git-ignored). A deliberately published README demo may live under `docs/demo/` only when it includes source-pinned provenance, exact non-claiming receipts, reproducible rendering, and an automated integrity check; it is documentation, never experiment evidence.
 - **Replay-only data**: anything in `src/cts/train/` or `src/cts/methods/` MUST consume `Quadruple` from a `ReplayShard` / `ReplayDir`. Never call a teacher API from a trainer.
 - **Two backend abstractions**: `LocalModelAPI` (smoke / unit tests) and `ScaleBackendAdapter` (Tunix / MaxText). Don't merge them.
 - **Method shape**: every method in `cts.methods` exposes `step(model, batch[, cfg]) -> (loss, metrics)`. Ablations are config toggles on top of `b5_cts`, never new methods.

README.md

@@ -141,4 +141,31 @@ PYTHONPATH=src JAX_PLATFORMS=cpu python -m cts.train.main \
   trainer.max_steps=2 trainer.warmup_steps=0 trainer.batch_size=2
 ```
 
-The SFT and DPO CLI paths now perform the requested Optax updates and emit `train.jsonl` plus `training-receipt.json` (steps, parameter change, finite loss, fairness settings, and decode shape). The receipt labels the tiny randomly initialized model explicitly. The full test suite also exercises joint CTS backbone/projection/critic optimization. CLI RL rollout/reward construction and scale execution remain unsupported and now fail explicitly instead of returning success from a stub. These local runs do not establish improved reasoning quality or reproduce paper-scale results.
+The SFT and DPO CLI paths perform the requested Optax updates and emit `train.jsonl` plus `training-receipt.json` (steps, parameter change, finite loss, fairness settings, and decode shape). The fixture-only CTS/GRPO path also runs locally: each comparison group contains the archived flawed attempt, archived corrected revision, and fresh nano-LM samples. The existing math final-answer verifier supplies transparent 0/1 rewards, while Critique-GRPO additionally reports token agreement with the archived revision as a fixture-only uptake proxy.
+
+```sh
+PYTHONPATH=src JAX_PLATFORMS=cpu python -m cts.train.main \
+  method=cts task=fixtures_math backend=local_nano \
+  trainer.max_steps=2 trainer.warmup_steps=0 trainer.batch_size=2 \
+  method_cfg.group_size=3 method_cfg.max_new_tokens=8 \
+  method_cfg.max_len=48 method_cfg.alignment.num_iters=5 \
+  method_cfg.alignment.use_ott=false out_dir=artifacts/demo/cts-fixture
+```
+
+This is an optimizer/integration fixture, not a benchmark. The model is tiny and randomly initialized; replay candidates deliberately guarantee a non-degenerate reward group, and fresh samples prove the rollout path executes. No local receipt establishes reasoning quality, scale behavior, or paper results.
+
+[![CTS fixture workflow: replay to rollout to parameter update](docs/demo/cts-fixture.gif)](docs/demo/cts-fixture.mp4)
+
+The checked-in [demo receipt](docs/demo/README.md) pins the exact source commit, input blobs, command, logs, artifact digests, and verification procedure.
+
+### Matrix evidence boundary
+
+`run_paper_matrix.py` currently trains each requested method, then evaluates archived
+replay pairs. It does not load the resulting model for evaluation. Fixture runs
+explicitly report human-audited archive labels; they are dispatch and optimizer
+checks, not a paper comparison. Other evaluation calls require a complete gold
+answer or test-bank map and fail when references are absent. Aggregation resamples
+each recorded seed summary once and reports seed counts; it does not manufacture
+instance counts by duplicating means. Large-model / TPU results and trained-model
+quality remain unverified until real checkpoint inference and independent test
+references are wired and executed.

docs/demo/README.md

@@ -0,0 +1,39 @@
+# Local CTS fixture demo
+
+[![Ten-second CTS fixture execution receipt](cts-fixture.gif)](cts-fixture.mp4)
+
+This is a visualization of an actual two-step CPU run from source tree `49d56d14413d3f6d5c539ef997a4766556c8ba3a`, published as commit `fddf9acfc3f3a16e338a3dd5b7e61c34cb1a34e6`. It shows the recorded comparison-group shapes, verifier rewards, optimizer logs, finite final loss, parameter delta, generated-token shape, and fairness fingerprint from `training-receipt.json` and `train.jsonl`.
+
+The run used archived math fixture data, a tiny randomly initialized nano-LM, and no downloads or provider calls. The replay `y0`/`y1` pair guarantees a non-degenerate local verifier group while the third candidate proves fresh sampling executes. This demo makes no policy-quality, benchmark, scale, or paper-result claim.
+
+These files are curated, source-pinned documentation assets under the narrow `AGENTS.md` demo exception. Runtime experiment outputs remain git-ignored under `artifacts/`.
+
+```sh
+PYTHONPATH=src JAX_PLATFORMS=cpu XLA_PYTHON_CLIENT_PREALLOCATE=false \
+python -m cts.train.main \
+  method=cts task=fixtures_math backend=local_nano \
+  trainer.max_steps=2 trainer.warmup_steps=0 trainer.batch_size=2 \
+  trainer.log_every=1 method_cfg.group_size=3 \
+  method_cfg.max_new_tokens=8 method_cfg.max_len=48 \
+  method_cfg.alignment.num_iters=5 method_cfg.alignment.use_ott=false \
+  out_dir=artifacts/demo/cts-fixture
+```
+
+The published media remains a **historical execution receipt** from the source above.
+The current source fixes the optional OTT solver contract and error handling;
+this recorded run explicitly used `use_ott=false` and does not verify OTT.
+The manifest records the changed current source blob separately, while the
+verifier checks both the historical execution blobs and current source.
+Actual installed-OTT balanced/unbalanced values, gradients, and failure
+propagation are verified by `tests/unit/test_losses.py`.
+
+Re-render and verify:
+
+```sh
+python scripts/render_fixture_demo.py \
+  --receipt docs/demo/training-receipt.json \
+  --log docs/demo/train.jsonl \
+  --gif docs/demo/cts-fixture.gif \
+  --mp4 docs/demo/cts-fixture.mp4
+python scripts/verify_fixture_demo.py
+```

docs/demo/console.txt

@@ -0,0 +1,4 @@
+[cts] method=cts backend=local_nano task=fixtures_math
+[cts] fairness fingerprint: FairnessFingerprint(base_ckpt_hash=None, total_train_tokens=None, total_opt_steps=2, prompt_template_hash='f6edf26d62fd', decode_temperature=1.0, verifier_id=None, revision_budget=1, test_compute_budget=None)
+[cts] trained steps=2 loss=5.6910 delta=14.0838
+[cts] metrics={'total': 5.691038608551025, 'grpo_loss': -0.0486871562898159, 'approx_kl': 0.020017830654978752, 'decode_anchor': 5.639525890350342, 'alignment': 8.985807653516531e-10, 'energy_loss': 1.0, 'reward_mean': 0.3333333432674408}

docs/demo/source-manifest.json

@@ -0,0 +1,33 @@
+{
+  "captured_source_commit": "fddf9acfc3f3a16e338a3dd5b7e61c34cb1a34e6",
+  "execution_local_commit": "5c7c08b5dc9bdd62fca5055f09eb26343665710a",
+  "captured_source_tree": "49d56d14413d3f6d5c539ef997a4766556c8ba3a",
+  "scope": "tiny random fixture on CPU; no policy-quality or scale claim",
+  "source_blobs": {
+    "src/cts/backends/local_api.py": "492eacf252a82a66b6b9434526cc204a2b165c85",
+    "src/cts/backends/nano_lm.py": "4ac62dba087fcdd8b889966709eb4abb1414f155",
+    "src/cts/configs/base.yaml": "c781268f007818552c8405d899f90facc04e05f8",
+    "src/cts/configs/method/cts.yaml": "b2b1078de75db05dc4091905eb3e40e5562349be",
+    "src/cts/configs/task/fixtures_math.yaml": "cfb0810ec5b47d574213e0372c095c8f70beb2f9",
+    "src/cts/data/fixtures/gsm8k_tiny.jsonl": "6004682ec253e91c993ccba09f9829e1489275f2",
+    "src/cts/losses/alignment.py": "345f0c031c5df897568344f613d15fa288b8c8be",
+    "src/cts/losses/sinkhorn.py": "aed5130f98f5c528a326c086d93587adf96b10d0",
+    "src/cts/methods/b2_grpo_outcome.py": "7669238aae7f31ef248882be5bfc4df37239ee21",
+    "src/cts/methods/b5_cts.py": "408bb6f75c286f47e480e38edaa574bbfbee31a4",
+    "src/cts/rl/rollout.py": "82558469f62f088bc60ef04463030e5ca3b4c904",
+    "src/cts/rl/verifier.py": "7cca2c5627ab1db4b05ebe813993a1361ce9d31a",
+    "src/cts/train/fixture_rl.py": "0d46e9376bcb32db8c50b4a1ed678b47db5e3327",
+    "src/cts/train/loop.py": "184bb9816b7446a15c989ad1da349d0318e27193",
+    "src/cts/train/main.py": "686065dc66e95900ea942c77413c7c864f6dc52c"
+  },
+  "artifact_sha256": {
+    "docs/demo/console.txt": "b47dd7c054fa95c02fefad3c35b0a138b25cad2349c16bb7932f3c1773306fb5",
+    "docs/demo/cts-fixture.gif": "e5c540b12b32914c49b4b3bdc2487c27b837eac5d8230e64a5b292d3af1fe192",
+    "docs/demo/cts-fixture.mp4": "ac7c3aa11505de339b7a17a35902e418354274bfb69af38df3f396678b8db82d",
+    "docs/demo/train.jsonl": "f2fcdb8680726252b7b1f45b4c16d66010f00b7f5e9fddf1356d7c16bafda448",
+    "docs/demo/training-receipt.json": "299cb7adfaa1e6703f446a8e7997182a1729b3dbc0a3772b280e418410fbd80d"
+  },
+  "current_source_overrides": {
+    "src/cts/losses/sinkhorn.py": "87fc282a4dbd9d411529191c5cacc9f7a3e04ef3"
+  }
+}

docs/demo/train.jsonl

@@ -0,0 +1,2 @@
+{"step": 0, "t": 1791874121.769764, "loss": 5.900531768798828, "alignment": 2.370143192820251e-09, "approx_kl": 0.0, "decode_anchor": 5.80053186416626, "energy_loss": 1.0, "grpo_loss": 3.973643103449831e-08, "reward_mean": 0.3333333432674408, "total": 5.900531768798828}
+{"step": 1, "t": 1791874125.073712, "loss": 5.7458343505859375, "alignment": 6.475784175563604e-08, "approx_kl": 0.00907925982028246, "decode_anchor": 5.695555210113525, "energy_loss": 1.0, "grpo_loss": -0.049811430275440216, "reward_mean": 0.3333333432674408, "total": 5.7458343505859375}

docs/demo/training-receipt.json

@@ -0,0 +1,42 @@
+{
+  "method": "cts",
+  "backend": "local_nano",
+  "steps": 2,
+  "parameter_delta_l1": 14.083788089745212,
+  "loss": 5.691038608551025,
+  "fairness_fingerprint": {
+    "base_ckpt_hash": null,
+    "total_train_tokens": null,
+    "total_opt_steps": 2,
+    "prompt_template_hash": "f6edf26d62fd",
+    "decode_temperature": 1.0,
+    "verifier_id": null,
+    "revision_budget": 1,
+    "test_compute_budget": null
+  },
+  "replay_path": "src/cts/data/fixtures/gsm8k_tiny.jsonl",
+  "replay_hash": "c53e0b2f08d3bb35",
+  "generated_token_shape": [
+    2,
+    2
+  ],
+  "scope": "tiny random fixture on CPU; no policy-quality or scale claim",
+  "fixture_rollout": {
+    "reward_id": "fixture_math_final_number_v1",
+    "group_size": 3,
+    "sampled_rollouts_per_prompt": 1,
+    "sampled_rollout_shape": [
+      2,
+      1,
+      8
+    ],
+    "candidate_group_shape": [
+      2,
+      3,
+      8
+    ],
+    "reward_min": 0.0,
+    "reward_max": 1.0,
+    "reward_mean": 0.3333333432674408
+  }
+}

scripts/aggregate_results.py

@@ -31,8 +31,12 @@ def _load_cells(root: Path) -> dict[tuple[str, str], list[dict]]:
         m = CELL_RE.match(p.name)
         if not m:
             continue
-        method, task, _seed = m.groups()
-        out[(method, task)].append(json.loads(p.read_text()))
+        method, task, seed = m.groups()
+        cell = json.loads(p.read_text())
+        cell["_seed"] = int(seed)
+        out[(method, task)].append(cell)
+    for runs in out.values():
+        runs.sort(key=lambda cell: cell["_seed"])
     return out
 
 
@@ -42,13 +46,13 @@ def _delta_means(cells: list[dict]) -> list[float]:
 
 def _format_table(rows: list[dict]) -> str:
     lines = [
-        "| Method | Task | Δ_critique mean ± std (seeds) | 95% paired-bootstrap CI (instances) | Wilcoxon p vs B3 | Holm reject |",
+        "| Method | Task | Δ_critique mean ± std (seeds) | 95% paired-bootstrap CI (seeds) | Wilcoxon p vs comparator | Holm reject |",
         "|---|---|---|---|---|---|",
     ]
     for r in rows:
         lines.append(
             f"| {r['method']} | {r['task']} | {r['mean']:+.4f} ± {r['std']:.4f} (n={r['n_seeds']}) "
-            f"| [{r['ci_lo']:+.4f}, {r['ci_hi']:+.4f}] (n={r['n_instances']}) "
+            f"| [{r['ci_lo']:+.4f}, {r['ci_hi']:+.4f}] (n={r['n_seeds']}) "
             f"| {r['p']:.3g} | {'yes' if r['reject'] else 'no'} |"
         )
     return "\n".join(lines)
@@ -80,21 +84,16 @@ def main() -> None:
     rows: list[dict] = []
     for (method, task), runs in cells.items():
         seed_means = _delta_means(runs)
-        # Reconstruct instance-level deltas by pooling y0/y1 pass rates per cell.
-        # The eval JSON exposes only headline + buckets, so use headline deltas
-        # repeated as the bootstrap source — coarse but defensible per-seed.
-        # For instance-level CIs the user must run aggregate against per-cell
-        # per-instance dumps (future extension).
-        pooled = seed_means * 50  # nominal resampling unit
-        ci = paired_bootstrap_ci([0.0] * len(pooled), pooled, n_resamples=2000)
+        # Resample actual independent seed summaries once each. Repeating them
+        # invents sample size and understates uncertainty.
+        ci = paired_bootstrap_ci([0.0] * len(seed_means), seed_means, n_resamples=2000)
         rows.append(
             {
                 "method": method,
                 "task": task,
                 "mean": mean(seed_means),
                 "std": pstdev(seed_means) if len(seed_means) > 1 else 0.0,
                 "n_seeds": len(seed_means),
-                "n_instances": len(pooled),
                 "ci_lo": ci["lo"],
                 "ci_hi": ci["hi"],
                 "_seed_means": seed_means,
@@ -106,14 +105,18 @@ def main() -> None:
     for task, per_method in by_task.items():
         base = per_method.get(args.base)
         if base is None:
-            continue
+            raise SystemExit(f"Comparator {args.base} missing for task {task}")
         contenders = [m for m in per_method if m != args.base]
         pvals: list[float] = []
         for m in contenders:
             other = per_method[m]
-            # pad to common length for paired test
-            n = min(len(base), len(other))
-            wx = wilcoxon_signed_rank(base[:n], other[:n])
+            base_seeds = [r["_seed"] for r in cells[(args.base, task)]]
+            other_seeds = [r["_seed"] for r in cells[(m, task)]]
+            if base_seeds != other_seeds:
+                raise SystemExit(
+                    f"Paired comparison requires identical seed IDs: {args.base} and {m} for {task}"
+                )
+            wx = wilcoxon_signed_rank(base, other)
             pvals.append(wx["p"])
         rejects = holm_bonferroni(pvals, alpha=args.alpha)
         for m, p, rej in zip(contenders, pvals, rejects, strict=True):
@@ -130,7 +133,9 @@ def main() -> None:
     table = _format_table(rows)
 
     out_text = (
-        f"# CTS paper matrix — aggregated results\n\n"
+        f"# CTS replay-pair diagnostics\n\n"
+        "These scores describe archived y0/y1 replay pairs, not trained-model performance. "
+        "Repeated fixture evaluations are not independent research evidence.\n\n"
         f"Comparator: **{args.base}**, α = {args.alpha} (Holm-corrected within task).\n\n"
         f"{table}\n"
     )