[stedrew]

Summary

Steve decided on 2026-10-16, after the meeting named L2 error as the primary metric, that the evaluation gate decides on relative l2 only.

• Default gate: {"enabled": true, "max_rel_l2_increase": 0.0, "max_peak_distance_increase_mm": null}. Any rise in mean relative l2 blocks the candidate.
• Peak distance, SSIM, PSNR, peak amplitude error and the -3 dB centroid distance are still in eval_report.json, overall and per region, but do not block.
• max_peak_distance_increase_mm is now an opt-in check. A config that sets it to a number behaves as before.
• This supersedes answer T8 of 2026-09-27. With T8, a workbench round was rejected even though peak distance fell from 1.81 to 0.80 mm.
• gate.py docstring and TASK_GUIDE.md updated.

Tests

• New DecideTest cases:
  • the demo case: relative l2 falls while peak distance rises, which is accepted;
  • a slight rise in relative l2, which is rejected;
  • a non-finite peak distance, which does not block by default;
  • the opt-in peak distance check.
• manage.py test starfish.controller.tasks.babel_brain_fno: 143 tests, OK. Run locally on Python 3.12 with CPU torch and a fakeredis server.
• Full controller suite: 583 tests. All pass except 47 R task tests, 19 failures and 28 errors. Those fail because this machine's R lacks jsonlite and mice; they touch no gate code. CI installs the R packages.

Follow-up

The demo on #86, workbench/babelbrain-demo/run_demo.py, still describes the old rule: its intro line, its --peak-margin-mm help text, which says "default 0, answer T8". Its README still reads correctly, since --peak-margin-mm now opts into the peak check. Its gate narration itself reads eval_report.json and will show the new decisions correctly.

🤖 Generated with Claude Code