Repository navigation
Practice tests return annotations #1692
Description
In new benchmarks are are no longer offering a "demo" test that returns annotations. Instead, there annotations will be returned with practice benchmarks. However, this is a flag to be sent in the SKU:
practice set with no annotations sent: airr_safety_1_2_en_us_practice
practice set with annotations: airr_safety_1_2_en_us_annotation
So "annotation" means "practice with annotations returned".
Activity
confiscate commentedon Oct 29, 2026
Hi @bollacker , I'd be interested in implementing this.
I noticed that ModelBench already supports annotation export through
--annotationsandBenchmarkRun.compile_annotations().Is this issue primarily about mapping the
airr_safety_1_2_en_us_annotationSKU to the existing annotation-export behavior, or does it require changes to the evaluator service as well?I'd be happy to take on the ModelBench changes if they can be implemented and tested locally.