Every optimization task in the suite starts from a seed that already works, so
what they measure is tuning: how much an optimizer adds to a competent program.
This variant asks a different question -- what an optimizer does when there is
nothing to tune and it has to write the program first.
`build.shell.yaml` runs the same benchmark against `target-shell/`. The skeleton
there satisfies the Harbor interface, resolves the model and constructs a
client, but its `run` writes an empty answer and returns. 72 lines against the
seed's 359.
The plumbing is deliberately kept. Model resolution, the
`removeprefix("openai/")` the gateway allow-list requires, and the client
construction are properties of this harness rather than of GAIA; making an
optimizer rediscover them by trial and error would spend budget on the wrong
thing and add variance unrelated to the question.
It writes an empty answer rather than doing nothing because a case that scores
zero and a case that errors are different events here -- errors count against
`error_rate_threshold`, and a wholly erroring evaluation comes back `invalid`.
Writing the file keeps every case scoreable, so the floor is a real 0.0 and a
half-built candidate gets a number rather than nothing.
`baseline_reward: 0.0` is a claim, not a measurement: it follows from the
skeleton writing an empty answer. Confirm it with one baseline round before
quoting deltas against it.
The shared instruction template opens with "Improve the program", which is the
wrong framing here and cannot change without touching every benchmark, so the
reframing lives entirely in `description`. It says the target is a skeleton, that
the first job is to make it work at all and the second is to make it good, and
leaves the approach open -- no mention of tools, search or turn budgets, which
would be an answer key.
The new invariant test pins the variant to the seeded config on task source,
partitions, target model and both gateway scopes, which is what makes the two
runs comparable, and asserts the skeleton never issues a model request. Nothing
else in the suite would notice an implementation reappearing in the shell, and a
shell that scores above zero is not a shell.
Rationale lives in the baseline README rather than the target, which the
optimizer mounts and would read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every optimization task in the suite starts from a seed that already works, so what they measure is tuning — how much an optimizer adds to a competent program. This variant asks a different question: what an optimizer does when there is nothing to tune and it has to write the program first.
build.shell.yamlruns the same benchmark againsttarget-shell/instead oftarget/. The skeleton satisfies the Harbor interface, resolves the model and constructs a client, but itsrunwrites an empty answer and returns. 72 lines against the seed's 359.Design choices worth reviewing
The plumbing is deliberately kept. Model resolution, the
removeprefix("openai/")the gateway allow-list requires, and the client construction are properties of this harness rather than of GAIA. Making an optimizer rediscover them by trial and error would spend budget on the wrong thing and add variance unrelated to the question being asked.It writes an empty answer rather than doing nothing. A case that scores zero and a case that errors are different events here: errors count against
error_rate_threshold, and a wholly erroring evaluation comes backinvalid. Writing the file keeps every case scoreable, so the floor is a real0.0and a half-built candidate gets a number rather than nothing.The reframing lives entirely in
description. The shared instruction template opens with "Improve the program", which is the wrong framing here and cannot change without touching every benchmark. The description says the target is a skeleton, that the first job is to make it work at all and the second is to make it good, and leaves the approach open — no mention of tools, search or turn budgets, which would be an answer key.Rationale lives in the baseline README, not the target. The optimizer mounts
target-shell/and would read anything placed there.Needs confirming before use
baseline_reward: 0.0is a claim, not a measurement — it follows from the skeleton writing an empty answer, but nothing has measured it. Worth one baseline round before quoting deltas against it. The config comment says so.Test coverage
Nothing else in the suite would notice an implementation reappearing in the shell, so
test_gaia_shell_variant_shares_the_measurement_substrate_and_stays_a_shellpins the variant to the seeded config on task source, partitions, target model and both gateway scopes — that is what makes the two runs comparable — and asserts the skeleton never issues a model request. Negative-controlled: adding aresponses.createcall fails it with the right message; reverting passes.One trap for anyone running these locally:
uv run pytestpicks Python 3.14 by default and litellm fails to build there. Use--python 3.12, which is whatharbor_python_versionalready specifies.Launching it
🤖 Generated with Claude Code
Greptile Summary
Adds a GAIA benchmark variant that starts from a scoreable empty agent.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains.
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[build.shell.yaml] --> B[Resolve target-shell and instruction template] B --> C[Compile benchmark] C --> D[Render custom framing] C --> E[Inherit shared workflow and rules] C --> F[Package empty GaiaAgent seed] F --> G[Run GAIA case] G --> H[Write empty answer.txt] H --> I[Scoreable zero-result baseline]Reviews (2): Last reviewed commit: "Let a build supply its own instruction t..." | Re-trigger Greptile