tau3: gpt-5.4-mini baseline + a benchmark-agnostic optimizer-commit analysis skill - #84
Open
yash-scaleai wants to merge 6 commits into
Open
tau3: gpt-5.4-mini baseline + a benchmark-agnostic optimizer-commit analysis skill#84yash-scaleai wants to merge 6 commits into
yash-scaleai wants to merge 6 commits into
Conversation
… turns Follows the variant convention swe-atlas-qna set: build.yaml stays on deepseek-v4-flash and build.gpt54mini.yaml retargets the benchmark, so the change is reversible and the two configs are comparable side by side. The variant carries no baseline_reward, because 0.7321 was measured on deepseek with no reasoning_effort at all and a delta against it would be a model comparison. It reports an absolute held-out score instead. Reasoning effort stays at the seed's medium. At xhigh the seed lost 16 of 150 held-out cases to context-window overflow; at medium it lost none. The seed raised RuntimeError when the model returned neither text nor a tool call, which killed 17 of 150 cases on this target and put the run above the 0.1 error_rate_threshold that aborts an evaluation. gaia took the same fix in 4e90dace and tau3 never got it; it stayed invisible while the target was deepseek, which does not emit reason-only turns. Producer allow-list gains the bare and aux slots, after 8 of 157 producer calls in a gaia opencode run came back 403 model_denied. rescore_candidate.py gains --config so a retargeted variant can be scored by the same path as the pinned baselines. Without it the script only ever reads build.yaml, and swe-atlas-qna's new variant cannot be pinned at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The producer allow-list grew from one slot to three (swe-atlas-qna, and tau3's gpt-5.4-mini variant), and the launcher could only fill the first. An anthropic-routed optimizer therefore had no way to declare its bare and aux model names and took a 403 model_denied on each. Trailing KEY=VALUE args now become --param pairs, validated at launch so a typo fails there rather than at the gateway. Note the slots must resolve to distinct strings: allowed_models is uniqueness-checked, so passing the wire form for both the prefixed and bare slot fails config validation before any container starts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
codex strips the provider prefix, so --model azure_ai/gpt-5.6-sol reaches the gateway as bare gpt-5.6-sol and is forwarded unchanged; the proxy then load-balances it across two Azure deployments and every turn after the first fails with invalid_encrypted_content, because the reasoning payload cannot be decrypted by the deployment that did not create it. That killed all four gpt-5.6 cells earlier today at 2-4 minutes. Uses the gateway model_aliases added in PR #81 to pin both bare names to their Azure deployment, which is the only place the prefix can be reattached since the harness will not carry it. Varun measured the split against the live proxy: bare failed 5 of 5 encrypted-content replays, azure_ai/-prefixed passed 5 of 5. My own 3-chain probe passed both ways and was too small to see it. Also why this cannot wait for the shared config: the OpenAI contract expired 2026-07-31, so every GPT model now has to route through Azure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hed cells Measured on the config the cells actually ran: 3 rounds x 3 attempts of the unmodified seed through rescore_candidate.py, n=1335, rounds 0.568/0.553/0.564, sd 0.0063. The old 0.7321 is not a valid comparator and the comment now says so in the imperative -- it was deepseek-v4-flash with no reasoning_effort, so a delta against it measures the model swap, not the optimizer. The 14 cells that finished before the pin existed wrote an empty baseline_rewards, which leaves tau3 the one benchmark whose gain cannot be derived from its own artifacts. backfill_baseline.py fills the field and pushes, so the analysis script reads it the same way it reads every other benchmark rather than carrying a tau3 special case. Provenance goes in a sibling file because VerificationResult forbids extra keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+180
to
+183
| # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). | ||
| # Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default | ||
| # 600000=10min), well under one inner eval, which pushed the officeqa optimizer | ||
| # into --detach + background-poll + end-turn -- and a headless --print run is |
There was a problem hiding this comment.
When an optimizer-generated candidate runs under this configuration, task_services_use_upstream places the raw upstream credential in its OPENAI_API_KEY while harness_user: null disables harness isolation, allowing the candidate to bypass the metered, model-allow-listed gateway and its spend controls. The Azure variant introduces the same combination. How this was verified: The configured credential was traced through the Harbor environment construction into the candidate process.
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/tau3/baseline/build.gpt54mini.yaml
Line: 180-183
Comment:
**Upstream credential reaches candidates**
When an optimizer-generated candidate runs under this configuration, `task_services_use_upstream` places the raw upstream credential in its `OPENAI_API_KEY` while `harness_user: null` disables harness isolation, allowing the candidate to bypass the metered, model-allow-listed gateway and its spend controls. The Azure variant introduces the same combination. **How this was verified:** The configured credential was traced through the Harbor environment construction into the candidate process.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Collaborator
Author
There was a problem hiding this comment.
Pre-existing pattern, not introduced here: tau3's own build.yaml (already on main) has the identical task_services_use_upstream: true / harness_user: null, and so do swe-atlas-qna and browsecomp-plus. It's needed because tau3 runs its own in-container user-simulator LLM service, which needs the real credential to call out. Documented as a known, accepted tradeoff in skills/run-benchmark/SKILL.md ("Both are known, deferred, and recorded... Treat the boundary as 'an honest optimizer cannot reach the key by accident,' not 'an adversarial one cannot reach it at all'"). This PR's two new configs (build.gpt54mini.yaml, build.azure.yaml) just carry the existing pattern forward for the new target model -- not a new risk surface.
There was a problem hiding this comment.
Acknowledged — you're right that this pattern predates the PR. The new configs carry forward an existing, documented tradeoff rather than introducing a new risk surface, and the SKILL.md callout makes the boundary explicit. I'll withdraw the flag.
Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.
Comment on lines
+124
to
+127
| log = [l.split("\t", 2) for l in | ||
| git(repo, "log", "--all", "--format=%H\t%at\t%s").splitlines() if l.strip()] | ||
| log.reverse() # oldest first: the seed is index 0 | ||
| for position, parts in enumerate(log): |
There was a problem hiding this comment.
When candidate versions contain multiple commits or independent histories, git log --all treats every reachable commit as a candidate and derives ordering from Git history rather than candidate creation metadata. This inflates candidate counts, corrupts shipped positions, and can label the wrong commit as the seed.
Prompt To Fix With AI
This is a comment left during a code review.
Path: harness-engineering-bench/skills/analyze-optimizer-commits/scripts/extract_candidates.py
Line: 124-127
Comment:
**Git history misidentifies candidates**
When candidate versions contain multiple commits or independent histories, `git log --all` treats every reachable commit as a candidate and derives ordering from Git history rather than candidate creation metadata. This inflates candidate counts, corrupts shipped positions, and can label the wrong commit as the seed.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.…y did Squashed from four commits written while building and then running it once on tau3: the initial version, a fix for a fabricated citation the first real run caught in itself, a cut of two stages (blind lenses, completeness critic) that turned out to reproduce what a 20-line script already gave for free, and a restructure of the output into attributed tables instead of prose. Benchmark-agnostic by construction, not just in name: both scripts take --benchmark as an argument and walk runs/<benchmark>/ generically, with no hardcoded target-repo path or model name anywhere in either. The tau3 mentions in SKILL.md and lenses.md are calibration examples for the kind of finding expected, explicitly labeled as such -- run it on officeqa or swe-atlas and it reads their cells the same way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
yash-scaleai
force-pushed
the
tau3-baseline-and-skill
branch
from
August 2, 2026 20:46
f2001ee to
22b87f6
Compare
…ount assumption Two issues from an automated review, both real. Legacy extraction (Python <3.12, where filter="data" is a TypeError) passed tarfile members straight to extractall with no path or symlink checks. Added the same containment check filter="data" does, so a traversal member or a symlink pointing outside dest is rejected on every Python version, not just 3.12+. The candidate-count concern was that `git log --all` walks every commit reachable from every ref, which is only a correct candidate enumeration if the repo is one linear lineage. Tried the principled-looking alternative -- refs/vero/candidates/*, one ref per candidate -- and it silently undercounted: two real, git-committed candidates in claude-opus-5-opencode-r1 have no ref at all, because they were superseded before ever being submitted for scoring (empty `scores` dict, which the analysis already reports as its own finding). Switching would have quietly corrupted a report that's already been verified and shared. Kept log --all, and instead made the assumption it depends on explicit: checked all 16 tau3 repos by hand (zero merges, one root commit each), and the script now checks that itself and refuses to guess on a repo where it doesn't hold, rather than silently mis-attributing positions on some other benchmark's history shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gpt-5.4-minibaseline at 0.5618 (rescore_candidate.py --seed, 3 rounds x 3 attempts, n=1335, sd 0.0063) and backfills it into every finalized cell'sfinalization.json, so the gain can be derived from S3 artifacts directly rather than added separately downstream.skills/analyze-optimizer-commits/: reads what an optimizer LLM actually did to a target harness across a benchmark's grid cells and produces a citation-bearing observations file. Benchmark-agnostic by construction — both scripts take--benchmarkas an argument and walkruns/<benchmark>/generically, no hardcoded target-repo path or model name anywhere. Run on tau3's 16 cells as its first real test;tau3/OPTIMIZER-BEHAVIOR.mdis that run's output.Known open issue, not fixed by this PR: two of the 16 cells shipped a harness byte-identical to the seed (confirmed by git tree hash) and scored ~0.12 below the pinned 0.5618 baseline for that same code. Documented in
OPTIMIZER-BEHAVIOR.md's "measurement-path gap" section. Resolving it needs a rescore of the seed through the finalization path, which needs a funded LiteLLM key.Test plan
extract_candidates.py --benchmark tau3reproduces the 83-candidate, 16-cell corpus from S3 artifactsOPTIMIZER-BEHAVIOR.mdpassed through an adversarial verification pass (12 confirmed as stated, 3 corrected)🤖 Generated with Claude Code
Greptile Summary
This PR pins and backfills tau3’s GPT-5.4-mini baseline while adding reusable optimizer-commit analysis tooling.
Confidence Score: 4/5
The PR is not yet safe to merge because candidate code can still access the upstream credential and branched Git histories can still corrupt optimizer-commit analysis.
The tau3 configurations continue to combine upstream task-service credentials with disabled harness isolation, leaving the raw key in the candidate environment, while the extractor’s merge and root checks do not establish the single linear lineage required by its
git log --allordering.Files Needing Attention: harness-engineering-bench/tau3/baseline/build.gpt54mini.yaml, harness-engineering-bench/tau3/baseline/build.azure.yaml, harness-engineering-bench/skills/analyze-optimizer-commits/scripts/extract_candidates.py
Important Files Changed
Reviews (3): Last reviewed commit: "extract_candidates: fix extraction conta..." | Re-trigger Greptile