Repository navigation
test: pin that a different-episode trace cannot complete a task - #571
Open
hacktivistxt wants to merge 2 commits into
Open
hacktivistxt wants to merge 2 commits into
hacktivistxt wants to merge 2 commits into
Conversation
When the Supervisor cited a basis observation whose 32-char hex ID it had mistyped, the deterministic validator rejected it with only the bad ID, so the blind retry mis-copied again and the run died on supervisor_contract after 2 attempts (observed: a near-miss ID differing in 6 of 44 characters). compile_plan now appends the closest valid observation IDs (difflib) plus the full available set to both the task-basis and finish-basis errors. These are fed back to the retry as validation_feedback, so it can copy the correct ID instead of being told only that its citation is unknown. Adds a focused test asserting the error names the near-miss's real observation ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completion reuse of a valid same-task observation succeeds when the lease and grounding trace share an episode. This adds the mirror case: the same exact-quote DONE delta and reusable ObservationRecord, but with the lease bound to a different trace_episode_id than the trace carries. The validator refuses it with "evidence trace belongs to a different episode" before any grounding is credited, so evidence captured in another episode cannot float a completion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Author
|
CI should pass on this branch. Locally against |
Author
|
Confirmed green: ran the same |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the missing execution-validator test for the episode-identity guard.
Completion by reusing a valid same-task observation already succeeds when the
lease and the grounding trace share an episode (see
test_executor_can_complete_by_exactly_reusing_same_task_evidence). This addsthe mirror negative case: the same exact-quote
DONEdelta and reusableObservationRecord, but with the lease bound to a differenttrace_episode_idthan the trace carries.
compile_executionrefuses it withevidence trace belongs to a different episodebefore any grounding iscredited — so evidence captured in another episode cannot float a completion.
Covers the
execution.py:301-305guard in theDONE-would-otherwise-succeedpath, which
test_evidence_trace_must_match_the_persisted_episode_identity(PROGRESS, empty events) did not exercise.
Verification:
uv run python -m pytest -q→ 120 passed, 1 skippeduv run ruff check src tests/ruff format --checkcleanuv run mypy srcclean🤖 Generated with Claude Code