Repository navigation
fix(queue): do not refuse a cancel once the worker has settled the job - #75
Merged
Merged
Conversation
A cancel repeated while a worker finishes could read the row as running and take its process listing after the worker exited. A descendant that ignores SIGINT kept the group alive, so no recorded member matched and the cancel failed with "Process group ownership cannot be verified" although the job was settling by itself. VerifyGroupIdentity now returns a named error that states the group and the members it compared. The daemon's signal, its tick for an adopted running row and the handover readers re-read the row on that error and accept an attempt that has published its outcome. A running row with no recorded member left still refuses, and nothing is signalled on unverified ownership. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
"real queue restarts after exit 0" waited only for the scheduler and phone helpers to restart, then asserted the daemon had too. On a slow Intel runner the daemon can lag behind, so the check failed with the supervisor working. Wait on everything the next checks assert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
… publishes Review of the settled-attempt check found a second path to the same refusal. executeWithAttempt stopped handling SIGINT on return, before the outcome was published, so a resend from Cancel could kill the worker with nothing written. The row then stayed running, a cancel in that interval refused, and the job reported the wrong status. The worker now holds a registration until it exits. Tests now wait before checking that a group was not signalled, since the old checks could not see a signal, and cover each part of settledAttempt's key. The handoff no longer states one cause as fact, because the CI log cannot separate the two paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
The separate SIGINT registration in ExecuteAttempt replaced the signal's default action before anything read the channel, so an acknowledged cancel arriving before the command started was discarded and the command ran. Drop it and leave executeWithAttempt's own handling registered until the worker exits, which also keeps a resend from killing the worker before it publishes. Add the done phase to the settled attempt test, and stop the handoff naming one cause or claiming evidence that does not separate the two paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
Dunky13
force-pushed
the
fix/cancel-ownership-race
branch
from
November 3, 2026 19:37
2a5f930 to
755ebaf
Compare
A cancel landing between executeWithAttempt's last check and the command, with no client left to resend, is still lost, as on main. Correct the reported 125, which only a raw SIGINT produces, and a reference to a rebased commit. Reset SIGINT after the in-process test that leaves it caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
TestCorruptedRowsCancelThroughClient/running_reading_Queued started its ticker before Cancel, so a tick could settle the row with 125 first. It now holds ticks until the daemon has taken the cancel: 2 of 60 race runs failed before, 0 of 60 after. TestInstallerNewerCopyBesideRunningRecordedAppStops read app-running as the first issue, but a helper holding scheduling.lock can add installer-busy ahead of it. It now finds the issue by ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
running_reading_Done had the same race as running_reading_Queued: a tick before Cancel's first read settled the row with 125, so the cancelled file recorded 125 instead of 7. It now waits for the first control reply. It never sends a cancel, so it cannot wait for that one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
Dunky13
added a commit
that referenced
this pull request
Nov 3, 2026
Since #75 a signal for a row the same worker has settled returns nil, so the test now expects that and covers errQueueChanged with another attempt taking the row. Drop the handoff's flaky test note, which #75 fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Went <marc@went.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
agent-heavy cancelcould refuse a job that was ending on its own:Process group ownership cannot be verified; shared slot retained without signalling. Seen as an intermittent failure ofTestRealDaemonCancellationRemovesDescendants(PR #74, run 41672401460,macOS tests (macos-15-intel, core-1)).docs/handoffs/process-census.mdalready recorded it on main.Cause
The daemon's
signalchecks a worker record read earlier, which names only the worker. A descendant that ignores SIGINT (sleep 60 &in a non-interactive shell) keeps the group alive after the worker is gone, so the group has a member but none the record names, and the check refuses. The old error text and missing job log cannot separate two ways to reach that state:Cancelresends every 200ms. A repeat can read the row as running, then list processes after the worker exited.executeWithAttemptstopped handling SIGINT on return, so a resend before the outcome was published killed the worker with nothing written. A review forced this with injected delays. With a raw SIGINT and no cancel recorded, the job also reported 125 instead of the command's status; a realCancelmarks the row first, so its wait reports 130.The refusal came from the daemon's
cancelreply:WaitCleanupnever returns the daemon's cleanup error and waits 12s, while the failure came 2.9s into the test. Birth instability is ruled out on arm64 (400 full-table reads undertaskpolicy -c backgroundnever changed a birth); #62 normalizes both sides.User impact:
agent-heavy cancel Non a job that is ending, with a background child still running, can print the refusal and exit nonzero; repeating succeeds. After a daemon restart, the same state madetickreturn an error for that tick.Fix
VerifyGroupIdentityreturns a named error,errGroupOwnershipUnverified, whose text states the group, the recorded members and the current members it compared.queueStore.settledAttemptre-reads the row. If the same attempt and worker has published its outcome, there is no worker to signal and the settled job's cleanup owns what is left. Used byqueueDaemon.signal,tickfor an adopted running row, and the three handover readers inqueue_handover.gothat already re-read on a vanished group.executeWithAttemptno longer stops its SIGINT handling on return, closing path 2. A resend after the command ended is discarded. One before the last check ahead of the command still cancels; one between that check and the command is lost unless a client resends, as onmain(window at most 1.7-6.1ms, recorded in the handoff).Cleanupandinstall_runtime_generations.goare untouched.install_managed.go,listenQueueSocketandsafeQueueSocketare not touched.Evidence
TestSignalFromAStaleRowAfterTheWorkerExited. With only thequeue_daemon.gochange reverted it fails withcompared group N, recorded members [N@...], current members [N+1@...]; with the fix it passes and the group is never signalled.Queue|Handover|Cancel|Ownership|Group|Lease|Signal|Census|Corrupt|RealDaemon: pass.go build ./... && go vet ./...: pass. Race-built real test plus the new test, 120 runs undertaskpolicy -c background, before path 2 was closed: pass.TestRealDaemonWorkerSurvivesInterruptsWhilePublishingfails withexit=125 want=7with the handling stopped on return.TestSettledAttemptKeysOnTheSameAttemptfails with thedonephase removed. Focused race suite: pass, 140s. An injected 800ms delay before the command showed a cancel still stops it (exit 130), in review of the fix; no test pins that window, because it needs a production hook.tickpath (it reads its own rows, so a stale snapshot cannot be forced without a hook).managedHeldByAppon Intel: no link found. The CI failure "An unowned queue process holds its lifetime lock" comes frommanagedDrainininstall_managed.go, which inspects locks and does not use process group ownership.Handoff:
docs/handoffs/cancel-ownership-race.md.Also on this branch:
native/tests/SupervisorTests.swiftnow waits for the daemon restart thatreal queue restarts after exit 0asserts, which fixes the Native failure on this PR's earlier run.Two more flaky tests from this PR's Intel
core-2run are fixed here.TestCorruptedRowsCancelThroughClient/running_reading_Queuedholds its ticks until the daemon has taken the cancel (2 of 60 race runs failed before, 0 of 60 after, on arm64).TestInstallerNewerCopyBesideRunningRecordedAppStopsfindsapp-runningby ID, because a helper holdingscheduling.lockcan putinstaller-busyahead of it (0 of 5 after; the failure was not reproduced locally).🤖 Generated with Claude Code