Skip to content

fix(daemon): survive a hub that moves under the readiness write instead of exiting - #115

Merged
magnificentlycursed merged 5 commits into
Corvidae-Coding-Projects:developfrom
magnificentlycursed:fix/daemon-housekeeping-reconciles
Oct 7, 2026
Merged

magnificentlycursed merged 5 commits into
Corvidae-Coding-Projects:developfrom
magnificentlycursed:fix/daemon-housekeeping-reconciles

Conversation

@magnificentlycursed

Copy link
Copy Markdown
Member

Closes #113.

A daemon whose housekeeping tick could not publish a ready record, because another checkout published to the hub between its validation and its write, exited with "refusing to publish ready state for a stale projection" and left the repository blocked until someone noticed. The same race, narrower, existed on the reconciliation path's own ready-record write.

Changes:

  • housekeeping_tick reports failures that come from the hub or the projection moving (an expired record, a hydration that cannot complete, a refused record write) as a reason to reconcile again instead of as errors. Identity and lease failures, and a projection that cannot be opened, stay fatal.
  • A hub that moves under the record write is first absorbed in place: hydrate again and refresh again, up to three times, under the same permit. A single publication from another checkout therefore costs no reconciling window at all; only a hub that keeps moving under every write hands the tick back to reconciliation.
  • The stale-projection refusal is a typed error (readiness::StaleProjection), also returned by write_ready_activation when the projection or the generation ref moved during activation. The reconciliation loop treats it as "derive again" with backoff rather than propagating to run_daemon.
  • The reconciliation path's own retry on a moving hub shares that budget: after five consecutive moves it parks with the cause rather than staying in reconciling indefinitely.
  • Reconciliations requested by housekeeping are counted. A completed tick in between resets the count; five in a row without one park the daemon as blocked_corrupt with the last cause, as the reconciliation path does for corruption, instead of flapping between ready and reconciling every thirty seconds. The reason is also written to the reconciling record so daemon status shows why the daemon left the ready state.

Tests: the single absorbed race; the hub moving under every write; the race on the reconciliation path; a hydration that cannot complete (Unix, via directory permissions); the budget; a quiet tick keeping the record fresh; the expired-record path from #107.

🤖 Generated with Claude Code

… current, instead of exiting

A housekeeping tick hydrates the projection, validates the record, and then
publishes a fresh ready record. When another checkout on the same .git
publishes between that validation and the write (a kickoff worktree agent
writing a comment and ending its session), `write_record` refuses with
"refusing to publish ready state for a stale projection" and the tick
propagated that out of the daemon. The daemon exited and every host command
was blocked until `daemon ensure` (Corvidae-Coding-Projects#113). A hydration failure took the same
exit.

The tick is now `housekeeping_tick`, which reports these conditions, along
with the expired-record case from Corvidae-Coding-Projects#107, as a reason to reconcile again.
`run_daemon` re-enters `reconcile_until_ready` under the same identity, which
already re-derives readiness from the hub and records waiting or blocked
states with their cause. Identity and lease failures remain fatal.

Tests: a before-write hook that publishes as another agent reproduces the
race; the tick returns Reconcile and reconciliation then reaches
ready_current under the same epoch. A quiet tick still refreshes the record.

Closes Corvidae-Coding-Projects#113.
…e path, and park persistent faults with their cause

The stale-projection refusal is now a typed error, and both places that
publish a ready record treat it as "the hub moved, derive again" instead
of ending the daemon:

- Housekeeping hydrates and refreshes again under the same permit, up to
  three times, so a single publication from another checkout is absorbed
  within the tick with no reconciling window. Only a hub that keeps
  moving under every write hands the tick back to reconciliation.
- The reconciliation path's own ready-record write, which activation and
  the record validation leave open to the same race, loops again with
  backoff rather than propagating to run_daemon and exiting.

Reconciliations requested by housekeeping are counted. A completed tick
in between resets the count; five in a row without one means the fault
is persistent, and the daemon records blocked_corrupt with the last cause
and holds there, as the reconciliation path does, instead of flapping
between ready and reconciling every thirty seconds with the reason only
on stdout. The reason is also written to the reconciling record so
`daemon status` shows why the daemon left the ready state.

A projection that cannot be opened stays fatal: it is not a hub condition
that reconciliation can repair.

Tests cover the single absorbed race, the hub moving under every write,
the race on the reconciliation path, a hydration that cannot complete
(Unix, via directory permissions), and the budget.
…the park consistently

Follow-ups from the verify round of the review:

- The reconciliation path's retry on a hub that moves under its ready
  record write had no bound, so sustained publication could hold the
  daemon in reconciling indefinitely. It now shares the housekeeping
  budget: after five consecutive moves it parks as blocked_corrupt with
  the last cause, as the housekeeping path does. A test drives the hub
  under every write and waits for the parked record.
- Parking writes the reconciling record before the blocked one, as the
  reconciliation path does, so the record history reads the same way.
- The hydration-failure test skips under a root runner, where directory
  permissions cannot inject the failure.
Adding the parking test moved the cfg(unix) attribute onto it, which left
the hydration-failure test's permission code compiling on Windows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant