Repository navigation
Container kickoff cannot complete a task on the readiness model: five defects with root causes, fixes on a fork PR, and the residual design findings #102
Description
Umbrella report from eight pilot runs of kickoff run --container on a downstream repository (vsdd-cli), 2026-09-24/25, against develop 29d0185. Five defects made the path fail in sequence; each is fixed on magnificentlycursed#6 (five commits, upstream PR to follow from that fork's develop). Four further findings are design-level and left open here. Related: #100 (CI red), #101 (private image).
Defects fixed (fork PR magnificentlycursed#6)
1. Worktree bootstrap has no readiness (repository readiness is missing). 0a2ac70 made readiness a per-checkout record and updated kickoff's driver side (run.rs: activate_repository) but not launch.rs::init_worktree_agent, which runs init, sync, session start in the fresh worktree; sync/session start fail closed. No test covers that function (its callers are end-to-end). Fix: daemon::ensure_and_wait after init. Commit 592aa49.
2. Lock confirmation bound predates verified authority (timed out after 34s (threshold 30s)). LOCK_CONFIRM_TIMEOUT_SECS = 30 dates from 4728766 (v2 cheap re-read); under v3, emit_compact_push_inner re-hydrates with per-event signature verification via a spawned process per event; a cold worktree on ~3,600 events measured 34 s (14 s warm). Fix: 120 s with the rationale on the constant. Commit 692cd1d.
3. Container readiness blocked_corrupt (worktree add --orphan -b crosslink/hub-v3-host … already used by worktree). The container mounts the host .git but not the host's hub-cache worktree; readiness observes the cache at the host path, finds nothing, tries to create it, and the shared .git refuses because the branch is checked out on the host. container start always mounted the cache; kickoff never did. Fix: mount .hub-cache and .knowledge-cache at their host paths. Commit a1fe1bb.
4. Hub publication from the container (could not read Username for 'https://github.com'). No credential helper inside. Fix: when the host env has GH_TOKEN/GITHUB_TOKEN, pass it and bind it via GIT_CONFIG_COUNT/KEY_0/VALUE_0 (x-access-token). Commit cd32023.
5. Writer sequence seeding vs the causal frontier (frontier prefix hash for agent '<id>' sequence 1169 is forged or rewritten). The reconciliation seeds each agent's frontier (state.json → frontier.agents.<id>.sequence) from the pinned pre-v3 agent tips; on a long-lived agent that is the old commit-per-event numbering (1169) while the ref tip's events.log holds 177 events. read_max_event_seq_from_ref seeded from the tip alone, so every new event (178) fell inside the frozen prefix and every reduce refused the agent's chain; the hub was unusable until the ref was force-reset (three times). Fix: ceiling = max(tip, checkpoint copy, recorded frontier); verified live (next event 1170, integrity 5/5). Commit f6d4a53. The migration-side question — whether the frontier should have been seeded from the tip's numbering instead — is yours to decide; the writer-side fix is safe either way.
Residual findings (open)
- Daemons exit shortly after publishing ready. Observed 7× across two repositories: the daemon's own housekeeping validation fails with
readiness record is stale(MAX_RECORD_AGE_SECONDS = 90; refresh deferred behind active mutation permits) and the process exits; every command then fails untildaemon ensure. - Reconciliation time and its bound. First reconciliation of a ~3,600-event hub took ~5 min (per-event signature verification); on later runs a small hub (~500 events) took ~20 min idle in
reconcile::check_repository/sync::core.ENSURE_DEADLINE_SECS = 120made--wait-readyreport a timeout while the daemon kept working; the fork split it (ensure_and_wait, 30 min). A transientgit ls-remoteSSL timeout at bootstrap was recorded asblocked_corrupt(terminal) rather than a remote wait. - Cold write path inside the container.
session endran > 120 s, so the provider harness terminated the run before.kickoff-statuswas written; the image's bundled binary also times out its owndaemon ensureat 120 s. - Durable fix suggestion. Cache verified event signatures per hub cache (an event's signature never changes); that removes the per-event subprocess from lock confirmation, first reconciliation, and in-container writes at once. Also worth documenting: readiness per checkout means one daemon per kickoff worktree and per CI run.
Authored by Claude Code on behalf of @magnificentlycursed.
Activity
magnificentlycursed commentedon Sep 29, 2026
Correction to "Residual findings", with two new findings
Further testing changed the first two residual findings. The defects fixed on PR #103 are not affected.
Daemons exit shortly after publishing ready. The exits were real, but the mechanism I gave is wrong, and the count is too high.
Error: readiness record is stale(02:40, 10:55 and 19:05 UTC) andError: hydrating current authority during daemon housekeeping(02:16).pkill -9 -f "crosslink daemon", which matches every checkout's daemon. Discount any reports from those days.Reconciliation time. The "~20 min idle in
reconcile::check_repository/sync::core" is more likely a hung fetch than slow work. On 2026-09-26 a daemon restart sat for 18+ minutes ingit fetch --no-tags origin +refs/heads/crosslink/reconciliation/*. While it did, it held the reconciliation transition, and every other start failed withrepository reconciliation transition is already active. I suspect the earlier ~20-minute case was the same. That shows two gaps:The 30-minute wait bound in a1fe1bb is wider than it needs to be. It applies to
daemon ensure --wait-readyfor every caller, not only to kickoff's worktree bootstrap, which is the one unattended caller that needed it. A narrower bound would make CI fail in 2 minutes instead of 30 on a hung reconcile, and it would keep this PR from changing the CLI's behaviour for other callers. I'll narrow it to kickoff's bootstrap in a follow-up commit on the fork's develop, which updates #103.Edited 2026-09-29: an earlier version said the shipped session-start hook is killed before crosslink can report why readiness isn't established. That was wrong. The hook catches its own 130 s timeout and reports it, and with either bound the message carries no readiness reason. The timeout was also never observed; I inferred it from the code.
Authored by Claude Code on behalf of @magnificentlycursed.
magnificentlycursed commentedon Sep 29, 2026
Where the residual findings stand
The five defects and their fixes in #103 are unchanged by any of this. On the residual findings:
target/releasebinary I was rebuilding. I can't tell them apart from my own interference, so I'm not asking anyone to act on this finding.refresh_generation_ref(reconcile/publication.rs, thegit fetch --no-tags <remote> +refs/heads/crosslink/reconciliation/current:…call) and the other reconcile fetches beside it runCommand::new("git")…output(), which blocks until git exits. Nothing setshttp.lowSpeedLimitor any other stall bound. A stalled HTTPS connection can therefore hold reconciliation indefinitely, which fits the 18-minute hang. That hang's refspec was thereconciliation/*form, and I haven't traced whether the transition permit is held across the call. The ask stands: bound these fetches and classify a timeout as a remote wait.--wait-ready's 120 s giving up while the daemon keeps working; the in-containersession endtaking over 120 s; and the suggestion to cache verified signatures.Authored by Claude Code on behalf of @magnificentlycursed.
magnificentlycursed commentedon Oct 2, 2026
Correction to "Residual findings". The five defects are unaffected.
gitprocesses per write.Authored by Claude Code on behalf of @magnificentlycursed.
magnificentlycursed commentedon Oct 5, 2026
Status after #107 and #108 merged into develop:
gitprocesses on a four-agent hub, which also makes the Windows readiness shards flake at their 30 s cap. That needs its own issue.Authored by Claude Code on behalf of @magnificentlycursed.
magnificentlycursed commentedon Oct 8, 2026
All five defects are now fixed or tracked on their own: defects 1, 3 and 5 by #107/#108, defect 4 by #114 (closing #109), and the transport and stale-projection exits found along the way by #112 and #115. Defect 2, the cost of a write, is now #118. Closing this umbrella in favour of #118.
Authored by Claude Code on behalf of @magnificentlycursed.