Skip to content

Each write spawns hundreds of git processes (~615 on bootstrap, ~350 per write), which trips fixed timeouts where git spawns are slow #118

Open

Description

@magnificentlycursed

A single crosslink write spawns hundreds of git processes, and bootstrapping a fresh one-commit repository with crosslink daemon ensure --wait-ready spawns about 615. Where a git spawn is slow, this makes writes and readiness slow enough to trip fixed timeouts.

Measured on macOS (2026-09-29, clean worktree of develop at the time, PATH shim counting git invocations):

Operation git spawns
Bootstrap a fresh repository to ready ~615
One write on a four-agent hub ~350

The bootstrap breaks down as 226 cat-file -t, 155 rev-parse --verify, 25 remote, 22 for-each-ref, 20 cat-file blob, 14 ls-remote ., 13 fetch --no-tags, and others. A stack sample during the stall sits in reconcile::publication::validate_source_objects → validate_object_type, which runs one git cat-file -t per object and is re-run on every pin_source call.

Why it matters: on macOS, /usr/bin/git is Apple's xcrun shim at about 35 ms per spawn, against about 9 ms for the Command Line Tools git called directly. At that rate:

Suggested fix: validate objects in one git cat-file --batch-check per pass instead of one cat-file -t per object, and memoize OIDs already validated across pin_source calls. The same per-object pattern likely explains the multi-minute first reconciliation on large hubs that motivated the long ensure --wait-ready deadline.

This is the remaining defect from #102.

Authored by Claude Code on behalf of @magnificentlycursed.

Activity

  1. magnificentlycursed commented on Oct 9, 2026

    @magnificentlycursed
    MemberAuthor

    Three more measurements from the same downstream project (vsdd-cli), 2026-10-08, macOS, binary built from develop at 388bed8 (the PR #115 merge), all against the 30 s lock-confirmation bound in shared_writer/locks.rs:

    write observed
    session work <issue> rebinding the orchestrating session to a new issue 58 s
    kickoff plan ... --issue <child> activating the child issue in the worktree (attempt 1) 72 s
    the same command retried after the first attempt's lock went stale (attempt 2) 86 s

    Two observations beyond the numbers:

    1. The guard measures the commit-and-push of the LockClaimed event after it has returned, so when it fires the event has already been pushed: the claim is reported as failed and the kickoff bails, but the lock lands on the hub under the worktree agent's identity with no process left to heartbeat it. Each timed-out activation therefore also produces a stranded lock (the shape in kickoff: a failed launch keeps the issue lock and worktree, and kickoff cleanup leaves the lock and branch behind #110).
    2. During the same window the daemon reported ready_current while the orchestrating session's readiness record went stale and then expired, and every lock on the host (including the orchestrating session's own, claimed within the hour) read STALE in locks list. Heartbeats and reconciliation appear to ride the same slow write path.

    No kickoff agent ran; nothing was spent. The worktree activation is the first write a kickoff makes, so at these latencies the container vehicle cannot start on this host at all.

    Authored by Claude Code on behalf of @magnificentlycursed.

  2. magnificentlycursed commented on Oct 9, 2026

    @magnificentlycursed
    MemberAuthor

    One bootstrap data point from the same host (macOS, develop at 388bed8), 2026-10-09: after the laptop woke from a night asleep and the main daemon was stopped to clear a cached readiness verdict, crosslink daemon ensure --wait-ready timed out at its 120 s wait ("unavailable: timed out waiting for daemon readiness after 120 seconds"), and daemon status reported ready_current a short while later. So on this host a readiness bootstrap now takes longer than two minutes, against the 16 to 27 s measured on 2026-09-29. The hub has grown since (about 900 issues and 80 knowledge pages), which fits the per-object validation cost the issue describes.

    Authored by Claude Code on behalf of @magnificentlycursed.

    Edited: added that the measurement followed a laptop wake, in case the first fetch after sleep inflates it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions