feat: checkpoint restore — revert code + agent memory to any prior turn, across local ↔ cloud

[Basit-Balogun10]

feat: checkpoint restore — revert code + agent memory to any prior turn, across local ↔ cloud

Closes #724 (consolidates closed dupes #317, #2328).
Depends on the backend PR (PostHog/posthog#70430)

A little history for context: this was originally split into two stacked PRs — the old PR A (restore within a session) and PR B (restore across local↔cloud handoffs) — deliberately kept separate to keep each diff small and easy to review. Partway through, a sync with main brought in the new modular package architecture (the packages/* split + DI/host-router refactor). That turned what should have been a routine rebase into a re-implementation on the new arch, and it reshuffled the very commit stack the A/B split depended on — enough conflicts that keeping them apart stopped making sense. So I folded both halves into this single, integrated PR.

---

Summary

Adds checkpoint restoration to a PostHog Code session: after any AI response turn the user can
roll the workspace and the agent's memory back to that point — a per-turn "restore" affordance in
the conversation, similar to Claude Code's rewind. Works on both adapters (Claude + Codex) and,
crucially, across local↔cloud handoffs.

Two guarantees have to hold together for that, and this PR owns both:

1. Checkpoints stay synced as a task moves between local and cloud — a checkpoint captured locally
   is still restorable after the task moves to the cloud, and a cloud checkpoint is still restorable
   after it comes back local.
2. You can rewind to any turn, regardless of which environment it happened in — pick any
   checkpoint (local- or cloud-origin) from the desktop and the code and the agent's memory both
   land on that turn.

Scope boundary: the restore action is always initiated from a local session (the desktop
performs the revert + truncation). You can restore to both local and cloud checkpoints and memory
stays correct across the handoff. Restoring while the task is still running in the cloud
(sandbox-side / cloud-origin restore) is a deferred follow-up — see roadmap.

Depends on the backend PR (PostHog/posthog#70430): restore truncates the run's durable S3
log at the checkpoint boundary via the truncate_log endpoint. Restore degrades gracefully
(client-side boundary trim) if the endpoint is absent, but the two should land together.

---

1. What pre-existed vs. what this PR adds

Already on main (upstream primitives):

• CaptureCheckpointSaga + RevertCheckpointSaga — packages/git/src/sagas/checkpoint.ts. Low-level
  git: snapshot a working tree to a ref; reset a working tree back to one.
• captureCheckpointState — packages/agent/src/server/agent-server.ts. Existed, but wired only for
  handoff state-packing (capture workspace → S3 during local→cloud handoff), not as a per-turn
  user-restorable stream.
• Handoff transport — packages/workspace-server/src/services/handoff/service.ts, handoff sagas.

New in this PR (absent from main):

• Per-turn cloud checkpoints — capture on every cloud TURN_COMPLETE, each uploaded to S3 with
  turn metadata.
• Cross-environment sync — syncCloudCheckpoints → materializeCheckpointRef +
  registerCloudCheckpoint: rebuild each cloud checkpoint's git ref locally at handoff so
  cloud-origin checkpoints become restorable on the desktop.
• The entire restore feature — packages/workspace-server/src/services/checkpoint/service.ts
  does not exist on main: runRestore, memory truncation (trimReseededCacheToCheckpoint,
  JSONL/rollout truncation, bounded resumeFromLog summary), plus UI (RestoreCheckpointDialog,
  useRestoreCheckpoint, restoreCheckpointReconnect).
• Adapter/model preservation across handoff; per-turn icon binding via turnCompletedAt;
  capture/restore concurrency; reload re-seeding; index.lock retry.

Value prop	Own it?	Nuance
Checkpoints syncable across environment handoffs	100% new	Handoff transport pre-existed; per-turn cloud capture + materialize/register into local git is new.
Restore to ANY checkpoint from local, regardless of origin	Ours, on one pre-existing primitive	RevertCheckpointSaga (raw revert) pre-existed. Picking a checkpoint in the UI, resolving cloud-origin refs locally, and truncating log + agent memory so the model doesn't "remember the future" — all ours.

One-liner: we don't reinvent git snapshot/revert — we turn it into a cross-environment,
memory-consistent, user-triggerable restore.

---

2. Background / the problem

A checkpoint must capture and later restore three things that can drift apart:

1. Working tree — files on disk (git).
2. Conversation log — the durable transcript (authoritative S3 copy + the local logs.ndjson
   cache mirror).
3. Agent memory — what the running agent believes happened (adapter-specific: Claude
   re-hydrates from the JSONL; Codex resumes from its rollout / a bounded summary).

Reverting files alone isn't enough — the agent would keep acting on turns the user just undid. This
PR keeps all three consistent, both for an in-session restore and across a handoff.

---

3. What this PR does

Capture

• Emit a git checkpoint after every completed turn by intercepting TURN_COMPLETE in the raw
  stream tap (not extNotification, which was lossy).
• Emit TURN_COMPLETE from the Claude adapter so Claude turns capture checkpoints too (parity
  with the Codex-first original).
• Persist checkpoints into logs.ndjson so the restore affordance survives an app restart.
• Cloud turns emit their own per-turn checkpoint (including the resume/init turn), each stamped
  with turnCompletedAt (the true turn boundary) so the desktop can bind its restore icon to the
  correct turn.

Restore

• Reverts the working tree to the checkpoint and truncates agent memory to that point.
• Truncates the durable log at the turn boundary (backend truncate_log) with a client-side
  prompt-boundary trim as the primary defense for Codex (whose events are all notifications).
• Re-adds surviving checkpoint markers the truncate dropped (async-late captures can land out of
  order) so every surviving turn stays restorable, live and after a reload.
• Replays the persisted model + adapter on restore-reconnect so the agent doesn't come back on the
  gateway default (the "model resets after restore" bug) or on the wrong runtime.
• Hardened for concurrency (restore lock, isReconnecting race guard), tail-checkpoint edge cases,
  and surfaced failures.

Two adapter-memory pieces that only surface across a handoff:

• Codex memory rebuild — turn-granularity truncation can't strip history baked into the
  post-handoff summary blob, so on a restore-reconnect Codex starts a fresh session seeded with a
  bounded context rebuilt from the truncated log (resumeFromLog + formatConversationForResume),
  and rollout replay on reload is suppressed so undone turns aren't re-persisted.
• Adapter-aware bypass-revert — the auto-revert-from-bypass effect hardcoded Claude's "default"
  mode, which a Codex session rejects → a toast loop after a cross-adapter handoff. Now resolved
  per-adapter (Claude→default, Codex→auto), unit-tested.

UX

• Per-turn restore icons on user messages, live while the agent is still responding (past
  turns stay restorable mid-stream); a restyled, destructive-action RestoreCheckpointDialog (Cancel
  locked while a restore is in flight; turn/adapter-aware copy) + an always-visible restore button.
• Restoring a past turn while the agent is mid-response stops the in-flight turn first.
• Restoring a cloud checkpoint while the task is still in the cloud is blocked with a guiding toast
  ("continue the task locally first") — matching the scope boundary above.
• Partial-failure surfacing: distinct toasts for "history trim failed" vs "reload to fully refresh the
  view," so a degraded restore is never silent.

---

4. End-to-end flow

Every node names the exact file » function. Renders on GitHub and at mermaid.live.

---
title: "A · Lifecycle — capture → handoff/sync → restore"
config:
  theme: base
  themeVariables:
    darkMode: true
    background: "#0f1430"
    primaryColor: "#232a52"
    primaryTextColor: "#eef1ff"
    primaryBorderColor: "#5661a8"
    lineColor: "#8b94dd"
    secondaryColor: "#2c2350"
    tertiaryColor: "#1b2340"
    noteBkgColor: "#2c3566"
    noteTextColor: "#eef1ff"
    noteBorderColor: "#5661a8"
    actorBkg: "#2a3160"
    actorTextColor: "#eef1ff"
    actorBorder: "#5661a8"
    actorLineColor: "#8b94dd"
    signalColor: "#cdd4ff"
    signalTextColor: "#eef1ff"
    labelBoxBkgColor: "#2a3160"
    labelBoxBorderColor: "#5661a8"
    labelTextColor: "#eef1ff"
    loopTextColor: "#eef1ff"
    sequenceNumberColor: "#0f1430"
---
sequenceDiagram
    autonumber

    participant UI as "UI · packages/ui"
    participant Core as "Core · sessionService.ts"
    participant WS as "host · workspace-server"
    participant Git as "git · packages/git"
    participant Cloud as "sandbox · agent-server.ts"
    participant S3 as "S3 run log"

    rect rgb(28,35,66)
        Note over Cloud,S3: CAPTURE — cloud turn

        Cloud->>Cloud: agent-server.ts » broadcastTurnComplete()
        Note right of Cloud: stamp turnCompletedAt (true boundary)

        Cloud->>Cloud: agent-server.ts » captureCheckpointState
        Note right of Cloud: inputs: turnIndex + turnCompletedAt
        Note right of Cloud: fire-and-forget
        Note right of Cloud: emits GIT_CHECKPOINT (promptId + turnCompletedAt)

        Cloud->>S3: handoff-checkpoint.ts » HandoffCheckpointTracker.captureForHandoff()
        Note right of S3: pack + upload (artifactPath)
        Note right of S3: marker persisted to run log
    end

    rect rgb(32,27,60)
        Note over WS,Git: CAPTURE — local turn

        WS->>Git: agent/agent.ts » captureLocalCheckpoint → CaptureCheckpointSaga
        Note right of Git: git-saga.ts » runsConcurrently
        Note right of Git: executeRead
        Note right of Git: temp index

        Git-->>WS: checkpointId + git ref

        WS->>WS: agent/agent.ts » sessionCheckpoints.set
        Note right of WS: promptId + ts + turnCompletedAt

        WS->>UI: GIT_CHECKPOINT notification
        Note right of UI: promptId + turnCompletedAt

        UI->>UI: buildConversationItems.ts » assignDeferredCheckpoints()
        Note right of UI: bind restore icon by turnCompletedAt
    end

    rect rgb(27,35,64)
        Note over UI,S3: HANDOFF local→cloud (pre-existing primitive)

        UI->>Core: sessionService.ts » handoffToCloud
        Note right of Core: taskId + repoPath

        Core->>WS: trpc handoff.execute
        Note right of WS: captureForHandoff packs local git state

        WS->>S3: agent-server.ts » captureCheckpointState
        Note right of S3: pendingHandoffGitState
        Note right of S3: pre-flight snapshot
        Note right of S3: promptId-less
        Note right of S3: iconless by design

        S3->>Cloud: sandbox resumes from packed state
    end

    rect rgb(43,30,63)
        Note over UI,S3: HANDOFF cloud→local

        UI->>Core: sessionService.ts » handoffToLocal
        Note right of Core: taskId + repoPath

        Core->>Core: sessionService.ts » runHandoffPreflight()
        Note right of Core: executeHandoff()
        Note right of Core: adapter = adapterStore.getAdapter(runId) or session.adapter

        Core->>WS: trpc handoff.execute
        Note right of WS: saga spawns local agent
        Note right of WS: applies git state

        WS->>S3: handoff/service.ts » syncCloudCheckpoints()
        Note right of S3: fetch run log

        WS->>Git: handoff-checkpoint.ts » HandoffCheckpointTracker.materializeCheckpointRef()
        Note right of Git: rebuild each cloud checkpoint git ref locally

        WS->>WS: agent/agent.ts » registerCloudCheckpoint
        Note right of WS: promptId + turnCompletedAt

        WS->>UI: agent/agent.ts » replayCheckpoints / emit
        Note right of UI: icons enabled

        Core->>WS: sessionService.ts » handoffToLocal
        Note right of WS: replay persisted config (model + mode)
    end

Loading

The story — a checkpoint is a snapshot of the working tree taken at a turn boundary (the moment
an agent finishes a turn). Two paths capture them; a third makes them portable:

• Cloud capture (top band): on a completed sandbox turn, broadcastTurnComplete stamps the true
  boundary time (turnCompletedAt) before firing a fire-and-forget capture → pack tree → upload to
  S3 → write a GIT_CHECKPOINT marker into the run log.
• Local capture (middle band): each turn runs CaptureCheckpointSaga against a temp git index
  (never blocks the agent) → records the checkpoint → notifies the UI, which pins the restore icon to
  the turn by turnCompletedAt — the one key immune to the async-late timestamp and the reused
  turn-index.
• Handoff local→cloud (pre-existing): packs the local git state and seeds the sandbox; its
  pre-flight snapshot is promptId-less and stays iconless by design. Shown for completeness — the
  primitive we build on, not part of what this PR adds.
• Handoff cloud→local (bottom band): carry over the run's real adapter
  (adapterStore ?? session.adapter — the fix that stops a codex task waking up as Claude) → saga
  spawns the local agent + applies git state → syncCloudCheckpoints walks the S3 log to rebuild
  each cloud checkpoint's git ref locally (materializeCheckpointRef) and re-register it → replay
  persisted model/mode.
• Net: every checkpoint, wherever it was born, ends up restorable on the desktop.

---
title: "B · Restore internals — click → code + memory consistent"
config:
  theme: base
  themeVariables:
    darkMode: true
    background: "#0f1430"
    primaryColor: "#232a52"
    primaryTextColor: "#eef1ff"
    primaryBorderColor: "#5661a8"
    lineColor: "#8b94dd"
    clusterBkg: "#161c38"
    clusterBorder: "#5661a8"
    edgeLabelBackground: "#161c38"
    titleColor: "#eef1ff"
---
flowchart TD
    A["UI click<br/>RestoreCheckpointDialog.tsx<br/>hooks/useRestoreCheckpoint.ts"] --> B["checkpoint/service.ts » runRestore()"]
    B --> C{"git/lock-detector » isLocked()?"}
    C -- yes --> C1["git/lock-detector » waitForUnlock()<br/>bounded retry"] --> D
    C -- no --> D["sagas/checkpoint.ts » RevertCheckpointSaga<br/>reset working tree to ref<br/>local OR materialized-cloud"]
    D --> E["Truncate agent memory to boundary"]
    E --> E1["S3 log — POST /truncate_log"]
    E --> E2["checkpoint/service.ts » trimReseededCacheToCheckpoint()<br/>local cache, keyed by turnCompletedAt"]
    E --> E3{"adapter?"}
    E3 -- claude --> E4["agent » jsonl-hydration.ts » hydrateSessionJsonl()<br/>(invoked on reconnect via agent/agent.ts)<br/>rebuild JSONL from truncated S3"]
    E3 -- codex --> E5["agent/agent.ts » fresh session +<br/>resume.ts » resumeFromLog() bounded summary"]
    E --> F["agent/agent.ts » truncateCheckpoints()<br/>drop orphaned refs after target"]
    F --> G["agent/agent.ts » markCheckpointRestore()<br/>→ sessionService.ts » restoreCheckpointReconnect()<br/>re-seed cache, replay adapter+model, isReconnecting UI"]
    G --> H["Session resumes at checkpoint<br/>code + memory consistent"]

    classDef blue fill:#232a52,stroke:#5661a8,color:#eef1ff;
    classDef purple fill:#2c2350,stroke:#6a4f9e,color:#eef1ff;
    classDef decide fill:#33285c,stroke:#7b64c9,color:#eef1ff;
    classDef done fill:#1d3a49,stroke:#4a8fb0,color:#eaffff;
    class A,B,C1,D blue;
    class E,E1,E2,E4,E5,F,G purple;
    class C,E3 decide;
    class H done;

Loading

The story — restoring isn't just git reset; the code and the agent's memory must land at the
same boundary, or the model "remembers the future." A click in RestoreCheckpointDialog →
runRestore, then:

• Protect the tree: if a git index.lock is held, wait (bounded) instead of failing.
• Revert code: RevertCheckpointSaga resets the working tree to the target ref — a
  locally-captured checkpoint or a materialized cloud-origin one, same code path (that's the
  point).
• Truncate memory to the same boundary, three fronts: authoritative S3 run log
  (POST /truncate_log) · local cache (trimmed by turnCompletedAt) · the agent's own conversation.
• Runtime split: Claude re-hydrates its JSONL from the truncated S3 log; codex starts a
  fresh session seeded with a bounded summary (resumeFromLog), because codex's handoff summary is
  one opaque rollout turn that turn-granular truncation can't slice.
• Finish: drop orphaned checkpoint refs after the target → restoreCheckpointReconnect re-seeds
  the cache + replays adapter/model → session resumes.
• Net: the session is genuinely at the checkpoint — tree, log, and memory agree.

---

5. Anatomy of the diff (~4.8k lines / ~59 files)

Where the change lives, so the file count reads at a glance:

Bucket	~lines	share	contents
Tests	~1,310	~27%	13 test files: restore service, handoff sync, icon binding, Claude re-hydration, adapter fallback, codex fresh-session reconnect.
New feature files	~775	~16%	5 new files that carry the feature: checkpoint/service.ts (restore orchestration — the largest single file), useRestoreCheckpoint + RestoreCheckpointDialog (UI), checkpoint.router (IPC), identifiers.
Cross-layer threading	~2,750	~57%	Edits to existing files that pass three new values — turnCompletedAt, adapter, checkpointId — through git → agent → workspace-server → core → host-router → ui → shared → api-client.

The change reaches into many packages because restore has to keep three things on the same turn
boundary — the code (git working tree), the run log (authoritative S3 copy + local cache), and
the agent's conversation memory — and they live in different packages. So it spans git (revert +
ref materialization), agent (capture + JSONL truncation / codex fresh-session), workspace-server
(orchestration), core (reconnect + adapter/model replay), and ui (the affordance). The diagrams above
trace that path.

Built on existing primitives — CaptureCheckpointSaga / RevertCheckpointSaga for git and the
existing handoff transport are reused as-is, so there's no new git plumbing or transport layer.

---

6. Design decisions & trade-offs

• Turn-granularity truncation, keyed by checkpoint position and timestamp — robust to log-format
  differences; the basis the cross-handoff re-seed logic builds on.
• Client-side boundary trim is authoritative for Codex; the backend truncate is the durable
  complement. Restore never hard-fails on a backend hiccup — it logs and falls back (only the git
  revert can fail hard; everything after it is non-fatal).
• materializeCheckpointRef is non-destructive — unpacks objects + rebuilds the ref only; the
  apply step already set the working tree, so the sync pass just needs refs.
• Re-seed the local cache from the truncated S3 log on restore, then timestamp-trim it so a
  reload can't resurrect restore-truncated turns.
• Codex restore always rebuilds from the truncated log (uniform path). A pure-local Codex restore
  becomes a faithful summary rather than verbatim rollout turns — accepted because a cross-handoff
  restore has no verbatim option anyway (the pre-handoff turns exist only as one summary blob).
• Restore is gated behind explicit user confirmation (destructive: mutates the working tree + log).

---

7. Roadmap / planned follow-ups

Each builds directly on the restore + memory-rebuild primitives shipped here:

• Cloud-origin restore — restore while still in the cloud (sandbox-side), vs. the current
  local-side restore.
• Edit a past turn (Allow editing any user message in chat (with undo/checkpoint) #2327) — restore to that turn's checkpoint, then re-prompt with edited content.
• Retry last turn (Retry message (re-run last user message with undo) #2329) — re-run the last user message with undo.
• Restore-modes menu — a rewind-style picker: restore code only / conversation only / both,
  plus summarize-from / up-to-here. A "soft" restore (revert code, keep memory) is one cell of
  this matrix.

---

8. Testing

Automated (in this PR, CI)

• packages/git/src/sagas/checkpoint.ts tests — RevertCheckpointSaga (renames, deletes, binary,
  worktree, submodules, large files).
• packages/git/src/handoff.test.ts — materializeCheckpointRef, pack/apply parity.
• packages/agent/src/adapters/claude/session/jsonl-hydration.hydrate.test.ts — Claude re-hydration
  from the truncated log.
• packages/agent/src/adapters/codex/{codex-agent,codex-client}.test.ts — codex rollout-replay
  suppression on loadSession (so a reload doesn't re-persist undone turns).
• packages/workspace-server/src/services/checkpoint/service.test.ts — restore concurrency lock +
  trimReseededCacheToCheckpoint.
• packages/workspace-server/src/services/local-logs/service.test.ts — prompt-boundary trim.
• packages/workspace-server/src/services/agent/agent.test.ts — codex fresh-session-on-restore
  reconnect.
• packages/core/src/sessions/handoffAdapter.test.ts — handoff adapter fallback (codex stays codex).
• packages/ui/…/buildConversationItems.test.ts, incrementalConversationItems.test.ts — per-turn
  restore-icon association + live/idle equivalence.

Run: pnpm --filter @posthog/code exec vitest run. Typecheck: tsc -p tsconfig.node.json --noEmit

• tsc -p tsconfig.web.json --noEmit.

Manual — the handoff matrix

Each row is deliberately a superset of the one above it, so a failure pinpoints which mechanism
actually broke rather than implicating the whole feature. A is the narrowest case — pure
local→cloud→local with zero cloud-side turns — so a failure there means the bug is in checkpoint
upload/sync across a handoff, not in cloud-side capture. B adds a cloud turn, bringing cloud-side
checkpoint capture into scope. C removes local history entirely and starts cold in the cloud
(nothing to upload, sync starts from zero) — a distinct failure mode from B's "there was already
something local to carry along." D stacks everything: two full round-trips plus a restore to a
cloud checkpoint sandwiched between two local ones — the only row that exercises restore against a
genuinely mixed-origin checkpoint history and upload de-dup on a second handoff.

For each: create a turn that writes a file, restore, confirm the file reverts and the agent's
memory drops the undone turn. Run on both Claude and Codex:

#	Flow	Restore at	Proves
A	local → cloud → local passthrough (no cloud turns)	a local turn	Local checkpoints survive an upload→sync round-trip untouched
B	local → cloud, cloud turns emit checkpoints	a cloud turn	Cloud-side turns capture their own checkpoints, and those sync back on handoff
C	cloud-native task (no local turns first)	a cloud turn	Cold-start cloud-native flow — zero local history, sync starts from nothing
D	full bidirectional local→cloud→local→cloud→local	a cloud checkpoint sandwiched between local turns	Everything above stacked together, plus restoring a mixed-origin history and de-dup on a second upload

Also verify: no Failed to change setting toast loop after a cross-adapter handoff; a reload after
restore doesn't resurrect truncated turns.

Optional — runnable smoke-test harness

Scenarios A–D above aren't just a manual checklist — they're also automated as a dev-console harness
(runA()…runD(), or runAll() for all four back-to-back) that drives the real running app
end-to-end: creates a task, sends real prompts to a real agent (Claude or Codex), waits for real
checkpoints, hands off to a real cloud sandbox and back, restores, and asserts file content + agent
memory at every step. It's the same four scenarios from the table above, scripted instead of clicked
through by hand — handy for re-running the full matrix after any future change to the handoff/restore
path without a manual pass.

It's intentionally not in this PR's diff (imports renderer internals directly; a dev instrument,
not a CI test), so it's published as a Gist instead:

Harness Gist: https://gist.github.com/Basit-Balogun10/59c767f5cbcc63068f9f262ee6f876bc

Run: pnpm dev → devtools → const m = await import('<raw-gist-url>') → await m.runA("claude")
(or "codex"), runB, runC, runD.

Validation status

All four scenarios pass on both Claude and Codex; automated suites green.

---

9. Risk / blast radius

• Capture adds a cheap per-turn git checkpoint (guarded, non-fatal on failure).
• Cloud sync + S3 truncate failures are non-fatal and logged (warn/error), never crashing a run.
• Restore mutates the working tree + truncates the log — gated behind explicit confirmation.
• No frontend schema/migration; backend is method-only (see backend PR).

10. Screenshots / recording

Demo — Scenario D: full bidirectional restore (local→cloud→local→cloud→local)

Local turn writes alpha → hand off to cloud → cloud turn adds beta (own checkpoint) → hand back to local (agent recalls both) → a new local turn adds gamma → hand off to cloud again → hand back to local once more — all three checkpoints, from both origins, survive the double round-trip. Then: restore to the cloud checkpoint sandwiched between the two local turns. File reverts to alpha+beta; gamma is gone from both the working tree and the agent's memory. This is the most exhaustive of the four scaffold scenarios — the one likeliest to catch a regression in any single leg of the round-trip, since it's the only row that restores against a genuinely mixed-origin history.

posthog-code-restore-checkpoint-demo-10x-github.mp4

Oh well, let's not mind the video quality, couldn't record at a better one due to excessive lag and yeah, the demo is sped up as well (and just under 5 minutes)

[trunk-io]

Merging to main in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here