Make restart preparation bounded and verify every task recovers

bin/validate

@@ -127,6 +127,8 @@ AGENT_WORKFLOWS_SOURCE_CHECKOUT=1 ruby skills/plan-pr-batch/scripts/check_goal_p
 ruby bin/check-control-tower-prompt-size-test.rb
 bin/check-control-tower-prompt-size
 ruby skills/pause/bin/pause-prompt-test.rb
+ruby skills/pause/bin/recovery-record-test.rb
+ruby skills/restart-codex-subagents/bin/recovery-status-test.rb
 ruby skills/close-batch/bin/close-batch-contract-test.rb
 ruby skills/post-merge-audit/bin/post-merge-audit-policy-test.rb
 ruby skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb

docs/agent-runner-restarts.md

@@ -1,10 +1,32 @@
-# Agent Runner Restart Prompts
+# Agent Runner Restarts
 
 Use these prompts when an operator needs to restart Codex, Claude Desktop,
 Claude Code, or another agent runner without losing useful handoff state.
 Use `$pause` when installed skills are available and you want these copy-paste
 prompts printed directly.
 
+## Restart Without Advance Notice
+
+Restartability is a normal operating requirement. Preparation is optional;
+recovery combines checkpoints, newer relevant logs and live state. A missing
+handoff is expected after a crash and must not prevent recovery.
+
+Use immediate restart when no time is available. When time permits, use one
+shared absolute UTC deadline, normally 60 seconds for the affected host,
+including dispatch and waits. Report missing acknowledgments and unknown or
+running operations when it expires; never extend it per task or call a timeout
+`RESTART_READY`. Planned preparation can use an explicitly longer deadline.
+These prompts cannot preempt a stalled host RPC or guarantee safe interruption.
+
+The installed [restart skill](../skills/restart-codex-subagents/SKILL.md) owns
+bounded parent/child preparation and fleet recovery. A standalone task can
+coordinate its host; no control tower or network service is required.
+[Combined recovery and local recording](../skills/pause/references/recovery.md)
+owns the recovery algorithm, recorder commands, limitations and recovery prompt.
+Use it when handoffs are missing or stale; the planned prompts below remain
+available when time permits. A state mismatch is first reconciled from evidence;
+only an unresolved conflict or changed authority requires operator direction.
+
 ## Which Prompt To Use
 
 - Use the **non-batch pause prompt** for a single thread that is not holding a
@@ -18,10 +40,10 @@ prompts printed directly.
   in [Cancelling Or Stopping A Batch](../workflows/pr-processing.md#cancelling-or-stopping-a-batch),
   then launch a new batch from a checkout that already has the desired files.
 - If a non-batch thread already exited before the pause prompt could be pasted,
-  resume from the last saved handoff and re-check branch, HEAD, local changes,
+  use available checkpoints plus newer logs and re-check branch, HEAD, local changes,
   and running processes before editing or pushing.
 - If a batch lane already exited before the pause prompt could be pasted, resume
-  from the last saved handoff and run
+  from available checkpoints plus newer relevant logs and run
   [bounded status recovery](../workflows/pr-processing.md#bounded-status-recovery)
   before editing, pushing, polling, or starting a new target.
 
@@ -64,11 +86,11 @@ current AGENTS.md first. Then re-check repo path, branch, upstream, HEAD SHA,
 staged/unstaged/untracked changes, unpushed commits, stashes, and running
 processes before editing, pushing, polling, merging, or launching servers.
 
-Reconstruct the current goal from the handoff and this request. Continue only
-from the recorded next resume step after the live state matches the handoff.
-If live state does not match the handoff, report the mismatch and stop for
-operator direction before editing, pushing, polling, merging, or launching
-servers.
+Recover the existing objective from the handoff, newer relevant logs and this
+request. Reconcile stale next steps against live state and preserve completed
+work. Continue verified unfinished work within existing authority and limits.
+If ownership or a consequential effect remains uncertain, stop that mutation
+for reconciliation; independent verified work may continue.
 
 Pasted restart handoff:
 <PASTE_RESTART_HANDOFF_HERE>

docs/skills.md

@@ -215,6 +215,13 @@ Use `$pause` before restarting Codex, Claude, or another agent runner. It
 produces restart-safe pause and resume prompts that preserve the current work
 and recovery steps.
 
+### [`$restart-codex-subagents`](../skills/restart-codex-subagents/SKILL.md)
+
+Use `$restart-codex-subagents` to prepare or recover tasks around a Codex
+restart, crash, or account change. Preparation is optional and bounded; recovery
+reconciles checkpoints, logs, and live state, then verifies that every intended
+task resumed or preserved its prior pause.
+
 ### [`$close-session`](../skills/close-session/SKILL.md)
 
 Use `$close-session` when a task may be finished and you want to know whether

skills/continue/SKILL.md

@@ -6,6 +6,12 @@ argument-hint: '[focus text or scope]'
 
 # Continue
 
+For crash or restart recovery, first read the installed
+[pause recovery procedure](../pause/references/recovery.md). Combine available
+checkpoints with newer relevant logs and live state; a missing handoff is not
+a blocker. Verify uncertain external effects before replaying them. Preserve
+intentional pauses, expired windows and budgets; recovery grants no new authority.
+
 Resume the current task. Before doing any new work, re-establish context so work does not drift or
 repeat:
 

skills/pause/SKILL.md

@@ -7,6 +7,16 @@ description: Print restart-safe copy-paste prompts for pausing an agent thread b
 
 Print operator prompts for safe agent-runner restarts.
 
+## Interruption And Short Grace
+
+For an unexpected interruption, a missing/stale handoff, or bounded restart
+preparation, read [the recovery procedure](references/recovery.md). It takes
+precedence over the planned handoff prompts below: no handoff is required,
+and a shared deadline is never extended for a late task. Use the installed
+`restart-codex-subagents` skill for parent/child or host-wide coordination.
+During ordinary work, use existing recording or its local recorder for compact
+checkpoints and consequential operation intent/results.
+
 ## Output Rules
 
 - If the user asks to "print", "show", "give me", or "copy/paste" prompts, do
@@ -97,11 +107,11 @@ current AGENTS.md first. Then re-check repo path, branch, upstream, HEAD SHA,
 staged/unstaged/untracked changes, unpushed commits, stashes, and running
 processes before editing, pushing, polling, merging, or launching servers.
 
-Reconstruct the current goal from the handoff and this request. Continue only
-from the recorded next resume step after the live state matches the handoff.
-If live state does not match the handoff, report the mismatch and stop for
-operator direction before editing, pushing, polling, merging, or launching
-servers.
+Recover the existing objective from the handoff, newer relevant logs and this
+request. Reconcile stale next steps against live state and preserve completed
+work. Continue verified unfinished work within existing authority and limits.
+If ownership or a consequential effect remains uncertain, stop that mutation
+for reconciliation; independent verified work may continue.
 
 Pasted restart handoff:
 <PASTE_RESTART_HANDOFF_HERE>

skills/pause/bin/recovery-record

@@ -0,0 +1,149 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "digest"
+require "fileutils"
+require "json"
+require "optparse"
+require "securerandom"
+require "tempfile"
+require "time"
+
+# Local recovery evidence, independent of the runner and coordination backend.
+module RecoveryRecord
+  module_function
+
+  def now
+    Time.now.utc.iso8601(6).sub(/Z\z/, "+00:00")
+  end
+
+  def atomic_write(path, value)
+    directory = File.dirname(path)
+    Tempfile.create([".pending-", ""], directory) do |stream|
+      stream.write("#{JSON.pretty_generate(value)}\n")
+      stream.flush
+      stream.fsync
+      File.rename(stream.path, path)
+      File.open(directory, File::RDONLY, &:fsync)
+    end
+  end
+
+  def task_directory(root, task, create:)
+    directory = File.join(root, Digest::SHA256.hexdigest(task))
+    [root, directory].each do |path|
+      raise ArgumentError, "recovery directories must not be symlinks" if File.symlink?(path)
+
+      FileUtils.mkdir_p(path, mode: 0o700) if create
+      next unless File.exist?(path)
+
+      stat = File.stat(path)
+      unless stat.directory? && stat.uid == Process.uid && (stat.mode & 0o077).zero?
+        raise ArgumentError, "recovery directories must be owner-only (0700)"
+      end
+    end
+    directory
+  end
+
+  def inspect_records(directory, task)
+    records = []
+    unreadable = []
+    Dir.glob("*.json", base: directory).sort.each do |name|
+      path = File.join(directory, name)
+
+      raise ArgumentError, "symlink" if File.symlink?(path)
+
+      source = File.read(path, encoding: Encoding::UTF_8)
+      raise EncodingError, "record must be UTF-8" unless source.valid_encoding?
+
+      record = JSON.parse(source)
+      unless record.is_a?(Hash) && record["task"] == task && %w[checkpoint operation].include?(record["kind"]) &&
+             %w[id recorded_at].all? { |key| record[key].is_a?(String) }
+        raise ArgumentError, "invalid record"
+      end
+
+      records << record
+    rescue SystemCallError, IOError, ArgumentError, EncodingError, JSON::ParserError
+      unreadable << File.basename(path)
+
+    end
+    records.sort_by! { |record| record["recorded_at"] }
+    { "records" => records,
+      "unresolved" => records.filter_map { |record| record["id"] if record["kind"] == "operation" && record["status"] != "exited" },
+      "unreadable" => unreadable }
+  end
+
+  def main(argv)
+    options = { root: File.join(Dir.home, ".local/state/agent-workflows/recovery") }
+    parser = OptionParser.new do |opts|
+      opts.banner = "Usage: recovery-record [--state-dir PATH] --task TASK_ID checkpoint|inspect|run [--label LABEL -- COMMAND ...]"
+      opts.on("--state-dir PATH") { |path| options[:root] = path }
+      opts.on("--task TASK_ID", "Stable task identity; never a credential") { |task| options[:task] = task }
+    end
+    parser.order!(argv)
+    raise ArgumentError, "--task must be nonempty" if !options[:task] || options[:task].empty?
+
+    action = argv.shift
+    raise ArgumentError, "expected checkpoint, inspect, or run" unless %w[checkpoint inspect run].include?(action)
+
+    if action == "run"
+      OptionParser.new do |opts|
+        opts.on("--label LABEL", "Non-secret target; argv/output are not saved") { |label| options[:label] = label }
+      end.order!(argv)
+      raise ArgumentError, "run requires --label and command argv after --" unless options[:label] && !argv.empty?
+    elsif !argv.empty?
+      raise ArgumentError, "unexpected arguments"
+    end
+
+    %i[task label].each do |key|
+      next unless options[key]
+
+      options[key] = options[key].dup.force_encoding(Encoding::UTF_8)
+      raise EncodingError, "#{key} must be UTF-8" unless options[key].valid_encoding?
+    end
+    raise ArgumentError, "--label must describe the operation" if action == "run" && options[:label].strip.empty?
+
+    directory = task_directory(File.expand_path(options[:root]), options[:task], create: action != "inspect")
+    if action == "inspect"
+      result = inspect_records(directory, options[:task])
+      puts JSON.pretty_generate(result)
+      return result["unreadable"].empty? ? 0 : 2
+    end
+
+    record = { "id" => SecureRandom.hex(16), "task" => options[:task], "recorded_at" => now }
+    path = File.join(directory, "#{record['id']}.json")
+    if action == "checkpoint"
+      source = $stdin.set_encoding(Encoding::UTF_8).read
+      raise EncodingError, "checkpoint must be UTF-8" unless source.valid_encoding?
+
+      state = JSON.parse(source)
+      raise ArgumentError, "checkpoint must be a JSON object" unless state.is_a?(Hash)
+
+      atomic_write(path, record.merge("kind" => "checkpoint", "state" => state))
+      puts path
+      return 0
+    end
+
+    cwd = Dir.pwd.force_encoding(Encoding::UTF_8)
+    raise EncodingError, "cwd must be UTF-8" unless cwd.valid_encoding?
+
+    record.merge!("kind" => "operation", "label" => options[:label], "cwd" => cwd, "status" => "intent")
+    atomic_write(path, record) # A failed intent write must prevent command execution.
+    warn "recovery intent: #{path}"
+    # The explicit executable/argv0 pair prevents Ruby's single-string shell fallback.
+    # Inherit streams; never capture output or replay an uncertain operation.
+    pid = Process.spawn([argv.first, argv.first], *argv.drop(1))
+    _, status = Process.wait2(pid)
+    code = status.signaled? ? -status.termsig : status.exitstatus
+    atomic_write(path, record.merge("status" => "exited", "exit_code" => code, "finished_at" => now))
+    code.negative? ? 128 - code : code
+  end
+end
+
+if $PROGRAM_NAME == __FILE__
+  begin
+    exit RecoveryRecord.main(ARGV)
+  rescue SystemCallError, IOError, ArgumentError, EncodingError, JSON::ParserError, OptionParser::ParseError => e
+    warn "recovery-record: #{e.class}: evidence unavailable; inspect live state"
+    exit 2
+  end
+end