herdr-crook

A shepherd's crook to reach the agents that wandered off into Slurm jobs and containers, and bring them back into the herd.

A herdr plugin that lets herdr detect and manage coding agents that run inside a Slurm allocation, inside a container — the classic HPC workflow:

1. open a pane on the cluster's login node,
2. request an interactive allocation (srun --pty, salloc),
3. start a container on the compute node (docker run -it, docker exec, docker compose run, a site wrapper script, enroot, apptainer, ...),
4. run an agent in it (claude, codex, opencode, ...).

Normally herdr on the login node only sees srun in that pane. With this plugin the pane shows the real agent, with herdr's usual idle / working / blocked / done states, notifications, herdr agent wait, agent read, and agent prompt / send-keys — also from a local herdr that reaches the login node with herdr machine add.

Nothing is installed or changed inside your images, and the plugin does not prescribe how you start containers.

How it works

 herdr pane (login node)
 └─ crook bridge ─ PTY relay ─ srun --pty ───────────► compute node: your shell
     │  marker process                               └─ docker run -it ... ─► container: claude
     │  HERDR_AGENT=claude                                         ▲
     │                                                             │ /proc + container engine API
     └─ srun --overlap (probe step) ────────────────► crook probe ─┘

• Bridge (crook run): runs your srun/salloc command under a transparent PTY relay on the login node. It exports a random CROOK_BRIDGE_ID into the job's environment.
• Probe: once the allocation runs, the bridge starts a lightweight extra job step (srun --jobid=<job> --overlap) running a self-contained Python probe on the allocated node(s). The probe finds your bridged terminal by CROOK_BRIDGE_ID, looks at its foreground process, and — if that is a container client — asks the container engine (Docker-compatible API: docker or podman) which container or exec session it is attached to and what runs in the foreground there. It recognises every agent herdr knows, by the same process-name rules herdr uses.
• Marker: herdr identifies agents from the processes in a pane's foreground job, honouring a HERDR_AGENT=<kind> hint. While an agent runs remotely, the bridge keeps a tiny stand-in process with that hint in the pane's foreground. herdr then runs its own screen detection for that agent on the relayed output — the same quality as a local agent — and accepts agent prompt for the pane. When the agent exits, the stand-in exits the same way a local agent would, so done notifications fire normally.
• Display: the pane shows where the agent runs, e.g. claude @ node001/dev (node, container and job are also available as sidebar tokens).

An agent started directly on the login node inside a bridged session (e.g. in an salloc shell) is detected too.

Requirements

• herdr 0.9.5 or newer on the login node (the plugin runs where the pane's herdr server runs).
• Python 3.6+ on the login node and on the compute nodes' host OS (standard library only; nothing is needed inside containers). crook ships no compiled code, so mixed x86_64 and aarch64 nodes work alike; it runs with each node's system python3 rather than one found through a shared PATH.
• Slurm with srun --overlap (20.11+); you must be allowed to start an extra step in your own job. The probe step runs with --overlap --gres=none --mem=0, so it shares CPUs and reserves no GPUs or memory.
• For container detection: a Docker-compatible engine socket you can query on the compute node (docker ps works without sudo), and a /proc that is not mounted with hidepid.

Install

On the login node (or locally with herdr --machine <label> ... where noted):

herdr plugin install denera/herdr-crook
~/.local/bin/crook setup        # or run the "Crook: set up shell integration" action

crook setup links crook into ~/.local/bin, writes a small shell snippet to ~/.local/share/crook/init.sh, and sources it from ~/.bashrc (and ~/.zshrc if present). In new herdr panes, interactive srun ... --pty ... and salloc then go through the bridge automatically. Remove it with crook setup --remove.

For development, link a checkout instead (herdr plugin link /abs/path; on a remote machine use herdr --machine <label> plugin link /abs/remote/path) and run bin/crook setup from it.

Use

Work as usual in a herdr pane on the login node:

$ sinfo -a
$ srun -p gpu --gres=gpu:1 -t 4:00:00 --pty bash      # plain srun, bridged by crook
node001$ docker run -it --rm -v $PWD:/work myimage bash
root@abc123:/work# claude

The pane turns into a claude agent within a second or two. Exit the agent and it goes back to a plain terminal.

After crook setup, you never type crook for this: in herdr panes, the shell hands interactive srun ... --pty ... and salloc to crook by itself. Everything else is untouched — srun without --pty, Slurm commands in scripts, shells outside herdr, and shells already inside a bridged session (on the compute node or in a container) run the real commands directly. To bypass crook once, use command srun ....

If you start allocations through a helper script (say gpu-shell, which runs srun -p gpu --gres=gpu:1 --pty bash for you), run the helper through crook: crook run gpu-shell, or add alias gpu-shell='crook run gpu-shell' to your shell rc. crook finds the srun inside the script and its job as usual. Helpers that start containers need nothing: the probe sees whatever the helper runs on the compute node.

Other ways in:

• crook srun ..., crook salloc ..., or crook run -- <any command> bridge one command explicitly (no shell integration needed).

• crook allocate (or the Crook: new allocation action) lists partitions from sinfo -a, asks for a partition and extra srun options, and starts a bridged srun --pty $SHELL. Bind it in the login node's herdr config.toml:

  [[keys.command]]
  key = "prefix+a"
  type = "plugin_action"
  command = "crook.allocate"
  description = "new Slurm allocation"

• Attaching to a running allocation works the same way: srun --jobid=<job> --overlap --pty bash.

• Containers started detached are picked up when you get back in with a terminal: docker compose up -d app followed by docker compose exec app bash, docker compose attach app, or docker attach <container> (the container name, e.g. proj-app-1, or container_name: from the compose file); the same for docker run -dit --name dev ... then docker exec -it dev bash or docker attach dev. attach needs a container started with a terminal (-it, or tty: true and stdin_open: true in compose).

Telling the bridge explicitly

If automatic detection cannot see your agent (no engine socket, hidepid, an unusual runtime), announce it from inside the session. Hints must carry the bridge's id, which the bridged session exports as CROOK_BRIDGE_ID (pass it into containers with docker run -e CROOK_BRIDGE_ID ...). With crook available:

crook signal agent claude -- claude    # hint while the command runs, then clear
crook signal agent claude              # or set it ...
crook signal agent                     # ... and clear it yourself

or from any shell, with no tools installed:

printf '\033]7717;crook;agent;id=%s;kind=claude\a' "$CROOK_BRIDGE_ID" > /dev/tty

The bridge strips these sequences from the output and drops a hint when the foreground command on the login node changes (e.g. srun exits); terminals without the bridge ignore them. Hints without the id are ignored unless accept_untagged_hints is set, so a file that happens to contain the sequence cannot claim the pane.

Configuration

Optional JSON at $(herdr plugin config-dir crook)/config.json (or $CROOK_CONFIG):

{
  "agents": {"my-claude-wrapper": "claude"},
  "probe_srun_args": ["--account=myproj"],
  "remote_python": "auto",
  "probe_interval": 1.0,
  "marker": "foreground",
  "metadata": true,
  "keep_herdr_env": false,
  "accept_untagged_hints": false
}

key	meaning
agents	extra executable names mapped to herdr agent kinds
probe_srun_args	extra options for the probe's srun --overlap step, if your site needs them
remote_python	interpreter for the probe on compute nodes; auto (default) uses the node's /usr/bin/python3, then /usr/local/bin/python3, then python3 on PATH, so a Python environment on a shared home built for another CPU architecture (x86_64 vs aarch64) is never picked by accident
probe_interval	seconds between probe scans
marker	foreground (default), member (no terminal hand-off; slower to update), or off
metadata	show <agent> @ <node>/<container> in herdr
keep_herdr_env	pass herdr's pane variables (HERDR_PANE_ID, socket path, ...) into the job; off by default so nested shells and compute nodes do not act as if they owned the pane (HERDR_ENV=1 is always kept, which stops herdr from starting a nested server there)
accept_untagged_hints	accept crook signal hints that lack the bridge id

Troubleshooting

• crook doctor checks the login-node setup; crook doctor --job <jobid> runs the probe once on that job's nodes and prints every terminal it sees, its foreground command and any detected agent. The Crook: doctor popup runs the first check.
• Bridge logs: ~/.local/state/crook/bridge-<pane>-*.log (probe stderr in *.log.probe).
• herdr agent explain <pane> shows how herdr classifies the agent's screen.

Limitations

• Only agents herdr knows (see herdr agent) can be surfaced.
• docker run without --name is matched to its container by start time. This assumes the node is not shared with other users starting containers at the same moment (e.g. exclusive allocations); --name, docker exec and compose services are matched by name instead.
• ssh to a compute node is not bridged; use srun --jobid=<job> --overlap --pty to open more shells in an allocation.
• A terminal multiplexer (tmux, screen) on the compute node or inside the container hides the agent from the probe; run the agent directly in the allocation's shell or the container's shell.
• The probe finds your terminal through CROOK_BRIDGE_ID in the job's environment, so srun --export=NONE (or SLURM_EXPORT_ENV=NONE) disables detection; crook doctor --job <id> warns about this.
• Do not start a herdr server on compute nodes or in containers that share your home directory with the login node: it would take over the login node's herdr socket.
• Plugins that automatically attach new tabs to an allocation bypass the bridge for those tabs.
• A herdr server restart ends the allocation's interactive step along with the pane, so agent sessions are not resumed automatically.

Development

python3 -m unittest discover -s tests -t .

tests/e2e/fakebin holds stand-ins for srun, squeue, scontrol, sinfo and an agent TUI, for exercising the bridge against a throw-away herdr server (XDG_CONFIG_HOME=<tmp> herdr server) on any machine.

License

Apache License 2.0. See LICENSE.