comfyui-mcp

A TypeScript MCP server that lets an AI agent drive a remote ComfyUI instance end-to-end — queue workflows, iterate on prompts and seeds, watch progress, view outputs as inline images, curate winners, download models, and author whole node graphs as Python — without ever touching the web canvas.

Built for and battle-tested on an NVIDIA DGX Spark (GB10, unified memory) that doubles as an LLM-serving box, which shaped some of the more interesting parts of the design (see GPU-yield handshake below).

Why

ComfyUI's node canvas is great for humans and terrible for agents. The HTTP API alone isn't enough either: workflow files live on the GPU host, outputs land on the GPU host, and model downloads need to happen on the GPU host. This server gives an agent a complete operational surface over both channels:

• HTTP to the ComfyUI API for queueing, progress, history, and image retrieval
• SSH to the GPU host for workflow file r/w, output listing, model downloads (via remote aria2c), and ComfyScript execution

Tool surface (16 tools)

Group	Tools
Stats	system_stats (RAM/VRAM, version, queue depth)
Viewing	list_outputs, view_image, view_latest — images come back as MCP image blocks
Workflows	list_workflows, read_workflow, write_workflow
Queueing	queue_workflow (with overrides), get_progress, cancel_queue
Waiting	wait_for_image — blocks until the render lands
Batching	queue_batch — up to 8 prompt/seed/param variants in one call
Models	list_models, download_model (Civitai + HuggingFace, token-aware)
Curation	pick_top — copy winners to a named folder remotely + scp them local
Scripting	run_comfyscript — author the graph as Python

The override engine

queue_workflow takes a saved workflow as a base and applies structured overrides — prompt, negative, checkpoint, seed, steps, cfg, guidance, sampler, scheduler, width, height, loras — by rewriting the graph before submission. The interesting cases:

• LoRA injection is architecture-aware: FLUX models load via UNETLoader and take CLIP from a separate DualCLIPLoader, so injected LoRAs use LoraLoaderModelOnly with no CLIP rewire; SDXL checkpoints get the classic LoraLoader treatment. The engine detects which world it's in.
• Unknown targets fail loudly: overriding a field the workflow has no node for returns override_target_not_found:<field> instead of silently generating the wrong thing.

Canonical starter templates for six model families (FLUX, FLUX.2, SDXL, Illustrious/anime, Qwen-Image, Z-Image Turbo) ship in templates/, generated deterministically from known-good source workflows by scripts/make_templates.py (strips LoRAs, normalizes titles, randomizes seeds, converts fixed-size upscales to relative ones so width/height overrides stay aspect-correct).

run_comfyscript — the escape hatch

JSON workflow + overrides covers routine text-to-image iteration. It does not cover a two-stage upscale pipeline, a video model with audio conditioning, or anything you'd actually need the canvas for. run_comfyscript accepts a Python snippet using ComfyScript, wraps it in the runtime preamble, executes it in the ComfyUI venv on the GPU host over SSH, and reports back only the newly created output files. Every node installed on the server (~2900 including custom nodes) is callable as a Python function, so the agent can author arbitrary graphs as code.

GPU-yield handshake

On a unified-memory box the image side and the LLM side compete for the same physical RAM, and a 35GB diffusion model colliding with a resident LLM is an OOM. Before any GPU work, src/yield.ts asks the LLM router (llama-swap) to unload its models — called from the shared queuePrompt() path so every current and future queue tool inherits it, and separately from run_comfyscript which bypasses that path. It's best-effort by design: if the router is down or slow, the server logs and proceeds rather than blocking a generation. The mirror-image policy (LLM loads freeing an idle ComfyUI) lives host-side.

Setup

npm install && npm run build

Register with your MCP client (Claude Code shown), pointing the env at your ComfyUI host:

// ~/.claude.json → mcpServers
"comfyui": {
  "command": "node",
  "args": ["/path/to/comfyui-mcp/dist/index.js"],
  "env": {
    "COMFYUI_HOST": "http://<gpu-host>:8188",
    "SPARK_SSH_ALIAS": "<ssh-alias>",            // from ~/.ssh/config
    "SPARK_COMFYUI_ROOT": "/path/to/ComfyUI",    // on the GPU host
    "SPARK_COMFYUI_PYTHON": "/path/to/venv/bin/python",
    "LLAMASWAP_HOST": "http://<gpu-host>:8089",  // optional, GPU-yield
    "CIVITAI_API_TOKEN": "${CIVITAI_API_TOKEN}", // optional, gated downloads
    "HF_TOKEN": "${HF_TOKEN}"
  }
}

SSH access to the GPU host (key-based, via the alias) is required for the workflow/model/comfyscript tools; the queue/view tools work over HTTP alone.

Testing

npm test          # 81 tests, fully offline
npm run smoke -- --live   # hits the real GPU host

Every tool handler is a pure function taking injectable deps (handleX(args, deps)), so the entire suite runs offline against fakes — no ComfyUI instance needed. The live smoke script is separate and explicit.

Docs

• docs/realism-findings.md — empirical findings on photorealistic portrait generation and character identity consistency with FLUX.1 Krea: what actually makes a generated image read as a modern photo, which prompt signals break identity across a set, and documented demographic biases in scene priors.

License

MIT