Add a ClawBench benchmark for Kernel MCP

[rgarcia]

What this adds

This PR adds a reproducible benchmark for the browser-control tools exposed by kernel-mcp-server.

ClawBench supplies 129 real-website tasks and decides whether each task was completed by inspecting the browser request that represents submission. Its clawbench-harbor-adapt CLI converts those task definitions into Harbor task directories: each contains a task.toml, the instruction, browser environment and runtime scripts, and the ClawBench verifier.

Harbor supplies the experiment runner. For each task, it installs a stock agent (currently Claude Code and Codex), gives that agent the instruction and MCP configuration, runs the verifier, and writes both the reward and an Agent Trajectory Interchange Format (ATIF) JSON record of the agent's messages, tool calls, and observations.

We use Hypeman as Harbor's execution backend. Each trial gets one isolated Hypeman VM containing the selected agent, the ClawBench browser/evaluator runtime, Redis, and the locally built Kernel MCP server.

The benchmark compares Kernel MCP with a Playwright MCP control arm without changing the agent, model, task, judge, or browser. Both arms control the same pre-created stealth Kernel browser with no forced viewport. The control agent receives stock @playwright/mcp tools connected to that browser through ClawBench's credential-free CDP bridge; the Kernel arm receives get_connection_context and execute_playwright_code from the local Kernel MCP build instead.

How one trial runs

1. build-image.sh creates the reusable Hypeman image. Starting from node:22-bookworm-slim, it installs Bun, the repository dependencies, Redis, mcp-remote, Python, and uv; copies this checkout; builds the Next.js MCP server; installs its launch/wrapper scripts; and records the Git SHA as /opt/kernel-mcp-server/SOURCE_SHA.
2. clawbench/run.sh invokes ClawBench's clawbench-harbor-adapt CLI. The CLI emits an ordinary Harbor task directory containing task.toml, the task instruction, ClawBench's browser/runtime assets, and its verifier.
3. prepare-task.py removes the generated task's Playwright MCP entry, selects the source-pinned Hypeman image, registers the local Kernel MCP stdio wrapper, and extends task setup to start the local MCP server.
4. Harbor creates one VM from that image and installs the selected Codex or Claude Code version. Task setup creates one stealth Kernel browser, copies in the generated ClawBench runtime, attaches its request evaluator, and starts Redis plus the local Next.js MCP server.
5. The agent controls that existing browser with execute_playwright_code. ClawBench scores the intercepted submission request, downloads the replay, deletes the browser, and writes the Harbor result directory.

ClawBench remains responsible for task scoring and browser cleanup. The only additional diagnostic is kernel_mcp_valid, which confirms that the agent used the local source build and the exact browser ClawBench created.

Run it

Build the source-pinned trial image:

./benchmarks/harbor/build-image.sh

Run one task:

export CLAWBENCH_REPO=../ClawBench
./benchmarks/harbor/clawbench/run.sh codex \
  v2-1134-chapter-finder-redcross

Run all 129 tasks with ten concurrent trials:

HARBOR_N_CONCURRENT=10 \
  ./benchmarks/harbor/clawbench/run.sh codex all

benchmarks/harbor/README.md documents credentials, model/version overrides, timeouts, and the Harbor result directory, including rewards, logs, the portable ATIF trajectory, and retained browser replay.

Tool-description result

The first full-suite run showed that agents often used execute_playwright_code without returning useful page state. We updated its description to ask for a relevant accessibility snapshot or compact result after every action, then reran the same 129-task suite with GPT-5.6 Luna:

Kernel MCP arm	Reward	Median browser calls	Median agent time	ariaSnapshot() calls
Original description	19/129	38	254s	367
Updated description	24/129	35	223s	2,460

The Playwright MCP control scored 15/129 in the same three-arm run. These are one-attempt-per-task results, so they establish the benchmark and the direction of the tool-description change rather than a leaderboard claim.

Scope and validation

• the benchmark exposes only get_connection_context and execute_playwright_code; browser lifecycle, coordinate actions, and managed auth are unavailable to the agent
• the previous benchmark-specific unit-test suite and multi-check verifier were removed; ClawBench remains the source of truth for reward, interception, replay finalization, and cleanup
• bun test — 249 passed
• Python compilation, Ruff, shell syntax checks, and a generated-task dry run pass
• the three-arm full-suite run retained and deletion-verified all 387 browser replays

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
mcp	Ready	Preview	Aug 24, 2026 11:54am

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 9e91f52. Configure here.}