Add a ClawBench benchmark for Kernel MCP

.dockerignore

@@ -0,0 +1,8 @@
+.git
+.github
+.next
+node_modules
+coverage
+.env*
+*.log
+benchmarks/harbor/.image.env

.gitignore

@@ -1,5 +1,7 @@
 # Dependencies
 node_modules/
+__pycache__/
+*.py[cod]
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
@@ -107,5 +109,9 @@ Makefile
 # private key
 mcp-key.pem
 
+# Harbor benchmark runtime data
+benchmarks/harbor/.image.env
+benchmarks/harbor/image/source-sha
+
 # TypeScript incremental build cache
 tsconfig.tsbuildinfo

README.md

@@ -261,7 +261,7 @@ Each Kernel feature has a single `manage_*` tool with an `action` parameter, kee
 
 One additional Managed Auth helper (`begin_auth_login`) is marked app-only (`_meta.ui.visibility: ["app"]`); it refuses to execute on hosts that do not declare MCP Apps support. The App forwards the server-issued signed flow checkpoint to the shared `manage_auth_connections` `wait` action, so flow identity and terminal-state decisions stay on the server.
 
-Self-hosted deployments can hide sensitive tool families by setting `KERNEL_MCP_DISABLED_TOOLSETS` to a comma-separated list. For example, `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` prevents `manage_api_keys` from being registered.
+Self-hosted deployments can select tool families with `KERNEL_MCP_ENABLED_TOOLSETS` or hide them with `KERNEL_MCP_DISABLED_TOOLSETS`. Both accept comma- or space-separated toolset names and standalone aliases. For example, `KERNEL_MCP_ENABLED_TOOLSETS="playwright computer"` exposes browser-control tools without browser lifecycle or managed-auth tools, while `KERNEL_MCP_DISABLED_TOOLSETS=api_keys` only removes `manage_api_keys`. `get_connection_context` remains available in either mode.
 
 Call `get_connection_context` before deciding whether to create or select a project. Its canonical `connection_scope` reports whether the connection is organization-wide or fixed to a project. Project-scoped tools advertise an optional `project` (name or ID) and a deprecated `project_id`: organization-wide connections may omit them to preserve organization-wide reads and API default-project behavior, while fixed-project connections may omit them or pass the matching project. Project resources use project-qualified `kernel://orgs/{organizationId}/projects/{projectId}/...` URIs. Authorization remains enforced by the Kernel API; selecting a project never grants access to it.
 

benchmarks/harbor/README.md

@@ -0,0 +1,66 @@
+# Benchmark Kernel MCP with ClawBench
+
+[ClawBench](https://github.com/TIGER-AI-Lab/ClawBench) is a suite of browser tasks. Each task describes work to complete on a real website and an evaluator that watches for the network request representing completion. ClawBench then judges the submitted request parameters.
+
+[Harbor](https://github.com/laude-institute/harbor) runs those tasks as reproducible agent trials. For each trial, Harbor creates an isolated environment, installs a stock agent such as Codex or Claude Code, gives it the task's MCP tools and instruction, runs the verifier, and writes the reward and ATIF trajectory to a job directory.
+
+This benchmark uses `harbor_hypeman:HypemanEnvironment` as Harbor's execution backend. For every trial, Harbor asks Hypeman to start an isolated VM from this repository's benchmark image. Everything for that trial runs inside that VM:
+
+1. ClawBench creates one stealth Kernel browser and attaches its request evaluator.
+2. The task setup starts Redis and the locally built `kernel-mcp-server` on port 3002.
+3. Harbor starts the stock agent with a stdio MCP command that connects to that local server.
+4. The agent controls ClawBench's existing browser through `execute_playwright_code`; it cannot create browsers or use managed auth.
+5. ClawBench scores the intercepted request, downloads the replay, and deletes the browser.
+
+The image records the current Git SHA, and the generated task records the ClawBench SHA and browser session ID. The additional `kernel_mcp_valid` result confirms that the agent called the local server with the browser ClawBench created. Task reward still comes directly from ClawBench.
+
+## Requirements
+
+- `uv`, Harbor 0.21.0, and `harbor-hypeman` 0.1.1
+- Hypeman CLI credentials
+- a ClawBench checkout containing `df6743f` from `kernel/ClawBench` PR #1
+- `KERNEL_MCP_BENCHMARK_API_KEY` scoped to an isolated evaluation project
+- `PURELY_MAIL_API_KEY` and `PURELY_MAIL_DOMAIN` for ClawBench account tasks
+- `OPENAI_API_KEY` for Codex, or Anthropic credentials for Claude Code
+- the ClawBench judge variables when using a hosted judge: `CLAWBENCH_JUDGE_BASE_URL`, `CLAWBENCH_JUDGE_API_KEY`, `CLAWBENCH_JUDGE_MODEL`, and `CLAWBENCH_JUDGE_API_TYPE`
+
+## Build the trial image
+
+From the `kernel-mcp-server` checkout:
+
+```bash
+./benchmarks/harbor/build-image.sh
+```
+
+This builds the current checkout with Bun and writes the image reference and Git SHA to the ignored `benchmarks/harbor/.image.env` file.
+
+## Run one task
+
+```bash
+export CLAWBENCH_REPO=../ClawBench
+./benchmarks/harbor/clawbench/run.sh codex \
+  v2-1134-chapter-finder-redcross
+```
+
+## Run the full suite
+
+```bash
+export CLAWBENCH_REPO=../ClawBench
+HARBOR_N_CONCURRENT=10 \
+  ./benchmarks/harbor/clawbench/run.sh codex all
+```
+
+Codex defaults to version `0.120.0` with `gpt-5.6-luna`. Claude Code defaults to version `2.1.238` with `claude-sonnet-5`. Override these with `CODEX_BENCHMARK_MODEL`, `CODEX_BENCHMARK_VERSION`, `CLAUDE_BENCHMARK_MODEL`, or `CLAUDE_BENCHMARK_VERSION`.
+
+Single-task runs have a 40-minute wall-clock limit. Full-suite runs default to six hours. Set `HARBOR_BENCHMARK_TIMEOUT` to override either limit. Set `HARBOR_JOBS_DIR` to choose where Harbor writes results.
+
+## Results
+
+Harbor writes its normal job directory, including:
+
+- `trajectory.json`: the agent's ATIF messages and tool calls
+- `reward.json`: ClawBench's reward plus the `kernel_mcp_valid` diagnostic
+- `clawbench-result.json`: evaluator details
+- `kernel-mcp-result.json`: local-source and same-browser wiring details
+- `recording.mp4`: the finalized Kernel replay
+- `kernel-mcp/`: local server logs and the source/session manifest

benchmarks/harbor/bin/kernel-mcp-local

@@ -0,0 +1,18 @@
+#!/bin/sh
+# Harbor launches MCP servers as stdio subprocesses of the benchmark agent.
+# This wrapper turns that stdio connection into an authenticated connection to
+# the kernel-mcp-server HTTP endpoint running locally in the same Hypeman task.
+# The setup script writes the project-scoped key to /run so it never appears in
+# the generated agent configuration.
+set -eu
+
+key_file=/run/kernel-mcp-benchmark/api-key
+if [ -z "${KERNEL_API_KEY:-}" ] && [ -r "$key_file" ]; then
+  KERNEL_API_KEY=$(cat "$key_file")
+  export KERNEL_API_KEY
+fi
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+exec mcp-remote \
+  http://127.0.0.1:3002/mcp \
+  --header "Authorization: Bearer ${KERNEL_API_KEY}"

benchmarks/harbor/bin/start-kernel-mcp-server

@@ -0,0 +1,69 @@
+#!/bin/bash
+# Start the services used by the source-pinned Kernel MCP build inside a Harbor
+# trial. ClawBench has already created the browser; this script starts Redis and
+# Next.js, stores the project-scoped key for the stdio wrapper, waits for MCP to
+# accept connections, and records which source build and browser the trial used.
+set -euo pipefail
+
+: "${KERNEL_API_KEY:?KERNEL_API_KEY is required}"
+
+log_dir=/logs/kernel-mcp
+key_dir=/run/kernel-mcp-benchmark
+mkdir -p "$log_dir" /logs/artifacts "$key_dir"
+chmod 0777 "$log_dir" /logs/artifacts
+chmod 0700 "$key_dir"
+printf '%s' "$KERNEL_API_KEY" >"$key_dir/api-key"
+chmod 0600 "$key_dir/api-key"
+
+redis-server --daemonize yes --bind 127.0.0.1 --port 6379 \
+  --logfile "$log_dir/redis.log" --dir /tmp
+
+export CLERK_SECRET_KEY=${CLERK_SECRET_KEY:-sk_test_kernel_mcp_benchmark_local_only}
+export NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY:-pk_test_YmVuY2htYXJrLmNsZXJrLmFjY291bnRzLmRldiQ}
+
+cd /opt/kernel-mcp-server
+nohup ./node_modules/.bin/next start -p 3002 \
+  >"$log_dir/server.stdout.log" \
+  2>"$log_dir/server.stderr.log" &
+echo $! >"$log_dir/server.pid"
+
+for _ in $(seq 1 90); do
+  if curl -fsS -X POST http://127.0.0.1:3002/mcp \
+    -H "Authorization: Bearer ${KERNEL_API_KEY}" \
+    -H "Content-Type: application/json" \
+    -H "Accept: application/json, text/event-stream" \
+    --data '{"jsonrpc":"2.0","id":"benchmark-healthcheck","method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"harbor-healthcheck","version":"1.0.0"}}}' \
+    >"$log_dir/initialize-response.txt"; then
+    break
+  fi
+  sleep 1
+done
+
+if [ ! -s "$log_dir/initialize-response.txt" ]; then
+  echo "Kernel MCP server did not become ready" >&2
+  tail -100 "$log_dir/server.stderr.log" >&2 || true
+  exit 1
+fi
+
+python3 - <<'PY'
+import json
+import os
+from pathlib import Path
+
+browser_path = Path("/my-info/kernel_browser.json")
+try:
+    browser = json.loads(browser_path.read_text())
+except (OSError, json.JSONDecodeError):
+    browser = {}
+
+manifest = {
+    "kernel_mcp_server_sha": Path("/opt/kernel-mcp-server/SOURCE_SHA").read_text().strip(),
+    "clawbench_source_sha": os.environ.get("CLAWBENCH_SOURCE_SHA", ""),
+    "browser_session_id": browser.get("session_id"),
+    "enabled_toolsets": os.environ.get("KERNEL_MCP_ENABLED_TOOLSETS", ""),
+    "image": os.environ.get("KERNEL_MCP_BENCHMARK_IMAGE", ""),
+}
+Path("/logs/kernel-mcp/run-manifest.json").write_text(json.dumps(manifest, indent=2))
+PY
+
+printf 'ready\n' >"$log_dir/ready"

benchmarks/harbor/build-image.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+set -euo pipefail
+
+repo_root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+cd "$repo_root"
+
+source_sha=$(git rev-parse HEAD)
+source_sha_file=benchmarks/harbor/image/source-sha
+build_log=$(mktemp)
+trap 'rm -f "$source_sha_file" "$build_log"' EXIT
+
+printf '%s\n' "$source_sha" >"$source_sha_file"
+
+set +e
+hypeman build \
+  --file benchmarks/harbor/image/Dockerfile \
+  --cpus 4 \
+  --memory 8192 \
+  --timeout 1800 \
+  . 2>&1 | tee "$build_log"
+build_status=${PIPESTATUS[0]}
+set -e
+
+build_id=$(sed -n -E 's/^Build (ID|started): //p' "$build_log" | tail -1)
+if [[ -z "$build_id" ]]; then
+  echo "Hypeman did not return a build ID" >&2
+  exit 1
+fi
+
+image_ref="docker.io/builds/$build_id:latest"
+if ((build_status != 0)); then
+  echo "Build record failed; checking for a delayed ready image for up to 5 minutes" >&2
+  image_ready=false
+  for _ in $(seq 1 30); do
+    if hypeman --format json image list | python3 -c '
+import json
+import sys
+
+image_ref = sys.argv[1]
+expected = {image_ref, image_ref.removeprefix("docker.io/")}
+images = json.load(sys.stdin)
+raise SystemExit(
+    0
+    if any(image.get("name") in expected and image.get("status") == "ready" for image in images)
+    else 1
+)
+' "$image_ref"
+    then
+      image_ready=true
+      break
+    fi
+    sleep 10
+  done
+  if [[ "$image_ready" != true ]]; then
+    exit "$build_status"
+  fi
+fi
+
+cat >benchmarks/harbor/.image.env <<EOF
+KERNEL_MCP_BENCHMARK_IMAGE=$image_ref
+KERNEL_MCP_SOURCE_SHA=$source_sha
+EOF
+chmod 0600 benchmarks/harbor/.image.env
+
+printf 'Wrote benchmarks/harbor/.image.env for %s\n' "$image_ref"