[NVIDIA #2] Add vLLM local backend with RadixAttention prefix caching to AHE eval

[AugustChaoTW]

Summary

Add vllm-local as a third AHE eval backend in owl.yaml, enabling hardware-level Prefix KV Cache via vLLM's RadixAttention/PagedAttention on the local GB10 GPU. This is the hardware-layer complement to the application-layer Prefix-KV issues (#42–#48).

Hardware Context

Resource	Value
GPU	NVIDIA GB10 (Grace Blackwell)
VRAM	121.6 GB
CUDA	SM 12.1, CUDA 13.0
vLLM	0.20.0 (installed, not running)

KV Cache pool estimation (Qwen2.5-32B-Instruct, INT8):

Model weights:  ~64 GB
KV Cache pool:  ~50 GB  (remaining after 85% GPU util threshold)
System:         ~8 GB
────────────────────────
Total:          122 GB ✓

50 GB KV Cache ÷ ~4 KB/token = ~12.5M tokens cacheable
Skill prefix typical size: < 5,000 tokens
→ prefix cache hit rate ≈ 100% after first request

Changes

1. owl.yaml — new backend entry

base_agent:
  backend: opencode
  backends:
    opencode:
      command: opencode
      args: ["-p", "{prompt}", "-f", "json"]
      env:
        OPENCODE_NO_AUTO_UPDATE: "1"
    claude-code:
      command: claude
      args: ["-p", "{prompt}", "--output-format", "json", "--dangerously-skip-permissions"]
      env: {}
    vllm-local:                              # ← NEW
      command: python3
      args:
        - "-c"
        - |
          import openai, sys
          client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="local")
          prompt = sys.argv[1] if len(sys.argv) > 1 else ""
          r = client.chat.completions.create(
            model="local",
            messages=[{"role":"user","content":prompt}],
            max_tokens=4096,
            temperature=0.2,
          )
          print(r.choices[0].message.content)
        - "{prompt}"
      env:
        VLLM_ENDPOINT: "http://localhost:8000"

2. ahe/run_eval.py — vLLM server lifecycle management

def ensure_vllm_server(model: str, gpu_util: float = 0.85) -> subprocess.Popen | None:
    """Start vLLM server if not already running. Returns process handle."""
    try:
        r = requests.get("http://localhost:8000/health", timeout=2)
        if r.ok:
            return None  # already running
    except Exception:
        pass

    cmd = [
        sys.executable, "-m", "vllm.entrypoints.openai.api_server",
        "--model", model,
        "--enable-prefix-caching",          # RadixAttention
        "--gpu-memory-utilization", str(gpu_util),
        "--dtype", "bfloat16",              # Blackwell native
        "--port", "8000",
        "--max-model-len", "32768",
    ]
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # Wait for /health to be ready (up to 120s)
    for _ in range(60):
        try:
            if requests.get("http://localhost:8000/health", timeout=2).ok:
                return proc
        except Exception:
            time.sleep(2)
    raise RuntimeError("vLLM server failed to start")

3. Trace schema v4 — add prefix_cache_hit_tokens

@dataclass
class Trace:
    ...
    prefix_cache_hit_tokens: int = 0   # from vLLM usage.prompt_tokens_details
    prefix_cache_miss_tokens: int = 0

4. AHE summary — prefix cache efficiency metrics

{
  "avg_prefix_cache_hit_rate": 0.94,
  "total_tokens_saved": 184320,
  "estimated_compute_saved_s": 47.2
}

Recommended Model for GB10

Qwen/Qwen2.5-32B-Instruct or Qwen/Qwen2.5-72B-Instruct-AWQ (fits in 121.6 GB at INT4/AWQ).

Acceptance Criteria

• python ahe/run_eval.py --suite terminal --backend vllm-local runs end-to-end
• vLLM server auto-starts if not running; shuts down gracefully after eval
• --enable-prefix-caching flag present in server startup
• Trace schema includes prefix_cache_hit_tokens
• Summary reports avg_prefix_cache_hit_rate
• skill_list() compatible — NVIDIA skills injected into vLLM prompts as prefix

Synergy with Prefix-KV Issues

This is the hardware-layer realization of the application-layer work in:

• [[Prefix-KV #1] Stable-first system prompt ordering for prefix cache efficiency #42] Stable-first prompt ordering (controls what goes into the cached prefix)
• [[Prefix-KV #3] MCP server semantic query result cache #44] AHE prefix sharing tracking (measures the hardware cache hit rate)

Effort: 1 day | Priority: P1

[AugustChaoTW]

GPU-agnostic requirement: This feature must degrade gracefully without an NVIDIA GPU. See #56 for the full compatibility contract, fallback implementation details, and CI requirements that must pass before this issue can be merged.