[NVIDIA #8] GPU-agnostic compatibility layer — all features must degrade gracefully without NVIDIA GPU

[AugustChaoTW]

Requirement

Every feature added in the NVIDIA integration series (#49–#55) and Prefix-KV series (#42–#48) must run correctly on a machine with no NVIDIA GPU. GPU acceleration is an optional performance enhancement, never a hard runtime dependency for core functionality.

Compatibility Matrix

Issue	Feature	GPU Required?	CPU Fallback
#42	Stable-first prompt ordering	No	N/A — pure logic
#43	Embedding LRU cache	No	Caches whatever backend returns
#44	MCP query result cache	No	N/A — in-memory Map
#45	AHE Trace v4 prefix metrics	No	N/A — pure metrics
#46	memory_prefix_report tool	No	N/A — reads SQLite only
#47	cache_control markers	No	No-op if provider doesn't support
#49	NVIDIA Skills install	No	SKILL.md are plain text
#50	vLLM AHE backend	Optional	Falls back to opencode / claude-code
#51	nv-embed-v2 embeddings	Optional	Falls back to nomic-embed-text via Ollama
#52	AI-Q Blueprint analysis	Optional	Falls back to local classifier; can use cloud API
#53	cuOpt MILP eviction	Optional	Falls back to LRU
#54	TensorRT-LLM backend	Optional	Falls back to opencode / claude-code

Required: GPU Detection Utility

Add ahe/gpu_detect.py — a shared utility imported by all GPU-optional components:

"""Lightweight GPU presence detection — no torch import required."""
from __future__ import annotations
import shutil, subprocess, functools

@functools.lru_cache(maxsize=1)
def has_nvidia_gpu() -> bool:
    """Return True if an NVIDIA GPU is present and nvidia-smi is available."""
    if shutil.which("nvidia-smi") is None:
        return False
    try:
        r = subprocess.run(["nvidia-smi", "--query-gpu=name",
                            "--format=csv,noheader"],
                           capture_output=True, text=True, timeout=5)
        return r.returncode == 0 and bool(r.stdout.strip())
    except Exception:
        return False

@functools.lru_cache(maxsize=1)
def cuda_version() -> str | None:
    """Return CUDA version string (e.g. '13.0') or None."""
    try:
        r = subprocess.run(["nvidia-smi", "--query-gpu=driver_version",
                            "--format=csv,noheader"],
                           capture_output=True, text=True, timeout=5)
        if r.returncode != 0:
            return None
        # Parse CUDA version from nvidia-smi header
        r2 = subprocess.run(["nvidia-smi"], capture_output=True, text=True, timeout=5)
        import re
        m = re.search(r"CUDA Version:\s*([\d.]+)", r2.stdout)
        return m.group(1) if m else None
    except Exception:
        return None

def vram_gb() -> float:
    """Return total VRAM in GB across all GPUs, or 0.0."""
    try:
        r = subprocess.run(
            ["nvidia-smi", "--query-gpu=memory.total", "--format=csv,noheader,nounits"],
            capture_output=True, text=True, timeout=5
        )
        if r.returncode != 0:
            return 0.0
        return sum(int(x.strip()) for x in r.stdout.strip().splitlines()) / 1024
    except Exception:
        return 0.0

Required: Graceful Degradation per Component

vLLM backend (#50)

# ahe/run_eval.py
from ahe.gpu_detect import has_nvidia_gpu

def load_backend(name: str | None = None) -> dict:
    cfg = _load_yaml(ROOT / "owl.yaml")
    target = name or cfg.get("base_agent", {}).get("backend", "opencode")

    if target == "vllm-local" and not has_nvidia_gpu():
        print("[warn] vllm-local requested but no NVIDIA GPU detected; "
              "falling back to 'opencode' backend")
        target = "opencode"
    ...

nv-embed-v2 (#51)

// src/index.ts / src/mcp-server.ts — EmbeddingService
const EMBED_MODEL    = process.env.MEMORY_EMBED_MODEL    ?? "nomic-embed-text";
const EMBED_ENDPOINT = process.env.MEMORY_EMBED_ENDPOINT ?? "http://localhost:11434/api/embed";
const EMBED_FORMAT   = process.env.MEMORY_EMBED_API_FORMAT ?? "ollama";

// No GPU check needed in TS: if nv-embed-v2 vLLM server is unreachable,
// EmbeddingService.available is set to false → graceful FTS5-only fallback.
// Users on CPU-only machines simply don't set MEMORY_EMBED_MODEL.

cuOpt eviction (#53)

# src/cuopt_eviction.py
from ahe.gpu_detect import has_nvidia_gpu

def solve_eviction_optimal(memories, slots):
    """GPU path: cuOpt MILP."""
    import cuopt
    ...

def solve_eviction_cpu(memories, slots):
    """CPU path: scipy linprog (LP relaxation, deterministic)."""
    try:
        from scipy.optimize import linprog
        n = len(memories)
        c = [-m["score"] for m in memories]   # minimize negative = maximize score
        A_ub = [[1] * n]
        b_ub = [slots]
        bounds = [(0, 1) if not m["pinned"] else (1, 1) for m in memories]
        res = linprog(c, A_ub=A_ub, b_ub=b_ub, bounds=bounds, method="highs")
        keep = {memories[i]["id"] for i, v in enumerate(res.x) if v > 0.5}
    except ImportError:
        # Ultimate fallback: sort by score, keep top-K
        keep = {m["id"] for m in sorted(memories, key=lambda x: -x["score"])[:slots]}
    return [m["id"] for m in memories if m["id"] not in keep]

def solve_eviction(memories, slots):
    if has_nvidia_gpu():
        try:
            return solve_eviction_optimal(memories, slots)
        except ImportError:
            pass   # cuOpt not installed even with GPU
    return solve_eviction_cpu(memories, slots)

TensorRT-LLM backend (#54)

# ahe/run_eval.py
if target == "trtllm-local" and not has_nvidia_gpu():
    print("[warn] trtllm-local requires NVIDIA GPU; falling back to 'opencode'")
    target = "opencode"

CI Requirement

Add a CI test that runs the full test suite with GPU explicitly disabled:

# .github/workflows/test-cpu-only.yml
name: CPU-only compatibility
on: [push, pull_request]
jobs:
  test-no-gpu:
    runs-on: ubuntu-latest   # GitHub-hosted, no GPU
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --timeout=60
        env:
          MEMORY_EMBED_MODEL: "nomic-embed-text"   # no nv-embed-v2
          MEMORY_EVICTION: "lru"                   # no cuOpt
          # No NVIDIA_API_KEY → AI-Q tests skipped

Acceptance Criteria

• ahe/gpu_detect.py implemented with has_nvidia_gpu(), cuda_version(), vram_gb()
• run_eval.py: vllm-local and trtllm-local backends warn + fallback if no GPU
• cuopt_eviction.py: scipy.optimize.linprog CPU path implemented + sort-by-score ultimate fallback
• EmbeddingService: no code change needed (already degrades to FTS5 if endpoint unreachable)
• All 157 existing pytest tests pass on CPU-only environment (no GPU)
• New CI workflow .github/workflows/test-cpu-only.yml added
• README.md updated: table showing which features require GPU vs CPU-only
• memory_status output reports gpuAvailable: true/false

README table to add

## GPU Requirements

| Feature | CPU-only | NVIDIA GPU | Benefit with GPU |
|---------|:--------:|:----------:|-----------------|
| Memory store (SQLite + FTS5) | ✓ | ✓ | — |
| Vector search (sqlite-vec) | ✓ | ✓ | — |
| Embedding (nomic-embed-text via Ollama) | ✓ | ✓ | — |
| Embedding (nv-embed-v2, 4096-dim) | — | ✓ | +15% recall |
| AHE eval (opencode / claude-code backends) | ✓ | ✓ | — |
| AHE eval (vLLM backend + RadixAttention) | slow | ✓ | 10× TTFT |
| AHE eval (TensorRT-LLM backend) | — | ✓ | 4–8× throughput |
| Memory eviction (LRU) | ✓ | ✓ | — |
| Memory eviction (cuOpt MILP) | scipy LP | ✓ | Optimal policy |
| NVIDIA Skills (SKILL.md) | ✓ | ✓ | — |
| AI-Q Blueprint analysis | cloud API | ✓ | Local inference |

Priority: High — blocks merging any NVIDIA series PR