[NVIDIA #7] Vertical integration tracking: Application-layer + Hardware-layer Prefix KV Cache

[AugustChaoTW]

Purpose

Tracking issue for the complete vertical integration of Prefix KV Cache across all layers in opencode-owl on the GB10 GPU. This issue documents how the application-layer Prefix-KV issues (#42–#48) and the hardware-layer NVIDIA issues (#49–#54) form a cohesive system.

Architecture Overview

┌──────────────────────────────────────────────────────────────────┐
│                    FULL STACK INTEGRATION                         │
├──────────────────────────────────────────────────────────────────┤
│  APPLICATION LAYER (opencode-owl)                                │
│  ├── #42 Stable-first prompt ordering                            │
│  │     → global memories → skills → facts → session context      │
│  ├── #43 Embedding LRU cache                                     │
│  │     → avoid repeated Ollama/vLLM embed calls                  │
│  ├── #44 MCP query result cache                                  │
│  │     → 30s TTL for repeated memory_query calls                 │
│  ├── #45 AHE Trace v4: prefix_hash + prefix_len_tokens          │
│  │     → measure what prefix is being cached                     │
│  ├── #46 memory_prefix_report tool                               │
│  │     → diagnose prefix stability, recommend ordering           │
│  └── #47 cache_control marker (blocked on OpenCode upstream)     │
├──────────────────────────────────────────────────────────────────┤
│  NVIDIA SKILLS LAYER                                              │
│  └── #49 NVIDIA Skills installed in registry                     │
│       → cuOpt, AI-Q skills injected as stable skill prefix       │
│       → AHE eval tasks auto-augmented with NVIDIA tool knowledge │
├──────────────────────────────────────────────────────────────────┤
│  HARDWARE LAYER (GB10 — 121.6 GB VRAM)                          │
│  ├── #50 vLLM + RadixAttention (--enable-prefix-caching)        │
│  │     → Skill prefix (< 5K tokens) cached as KV tensors        │
│  │     → TTFT: ~2s → ~30ms for cached prefix                    │
│  ├── #51 nv-embed-v2 (4096-dim) via vLLM                        │
│  │     → Better semantic search → better memory retrieval        │
│  ├── #52 AI-Q Blueprint deep analysis                            │
│  │     → Multi-agent failure root-cause synthesis                │
│  ├── #53 cuOpt MILP eviction (GPU-optimal cache replacement)     │
│  │     → Replace LRU with globally-optimal eviction policy       │
│  └── #54 TensorRT-LLM FP8 backend                               │
│       → 3–8× faster AHE eval, paged KV cache                    │
└──────────────────────────────────────────────────────────────────┘

Data Flow: Prefix KV Cache End-to-End

1. memory_query("nvidia cuopt")
        ↓ [#43 cache miss → SQLite FTS5 + nv-embed-v2 KNN]
2. retrieve: [cuopt-install skill, cuopt-routing skill, ...]
        ↓
3. chat.system.transform [#42 stable-first ordering]
   ┌─────────────────────────────────────┐
   │ [STABLE PREFIX — cached by vLLM]   │
   │  global/preference: ...             │
   │  global/skill: ...                  │
   │  project/skill: cuopt-install ...   │  ← NVIDIA skill injected
   │              cuopt-routing ...      │
   ├─────────────────────────────────────┤
   │ [DYNAMIC — recomputed every turn]  │  ← cache breakpoint here
   │  project/fact: ...                  │
   │  <current_task>: ...               │
   └─────────────────────────────────────┘
        ↓
4. vLLM RadixAttention lookup [#50]
   prefix_hash matches? → KV cache HIT → skip prefill
   → TTFT drops from 2s → 30ms
        ↓
5. AHE trace records:                   [#45]
   prefix_hash: "a3f8c102"
   prefix_cache_hit_tokens: 4821
   prefix_cache_miss_tokens: 0
        ↓
6. AHE summary:
   avg_prefix_cache_hit_rate: 0.97
   total_tokens_saved: 2.1M
   estimated_speedup: 4.3×

Key Metrics to Track

Metric	Baseline	Target	Measured By
TTFT (with prefix)	~2000ms	< 50ms	vLLM stats
Memory hit rate	~40%	> 80%	memory_hitrate
AHE eval throughput	1×	4–8×	avg_latency_s
Prefix cache hit rate	0%	> 90%	prefix_cache_hit_tokens
Embedding recall@10	baseline	+15%	ablation test

Implementation Order

Week 1:  #49 (10 min) → #50 (1 day) → #42 (1h) → #51 (2h)
Week 2:  #43 (2h)     → #44 (1h)   → #45 (2h) → #46 (3h)
Week 3+: #52 (3 days) → #53 (2 wk) → #54 (1 wk)

Success Definition

This vertical integration is complete when:

• All P0/P1 issues ([NVIDIA #1] Install NVIDIA Skills into opencode-owl skill registry (P0) #49, [NVIDIA #2] Add vLLM local backend with RadixAttention prefix caching to AHE eval #50, [NVIDIA #3] Upgrade embedding backend: Ollama nomic-embed-text → nv-embed-v2 on GB10 #51) closed
• AHE eval pass@1 improves by ≥ 10% due to richer NVIDIA skill injection
• avg_prefix_cache_hit_rate > 90% in AHE runs
• avg_latency_s reduces by ≥ 3× compared to Ollama backend baseline
• memory_prefix_report shows cache_eligible: true, current_order_is_optimal: true

Related

• Prefix-KV series: [Prefix-KV #1] Stable-first system prompt ordering for prefix cache efficiency #42 [Prefix-KV #2] Embedding LRU cache in EmbeddingService (avoid repeated Ollama calls) #43 [Prefix-KV #3] MCP server semantic query result cache #44 [Prefix-KV #4] AHE eval prefix sharing tracking in Trace schema v4 #45 [Prefix-KV #5] New MCP tool: memory_prefix_report #46 [Prefix-KV #6] Anthropic cache_control integration via structured system prompt blocks #47 [Prefix-KV #7] Research: Radix Tree memory index for prefix-aware retrieval #48
• NVIDIA series: [NVIDIA #1] Install NVIDIA Skills into opencode-owl skill registry (P0) #49 [NVIDIA #2] Add vLLM local backend with RadixAttention prefix caching to AHE eval #50 [NVIDIA #3] Upgrade embedding backend: Ollama nomic-embed-text → nv-embed-v2 on GB10 #51 [NVIDIA #4] Integrate NVIDIA AI-Q Blueprint as AHE deep-research analysis backend #52 [NVIDIA #5] GPU-accelerated memory eviction scheduling via cuOpt (replace LRU) #53 [NVIDIA #6] TensorRT-LLM as ultra-low-latency AHE eval backend on GB10 #54