[Benchmark] Multi-dimensional efficiency & effectiveness measurement framework

## Motivation

Before merging any Prefix-KV (#42–#48) or NVIDIA integration (#49–#55) PR, we need objective evidence that:
1. **Efficiency** improves (latency ↓, throughput ↑, cost ↓)
2. **Effectiveness** is preserved or improves (pass@1 ≥ baseline, memory recall ↑)
3. **CPU-only parity** — no regression without GPU (#56)

Current baseline (2 tasks only — insufficient for statistical significance):
- `pass@1 = 1.0`, `avg_latency_s = 11.45s`

---

## Measurement Dimensions

### D1 — Latency & Throughput
| Metric | Unit | Tool |
|--------|------|------|
| TTFT (time-to-first-token) | ms | vLLM `usage.prompt_tokens_details` / timing |
| End-to-end task latency | s | `trace.latency_s` |
| AHE eval throughput | tasks/hour | `len(tasks) / total_wall_time` |
| Tokens/second (generation) | tok/s | vLLM stats endpoint |

### D2 — Cache Efficiency
| Metric | Unit | Tool |
|--------|------|------|
| Prefix cache hit rate | % | `trace.prefix_cache_hit_tokens / total_prompt_tokens` |
| Tokens saved by prefix cache | tokens | sum of `prefix_cache_hit_tokens` |
| Memory query cache hit rate | % | `memory_access_analytics.hitRate` |
| Embedding cache hit rate | % | `EmbeddingService` internal counter |

### D3 — Quality
| Metric | Unit | Tool |
|--------|------|------|
| AHE pass@1 | 0–1 | `summary.pass_at_1` |
| Memory retrieval recall@5 | 0–1 | offline eval against labeled queries |
| Memory retrieval NDCG@10 | 0–1 | offline eval |
| Embedding similarity correlation | Pearson r | nomic vs nv-embed-v2 comparison |

### D4 — Resource Usage
| Metric | Unit | Tool |
|--------|------|------|
| GPU VRAM peak | GB | `nvidia-smi` |
| GPU utilization | % | `nvidia-smi` |
| CPU utilization (CPU-only mode) | % | `psutil` |
| SQLite query time | ms | instrumented `MemoryStore` |

### D5 — Cost (if using API backend)
| Metric | Unit | Tool |
|--------|------|------|
| Input tokens per task | tokens | trace |
| Cost per task (USD) | $ | `trace.cost_usd` |
| Estimated savings from prefix cache | $ | `hit_tokens × (1 - 0.1) × price/M` |

---

## Configuration Matrix (A/B/C/D)

```
Config  │ Backend      │ Prefix   │ Embed      │ Eviction │ Query Cache
────────┼──────────────┼──────────┼────────────┼──────────┼────────────
BASE    │ opencode     │ random   │ nomic/768  │ LRU      │ none
OPT-A   │ opencode     │ stable   │ nomic/768  │ LRU      │ 30s TTL
OPT-B   │ vLLM+RadixA  │ stable   │ nomic/768  │ LRU      │ 30s TTL
OPT-C   │ vLLM+RadixA  │ stable   │ nv-embed/  │ LRU      │ 30s TTL
         │              │          │ 4096       │          │
OPT-D   │ vLLM+RadixA  │ stable   │ nv-embed/  │ cuOpt    │ 30s TTL
         │              │          │ 4096       │          │
CPU-ONLY│ opencode     │ stable   │ nomic/768  │ scipy LP │ 30s TTL
```

Each config runs the **same eval suite** under the **same random seed**.

---

## Expanded Eval Task Suite

Current 2-task suite is insufficient. Add `evals/tasks/benchmark/` with 20 tasks:

| Task ID | Type | Description |
|---------|------|-------------|
| b001–b005 | TypeScript coding | Add function, fix bug, refactor |
| b006–b010 | Python coding | Implement algorithm, fix test |
| b011–b015 | File manipulation | Create/edit/move files per spec |
| b016–b018 | Multi-step | Requires memory of previous steps |
| b019–b020 | NVIDIA-specific | cuOpt install check, AI-Q query |

**Rationale:** 20 tasks → 95% CI width ≈ ±22% for pass@1 — acceptable for trend detection.

---

## Benchmark Runner Script

**File:** `ahe/benchmark.py`

```python
#!/usr/bin/env python3
"""
Multi-configuration benchmark runner.

Usage:
  python ahe/benchmark.py --suite benchmark --configs BASE OPT-A OPT-B
  python ahe/benchmark.py --suite benchmark --configs all --repeats 3
"""
from __future__ import annotations
import argparse, json, time, statistics
from pathlib import Path
from dataclasses import dataclass, asdict

ROOT = Path(__file__).parent.parent

CONFIGS = {
    "BASE":     {"backend": "opencode",   "prefix_order": "random", "embed": "nomic",    "eviction": "lru",   "query_cache": False},
    "OPT-A":    {"backend": "opencode",   "prefix_order": "stable", "embed": "nomic",    "eviction": "lru",   "query_cache": True},
    "OPT-B":    {"backend": "vllm-local", "prefix_order": "stable", "embed": "nomic",    "eviction": "lru",   "query_cache": True},
    "OPT-C":    {"backend": "vllm-local", "prefix_order": "stable", "embed": "nv-embed", "eviction": "lru",   "query_cache": True},
    "OPT-D":    {"backend": "vllm-local", "prefix_order": "stable", "embed": "nv-embed", "eviction": "cuopt", "query_cache": True},
    "CPU-ONLY": {"backend": "opencode",   "prefix_order": "stable", "embed": "nomic",    "eviction": "scipy", "query_cache": True},
}

@dataclass
class BenchResult:
    config: str
    suite: str
    repeat: int
    pass_at_1: float
    avg_latency_s: float
    p50_latency_s: float
    p95_latency_s: float
    prefix_cache_hit_rate: float
    memory_hit_rate: float
    tokens_saved: int
    gpu_vram_peak_gb: float
    wall_time_s: float

def run_config(config_name: str, cfg: dict, suite: str, repeat: int) -> BenchResult:
    """Run one eval suite under a specific configuration."""
    import subprocess, os
    env = {
        **os.environ,
        "MEMORY_PREFIX_ORDER": cfg["prefix_order"],
        "MEMORY_EMBED_MODEL":  "nomic-embed-text" if cfg["embed"] == "nomic" else "nvidia/NV-Embed-v2",
        "MEMORY_EVICTION":     cfg["eviction"],
        "MEMORY_QUERY_CACHE":  "true" if cfg["query_cache"] else "false",
        "OPENCODE_NO_AUTO_UPDATE": "1",
    }
    start = time.monotonic()
    result = subprocess.run(
        ["python3", "ahe/run_eval.py", "--suite", suite,
         "--backend", cfg["backend"], "--json-summary"],
        capture_output=True, text=True, timeout=3600,
        cwd=ROOT, env=env
    )
    wall_time = time.monotonic() - start

    summary = json.loads(result.stdout) if result.returncode == 0 else {}
    return BenchResult(
        config=config_name, suite=suite, repeat=repeat,
        pass_at_1=summary.get("pass_at_1", 0.0),
        avg_latency_s=summary.get("avg_latency_s", 0.0),
        p50_latency_s=summary.get("p50_latency_s", 0.0),
        p95_latency_s=summary.get("p95_latency_s", 0.0),
        prefix_cache_hit_rate=summary.get("avg_prefix_cache_hit_rate", 0.0),
        memory_hit_rate=summary.get("memory_hit_rate", 0.0),
        tokens_saved=summary.get("total_tokens_saved", 0),
        gpu_vram_peak_gb=summary.get("gpu_vram_peak_gb", 0.0),
        wall_time_s=wall_time,
    )

def print_report(results: list[BenchResult]) -> None:
    from collections import defaultdict
    by_config = defaultdict(list)
    for r in results:
        by_config[r.config].append(r)

    print(f"\n{'Config':<10} {'pass@1':>7} {'latency(s)':>12} {'p95(s)':>8} "
          f"{'cache_hit%':>11} {'mem_hit%':>9} {'tok_saved':>10}")
    print("─" * 75)

    base_latency = statistics.mean(r.avg_latency_s for r in by_config.get("BASE", [{"avg_latency_s": 1}]))

    for cfg_name, runs in sorted(by_config.items()):
        pass_avg  = statistics.mean(r.pass_at_1 for r in runs)
        lat_avg   = statistics.mean(r.avg_latency_s for r in runs)
        p95_avg   = statistics.mean(r.p95_latency_s for r in runs)
        cache_avg = statistics.mean(r.prefix_cache_hit_rate for r in runs)
        mem_avg   = statistics.mean(r.memory_hit_rate for r in runs)
        tok_total = sum(r.tokens_saved for r in runs)
        speedup   = base_latency / lat_avg if lat_avg > 0 else 0

        print(f"{cfg_name:<10} {pass_avg:>7.3f} {lat_avg:>10.2f}s "
              f"({speedup:+.1f}×) {p95_avg:>6.2f}s "
              f"{cache_avg*100:>9.1f}% {mem_avg*100:>7.1f}% {tok_total:>10,}")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", default="benchmark")
    parser.add_argument("--configs", nargs="+", default=["BASE", "OPT-A", "OPT-B"])
    parser.add_argument("--repeats", type=int, default=3,
                        help="Runs per config for statistical stability")
    args = parser.parse_args()

    selected = CONFIGS if args.configs == ["all"] else {k: CONFIGS[k] for k in args.configs}
    results: list[BenchResult] = []

    for cfg_name, cfg in selected.items():
        for rep in range(args.repeats):
            print(f"Running {cfg_name} repeat {rep+1}/{args.repeats}...")
            r = run_config(cfg_name, cfg, args.suite, rep)
            results.append(r)

    # Save raw results
    out = ROOT / "traces" / "benchmarks" / f"bench_{int(time.time())}.json"
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps([asdict(r) for r in results], indent=2))
    print(f"\nRaw results saved to {out}")

    print_report(results)

if __name__ == "__main__":
    main()
```

---

## Memory Retrieval Quality Benchmark

**File:** `ahe/bench_memory_retrieval.py`

```python
"""
Offline retrieval quality benchmark.
Compares nomic-embed-text (768-dim) vs nv-embed-v2 (4096-dim) and
FTS5-only vs hybrid search.

Requires: a labeled query→memory relevance dataset.
"""

LABELED_QUERIES = [
    {
        "query": "how to run bun test",
        "relevant_memory_ids": ["mem_bun_test_001", "mem_bun_test_002"],
    },
    {
        "query": "prefix kv cache stability",
        "relevant_memory_ids": ["mem_prefix_kv_001"],
    },
    # ... 50 labeled pairs
]

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    return len(set(retrieved[:k]) & relevant) / len(relevant)

def ndcg_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    import math
    dcg  = sum((1 / math.log2(i + 2)) for i, r in enumerate(retrieved[:k]) if r in relevant)
    idcg = sum((1 / math.log2(i + 2)) for i in range(min(len(relevant), k)))
    return dcg / idcg if idcg > 0 else 0.0

def run_retrieval_benchmark(store, queries, k=5):
    results = {"recall@5": [], "ndcg@10": [], "latency_ms": []}
    for q in queries:
        t0 = time.monotonic()
        retrieved = store.queryMemories({"search": q["query"], "limit": k})
        latency_ms = (time.monotonic() - t0) * 1000
        retrieved_ids = [m.id for m in retrieved]
        relevant = set(q["relevant_memory_ids"])
        results["recall@5"].append(recall_at_k(retrieved_ids, relevant, 5))
        results["ndcg@10"].append(ndcg_at_k(retrieved_ids, relevant, 10))
        results["latency_ms"].append(latency_ms)
    return {k: statistics.mean(v) for k, v in results.items()}
```

---

## Statistical Significance Requirements

For a result to be accepted as a real improvement (not noise):

| Comparison | Minimum Δ | Test | Significance |
|------------|-----------|------|-------------|
| pass@1 improvement | ≥ +0.05 | Proportion z-test | p < 0.05 |
| Latency reduction | ≥ -20% | Welch t-test | p < 0.05 |
| Cache hit rate | ≥ 80% | Point estimate | — |
| Recall@5 improvement | ≥ +0.05 | Wilcoxon | p < 0.05 |

Minimum sample size: **20 tasks × 3 repeats = 60 observations** per config.

---

## Expected Results Table (Hypothesis)

| Config | pass@1 | avg_latency_s | prefix_hit% | mem_hit% | speedup |
|--------|:------:|:-------------:|:-----------:|:--------:|:-------:|
| BASE | 0.80 | 11.5s | 0% | 40% | 1.0× |
| OPT-A | 0.80 | 10.5s | 0% | 65% | 1.1× |
| OPT-B | 0.82 | 1.2s | 95% | 65% | 9.6× |
| OPT-C | 0.85 | 1.1s | 95% | 75% | 10.5× |
| OPT-D | 0.85 | 1.1s | 95% | 80% | 10.5× |
| CPU-ONLY | 0.80 | 10.8s | 0% | 65% | 1.0× |

Rationale:
- OPT-A: query cache improves memory hit rate; slight latency from stable prefix
- OPT-B: vLLM RadixAttention dominates — 9× speedup from prefix caching
- OPT-C: better embedding improves recall → higher pass@1
- CPU-ONLY: must match OPT-A quality with no GPU (validates #56/#57)

---

## CI Integration

```yaml
# .github/workflows/benchmark.yml
name: Benchmark (manual)
on:
  workflow_dispatch:
    inputs:
      configs:
        description: 'Configs to benchmark (space-separated)'
        default: 'BASE OPT-A CPU-ONLY'
      repeats:
        description: 'Repeats per config'
        default: '3'

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt scipy
      - run: python ahe/benchmark.py
               --suite benchmark
               --configs ${{ inputs.configs }}
               --repeats ${{ inputs.repeats }}
      - uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: traces/benchmarks/
```

GPU configs (OPT-B/C/D) require self-hosted runner with GB10 — run manually.

---

## Acceptance Criteria

- [ ] `evals/tasks/benchmark/` — 20 tasks added
- [ ] `ahe/benchmark.py` — runner implemented, `--json-summary` flag in `run_eval.py`
- [ ] `ahe/bench_memory_retrieval.py` — 50 labeled query pairs, recall@5 + NDCG@10
- [ ] Statistical significance test included in report output
- [ ] OPT-B shows ≥ 5× latency improvement over BASE on GB10
- [ ] CPU-ONLY pass@1 within ±0.05 of BASE
- [ ] `traces/benchmarks/` report auto-uploaded as CI artifact
- [ ] `.github/workflows/benchmark.yml` — manual trigger for full benchmark matrix

## Related

- Measures: #42–#48 (Prefix-KV), #49–#54 (NVIDIA integration)
- Prerequisite for: all NVIDIA series PRs — must show positive result before merge
- Blocked by: #56 (compatibility), #57 (CPU tests)

Navigation Menu

[Benchmark] Multi-dimensional efficiency & effectiveness measurement framework #58

Motivation

Measurement Dimensions

D1 — Latency & Throughput

D2 — Cache Efficiency

D3 — Quality

D4 — Resource Usage

D5 — Cost (if using API backend)

Configuration Matrix (A/B/C/D)

Expanded Eval Task Suite

Benchmark Runner Script

Memory Retrieval Quality Benchmark

Statistical Significance Requirements

Expected Results Table (Hypothesis)

CI Integration

Acceptance Criteria

Related

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Metric	Unit	Tool
TTFT (time-to-first-token)	ms	vLLM `usage.prompt_tokens_details` / timing
End-to-end task latency	s	`trace.latency_s`
AHE eval throughput	tasks/hour	`len(tasks) / total_wall_time`
Tokens/second (generation)	tok/s	vLLM stats endpoint

Metric	Unit	Tool
Prefix cache hit rate	%	`trace.prefix_cache_hit_tokens / total_prompt_tokens`
Tokens saved by prefix cache	tokens	sum of `prefix_cache_hit_tokens`
Memory query cache hit rate	%	`memory_access_analytics.hitRate`
Embedding cache hit rate	%	`EmbeddingService` internal counter

Metric	Unit	Tool
AHE pass@1	0–1	`summary.pass_at_1`
Memory retrieval recall@5	0–1	offline eval against labeled queries
Memory retrieval NDCG@10	0–1	offline eval
Embedding similarity correlation	Pearson r	nomic vs nv-embed-v2 comparison

Metric	Unit	Tool
GPU VRAM peak	GB	`nvidia-smi`
GPU utilization	%	`nvidia-smi`
CPU utilization (CPU-only mode)	%	`psutil`
SQLite query time	ms	instrumented `MemoryStore`

Metric	Unit	Tool
Input tokens per task	tokens	trace
Cost per task (USD)	$	`trace.cost_usd`
Estimated savings from prefix cache	$	`hit_tokens × (1 - 0.1) × price/M`

Task ID	Type	Description
b001–b005	TypeScript coding	Add function, fix bug, refactor
b006–b010	Python coding	Implement algorithm, fix test
b011–b015	File manipulation	Create/edit/move files per spec
b016–b018	Multi-step	Requires memory of previous steps
b019–b020	NVIDIA-specific	cuOpt install check, AI-Q query

Comparison	Minimum Δ	Test	Significance
pass@1 improvement	≥ +0.05	Proportion z-test	p < 0.05
Latency reduction	≥ -20%	Welch t-test	p < 0.05
Cache hit rate	≥ 80%	Point estimate	—
Recall@5 improvement	≥ +0.05	Wilcoxon	p < 0.05

Config	pass@1	avg_latency_s	prefix_hit%	mem_hit%	speedup
BASE	0.80	11.5s	0%	40%	1.0×
OPT-A	0.80	10.5s	0%	65%	1.1×
OPT-B	0.82	1.2s	95%	65%	9.6×
OPT-C	0.85	1.1s	95%	75%	10.5×
OPT-D	0.85	1.1s	95%	80%	10.5×
CPU-ONLY	0.80	10.8s	0%	65%	1.0×

[Benchmark] Multi-dimensional efficiency & effectiveness measurement framework #58

Description

Motivation

Measurement Dimensions

D1 — Latency & Throughput

D2 — Cache Efficiency

D3 — Quality

D4 — Resource Usage

D5 — Cost (if using API backend)

Configuration Matrix (A/B/C/D)

Expanded Eval Task Suite

Benchmark Runner Script

Memory Retrieval Quality Benchmark

Statistical Significance Requirements

Expected Results Table (Hypothesis)

CI Integration

Acceptance Criteria

Related

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions