feat(phase-60): vector and topic quality fixtures (60-03)

.planning/REQUIREMENTS.md

@@ -59,13 +59,13 @@ exactly one plan in `docs/plans/v3.2-prove-it-plan.md`.
 - [x] **BENCH-10**: Per-conversation isolation on `--backend cli` (60-01)
 - [x] **BENCH-11**: Deterministic drain wait (poll checkpoints, no blind sleep) (60-01)
 - [ ] **BENCH-12**: Committed `locomo_llm_judge` full-dataset result (60-02 / #39)
-- [ ] **BENCH-13**: Layer switch `bm25|vector|hybrid` on the custom harness (60-03)
+- [x] **BENCH-13**: Layer switch `bm25|vector|hybrid` on the custom harness (60-03)
 
 ### Quality evidence (QUAL)
 
-- [ ] **QUAL-01**: Semantic fixture set, ≥15 tests (60-03 / #40)
-- [ ] **QUAL-02**: Topic clustering purity + ARI artifact (60-03 / #47)
-- [ ] **QUAL-03**: README "Solid" rows cite committed artifacts (60-03)
+- [x] **QUAL-01**: Semantic fixture set, ≥15 tests (60-03 / #40)
+- [x] **QUAL-02**: Topic clustering purity + ARI artifact (60-03 / #47)
+- [x] **QUAL-03**: README "Solid" rows cite committed artifacts (60-03)
 
 ### Operate it (OPS)
 
@@ -122,10 +122,10 @@ exactly one plan in `docs/plans/v3.2-prove-it-plan.md`.
 | BENCH-10 | 60-01 | In progress |
 | BENCH-11 | 60-01 | In progress |
 | BENCH-12 | 60-02 | Open (#39) |
-| BENCH-13 | 60-03 | Open (#40) |
-| QUAL-01 | 60-03 | Open (#40) |
-| QUAL-02 | 60-03 | Open (#47) |
-| QUAL-03 | 60-03 | Open |
+| BENCH-13 | 60-03 | This PR |
+| QUAL-01 | 60-03 | This PR |
+| QUAL-02 | 60-03 | This PR |
+| QUAL-03 | 60-03 | This PR |
 | OPS-01 | 61-01 | Open (#41) |
 | OPS-02 | 61-02 | Open (#42) |
 | OPS-03 | 61-03 | Open |

.planning/ROADMAP.md

@@ -321,11 +321,11 @@ Nothing on this list is a new capability except Phase 62, which is conditional.
 - [x] 59-02: Orphan branch triage (`docs/plans/phase-59-orphan-branch-triage.md`); OpenCode branch deleted
 - [x] 59-03: Planning truth (PROJECT.md, ROADMAP, STATE, 8 GitHub issues labelled `v3.2`)
 
-### Phase 60: Real Numbers (0/3) — 60-01 executing
+### Phase 60: Real Numbers (2/3) — 60-03 this PR; 60-02 maintainer
 
-- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli` (this PR)
+- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli` (PR #49)
 - [ ] 60-02: The run — maintainer, needs API key + documented machine (#39)
-- [ ] 60-03: Vector (#40) and topic (#47) quality fixtures
+- [x] 60-03: Vector (#40) and topic (#47) quality fixtures
 
 ### Phase 61: Operate It (0/5)
 

.planning/STATE.md

@@ -3,14 +3,14 @@ gsd_state_version: 1.0
 milestone_name: Prove It
 status: executing
 stopped_at: null
-last_updated: "2026-09-02T01:40:00.000Z"
-last_activity: 2026-09-02 — Phase 60-01 live-backend isolation verified (spawn-per-conversation, checkpoint drain, CLI smoke)
+last_updated: "2026-09-02T15:30:00.000Z"
+last_activity: 2026-09-02 — Phase 60-03 vector/topic quality fixtures (paraphrase set + purity/ARI)
 progress:
   total_phases: 4
   completed_phases: 1
   total_plans: 13
-  completed_plans: 3
-  percent: 23
+  completed_plans: 4
+  percent: 31
 ---
 
 # Project State
@@ -20,23 +20,24 @@ progress:
 See: .planning/PROJECT.md (updated 2026-09-01)
 
 **Core value:** Agent can answer "what were we talking about last week?" without scanning everything
-**Current focus:** v3.2 Phase 60-01 live-backend isolation ready to merge. Phase 59 Guardrails and Inventory is on `main` (#45).
+**Current focus:** v3.2 Phase 60-03 quality fixtures on `feature/phase-60-quality-fixtures`. Phase 60-01 isolation is PR #49. Phase 59 Guardrails and Inventory is on `main` (#45).
 
 ## Current Position
 
-Phase: 60 of 62 (Real Numbers) — plan 60-01 verified locally
+Phase: 60 of 62 (Real Numbers) — plan 60-03 executing; 60-01 in PR #49
 Status: v3.1.0 shipped 2026-09-01 (5 of 5 platforms). v3.2 Prove It adopted (expanded spec). Phase 59 complete.
-Last activity: 2026-09-02 — 60-01 spawn-per-conversation + GetIndexCheckpoints drain + live CLI smoke + isolation bleed test
+Last activity: 2026-09-02 — 60-03 paraphrase fixtures (BM25 recall@5 = 0.00, hybrid 1.00) + topic purity/ARI artifact
 
-Progress: [███░░░░░░░] 3/13 plans (Phase 59 complete). Phase 60-01 verified, awaiting merge.
+Progress: [████░░░░░░] 4/13 plans (Phase 59 complete). Phase 60-01 in #49; 60-03 this branch.
 
 ## Out-of-band Work
 
 ### Open PRs
 
 | PR | What | Status |
 |---|---|---|
-| _(this branch)_ | Phase 60-01 live-backend isolation + remaining 59 gaps | Open |
+| #49 | Phase 60-01 live-backend isolation | Open |
+| _(this branch)_ | Phase 60-03 vector/topic quality fixtures | Open |
 
 ### Open issues (the v3.2 backlog)
 

.planning/phases/60-real-numbers/60-03-PLAN.md

@@ -0,0 +1,46 @@
+---
+phase: 60-real-numbers
+plan: 03
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - benchmarks/fixtures/semantic-001.toml
+  - benchmarks/fixtures/sessions/semantic-*.jsonl
+  - crates/memory-bench/src/layers.rs
+  - crates/memory-bench/src/lexicon.rs
+  - crates/memory-bench/src/cli.rs
+  - crates/memory-bench/src/main.rs
+  - crates/memory-bench/src/runner.rs
+  - crates/memory-topics/src/metrics.rs
+  - crates/memory-topics/src/quality.rs
+  - crates/memory-topics/src/tfidf.rs
+  - crates/memory-topics/examples/topics_quality.rs
+  - benchmarks/results/semantic-bm25.json
+  - benchmarks/results/semantic-vector.json
+  - benchmarks/results/semantic-hybrid.json
+  - benchmarks/results/topics-quality.json
+  - README.md
+  - docs/positioning/agent-memory-vs-competition.md
+  - docs/benchmarks.md
+autonomous: true
+requirements: [BENCH-13, QUAL-01, QUAL-02, QUAL-03]
+must_haves:
+  truths:
+    - "semantic fixture set has ≥15 tests whose relevant sessions share meaning but not tokens with the query"
+    - "BM25-only mock recall@5 on the semantic set is < 0.4"
+    - "--layers bm25|vector|hybrid is a custom-harness switch"
+    - "topics-quality.json records purity and ARI from TopicExtractor::cluster"
+    - "README vector and topic rows cite the committed artifacts"
+  artifacts:
+    - path: benchmarks/results/semantic-hybrid.json
+      provides: hybrid recall@5 on the paraphrase set
+      contains: layers
+    - path: benchmarks/results/topics-quality.json
+      provides: purity and ARI
+      contains: adjusted_rand_index
+---
+
+<objective>
+Vector and topic quality fixtures: a paraphrase set BM25 cannot solve, a layer switch, and a labelled clustering artifact behind the README status table.
+</objective>

.planning/phases/60-real-numbers/60-CONTEXT.md

@@ -1,7 +1,7 @@
 # Phase 60: Real Numbers - Context
 
 **Gathered:** 2026-09-02
-**Status:** 60-01 in execution
+**Status:** 60-01 in PR #49; 60-03 executing
 **Source:** docs/plans/v3.2-prove-it-plan.md
 
 ## Phase Boundary
@@ -21,3 +21,15 @@ vector/topic fixtures.
   BM25 only).
 - Poll interval is `mpsc::recv_timeout`, not `std::thread::sleep`.
 - `--limit-questions` exists so 60-02 can dry-run.
+
+## 60-03 decisions
+
+- `--layers` is a **mock-backend** switch. Live `memory search` is always
+  RouteQuery hybrid (indexing on a fresh daemon is BM25-only).
+- Mock BM25 = token overlap. Mock vector = committed paraphrase lexicon +
+  TF-IDF cosine (not Candle). Mock hybrid = RRF k=60.
+- `memory-bench all` excludes `semantic` so `custom-harness-mock.json` is
+  not tanked.
+- Topic quality is `TopicExtractor::cluster` on capped TF-IDF (top 32,
+  df≥2) of an 80-doc / 8-cluster synthetic corpus, not live TOC / Candle.
+- Honest caveats are required in result JSON and README.

.planning/phases/60-real-numbers/60-VERIFICATION.md

@@ -1,12 +1,12 @@
 ---
 phase: 60-real-numbers
 verified: 2026-09-02
-status: 60-01-verified
+status: 60-01-verified; 60-03-executing
 ---
 
 # Phase 60: Real Numbers Verification
 
-## 60-01 (this PR)
+## 60-01 (PR #49)
 
 | # | Truth | Status | Evidence |
 |---|-------|--------|----------|
@@ -18,3 +18,14 @@ status: 60-01-verified
 | 6 | `drain_wait_ms` per conversation | CODE | `LocomoConversationResult.drain_wait_ms` |
 | 7 | Query reader sees indexer commits | CODE | `TeleportSearcher::search` calls `reload()` |
 | 8 | Sequence 0 advances BM25 checkpoint | UNIT | `test_process_batch_sequence_zero_advances_checkpoint` |
+
+## 60-03 (this branch)
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | ≥15 semantic tests; hit files share no query tokens | UNIT | `semantic_hits_do_not_contain_query_tokens` |
+| 2 | BM25 recall@5 < 0.4 on the paraphrase set | RUN | `semantic-bm25.json` recall@5 = 0.00 (0/16); `semantic_fixtures_bm25_below_point_four_vector_wins` |
+| 3 | Vector / hybrid beat BM25 | RUN | `semantic-vector.json` 1.00 16/16; `semantic-hybrid.json` 1.00 16/16 |
+| 4 | `--layers` is a custom-harness mock switch | CODE | `RetrievalLayer`; CLI search still RouteQuery |
+| 5 | Purity + ARI on `TopicExtractor::cluster` | RUN | `topics-quality.json`; `metrics` hand-computed 3-cluster (purity 8/9, ARI 4.5/7) |
+| 6 | README vector/topic rows cite artifacts | DOCS | README status table; positioning Claims Ledger |

README.md

@@ -164,8 +164,8 @@ is experimental.
 | TOC build and drill-down navigation | **Solid** | Year → Month → Week → Day → Segment → Grip |
 | Grips / provenance | **Solid** | Excerpts link back to the events they came from |
 | BM25 keyword search (Tantivy) | **Solid** | Exact tokens, no stemming (`jwt` does not match `JWTs`). Events indexed before v3.1 have empty `text_preview` and there is no backfill command — see [UPGRADING](docs/UPGRADING.md) and [#41](https://github.com/SpillwaveSolutions/agent-memory/issues/41) |
-| Vector search (HNSW + Candle) | **Solid** | Mechanism is wired; retrieval *quality* is not yet measured ([#40](https://github.com/SpillwaveSolutions/agent-memory/issues/40)). First daemon start downloads the embedding model; with no network the daemon warns and runs BM25-only |
-| Topic graph | **Works** | Clustering quality is not benchmarked ([#47](https://github.com/SpillwaveSolutions/agent-memory/issues/47)) |
+| Vector search (HNSW + Candle) | **Solid** | Mechanism is wired. Mock-harness hybrid recall@5 = **1.00** vs BM25 **0.00** on the 16-test paraphrase set ([`semantic-hybrid.json`](benchmarks/results/semantic-hybrid.json)); the vector *layer in that artifact* is a committed lexicon + TF-IDF cosine, not Candle ([#40](https://github.com/SpillwaveSolutions/agent-memory/issues/40)). First daemon start downloads the embedding model; with no network the daemon warns and runs BM25-only |
+| Topic graph | **Works** | ARI **1.00** / purity **1.00** on a synthetic 80-doc TF-IDF corpus ([`topics-quality.json`](benchmarks/results/topics-quality.json)), not Candle TOC embeddings ([#47](https://github.com/SpillwaveSolutions/agent-memory/issues/47)) |
 | Hybrid fusion + `RouteQuery` orchestration | **Works** | Wired end-to-end in Phase 54; explainability reports what actually ran |
 | LLM summarization / LLM rerank | **Experimental** | Needs an API key; fails open to the heuristic ranker and reports `rerank=heuristic` when it does |
 | Cross-project federated query | **Experimental** | Implemented; not performance-characterised |
@@ -180,9 +180,11 @@ is experimental.
 the old "65 second TOC" number was a harness defect (it timed ingest-time
 rollup and labelled it navigation).
 
-Committed results live in `benchmarks/results/`. Today both are **mock-backend**
-runs — a mock retrieval backend and a mock judge — so they demonstrate the
-harness, not competitive quality. **There is deliberately no comparison
+Committed results live in `benchmarks/results/`. `custom-harness-mock.json` and
+`locomo-smoke.json` are **mock-backend** runs — they demonstrate the harness,
+not competitive quality. `semantic-{bm25,vector,hybrid}.json` are also mock
+(token-overlap vs a committed paraphrase lexicon); they exist so the vector
+row has a BM25-cannot-solve baseline. **There is deliberately no comparison
 marketing in this repo**, and there will not be until a real-backend,
 real-judge run is committed next to the claim
 ([#39](https://github.com/SpillwaveSolutions/agent-memory/issues/39)).