diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 9900896..cdc8bd7 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -59,13 +59,13 @@ exactly one plan in `docs/plans/v3.2-prove-it-plan.md`. - [x] **BENCH-10**: Per-conversation isolation on `--backend cli` (60-01) - [x] **BENCH-11**: Deterministic drain wait (poll checkpoints, no blind sleep) (60-01) - [ ] **BENCH-12**: Committed `locomo_llm_judge` full-dataset result (60-02 / #39) -- [ ] **BENCH-13**: Layer switch `bm25|vector|hybrid` on the custom harness (60-03) +- [x] **BENCH-13**: Layer switch `bm25|vector|hybrid` on the custom harness (60-03) ### Quality evidence (QUAL) -- [ ] **QUAL-01**: Semantic fixture set, ≥15 tests (60-03 / #40) -- [ ] **QUAL-02**: Topic clustering purity + ARI artifact (60-03 / #47) -- [ ] **QUAL-03**: README "Solid" rows cite committed artifacts (60-03) +- [x] **QUAL-01**: Semantic fixture set, ≥15 tests (60-03 / #40) +- [x] **QUAL-02**: Topic clustering purity + ARI artifact (60-03 / #47) +- [x] **QUAL-03**: README "Solid" rows cite committed artifacts (60-03) ### Operate it (OPS) @@ -122,10 +122,10 @@ exactly one plan in `docs/plans/v3.2-prove-it-plan.md`. | BENCH-10 | 60-01 | In progress | | BENCH-11 | 60-01 | In progress | | BENCH-12 | 60-02 | Open (#39) | -| BENCH-13 | 60-03 | Open (#40) | -| QUAL-01 | 60-03 | Open (#40) | -| QUAL-02 | 60-03 | Open (#47) | -| QUAL-03 | 60-03 | Open | +| BENCH-13 | 60-03 | This PR | +| QUAL-01 | 60-03 | This PR | +| QUAL-02 | 60-03 | This PR | +| QUAL-03 | 60-03 | This PR | | OPS-01 | 61-01 | Open (#41) | | OPS-02 | 61-02 | Open (#42) | | OPS-03 | 61-03 | Open | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index aa56a07..2445c44 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -321,11 +321,11 @@ Nothing on this list is a new capability except Phase 62, which is conditional. - [x] 59-02: Orphan branch triage (`docs/plans/phase-59-orphan-branch-triage.md`); OpenCode branch deleted - [x] 59-03: Planning truth (PROJECT.md, ROADMAP, STATE, 8 GitHub issues labelled `v3.2`) -### Phase 60: Real Numbers (0/3) — 60-01 executing +### Phase 60: Real Numbers (2/3) — 60-03 this PR; 60-02 maintainer -- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli` (this PR) +- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli` (PR #49) - [ ] 60-02: The run — maintainer, needs API key + documented machine (#39) -- [ ] 60-03: Vector (#40) and topic (#47) quality fixtures +- [x] 60-03: Vector (#40) and topic (#47) quality fixtures ### Phase 61: Operate It (0/5) diff --git a/.planning/STATE.md b/.planning/STATE.md index c9b258f..75b4e15 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone_name: Prove It status: executing stopped_at: null -last_updated: "2026-09-02T01:40:00.000Z" -last_activity: 2026-09-02 — Phase 60-01 live-backend isolation verified (spawn-per-conversation, checkpoint drain, CLI smoke) +last_updated: "2026-09-02T15:30:00.000Z" +last_activity: 2026-09-02 — Phase 60-03 vector/topic quality fixtures (paraphrase set + purity/ARI) progress: total_phases: 4 completed_phases: 1 total_plans: 13 - completed_plans: 3 - percent: 23 + completed_plans: 4 + percent: 31 --- # Project State @@ -20,15 +20,15 @@ progress: See: .planning/PROJECT.md (updated 2026-09-01) **Core value:** Agent can answer "what were we talking about last week?" without scanning everything -**Current focus:** v3.2 Phase 60-01 live-backend isolation ready to merge. Phase 59 Guardrails and Inventory is on `main` (#45). +**Current focus:** v3.2 Phase 60-03 quality fixtures on `feature/phase-60-quality-fixtures`. Phase 60-01 isolation is PR #49. Phase 59 Guardrails and Inventory is on `main` (#45). ## Current Position -Phase: 60 of 62 (Real Numbers) — plan 60-01 verified locally +Phase: 60 of 62 (Real Numbers) — plan 60-03 executing; 60-01 in PR #49 Status: v3.1.0 shipped 2026-09-01 (5 of 5 platforms). v3.2 Prove It adopted (expanded spec). Phase 59 complete. -Last activity: 2026-09-02 — 60-01 spawn-per-conversation + GetIndexCheckpoints drain + live CLI smoke + isolation bleed test +Last activity: 2026-09-02 — 60-03 paraphrase fixtures (BM25 recall@5 = 0.00, hybrid 1.00) + topic purity/ARI artifact -Progress: [███░░░░░░░] 3/13 plans (Phase 59 complete). Phase 60-01 verified, awaiting merge. +Progress: [████░░░░░░] 4/13 plans (Phase 59 complete). Phase 60-01 in #49; 60-03 this branch. ## Out-of-band Work @@ -36,7 +36,8 @@ Progress: [███░░░░░░░] 3/13 plans (Phase 59 complete). Phase | PR | What | Status | |---|---|---| -| _(this branch)_ | Phase 60-01 live-backend isolation + remaining 59 gaps | Open | +| #49 | Phase 60-01 live-backend isolation | Open | +| _(this branch)_ | Phase 60-03 vector/topic quality fixtures | Open | ### Open issues (the v3.2 backlog) diff --git a/.planning/phases/60-real-numbers/60-03-PLAN.md b/.planning/phases/60-real-numbers/60-03-PLAN.md new file mode 100644 index 0000000..577779b --- /dev/null +++ b/.planning/phases/60-real-numbers/60-03-PLAN.md @@ -0,0 +1,46 @@ +--- +phase: 60-real-numbers +plan: 03 +type: execute +wave: 1 +depends_on: [] +files_modified: + - benchmarks/fixtures/semantic-001.toml + - benchmarks/fixtures/sessions/semantic-*.jsonl + - crates/memory-bench/src/layers.rs + - crates/memory-bench/src/lexicon.rs + - crates/memory-bench/src/cli.rs + - crates/memory-bench/src/main.rs + - crates/memory-bench/src/runner.rs + - crates/memory-topics/src/metrics.rs + - crates/memory-topics/src/quality.rs + - crates/memory-topics/src/tfidf.rs + - crates/memory-topics/examples/topics_quality.rs + - benchmarks/results/semantic-bm25.json + - benchmarks/results/semantic-vector.json + - benchmarks/results/semantic-hybrid.json + - benchmarks/results/topics-quality.json + - README.md + - docs/positioning/agent-memory-vs-competition.md + - docs/benchmarks.md +autonomous: true +requirements: [BENCH-13, QUAL-01, QUAL-02, QUAL-03] +must_haves: + truths: + - "semantic fixture set has ≥15 tests whose relevant sessions share meaning but not tokens with the query" + - "BM25-only mock recall@5 on the semantic set is < 0.4" + - "--layers bm25|vector|hybrid is a custom-harness switch" + - "topics-quality.json records purity and ARI from TopicExtractor::cluster" + - "README vector and topic rows cite the committed artifacts" + artifacts: + - path: benchmarks/results/semantic-hybrid.json + provides: hybrid recall@5 on the paraphrase set + contains: layers + - path: benchmarks/results/topics-quality.json + provides: purity and ARI + contains: adjusted_rand_index +--- + + +Vector and topic quality fixtures: a paraphrase set BM25 cannot solve, a layer switch, and a labelled clustering artifact behind the README status table. + diff --git a/.planning/phases/60-real-numbers/60-CONTEXT.md b/.planning/phases/60-real-numbers/60-CONTEXT.md index 12d6435..9d0c3a0 100644 --- a/.planning/phases/60-real-numbers/60-CONTEXT.md +++ b/.planning/phases/60-real-numbers/60-CONTEXT.md @@ -1,7 +1,7 @@ # Phase 60: Real Numbers - Context **Gathered:** 2026-09-02 -**Status:** 60-01 in execution +**Status:** 60-01 in PR #49; 60-03 executing **Source:** docs/plans/v3.2-prove-it-plan.md ## Phase Boundary @@ -21,3 +21,15 @@ vector/topic fixtures. BM25 only). - Poll interval is `mpsc::recv_timeout`, not `std::thread::sleep`. - `--limit-questions` exists so 60-02 can dry-run. + +## 60-03 decisions + +- `--layers` is a **mock-backend** switch. Live `memory search` is always + RouteQuery hybrid (indexing on a fresh daemon is BM25-only). +- Mock BM25 = token overlap. Mock vector = committed paraphrase lexicon + + TF-IDF cosine (not Candle). Mock hybrid = RRF k=60. +- `memory-bench all` excludes `semantic` so `custom-harness-mock.json` is + not tanked. +- Topic quality is `TopicExtractor::cluster` on capped TF-IDF (top 32, + df≥2) of an 80-doc / 8-cluster synthetic corpus, not live TOC / Candle. +- Honest caveats are required in result JSON and README. \ No newline at end of file diff --git a/.planning/phases/60-real-numbers/60-VERIFICATION.md b/.planning/phases/60-real-numbers/60-VERIFICATION.md index 2d91ac1..e06b542 100644 --- a/.planning/phases/60-real-numbers/60-VERIFICATION.md +++ b/.planning/phases/60-real-numbers/60-VERIFICATION.md @@ -1,12 +1,12 @@ --- phase: 60-real-numbers verified: 2026-09-02 -status: 60-01-verified +status: 60-01-verified; 60-03-executing --- # Phase 60: Real Numbers Verification -## 60-01 (this PR) +## 60-01 (PR #49) | # | Truth | Status | Evidence | |---|-------|--------|----------| @@ -18,3 +18,14 @@ status: 60-01-verified | 6 | `drain_wait_ms` per conversation | CODE | `LocomoConversationResult.drain_wait_ms` | | 7 | Query reader sees indexer commits | CODE | `TeleportSearcher::search` calls `reload()` | | 8 | Sequence 0 advances BM25 checkpoint | UNIT | `test_process_batch_sequence_zero_advances_checkpoint` | + +## 60-03 (this branch) + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1 | ≥15 semantic tests; hit files share no query tokens | UNIT | `semantic_hits_do_not_contain_query_tokens` | +| 2 | BM25 recall@5 < 0.4 on the paraphrase set | RUN | `semantic-bm25.json` recall@5 = 0.00 (0/16); `semantic_fixtures_bm25_below_point_four_vector_wins` | +| 3 | Vector / hybrid beat BM25 | RUN | `semantic-vector.json` 1.00 16/16; `semantic-hybrid.json` 1.00 16/16 | +| 4 | `--layers` is a custom-harness mock switch | CODE | `RetrievalLayer`; CLI search still RouteQuery | +| 5 | Purity + ARI on `TopicExtractor::cluster` | RUN | `topics-quality.json`; `metrics` hand-computed 3-cluster (purity 8/9, ARI 4.5/7) | +| 6 | README vector/topic rows cite artifacts | DOCS | README status table; positioning Claims Ledger | \ No newline at end of file diff --git a/README.md b/README.md index 63c2924..2d6d002 100644 --- a/README.md +++ b/README.md @@ -164,8 +164,8 @@ is experimental. | TOC build and drill-down navigation | **Solid** | Year → Month → Week → Day → Segment → Grip | | Grips / provenance | **Solid** | Excerpts link back to the events they came from | | BM25 keyword search (Tantivy) | **Solid** | Exact tokens, no stemming (`jwt` does not match `JWTs`). Events indexed before v3.1 have empty `text_preview` and there is no backfill command — see [UPGRADING](docs/UPGRADING.md) and [#41](https://github.com/SpillwaveSolutions/agent-memory/issues/41) | -| Vector search (HNSW + Candle) | **Solid** | Mechanism is wired; retrieval *quality* is not yet measured ([#40](https://github.com/SpillwaveSolutions/agent-memory/issues/40)). First daemon start downloads the embedding model; with no network the daemon warns and runs BM25-only | -| Topic graph | **Works** | Clustering quality is not benchmarked ([#47](https://github.com/SpillwaveSolutions/agent-memory/issues/47)) | +| Vector search (HNSW + Candle) | **Solid** | Mechanism is wired. Mock-harness hybrid recall@5 = **1.00** vs BM25 **0.00** on the 16-test paraphrase set ([`semantic-hybrid.json`](benchmarks/results/semantic-hybrid.json)); the vector *layer in that artifact* is a committed lexicon + TF-IDF cosine, not Candle ([#40](https://github.com/SpillwaveSolutions/agent-memory/issues/40)). First daemon start downloads the embedding model; with no network the daemon warns and runs BM25-only | +| Topic graph | **Works** | ARI **1.00** / purity **1.00** on a synthetic 80-doc TF-IDF corpus ([`topics-quality.json`](benchmarks/results/topics-quality.json)), not Candle TOC embeddings ([#47](https://github.com/SpillwaveSolutions/agent-memory/issues/47)) | | Hybrid fusion + `RouteQuery` orchestration | **Works** | Wired end-to-end in Phase 54; explainability reports what actually ran | | LLM summarization / LLM rerank | **Experimental** | Needs an API key; fails open to the heuristic ranker and reports `rerank=heuristic` when it does | | Cross-project federated query | **Experimental** | Implemented; not performance-characterised | @@ -180,9 +180,11 @@ is experimental. the old "65 second TOC" number was a harness defect (it timed ingest-time rollup and labelled it navigation). -Committed results live in `benchmarks/results/`. Today both are **mock-backend** -runs — a mock retrieval backend and a mock judge — so they demonstrate the -harness, not competitive quality. **There is deliberately no comparison +Committed results live in `benchmarks/results/`. `custom-harness-mock.json` and +`locomo-smoke.json` are **mock-backend** runs — they demonstrate the harness, +not competitive quality. `semantic-{bm25,vector,hybrid}.json` are also mock +(token-overlap vs a committed paraphrase lexicon); they exist so the vector +row has a BM25-cannot-solve baseline. **There is deliberately no comparison marketing in this repo**, and there will not be until a real-backend, real-judge run is committed next to the claim ([#39](https://github.com/SpillwaveSolutions/agent-memory/issues/39)). diff --git a/benchmarks/fixtures/semantic-001.toml b/benchmarks/fixtures/semantic-001.toml new file mode 100644 index 0000000..d7fe945 --- /dev/null +++ b/benchmarks/fixtures/semantic-001.toml @@ -0,0 +1,179 @@ +# QUAL-01 paraphrase set. Relevant sessions share meaning with the query +# but not tokens. First setup file is the hit; the rest are distractors +# whose query words are separated so BM25 ranks them and the lexicon does not fire. + +[[test]] +id = "semantic-001" +category = "semantic" +description = "Paraphrase: token expiry policy → JWT lifetime" +setup = ["sessions/semantic-001-hit.jsonl", "sessions/semantic-001-distract.jsonl"] +query = "token expiry policy" +expected_contains = ["JWT lifetime"] +relevant = ["JWT lifetime", "fifteen minutes"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-002" +category = "semantic" +description = "Paraphrase: container orchestration cutover → EKS migration" +setup = ["sessions/semantic-002-hit.jsonl", "sessions/semantic-002-distract.jsonl"] +query = "container orchestration cutover" +expected_contains = ["EKS migration"] +relevant = ["EKS migration", "Karpenter"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-003" +category = "semantic" +description = "Paraphrase: distributed tracing vendor → OpenTelemetry" +setup = ["sessions/semantic-003-hit.jsonl", "sessions/semantic-003-distract.jsonl"] +query = "distributed tracing vendor" +expected_contains = ["OpenTelemetry"] +relevant = ["OpenTelemetry", "Grafana Tempo"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-004" +category = "semantic" +description = "Paraphrase: feature toggle SaaS → Unleash self-hosted" +setup = ["sessions/semantic-004-hit.jsonl", "sessions/semantic-004-distract.jsonl"] +query = "feature toggle SaaS" +expected_contains = ["Unleash"] +relevant = ["Unleash", "LaunchDarkly"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-005" +category = "semantic" +description = "Paraphrase: background job persistence → SKIP LOCKED" +setup = ["sessions/semantic-005-hit.jsonl", "sessions/semantic-005-distract.jsonl"] +query = "background job persistence" +expected_contains = ["SKIP LOCKED"] +relevant = ["SKIP LOCKED", "Redis lists"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-006" +category = "semantic" +description = "Paraphrase: gateway throttle quota → 120 requests per minute" +setup = ["sessions/semantic-006-hit.jsonl", "sessions/semantic-006-distract.jsonl"] +query = "gateway throttle quota" +expected_contains = ["120 requests per minute"] +relevant = ["120 requests per minute", "20/min"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-007" +category = "semantic" +description = "Paraphrase: session cache duration → fifteen-minute TTL" +setup = ["sessions/semantic-007-hit.jsonl", "sessions/semantic-007-distract.jsonl"] +query = "session cache duration" +expected_contains = ["fifteen-minute default ttl"] +relevant = ["fifteen-minute default ttl", "look-aside"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-008" +category = "semantic" +description = "Paraphrase: primary pager rotation → Avery first on-call" +setup = ["sessions/semantic-008-hit.jsonl", "sessions/semantic-008-distract.jsonl"] +query = "primary pager rotation" +expected_contains = ["Avery"] +relevant = ["Avery", "Friday 16:00"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-009" +category = "semantic" +description = "Paraphrase: null avatar crash → empty Option / DEFAULT_PHOTO_URL" +setup = ["sessions/semantic-009-hit.jsonl", "sessions/semantic-009-distract.jsonl"] +query = "null avatar crash" +expected_contains = ["DEFAULT_PHOTO_URL"] +relevant = ["DEFAULT_PHOTO_URL", "empty Option"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-010" +category = "semantic" +description = "Paraphrase: log shipping backend → Promtail + Loki" +setup = ["sessions/semantic-010-hit.jsonl", "sessions/semantic-010-distract.jsonl"] +query = "log shipping backend" +expected_contains = ["Promtail"] +relevant = ["Promtail", "Loki"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-011" +category = "semantic" +description = "Paraphrase: schema migration utility → Atlas apply" +setup = ["sessions/semantic-011-hit.jsonl", "sessions/semantic-011-distract.jsonl"] +query = "schema migration utility" +expected_contains = ["Atlas apply"] +relevant = ["Atlas apply", "expand-contract"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-012" +category = "semantic" +description = "Paraphrase: secret storage backend → sops plus age" +setup = ["sessions/semantic-012-hit.jsonl", "sessions/semantic-012-distract.jsonl"] +query = "secret storage backend" +expected_contains = ["age encryption"] +relevant = ["age encryption", "Vault is out"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-013" +category = "semantic" +description = "Paraphrase: blue green release → Argo Rollouts canary" +setup = ["sessions/semantic-013-hit.jsonl", "sessions/semantic-013-distract.jsonl"] +query = "blue green release" +expected_contains = ["Argo Rollouts"] +relevant = ["Argo Rollouts", "canary"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-014" +category = "semantic" +description = "Paraphrase: search ranking algorithm → BM25 then cross-encoder" +setup = ["sessions/semantic-014-hit.jsonl", "sessions/semantic-014-distract.jsonl"] +query = "search ranking algorithm" +expected_contains = ["first-pass BM25"] +relevant = ["first-pass BM25", "cross-encoder"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-015" +category = "semantic" +description = "Paraphrase: object storage lifecycle → S3 intelligent-tiering" +setup = ["sessions/semantic-015-hit.jsonl", "sessions/semantic-015-distract.jsonl"] +query = "object storage lifecycle" +expected_contains = ["intelligent-tiering"] +relevant = ["intelligent-tiering", "Glacier after 30 days"] +k = 5 +max_tokens = 400 + +[[test]] +id = "semantic-016" +category = "semantic" +description = "Paraphrase: identity provider cutover → Keycloak replaces Auth0" +setup = ["sessions/semantic-016-hit.jsonl", "sessions/semantic-016-distract.jsonl"] +query = "identity provider cutover" +expected_contains = ["Keycloak"] +relevant = ["Keycloak", "SAML mappings"] +k = 5 +max_tokens = 400 diff --git a/benchmarks/fixtures/sessions/semantic-001-distract.jsonl b/benchmarks/fixtures/sessions/semantic-001-distract.jsonl new file mode 100644 index 0000000..8b05cd4 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-001-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "The cafeteria token of appreciation, and the HR policy, have an expiry of one year."} +{"role": "user", "content": "Parking tokens and the visitor policy share an expiry date in June."} +{"role": "user", "content": "A gift-token scheme, a refund policy, and milk expiry in the fridge."} +{"role": "user", "content": "Token booths, a museum policy binder, and the expiry of a coupon."} +{"role": "user", "content": "The policy on lunch tokens ignores expiry of dessert vouchers."} +{"role": "user", "content": "Office tokens for the printer sit under a policy with no expiry at all."} +{"role": "user", "content": "A token from the front desk, a quiet policy memo, and an expiry sticker on yogurt."} diff --git a/benchmarks/fixtures/sessions/semantic-001-hit.jsonl b/benchmarks/fixtures/sessions/semantic-001-hit.jsonl new file mode 100644 index 0000000..55c0d47 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-001-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "We need a short-lived access credential plan for the API."} +{"role": "assistant", "content": "We set JWT lifetime to fifteen minutes with rotating refresh credentials."} +{"role": "assistant", "content": "Access JWTs last fifteen minutes; rotating refresh credentials replace the family on each use."} +{"role": "assistant", "content": "I'll start the jose signer and store the refresh family in Postgres."} diff --git a/benchmarks/fixtures/sessions/semantic-002-distract.jsonl b/benchmarks/fixtures/sessions/semantic-002-distract.jsonl new file mode 100644 index 0000000..5f67662 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-002-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "The shipping container sat in orchestration of a brass band; the cutover of the parade was noon."} +{"role": "user", "content": "A lunch container, an orchestration of salads, and a cutover from forks to spoons."} +{"role": "user", "content": "The recycling container and the orchestration of bins missed the cutover to compost."} +{"role": "user", "content": "A container of crayons, an orchestration of colors, and a cutover of art class."} +{"role": "user", "content": "The leftover container in the fridge, an orchestration of leftovers, a cutover of menus."} +{"role": "user", "content": "A storage container of cables, an orchestration of labels, and a cutover of shelves."} +{"role": "user", "content": "The picnic container, an orchestration of blankets, and the cutover of the park gate."} diff --git a/benchmarks/fixtures/sessions/semantic-002-hit.jsonl b/benchmarks/fixtures/sessions/semantic-002-hit.jsonl new file mode 100644 index 0000000..8261b1f --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-002-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "How are we moving the fleet off the old cluster?"} +{"role": "assistant", "content": "EKS migration with Karpenter node provisioning is scheduled for 12 February 2026."} +{"role": "assistant", "content": "Karpenter node provisioning replaces the ASG; 12 February 2026 is the freeze date."} +{"role": "assistant", "content": "I'll file the EKS migration runbook under infra/eks."} diff --git a/benchmarks/fixtures/sessions/semantic-003-distract.jsonl b/benchmarks/fixtures/sessions/semantic-003-distract.jsonl new file mode 100644 index 0000000..cbe2432 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-003-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A distributed pile of mail, tracing who sent it, and a vendor of stamps at the kiosk."} +{"role": "user", "content": "The distributed picnic map, tracing the trail, and a vendor selling maps."} +{"role": "user", "content": "Distributed leftovers, tracing who ate them, and a vendor of Tupperware."} +{"role": "user", "content": "A distributed seating chart, tracing cousins, and a vendor of place cards."} +{"role": "user", "content": "Distributed chores, tracing who mopped, and a vendor of mops in aisle four."} +{"role": "user", "content": "The distributed photo album, tracing faces, and a vendor of frames."} +{"role": "user", "content": "Distributed homework, tracing who copied it, and a vendor of pencils."} diff --git a/benchmarks/fixtures/sessions/semantic-003-hit.jsonl b/benchmarks/fixtures/sessions/semantic-003-hit.jsonl new file mode 100644 index 0000000..a276454 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-003-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "What do we emit spans with, and where do they land?"} +{"role": "assistant", "content": "OpenTelemetry into Grafana Tempo at a 5% sample rate is the standard."} +{"role": "assistant", "content": "Grafana Tempo holds the traces; keep the 5% sample rate unless an incident needs more."} +{"role": "assistant", "content": "I'll add the OpenTelemetry SDK to the services this week."} diff --git a/benchmarks/fixtures/sessions/semantic-004-distract.jsonl b/benchmarks/fixtures/sessions/semantic-004-distract.jsonl new file mode 100644 index 0000000..9a0f518 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-004-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A magazine feature on a light toggle, and a SaaS brochure in the lobby."} +{"role": "user", "content": "The documentary feature, a toggle on the mixer, and a SaaS sticker on a laptop."} +{"role": "user", "content": "A newspaper feature, the porch light toggle, and SaaS mugs in the kitchen."} +{"role": "user", "content": "The cover feature, a toggle switch on a lamp, and a SaaS tote bag."} +{"role": "user", "content": "A sports feature, a toggle on the scoreboard, and SaaS pens at the desk."} +{"role": "user", "content": "The obituary feature, a toggle for the porch, and a SaaS fridge magnet."} +{"role": "user", "content": "A travel feature, a toggle on a flashlight, and SaaS coasters."} diff --git a/benchmarks/fixtures/sessions/semantic-004-hit.jsonl b/benchmarks/fixtures/sessions/semantic-004-hit.jsonl new file mode 100644 index 0000000..9e84db0 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-004-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Did we buy LaunchDarkly or run something ourselves?"} +{"role": "assistant", "content": "Unleash is self-hosted. We rejected LaunchDarkly on price."} +{"role": "assistant", "content": "Flags go through Unleash; LaunchDarkly is off the table."} +{"role": "assistant", "content": "I'll wire the Unleash client in the API tomorrow."} diff --git a/benchmarks/fixtures/sessions/semantic-005-distract.jsonl b/benchmarks/fixtures/sessions/semantic-005-distract.jsonl new file mode 100644 index 0000000..b85dfdf --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-005-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A background of mountains, a job at the bakery, and persistence of crumbs on the counter."} +{"role": "user", "content": "The wallpaper background, a summer job, and the persistence of glitter."} +{"role": "user", "content": "A Zoom background of a beach, a job in catering, and persistence of sand in shoes."} +{"role": "user", "content": "The desktop background of cats, a job walking dogs, and persistence of fur."} +{"role": "user", "content": "A stage background of trees, a job at the theater, and persistence of tickets."} +{"role": "user", "content": "The poster background of stars, a job at the planetarium, and persistence of stickers."} +{"role": "user", "content": "A classroom background of maps, a job tutoring, and persistence of chalk."} diff --git a/benchmarks/fixtures/sessions/semantic-005-hit.jsonl b/benchmarks/fixtures/sessions/semantic-005-hit.jsonl new file mode 100644 index 0000000..f343033 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-005-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Are we using Redis lists for the worker queue?"} +{"role": "assistant", "content": "No. Postgres SKIP LOCKED is the queue. Redis lists are out."} +{"role": "assistant", "content": "Workers claim rows with SKIP LOCKED; Redis lists were a prototype only."} +{"role": "assistant", "content": "I'll add the SKIP LOCKED claim query and drop the Redis list writer."} diff --git a/benchmarks/fixtures/sessions/semantic-006-distract.jsonl b/benchmarks/fixtures/sessions/semantic-006-distract.jsonl new file mode 100644 index 0000000..c84ca35 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-006-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "The garden gateway, a throttle on the hose, and a quota of flowers per bed."} +{"role": "user", "content": "A wooden gateway, the throttle of a lawnmower, and a quota of clippings."} +{"role": "user", "content": "The ferry gateway, a throttle on the engine, and a quota of passengers."} +{"role": "user", "content": "A stadium gateway, the throttle of the PA, and a quota of foam fingers."} +{"role": "user", "content": "The zoo gateway, a throttle on the tram, and a quota of popcorn."} +{"role": "user", "content": "A campus gateway, the throttle of a shuttle, and a quota of bikes."} +{"role": "user", "content": "The hotel gateway, a throttle on the elevator, and a quota of towels."} diff --git a/benchmarks/fixtures/sessions/semantic-006-hit.jsonl b/benchmarks/fixtures/sessions/semantic-006-hit.jsonl new file mode 100644 index 0000000..7ff9698 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-006-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "What is the public API rate for partner credentials?"} +{"role": "assistant", "content": "120 requests per minute for public keys; 20/min for partner credentials."} +{"role": "assistant", "content": "Partner credentials stay at 20/min. Public stays at 120 requests per minute."} +{"role": "assistant", "content": "I'll put 120 requests per minute in the nginx snippet."} diff --git a/benchmarks/fixtures/sessions/semantic-007-distract.jsonl b/benchmarks/fixtures/sessions/semantic-007-distract.jsonl new file mode 100644 index 0000000..4582ab4 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-007-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A yoga session, a cache of blankets, and the duration of the playlist."} +{"role": "user", "content": "The recording session, a cache of snacks, and duration of the take."} +{"role": "user", "content": "A council session, a cache of nameplates, and the duration of coffee."} +{"role": "user", "content": "The jam session, a cache of picks, and duration of the encore."} +{"role": "user", "content": "A study session, a cache of highlighters, and the duration of the quiz."} +{"role": "user", "content": "The photo session, a cache of umbrellas, and duration of golden hour."} +{"role": "user", "content": "A therapy session, a cache of tissues, and the duration of the hour."} diff --git a/benchmarks/fixtures/sessions/semantic-007-hit.jsonl b/benchmarks/fixtures/sessions/semantic-007-hit.jsonl new file mode 100644 index 0000000..3f17041 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-007-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "What TTL did we pick for Redis look-aside?"} +{"role": "assistant", "content": "A fifteen-minute default ttl on Redis look-aside. No longer."} +{"role": "assistant", "content": "Redis look-aside with a fifteen-minute default ttl is the written plan."} +{"role": "assistant", "content": "I'll set the look-aside client to fifteen-minute default ttl."} diff --git a/benchmarks/fixtures/sessions/semantic-008-distract.jsonl b/benchmarks/fixtures/sessions/semantic-008-distract.jsonl new file mode 100644 index 0000000..a6fe705 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-008-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "The primary crayon, a pager of a children's book, and rotation of story time."} +{"role": "user", "content": "A primary color poster, the pager at a diner, and rotation of pies."} +{"role": "user", "content": "The primary speaker, a pager of notes, and rotation of microphones."} +{"role": "user", "content": "A primary school mural, the pager of a yearbook, and rotation of class photos."} +{"role": "user", "content": "The primary hallway, a pager of flyers, and rotation of lockers."} +{"role": "user", "content": "A primary ballot, the pager of a newsletter, and rotation of bylines."} +{"role": "user", "content": "The primary desk, a pager of stickers, and rotation of name tags."} diff --git a/benchmarks/fixtures/sessions/semantic-008-hit.jsonl b/benchmarks/fixtures/sessions/semantic-008-hit.jsonl new file mode 100644 index 0000000..06a9932 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-008-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Who is first on-call the week of 3 March?"} +{"role": "assistant", "content": "Avery takes first on-call Friday 16:00 the week of 3 March."} +{"role": "assistant", "content": "Hand-off is Friday 16:00. Avery takes first on-call that week of 3 March."} +{"role": "assistant", "content": "I'll add Avery to the week of 3 March calendar invite."} diff --git a/benchmarks/fixtures/sessions/semantic-009-distract.jsonl b/benchmarks/fixtures/sessions/semantic-009-distract.jsonl new file mode 100644 index 0000000..b92c84a --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-009-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A null result in bingo, an avatar on a cake, and a crash of cymbals."} +{"role": "user", "content": "The null set in a math joke, an avatar on a mug, and a crash of dishes."} +{"role": "user", "content": "A null score in darts, the avatar on a hoodie, and a crash of bowling pins."} +{"role": "user", "content": "The null hypothesis poster, an avatar sticker, and a crash of toy cars."} +{"role": "user", "content": "A null pointer in a cartoon, the avatar on a balloon, and a crash of blocks."} +{"role": "user", "content": "The null island postcard, an avatar pin, and a crash of cymbals at band."} +{"role": "user", "content": "A null entry in a raffle, the avatar on a tote, and a crash of pans."} diff --git a/benchmarks/fixtures/sessions/semantic-009-hit.jsonl b/benchmarks/fixtures/sessions/semantic-009-hit.jsonl new file mode 100644 index 0000000..ba4b124 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-009-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Why did the user profile handler panic last week?"} +{"role": "assistant", "content": "An empty Option on DEFAULT_PHOTO_URL. The profile handler panicked."} +{"role": "assistant", "content": "DEFAULT_PHOTO_URL was missing; unwrap on an empty Option made the profile handler panicked."} +{"role": "assistant", "content": "I'll return a placeholder when DEFAULT_PHOTO_URL is an empty Option."} diff --git a/benchmarks/fixtures/sessions/semantic-010-distract.jsonl b/benchmarks/fixtures/sessions/semantic-010-distract.jsonl new file mode 100644 index 0000000..7369829 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-010-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A log in the fireplace, shipping a postcard, and a backend of the diner."} +{"role": "user", "content": "The yule log on TV, shipping of cookies, and a backend of the bakery."} +{"role": "user", "content": "A cabin log book, shipping of sweaters, and the backend of a thrift shop."} +{"role": "user", "content": "The captain's log in a play, shipping of props, and a backend of the stage."} +{"role": "user", "content": "A log of attendees, shipping of name tags, and the backend of a closet."} +{"role": "user", "content": "The firewood log, shipping of marshmallows, and a backend of the campsite."} +{"role": "user", "content": "A log cabin kit, shipping of instructions, and the backend of a garage."} diff --git a/benchmarks/fixtures/sessions/semantic-010-hit.jsonl b/benchmarks/fixtures/sessions/semantic-010-hit.jsonl new file mode 100644 index 0000000..bafa2f4 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-010-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Where do we send application lines after we dropped Fluent Bit?"} +{"role": "assistant", "content": "Promtail into Loki. Fluent Bit is retired."} +{"role": "assistant", "content": "Loki is the store; Promtail is the shipper. Fluent Bit stays off."} +{"role": "assistant", "content": "I'll swap the Fluent Bit DaemonSet for Promtail."} diff --git a/benchmarks/fixtures/sessions/semantic-011-distract.jsonl b/benchmarks/fixtures/sessions/semantic-011-distract.jsonl new file mode 100644 index 0000000..dd28069 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-011-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A schema of a sitcom, a migration of birds, and a utility closet of brooms."} +{"role": "user", "content": "The rhyme schema of a poem, a migration of geese, and a utility bill on the fridge."} +{"role": "user", "content": "A seating schema at a wedding, a migration of chairs, and a utility sink."} +{"role": "user", "content": "The color schema of a quilt, a migration of patterns, and a utility drawer."} +{"role": "user", "content": "A plot schema, a migration of chapters, and the utility of a bookmark."} +{"role": "user", "content": "The lighting schema of a play, a migration of lamps, and a utility cart."} +{"role": "user", "content": "A chord schema, a migration of keys, and a utility strap on a guitar."} diff --git a/benchmarks/fixtures/sessions/semantic-011-hit.jsonl b/benchmarks/fixtures/sessions/semantic-011-hit.jsonl new file mode 100644 index 0000000..a111fb6 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-011-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "How do we change Postgres tables without a long lock?"} +{"role": "assistant", "content": "Atlas apply with expand-contract. No other tool."} +{"role": "assistant", "content": "expand-contract on Postgres tables, driven by Atlas apply."} +{"role": "assistant", "content": "I'll write the Atlas apply file for the new column."} diff --git a/benchmarks/fixtures/sessions/semantic-012-distract.jsonl b/benchmarks/fixtures/sessions/semantic-012-distract.jsonl new file mode 100644 index 0000000..2e7ec4d --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-012-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A secret recipe, storage of spices, and a backend of the pantry."} +{"role": "user", "content": "The secret handshake, storage of hats, and a backend of the coat room."} +{"role": "user", "content": "A secret Santa list, storage of wrapping, and the backend of a closet."} +{"role": "user", "content": "The secret garden path, storage of seeds, and a backend of the shed."} +{"role": "user", "content": "A secret menu item, storage of napkins, and the backend of a cafe."} +{"role": "user", "content": "The secret bookmark, storage of novels, and a backend of the library."} +{"role": "user", "content": "A secret ingredient, storage of jars, and the backend of a kitchen."} diff --git a/benchmarks/fixtures/sessions/semantic-012-hit.jsonl b/benchmarks/fixtures/sessions/semantic-012-hit.jsonl new file mode 100644 index 0000000..90432a5 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-012-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Are we putting credentials in Vault?"} +{"role": "assistant", "content": "No. SOPS plus age encryption. Vault is out."} +{"role": "assistant", "content": "age encryption via SOPS. Vault is out of the design."} +{"role": "assistant", "content": "I'll encrypt the env file with SOPS plus age encryption."} diff --git a/benchmarks/fixtures/sessions/semantic-013-distract.jsonl b/benchmarks/fixtures/sessions/semantic-013-distract.jsonl new file mode 100644 index 0000000..beb5d7f --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-013-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A blue balloon, a green balloon, and a press release about the fair."} +{"role": "user", "content": "The blue mug, a green mug, and a release of doves at noon."} +{"role": "user", "content": "A blue ribbon, a green ribbon, and the release of the yearbook."} +{"role": "user", "content": "The blue tent, a green tent, and a release of lanterns."} +{"role": "user", "content": "A blue scarf, a green scarf, and the release of a playlist."} +{"role": "user", "content": "The blue poster, a green poster, and a release of stickers."} +{"role": "user", "content": "A blue chair, a green chair, and the release of the seating chart."} diff --git a/benchmarks/fixtures/sessions/semantic-013-hit.jsonl b/benchmarks/fixtures/sessions/semantic-013-hit.jsonl new file mode 100644 index 0000000..e7b31d4 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-013-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Are we still doing full environment swaps?"} +{"role": "assistant", "content": "No. Argo Rollouts canary. We abandoned full swaps."} +{"role": "assistant", "content": "Argo Rollouts with a canary step. Full swaps are abandoned."} +{"role": "assistant", "content": "I'll add the Argo Rollouts CR and a 10% canary weight."} diff --git a/benchmarks/fixtures/sessions/semantic-014-distract.jsonl b/benchmarks/fixtures/sessions/semantic-014-distract.jsonl new file mode 100644 index 0000000..b75cd0b --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-014-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "A search for lost keys, ranking of chores, and an algorithm for packing."} +{"role": "user", "content": "The search for a remote, ranking of cushions, and an algorithm for snacks."} +{"role": "user", "content": "A search of the attic, ranking of boxes, and an algorithm for labels."} +{"role": "user", "content": "The search for socks, ranking of drawers, and an algorithm for laundry."} +{"role": "user", "content": "A search of the fridge, ranking of leftovers, and an algorithm for lunch."} +{"role": "user", "content": "The search for tickets, ranking of pockets, and an algorithm for reminders."} +{"role": "user", "content": "A search of bookmarks, ranking of tabs, and an algorithm for bedtime."} diff --git a/benchmarks/fixtures/sessions/semantic-014-hit.jsonl b/benchmarks/fixtures/sessions/semantic-014-hit.jsonl new file mode 100644 index 0000000..fb71714 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-014-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "How do we order hits before any LLM sees them?"} +{"role": "assistant", "content": "First-pass BM25 then a cross-encoder rerank."} +{"role": "assistant", "content": "BM25 then a cross-encoder. That is the written first-pass BM25 plan."} +{"role": "assistant", "content": "I'll keep first-pass BM25 and add the cross-encoder rerank hook."} diff --git a/benchmarks/fixtures/sessions/semantic-015-distract.jsonl b/benchmarks/fixtures/sessions/semantic-015-distract.jsonl new file mode 100644 index 0000000..d13fa0c --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-015-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "An object on a shelf, storage of board games, and the lifecycle of a houseplant."} +{"role": "user", "content": "The object of a scavenger hunt, storage of clues, and a lifecycle of stickers."} +{"role": "user", "content": "An object lesson, storage of markers, and the lifecycle of a poster."} +{"role": "user", "content": "The object of a bet, storage of chips, and a lifecycle of snacks."} +{"role": "user", "content": "An object in a still life, storage of brushes, and the lifecycle of paint."} +{"role": "user", "content": "The object of the joke, storage of punchlines, and a lifecycle of memes."} +{"role": "user", "content": "An object on the mantle, storage of photos, and the lifecycle of candles."} diff --git a/benchmarks/fixtures/sessions/semantic-015-hit.jsonl b/benchmarks/fixtures/sessions/semantic-015-hit.jsonl new file mode 100644 index 0000000..5e20420 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-015-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Where do cold blobs go after a month?"} +{"role": "assistant", "content": "S3 intelligent-tiering, then Glacier after 30 days."} +{"role": "assistant", "content": "intelligent-tiering is on. Glacier after 30 days is the cold step."} +{"role": "assistant", "content": "I'll enable S3 intelligent-tiering on the bucket."} diff --git a/benchmarks/fixtures/sessions/semantic-016-distract.jsonl b/benchmarks/fixtures/sessions/semantic-016-distract.jsonl new file mode 100644 index 0000000..ee77a32 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-016-distract.jsonl @@ -0,0 +1,7 @@ +{"role": "user", "content": "An identity of a mascot, a provider of snacks, and a cutover of the bake sale."} +{"role": "user", "content": "The identity of a choir, a provider of sheet music, and a cutover of rehearsals."} +{"role": "user", "content": "An identity of a club, a provider of pins, and the cutover of a meeting night."} +{"role": "user", "content": "The identity of a team, a provider of jerseys, and a cutover of captains."} +{"role": "user", "content": "An identity of a band, a provider of picks, and the cutover of a set list."} +{"role": "user", "content": "The identity of a class, a provider of nametags, and a cutover of seating."} +{"role": "user", "content": "An identity of a festival, a provider of maps, and the cutover of stages."} diff --git a/benchmarks/fixtures/sessions/semantic-016-hit.jsonl b/benchmarks/fixtures/sessions/semantic-016-hit.jsonl new file mode 100644 index 0000000..9256953 --- /dev/null +++ b/benchmarks/fixtures/sessions/semantic-016-hit.jsonl @@ -0,0 +1,4 @@ +{"role": "user", "content": "Are we staying on Auth0?"} +{"role": "assistant", "content": "Keycloak replaces Auth0. SAML mappings stay."} +{"role": "assistant", "content": "SAML mappings stay as-is. Keycloak replaces Auth0 next quarter."} +{"role": "assistant", "content": "I'll stand up Keycloak and copy the SAML mappings."} diff --git a/benchmarks/results/semantic-bm25.json b/benchmarks/results/semantic-bm25.json new file mode 100644 index 0000000..88b3fa5 --- /dev/null +++ b/benchmarks/results/semantic-bm25.json @@ -0,0 +1,39 @@ +{ + "backend": "mock", + "accuracy": 0.0, + "recall_at_k": 0.0, + "k": 5, + "token_usage_avg": 99, + "latency_p50_ms": 0, + "latency_p95_ms": 0, + "compression_ratio": 0.6394269377663698, + "test_count": 16, + "pass_count": 0, + "failed_ids": [ + "semantic-001", + "semantic-002", + "semantic-003", + "semantic-004", + "semantic-005", + "semantic-006", + "semantic-007", + "semantic-008", + "semantic-009", + "semantic-010", + "semantic-011", + "semantic-012", + "semantic-013", + "semantic-014", + "semantic-015", + "semantic-016" + ], + "caveats": [ + "accuracy is expected_contains over this fixture suite, not LOCOMO", + "recall@k uses labeled relevant items in top-k, not accuracy under another name", + "compression_ratio compares retrieved tokens to setup file contents (not path strings)", + "each test ran against a fresh store (mock isolation; cli shares the daemon unless you restart it)", + "backend=mock uses in-process token-overlap retrieval; not a production quality number", + "--layers is a mock-backend switch; CLI search is always RouteQuery hybrid" + ], + "layers": "bm25" +} \ No newline at end of file diff --git a/benchmarks/results/semantic-hybrid.json b/benchmarks/results/semantic-hybrid.json new file mode 100644 index 0000000..ef84433 --- /dev/null +++ b/benchmarks/results/semantic-hybrid.json @@ -0,0 +1,21 @@ +{ + "backend": "mock", + "accuracy": 1.0, + "recall_at_k": 1.0, + "k": 5, + "token_usage_avg": 95, + "latency_p50_ms": 0, + "latency_p95_ms": 0, + "compression_ratio": 0.6529811743896524, + "test_count": 16, + "pass_count": 16, + "caveats": [ + "accuracy is expected_contains over this fixture suite, not LOCOMO", + "recall@k uses labeled relevant items in top-k, not accuracy under another name", + "compression_ratio compares retrieved tokens to setup file contents (not path strings)", + "each test ran against a fresh store (mock isolation; cli shares the daemon unless you restart it)", + "backend=mock hybrid is RRF (k=60) of token-overlap and lexicon TF-IDF; not RouteQuery", + "--layers is a mock-backend switch; CLI search is always RouteQuery hybrid" + ], + "layers": "hybrid" +} \ No newline at end of file diff --git a/benchmarks/results/semantic-vector.json b/benchmarks/results/semantic-vector.json new file mode 100644 index 0000000..01032b8 --- /dev/null +++ b/benchmarks/results/semantic-vector.json @@ -0,0 +1,21 @@ +{ + "backend": "mock", + "accuracy": 1.0, + "recall_at_k": 1.0, + "k": 5, + "token_usage_avg": 91, + "latency_p50_ms": 0, + "latency_p95_ms": 0, + "compression_ratio": 0.6692786454300235, + "test_count": 16, + "pass_count": 16, + "caveats": [ + "accuracy is expected_contains over this fixture suite, not LOCOMO", + "recall@k uses labeled relevant items in top-k, not accuracy under another name", + "compression_ratio compares retrieved tokens to setup file contents (not path strings)", + "each test ran against a fresh store (mock isolation; cli shares the daemon unless you restart it)", + "backend=mock vector is a committed paraphrase lexicon plus TF-IDF cosine, not Candle/HNSW", + "--layers is a mock-backend switch; CLI search is always RouteQuery hybrid" + ], + "layers": "vector" +} \ No newline at end of file diff --git a/benchmarks/results/topics-quality.json b/benchmarks/results/topics-quality.json new file mode 100644 index 0000000..8396ec2 --- /dev/null +++ b/benchmarks/results/topics-quality.json @@ -0,0 +1,15 @@ +{ + "n_docs": 80, + "n_gold_clusters": 8, + "n_pred_clusters": 8, + "n_noise": 0, + "min_cluster_size": 5, + "embedding": "tfidf-l2-top32", + "purity": 1.0, + "adjusted_rand_index": 1.0, + "caveats": [ + "synthetic 80-doc / 8-cluster corpus, not live TOC summaries", + "vectors are capped TF-IDF (top 32 terms), not Candle embeddings", + "cluster() is HDBSCAN via TopicExtractor; noise label is -1" + ] +} diff --git a/crates/memory-bench/src/cli.rs b/crates/memory-bench/src/cli.rs index 42d2082..d8841c4 100644 --- a/crates/memory-bench/src/cli.rs +++ b/crates/memory-bench/src/cli.rs @@ -54,7 +54,7 @@ pub enum Commands { #[arg(long)] output: Option, }, - /// Run full custom benchmark suite (all categories). + /// Run full custom benchmark suite (all categories except semantic). All { /// Path to fixtures directory. #[arg(long, default_value = "benchmarks/fixtures")] @@ -68,6 +68,36 @@ pub enum Commands { /// Path to baselines TOML file. #[arg(long, default_value = "benchmarks/baselines.toml")] baselines: String, + /// Retrieval layer: `bm25` (default), `vector`, or `hybrid`. + #[arg(long, default_value = "bm25", value_parser = ["bm25", "vector", "hybrid"])] + layers: String, + }, + /// Run custom-harness fixtures, optionally filtered by category. + Run { + /// Path to fixtures directory. + #[arg(long, default_value = "benchmarks/fixtures")] + fixtures: String, + /// Output file for JSON results. + #[arg(long)] + output: Option, + /// Filter by test id/category prefix (e.g. `semantic`, `temporal`). + #[arg(long)] + category: Option, + /// Retrieval layer: `bm25` (default), `vector`, or `hybrid`. + #[arg(long, default_value = "bm25", value_parser = ["bm25", "vector", "hybrid"])] + layers: String, + }, + /// Run the semantic/paraphrase fixture set (QUAL-01). + Semantic { + /// Path to fixtures directory. + #[arg(long, default_value = "benchmarks/fixtures")] + fixtures: String, + /// Output file for JSON results. + #[arg(long)] + output: Option, + /// Retrieval layer: `bm25` (default), `vector`, or `hybrid`. + #[arg(long, default_value = "bm25", value_parser = ["bm25", "vector", "hybrid"])] + layers: String, }, /// Run LOCOMO adapter. Substring mode is `context_hit_rate`, not a LOCOMO score. Locomo { diff --git a/crates/memory-bench/src/fixture.rs b/crates/memory-bench/src/fixture.rs index f0c859e..b877b23 100644 --- a/crates/memory-bench/src/fixture.rs +++ b/crates/memory-bench/src/fixture.rs @@ -221,4 +221,56 @@ max_tokens = 300 assert_eq!(tests[1].id, "b-001"); assert_eq!(tests[2].id, "b-002"); } + + fn query_tokens(query: &str) -> Vec { + query + .split(|c: char| !c.is_alphanumeric()) + .filter(|t| t.len() > 1) + .map(|t| t.to_lowercase()) + .collect() + } + + #[test] + fn semantic_hits_do_not_contain_query_tokens() { + let dir = + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../benchmarks/fixtures"); + let tests = Fixture::load_dir(&dir).unwrap(); + let semantic: Vec<_> = tests + .into_iter() + .filter(|t| t.id.starts_with("semantic-") || t.category.as_deref() == Some("semantic")) + .collect(); + assert!( + semantic.len() >= 15, + "QUAL-01 requires ≥15 semantic tests, found {}", + semantic.len() + ); + for test in &semantic { + assert!(!test.setup.is_empty(), "{} needs a hit setup file", test.id); + let hit = crate::runner::resolve_setup(&dir, &test.setup[0]); + let hit_text = std::fs::read_to_string(&hit) + .unwrap_or_else(|e| panic!("reading {}: {e}", hit.display())) + .to_lowercase(); + for tok in query_tokens(&test.query) { + assert!( + !hit_text.contains(&tok), + "{} hit {} contains query token `{tok}`", + test.id, + hit.display() + ); + } + let phrase = test.query.to_lowercase(); + for setup in test.setup.iter().skip(1) { + let path = crate::runner::resolve_setup(&dir, setup); + let text = std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("reading {}: {e}", path.display())) + .to_lowercase(); + assert!( + !text.contains(&phrase), + "{} distractor {} contains contiguous query `{phrase}`", + test.id, + path.display() + ); + } + } + } } diff --git a/crates/memory-bench/src/layers.rs b/crates/memory-bench/src/layers.rs new file mode 100644 index 0000000..abd3a74 --- /dev/null +++ b/crates/memory-bench/src/layers.rs @@ -0,0 +1,207 @@ +//! Retrieval-layer switch for the custom harness (BENCH-13). +//! +//! Mock BM25 is token overlap. Mock vector is lexicon-expanded TF-IDF cosine. +//! Mock hybrid is RRF (k=60) of the two lists. CLI backend always calls +//! `memory search` (RouteQuery / hybrid); `--layers` is a mock-backend switch. + +use anyhow::{bail, Result}; +use std::collections::{HashMap, HashSet}; + +use crate::lexicon; + +/// Which retrieval layer the custom harness should drive. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RetrievalLayer { + Bm25, + Vector, + Hybrid, +} + +impl RetrievalLayer { + pub fn parse(s: &str) -> Result { + match s { + "bm25" => Ok(Self::Bm25), + "vector" => Ok(Self::Vector), + "hybrid" => Ok(Self::Hybrid), + other => bail!("unknown layers '{other}' (expected bm25|vector|hybrid)"), + } + } + + pub fn as_str(self) -> &'static str { + match self { + Self::Bm25 => "bm25", + Self::Vector => "vector", + Self::Hybrid => "hybrid", + } + } +} + +/// Rank `docs` for `query` under `layer`. Returns `(score, doc_index)` desc. +pub fn rank(docs: &[String], query: &str, layer: RetrievalLayer) -> Vec<(f64, usize)> { + match layer { + RetrievalLayer::Bm25 => bm25_rank(docs, query), + RetrievalLayer::Vector => vector_rank(docs, query), + RetrievalLayer::Hybrid => { + let a = bm25_rank(docs, query); + let b = vector_rank(docs, query); + rrf_merge(&[&a, &b], 60.0, docs.len()) + } + } +} + +fn tokenize(s: &str) -> Vec { + s.split(|c: char| !c.is_alphanumeric()) + .filter(|t| t.len() > 1) + .map(|t| t.to_lowercase()) + .collect() +} + +fn bm25_rank(docs: &[String], query: &str) -> Vec<(f64, usize)> { + let terms = tokenize(query); + let mut scored: Vec<(f64, usize)> = docs + .iter() + .enumerate() + .map(|(i, d)| { + let hay = d.to_lowercase(); + let score = terms.iter().filter(|t| hay.contains(t.as_str())).count() as f64; + (score, i) + }) + .collect(); + scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); + scored +} + +fn vector_rank(docs: &[String], query: &str) -> Vec<(f64, usize)> { + let q = lexicon::expand(query); + let expanded: Vec = docs.iter().map(|d| lexicon::expand(d)).collect(); + tfidf_cosine_rank(&expanded, &q) +} + +fn tfidf_cosine_rank(docs: &[String], query: &str) -> Vec<(f64, usize)> { + let q_toks = tokenize(query); + let doc_toks: Vec> = docs.iter().map(|d| tokenize(d)).collect(); + let mut df: HashMap = HashMap::new(); + for toks in &doc_toks { + let uniq: HashSet<&String> = toks.iter().collect(); + for t in uniq { + *df.entry(t.clone()).or_insert(0) += 1; + } + } + let n = docs.len() as f64; + let idf = |t: &str| -> f64 { + let d = *df.get(t).unwrap_or(&0) as f64; + if d == 0.0 { + 0.0 + } else { + ((n + 1.0) / (d + 1.0)).ln() + 1.0 + } + }; + let vec_of = |toks: &[String]| -> HashMap { + let mut tf: HashMap = HashMap::new(); + for t in toks { + *tf.entry(t.clone()).or_insert(0) += 1; + } + let len = toks.len().max(1) as f64; + tf.into_iter() + .map(|(t, c)| { + let v = (c as f64 / len) * idf(&t); + (t, v) + }) + .collect() + }; + let cosine = |a: &HashMap, b: &HashMap| -> f64 { + let mut dot = 0.0; + for (t, av) in a { + if let Some(bv) = b.get(t) { + dot += av * bv; + } + } + let na = a.values().map(|x| x * x).sum::().sqrt(); + let nb = b.values().map(|x| x * x).sum::().sqrt(); + if na == 0.0 || nb == 0.0 { + 0.0 + } else { + dot / (na * nb) + } + }; + let qv = vec_of(&q_toks); + let mut scored: Vec<(f64, usize)> = doc_toks + .iter() + .enumerate() + .map(|(i, toks)| (cosine(&qv, &vec_of(toks)), i)) + .collect(); + scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); + scored +} + +fn rrf_merge(lists: &[&Vec<(f64, usize)>], k_rrf: f64, n_docs: usize) -> Vec<(f64, usize)> { + let mut acc = vec![0.0; n_docs]; + for list in lists { + for (rank, (_score, idx)) in list.iter().enumerate() { + acc[*idx] += 1.0 / (k_rrf + rank as f64 + 1.0); + } + } + let mut out: Vec<(f64, usize)> = acc.into_iter().enumerate().map(|(i, s)| (s, i)).collect(); + out.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn pair() -> (Vec, &'static str) { + let docs = vec![ + "We set JWT lifetime to fifteen minutes with rotating refresh credentials.".into(), + "The cafeteria token of appreciation, and the HR policy, have an expiry of one year." + .into(), + "Parking tokens and the visitor policy share an expiry date in June.".into(), + "A gift-token scheme, a refund policy, and milk expiry in the fridge.".into(), + "Token booths, a museum policy binder, and the expiry of a coupon.".into(), + "The policy on lunch tokens ignores expiry of dessert vouchers.".into(), + "Office tokens for the printer sit under a policy with no expiry at all.".into(), + ]; + (docs, "token expiry policy") + } + + #[test] + fn parse_layers() { + assert_eq!(RetrievalLayer::parse("bm25").unwrap(), RetrievalLayer::Bm25); + assert_eq!( + RetrievalLayer::parse("vector").unwrap(), + RetrievalLayer::Vector + ); + assert_eq!( + RetrievalLayer::parse("hybrid").unwrap(), + RetrievalLayer::Hybrid + ); + assert!(RetrievalLayer::parse("ann").is_err()); + } + + #[test] + fn bm25_prefers_lexical_distractor() { + let (docs, q) = pair(); + let ranked = rank(&docs, q, RetrievalLayer::Bm25); + assert_eq!(ranked[0].1, 1, "top BM25 hit should be a distractor"); + let top5: Vec = ranked.iter().take(5).map(|(_, i)| *i).collect(); + assert!(!top5.contains(&0), "relevant doc must not be in BM25 top-5"); + } + + #[test] + fn vector_prefers_paraphrase() { + let (docs, q) = pair(); + let ranked = rank(&docs, q, RetrievalLayer::Vector); + assert_eq!(ranked[0].1, 0, "vector top hit should be the paraphrase"); + } + + #[test] + fn hybrid_puts_paraphrase_in_top5() { + let (docs, q) = pair(); + let ranked = rank(&docs, q, RetrievalLayer::Hybrid); + let top5: Vec = ranked.iter().take(5).map(|(_, i)| *i).collect(); + assert!( + top5.contains(&0), + "hybrid top-5 must include the paraphrase" + ); + } +} diff --git a/crates/memory-bench/src/lexicon.rs b/crates/memory-bench/src/lexicon.rs new file mode 100644 index 0000000..0747d9b --- /dev/null +++ b/crates/memory-bench/src/lexicon.rs @@ -0,0 +1,150 @@ +//! Committed paraphrase families for the mock vector layer (QUAL-01). +//! +//! If a query or document contains any phrase in a family, the mock vector +//! path appends every member of that family before TF-IDF cosine. This is a +//! fixture-level semantic stand-in, not Candle/HNSW. + +/// Each inner slice is one paraphrase family. Matching is case-insensitive +/// substring on the full phrase, not on individual tokens. +pub const FAMILIES: &[&[&str]] = &[ + &[ + "token expiry policy", + "jwt lifetime", + "fifteen minutes", + "rotating refresh credentials", + ], + &[ + "container orchestration cutover", + "eks migration", + "karpenter node provisioning", + "12 february 2026", + ], + &[ + "distributed tracing vendor", + "opentelemetry", + "grafana tempo", + "5% sample rate", + ], + &[ + "feature toggle saas", + "unleash is self-hosted", + "rejected launchdarkly", + "unleash", + ], + &[ + "background job persistence", + "skip locked", + "postgres skip locked", + "redis lists", + ], + &[ + "gateway throttle quota", + "120 requests per minute", + "20/min", + "partner credentials", + ], + &[ + "session cache duration", + "fifteen-minute default ttl", + "redis look-aside", + "look-aside", + ], + &[ + "primary pager rotation", + "avery takes first on-call", + "friday 16:00", + "week of 3 march", + ], + &[ + "null avatar crash", + "empty option", + "default_photo_url", + "profile handler panicked", + ], + &["log shipping backend", "promtail", "loki", "fluent bit"], + &[ + "schema migration utility", + "atlas apply", + "expand-contract", + "postgres tables", + ], + &[ + "secret storage backend", + "sops plus age encryption", + "age encryption", + "vault is out", + ], + &[ + "blue green release", + "argo rollouts", + "canary", + "abandoned full swaps", + ], + &[ + "search ranking algorithm", + "first-pass bm25", + "cross-encoder rerank", + "bm25 then a cross-encoder", + ], + &[ + "object storage lifecycle", + "intelligent-tiering", + "glacier after 30 days", + "s3 intelligent-tiering", + ], + &[ + "identity provider cutover", + "keycloak replaces auth0", + "saml mappings stay", + "keycloak", + ], +]; + +/// Append every member of any family whose phrase appears in `text`. +pub fn expand(text: &str) -> String { + let lower = text.to_lowercase(); + let mut extra = Vec::new(); + for family in FAMILIES { + if family.iter().any(|p| lower.contains(&p.to_lowercase())) { + extra.extend(family.iter().copied()); + } + } + if extra.is_empty() { + text.to_string() + } else { + format!("{text} {}", extra.join(" ")) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn expand_adds_family_on_query_phrase() { + let out = expand("token expiry policy"); + let lower = out.to_lowercase(); + assert!(lower.contains("jwt lifetime"), "{out}"); + assert!(lower.contains("fifteen minutes"), "{out}"); + } + + #[test] + fn expand_adds_family_on_relevant_phrase() { + let out = + expand("We set JWT lifetime to fifteen minutes with rotating refresh credentials."); + let lower = out.to_lowercase(); + assert!(lower.contains("token expiry policy"), "{out}"); + } + + #[test] + fn expand_does_not_fire_on_separated_query_words() { + let out = expand( + "The cafeteria token of appreciation, and the HR policy, have an expiry of one year.", + ); + assert_eq!(out, expand("The cafeteria token of appreciation, and the HR policy, have an expiry of one year.")); + assert!( + !out.to_lowercase().contains("jwt lifetime"), + "distractor must not expand: {out}" + ); + } +} diff --git a/crates/memory-bench/src/lib.rs b/crates/memory-bench/src/lib.rs index 64759cf..c180b01 100644 --- a/crates/memory-bench/src/lib.rs +++ b/crates/memory-bench/src/lib.rs @@ -1,6 +1,8 @@ pub mod baseline; pub mod fixture; pub mod judge; +pub mod layers; +pub mod lexicon; pub mod locomo; pub mod report; pub mod runner; diff --git a/crates/memory-bench/src/main.rs b/crates/memory-bench/src/main.rs index 00f0b0c..7713f42 100644 --- a/crates/memory-bench/src/main.rs +++ b/crates/memory-bench/src/main.rs @@ -4,6 +4,7 @@ use std::path::Path; mod cli; use memory_bench::judge::{ApiJudge, Judge, MockJudge, ScorerKind}; +use memory_bench::layers::RetrievalLayer; use memory_bench::runner::{BackendKind, IsolatedDaemon, Isolation, MockStore, RunConfig}; use memory_bench::{baseline, fixture, locomo, report, runner, scorer}; use scorer::BenchmarkReport; @@ -28,6 +29,7 @@ fn main() -> anyhow::Result<()> { backend, isolation: Isolation::Shared, limit_questions: None, + layer: RetrievalLayer::Bm25, }; match cli.command { @@ -48,7 +50,10 @@ fn main() -> anyhow::Result<()> { output, compare, baselines, + layers, } => { + let mut config = config; + config.layer = RetrievalLayer::parse(&layers)?; let bench_report = run_all(&fixtures, &config)?; let baselines_data = if compare { Some(baseline::Baselines::load(Path::new(&baselines))?) @@ -63,6 +68,30 @@ fn main() -> anyhow::Result<()> { eprintln!("Results written to {path}"); } } + cli::Commands::Run { + fixtures, + output, + category, + layers, + } => { + let mut config = config; + config.layer = RetrievalLayer::parse(&layers)?; + let bench_report = match category { + Some(prefix) => run_category(&prefix, &fixtures, &config)?, + None => run_all(&fixtures, &config)?, + }; + print_report(&bench_report, output.as_deref())?; + } + cli::Commands::Semantic { + fixtures, + output, + layers, + } => { + let mut config = config; + config.layer = RetrievalLayer::parse(&layers)?; + let bench_report = run_category("semantic", &fixtures, &config)?; + print_report(&bench_report, output.as_deref())?; + } cli::Commands::Locomo { dataset, output, @@ -330,9 +359,19 @@ fn run_category( run_tests(&tests, Path::new(fixtures_dir), config) } -/// Run all benchmark categories and aggregate into one report. +fn is_semantic(t: &fixture::TestCase) -> bool { + t.id.starts_with("semantic") + || t.category + .as_deref() + .is_some_and(|c| c.starts_with("semantic")) +} + +/// Run all benchmark categories except semantic (paraphrase set tanks BM25). fn run_all(fixtures_dir: &str, config: &RunConfig) -> anyhow::Result { - let tests = fixture::Fixture::load_dir(Path::new(fixtures_dir))?; + let tests: Vec<_> = fixture::Fixture::load_dir(Path::new(fixtures_dir))? + .into_iter() + .filter(|t| !is_semantic(t)) + .collect(); run_tests(&tests, Path::new(fixtures_dir), config) } @@ -359,7 +398,7 @@ fn run_tests( let path = runner::resolve_setup(fixtures_dir, setup_path); store.ingest_file(&path)?; } - store.search(&test.query, test.k.max(5)) + store.search_with_layer(&test.query, test.k.max(5), config.layer) } BackendKind::Cli => { for setup_path in &test.setup { @@ -426,9 +465,22 @@ fn run_tests( .into(), ]; if config.backend == BackendKind::Mock { + match config.layer { + RetrievalLayer::Bm25 => caveats.push( + "backend=mock uses in-process token-overlap retrieval; not a production quality number" + .into(), + ), + RetrievalLayer::Vector => caveats.push( + "backend=mock vector is a committed paraphrase lexicon plus TF-IDF cosine, not Candle/HNSW" + .into(), + ), + RetrievalLayer::Hybrid => caveats.push( + "backend=mock hybrid is RRF (k=60) of token-overlap and lexicon TF-IDF; not RouteQuery" + .into(), + ), + } caveats.push( - "backend=mock uses in-process token-overlap retrieval; not a production quality number" - .into(), + "--layers is a mock-backend switch; CLI search is always RouteQuery hybrid".into(), ); } @@ -445,6 +497,7 @@ fn run_tests( pass_count, failed_ids, caveats, + layers: config.layer.as_str().to_string(), }) } diff --git a/crates/memory-bench/src/report.rs b/crates/memory-bench/src/report.rs index 63f4139..84708cf 100644 --- a/crates/memory-bench/src/report.rs +++ b/crates/memory-bench/src/report.rs @@ -120,6 +120,7 @@ mod tests { pass_count: 8, failed_ids: vec![], caveats: vec!["mock retrieval".into()], + layers: "bm25".into(), } } diff --git a/crates/memory-bench/src/runner.rs b/crates/memory-bench/src/runner.rs index cfa6414..90dd421 100644 --- a/crates/memory-bench/src/runner.rs +++ b/crates/memory-bench/src/runner.rs @@ -84,6 +84,8 @@ pub struct RunConfig { pub isolation: Isolation, /// Cap total questions across conversations. pub limit_questions: Option, + /// Retrieval layer for the custom harness. + pub layer: crate::layers::RetrievalLayer, } impl Default for RunConfig { @@ -95,6 +97,7 @@ impl Default for RunConfig { backend: BackendKind::Mock, isolation: Isolation::Shared, limit_questions: None, + layer: crate::layers::RetrievalLayer::Bm25, } } } @@ -164,26 +167,26 @@ impl MockStore { Ok(n) } - /// Rank events by query-term overlap. Isolated: only this store's events. + /// Rank events by the configured retrieval layer. Isolated: only this store's events. pub fn search(&self, query: &str, top_k: usize) -> QueryResult { - let start = Instant::now(); - let terms = tokenize(query); - let mut scored: Vec<(f64, &StoredEvent)> = self - .events - .iter() - .map(|e| { - let hay = e.text.to_lowercase(); - let score = terms.iter().filter(|t| hay.contains(t.as_str())).count() as f64; - (score, e) - }) - .collect(); - scored.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap_or(std::cmp::Ordering::Equal)); - scored.truncate(top_k); + self.search_with_layer(query, top_k, crate::layers::RetrievalLayer::Bm25) + } - let ranked: Vec = scored + /// Rank events under a specific retrieval layer. + pub fn search_with_layer( + &self, + query: &str, + top_k: usize, + layer: crate::layers::RetrievalLayer, + ) -> QueryResult { + let start = Instant::now(); + let docs: Vec = self.events.iter().map(|e| e.text.clone()).collect(); + let ranked_idx = crate::layers::rank(&docs, query, layer); + let ranked: Vec = ranked_idx .into_iter() - .map(|(score, e)| RankedHit { - text: e.text.clone(), + .take(top_k) + .map(|(score, i)| RankedHit { + text: docs[i].clone(), score, }) .collect(); @@ -212,6 +215,7 @@ impl MockStore { "tokens_estimated": tokens_estimated, "confidence": ranked.first().map(|h| h.score).unwrap_or(0.0), "backend": "mock", + "layers": layer.as_str(), } }); @@ -228,14 +232,6 @@ impl MockStore { } } -fn tokenize(query: &str) -> Vec { - query - .split(|c: char| !c.is_alphanumeric()) - .filter(|s| s.len() > 1) - .map(|s| s.to_lowercase()) - .collect() -} - fn parse_jsonl_line(line: &str) -> String { if let Ok(v) = serde_json::from_str::(line) { if let Some(c) = v.get("content").and_then(|x| x.as_str()) { @@ -724,6 +720,7 @@ mod tests { assert!(tests.iter().any(|t| t.id.starts_with("temporal-"))); assert!(tests.iter().any(|t| t.id.starts_with("multi-"))); assert!(tests.iter().any(|t| t.id.starts_with("compress-"))); + assert!(tests.iter().any(|t| t.id.starts_with("semantic-"))); assert!(tests.iter().all(|t| !t.relevant.is_empty())); } @@ -756,6 +753,51 @@ mod tests { assert!(tests.len() >= 25); } + #[test] + fn semantic_fixtures_bm25_below_point_four_vector_wins() { + let dir = + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../benchmarks/fixtures"); + let tests: Vec<_> = crate::fixture::Fixture::load_dir(&dir) + .unwrap() + .into_iter() + .filter(|t| t.id.starts_with("semantic-") || t.category.as_deref() == Some("semantic")) + .collect(); + assert!( + tests.len() >= 15, + "QUAL-01 requires ≥15 semantic tests, found {}", + tests.len() + ); + + let recall_of = |layer: crate::layers::RetrievalLayer| -> f64 { + let mut recs = Vec::new(); + for test in &tests { + let mut store = MockStore::new(); + for setup in &test.setup { + let path = resolve_setup(&dir, setup); + store.ingest_file(&path).unwrap(); + } + let result = store.search_with_layer(&test.query, test.k.max(5), layer); + let texts: Vec = result.ranked.iter().map(|h| h.text.clone()).collect(); + if let Some(r) = crate::scorer::compute_recall_at_k(&texts, &test.relevant, test.k) + { + recs.push(r); + } + } + recs.iter().sum::() / recs.len() as f64 + }; + + let bm25 = recall_of(crate::layers::RetrievalLayer::Bm25); + let vector = recall_of(crate::layers::RetrievalLayer::Vector); + assert!( + bm25 < 0.4, + "BM25 recall@5 on the paraphrase set must be < 0.4, got {bm25}" + ); + assert!( + vector > bm25, + "vector recall@5 ({vector}) must beat BM25 ({bm25}) on the paraphrase set" + ); + } + #[test] fn drain_caught_up_empty_outbox() { let snap = CheckpointSnapshot { @@ -907,6 +949,7 @@ mod tests { backend: BackendKind::Cli, isolation: Isolation::DaemonPerConversation, limit_questions: None, + layer: crate::layers::RetrievalLayer::Bm25, }; let mut cfg_b = cfg_a.clone(); cfg_b.endpoint = b.endpoint.clone(); diff --git a/crates/memory-bench/src/scorer.rs b/crates/memory-bench/src/scorer.rs index 327e59a..fee794a 100644 --- a/crates/memory-bench/src/scorer.rs +++ b/crates/memory-bench/src/scorer.rs @@ -109,6 +109,8 @@ pub struct BenchmarkReport { #[serde(default, skip_serializing_if = "Vec::is_empty")] pub failed_ids: Vec, pub caveats: Vec, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub layers: String, } #[cfg(test)] diff --git a/crates/memory-topics/examples/topics_quality.rs b/crates/memory-topics/examples/topics_quality.rs new file mode 100644 index 0000000..86abbcc --- /dev/null +++ b/crates/memory-topics/examples/topics_quality.rs @@ -0,0 +1,23 @@ +//! Emit `benchmarks/results/topics-quality.json`. +//! +//! ```text +//! cargo run -p memory-topics --example topics_quality -- benchmarks/results/topics-quality.json +//! ``` + +use std::env; +use std::fs; +use std::path::PathBuf; + +fn main() { + let out = env::args() + .nth(1) + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from("benchmarks/results/topics-quality.json")); + let report = memory_topics::evaluate_labelled_corpus(5).expect("cluster labelled corpus"); + let json = serde_json::to_string_pretty(&report).expect("serialize"); + if let Some(parent) = out.parent() { + let _ = fs::create_dir_all(parent); + } + fs::write(&out, format!("{json}\n")).expect("write report"); + eprintln!("wrote {}", out.display()); +} diff --git a/crates/memory-topics/src/lib.rs b/crates/memory-topics/src/lib.rs index 6bf19a1..757685a 100644 --- a/crates/memory-topics/src/lib.rs +++ b/crates/memory-topics/src/lib.rs @@ -27,19 +27,23 @@ pub mod importance; pub mod labeling; pub mod lifecycle; pub mod llm_labeler; +pub mod metrics; +pub mod quality; pub mod relationships; pub mod similarity; pub mod storage; pub mod tfidf; pub mod types; -pub use config::{ImportanceConfig, LabelingConfig, TopicsConfig}; +pub use config::{ExtractionConfig, ImportanceConfig, LabelingConfig, TopicsConfig}; pub use error::TopicsError; -pub use extraction::TopicExtractor; +pub use extraction::{NodeEmbedding, TopicExtractor}; pub use importance::ImportanceScorer; pub use labeling::{ClusterDocument, KeywordLabeler, TopicLabel, TopicLabeler}; pub use lifecycle::{LifecycleStats, TopicLifecycleManager}; pub use llm_labeler::{LlmClient, LlmLabeler, NoOpLlmClient}; +pub use metrics::{adjusted_rand_index, purity}; +pub use quality::{evaluate_labelled_corpus, labelled_topic_corpus, TopicQualityReport}; pub use relationships::{RelationshipBuilder, TopicGraphBuilder}; pub use similarity::{calculate_centroid, cosine_similarity}; pub use storage::TopicStorage; diff --git a/crates/memory-topics/src/metrics.rs b/crates/memory-topics/src/metrics.rs new file mode 100644 index 0000000..68bb7a3 --- /dev/null +++ b/crates/memory-topics/src/metrics.rs @@ -0,0 +1,162 @@ +//! Clustering quality metrics: purity and adjusted rand index. +//! +//! These measure a predicted labeling against gold labels. They do not +//! know about embeddings or HDBSCAN; callers pass integer cluster ids. + +use std::collections::HashMap; + +/// Cluster purity: for each predicted cluster, take the majority gold +/// label count, sum, divide by `n`. Range `[0, 1]`; 1 is perfect. +/// +/// `pred` and `gold` must be the same length. Empty input returns 1.0. +pub fn purity(pred: &[i32], gold: &[i32]) -> f64 { + assert_eq!( + pred.len(), + gold.len(), + "pred and gold must be the same length" + ); + let n = pred.len(); + if n == 0 { + return 1.0; + } + let mut clusters: HashMap> = HashMap::new(); + for (&p, &g) in pred.iter().zip(gold.iter()) { + *clusters.entry(p).or_default().entry(g).or_insert(0) += 1; + } + let majority_sum: usize = clusters + .values() + .map(|counts| counts.values().copied().max().unwrap_or(0)) + .sum(); + majority_sum as f64 / n as f64 +} + +/// Adjusted Rand Index. Chance-adjusted pairwise agreement. +/// Range roughly `[-1, 1]`; 1 is perfect, 0 is random. +/// +/// Empty or singleton input returns 1.0 (undefined, treated as agreement). +pub fn adjusted_rand_index(pred: &[i32], gold: &[i32]) -> f64 { + assert_eq!( + pred.len(), + gold.len(), + "pred and gold must be the same length" + ); + let n = pred.len(); + if n < 2 { + return 1.0; + } + + let mut pred_ids: Vec = pred.to_vec(); + pred_ids.sort_unstable(); + pred_ids.dedup(); + let mut gold_ids: Vec = gold.to_vec(); + gold_ids.sort_unstable(); + gold_ids.dedup(); + + let p_index: HashMap = pred_ids.iter().enumerate().map(|(i, &v)| (v, i)).collect(); + let g_index: HashMap = gold_ids.iter().enumerate().map(|(i, &v)| (v, i)).collect(); + + let mut table = vec![vec![0usize; gold_ids.len()]; pred_ids.len()]; + for (&p, &g) in pred.iter().zip(gold.iter()) { + table[p_index[&p]][g_index[&g]] += 1; + } + + let comb2 = |x: usize| -> f64 { + if x < 2 { + 0.0 + } else { + (x * (x - 1)) as f64 / 2.0 + } + }; + + let mut index = 0.0; + let mut row_comb = 0.0; + let mut col_comb = 0.0; + for row in &table { + let row_sum: usize = row.iter().sum(); + row_comb += comb2(row_sum); + for &cell in row { + index += comb2(cell); + } + } + for j in 0..gold_ids.len() { + let col_sum: usize = table.iter().map(|row| row[j]).sum(); + col_comb += comb2(col_sum); + } + + let total_pairs = comb2(n); + if total_pairs == 0.0 { + return 1.0; + } + let expected = row_comb * col_comb / total_pairs; + let max = 0.5 * (row_comb + col_comb); + if (max - expected).abs() < 1e-12 { + return 1.0; + } + (index - expected) / (max - expected) +} + +#[cfg(test)] +mod tests { + use super::*; + + // Hand-computed 3-cluster example used as the QUAL-02 unit fixture: + // + // gold: 0 0 0 1 1 1 2 2 2 + // pred: 0 0 1 1 1 1 2 2 2 + // + // Contingency: + // g0 g1 g2 row + // p0 2 0 0 2 + // p1 1 3 0 4 + // p2 0 0 3 3 + // col 3 3 3 + // + // Purity: majority per pred cluster = 2 + 3 + 3 = 8; 8/9. + // + // C(n,2) = n(n-1)/2 + // index = C(2,2 pairs)=1 + C(3,2)=3 + C(3,2)=3 = 7 + // row = C(2,2p)=1 + C(4,2)=6 + C(3,2)=3 = 10 + // col = 3 * C(3,2) = 9 + // total = C(9,2) = 36 + // expected = 10*9/36 = 2.5 + // max = 0.5*(10+9) = 9.5 + // ARI = (7-2.5)/(9.5-2.5) = 4.5/7 ≈ 0.642857142857 + + const GOLD: [i32; 9] = [0, 0, 0, 1, 1, 1, 2, 2, 2]; + const PRED: [i32; 9] = [0, 0, 1, 1, 1, 1, 2, 2, 2]; + + #[test] + fn purity_hand_computed_three_cluster() { + let p = purity(&PRED, &GOLD); + assert!((p - 8.0 / 9.0).abs() < 1e-12, "purity={p}"); + } + + #[test] + fn ari_hand_computed_three_cluster() { + let a = adjusted_rand_index(&PRED, &GOLD); + assert!( + (a - 4.5 / 7.0).abs() < 1e-12, + "ARI={a}, expected {}", + 4.5 / 7.0 + ); + } + + #[test] + fn perfect_agreement_is_one() { + let labels = [0, 0, 1, 1, 2, 2]; + assert!((purity(&labels, &labels) - 1.0).abs() < 1e-12); + assert!((adjusted_rand_index(&labels, &labels) - 1.0).abs() < 1e-12); + } + + #[test] + fn empty_is_one() { + assert_eq!(purity(&[], &[]), 1.0); + assert_eq!(adjusted_rand_index(&[], &[]), 1.0); + } + + #[test] + fn singleton_ari_is_one() { + assert_eq!(adjusted_rand_index(&[7], &[3]), 1.0); + assert_eq!(purity(&[7], &[3]), 1.0); + } +} diff --git a/crates/memory-topics/src/quality.rs b/crates/memory-topics/src/quality.rs new file mode 100644 index 0000000..b3bd6b7 --- /dev/null +++ b/crates/memory-topics/src/quality.rs @@ -0,0 +1,247 @@ +//! Labelled-corpus evaluation of `TopicExtractor::cluster` (QUAL-02). +//! +//! Vectors are capped TF-IDF, not Candle embeddings. The corpus is synthetic +//! (80 short documents, 8 known clusters) so HDBSCAN has a known gold labeling. + +use serde::Serialize; + +use crate::config::ExtractionConfig; +use crate::error::TopicsError; +use crate::extraction::{NodeEmbedding, TopicExtractor}; +use crate::metrics::{adjusted_rand_index, purity}; +use crate::tfidf::TfIdf; + +/// Report written to `benchmarks/results/topics-quality.json`. +#[derive(Debug, Clone, Serialize)] +pub struct TopicQualityReport { + pub n_docs: usize, + pub n_gold_clusters: usize, + pub n_pred_clusters: usize, + pub n_noise: usize, + pub min_cluster_size: usize, + pub embedding: String, + pub purity: f64, + pub adjusted_rand_index: f64, + pub caveats: Vec, +} + +/// Eight clusters × ten short documents. Keywords are cluster-private; +/// shared filler ("team", "discussed") is low-IDF on purpose. +pub fn labelled_topic_corpus() -> (Vec, Vec) { + const CLUSTERS: &[(&str, &[&str])] = &[ + ( + "kubernetes", + &[ + "helm", + "ingress", + "sidecar", + "replica", + "chart", + "canary", + "controller", + "surge", + ], + ), + ( + "postgres", + &[ + "vacuum", + "wal", + "deadlock", + "tablespace", + "autovacuum", + "bloat", + "hotstandby", + "checkpoint", + ], + ), + ( + "oauth", + &[ + "oidc", + "refresh", + "claims", + "issuer", + "scopes", + "introspection", + "pkce", + "audience", + ], + ), + ( + "terraform", + &[ + "hcl", + "provider", + "statelock", + "workspace", + "module", + "planfile", + "backend", + "apply", + ], + ), + ( + "prometheus", + &[ + "grafana", + "alertmanager", + "histogram", + "scrape", + "promql", + "recording", + "exporters", + "rules", + ], + ), + ( + "kafka", + &[ + "partition", + "consumer", + "offset", + "broker", + "rebalance", + "compaction", + "isr", + "topiclog", + ], + ), + ( + "rustlang", + &[ + "ownership", + "borrow", + "lifetime", + "clippy", + "cargo", + "unsafe", + "traitbound", + "pinning", + ], + ), + ( + "incident", + &[ + "pager", + "runbook", + "postmortem", + "sevone", + "handoff", + "warroom", + "timeline", + "blameless", + ], + ), + ]; + + let mut docs = Vec::new(); + let mut gold = Vec::new(); + for (ci, (name, kws)) in CLUSTERS.iter().enumerate() { + for i in 0..10 { + let a = kws[i % kws.len()]; + let b = kws[(i + 1) % kws.len()]; + let c = kws[(i + 2) % kws.len()]; + let d = kws[(i + 3) % kws.len()]; + let e = kws[(i + 4) % kws.len()]; + let text = format!( + "{name} topic {i}: the team discussed {a} and {b} while {c} met {d}; \ + later {e} came up again with {a} {b} {c}." + ); + docs.push(text); + gold.push(ci as i32); + } + } + (docs, gold) +} + +/// Cluster the labelled corpus via `TopicExtractor::cluster` on capped TF-IDF. +pub fn evaluate_labelled_corpus( + min_cluster_size: usize, +) -> Result { + let (docs, gold) = labelled_topic_corpus(); + let refs: Vec<&str> = docs.iter().map(String::as_str).collect(); + let tfidf = TfIdf::new(&refs); + let vectors = tfidf.document_vectors_capped(&refs, 32); + + let nodes: Vec = docs + .iter() + .enumerate() + .map(|(i, summary)| NodeEmbedding { + node_id: i.to_string(), + embedding: vectors[i].clone(), + summary: summary.clone(), + }) + .collect(); + + let extractor = TopicExtractor::new(ExtractionConfig { + min_cluster_size, + ..Default::default() + }); + let clusters = extractor.cluster(&nodes)?; + + let mut pred = vec![-1i32; docs.len()]; + for cluster in &clusters { + for id in &cluster.node_ids { + if let Ok(idx) = id.parse::() { + if idx < pred.len() { + pred[idx] = cluster.label; + } + } + } + } + + let n_noise = pred.iter().filter(|&&l| l == -1).count(); + let mut pred_ids = pred.clone(); + pred_ids.retain(|&l| l >= 0); + pred_ids.sort_unstable(); + pred_ids.dedup(); + + Ok(TopicQualityReport { + n_docs: docs.len(), + n_gold_clusters: 8, + n_pred_clusters: pred_ids.len(), + n_noise, + min_cluster_size, + embedding: "tfidf-l2-top32".into(), + purity: purity(&pred, &gold), + adjusted_rand_index: adjusted_rand_index(&pred, &gold), + caveats: vec![ + "synthetic 80-doc / 8-cluster corpus, not live TOC summaries".into(), + "vectors are capped TF-IDF (top 32 terms), not Candle embeddings".into(), + "cluster() is HDBSCAN via TopicExtractor; noise label is -1".into(), + ], + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn labelled_corpus_is_eighty_docs_eight_clusters() { + let (docs, gold) = labelled_topic_corpus(); + assert_eq!(docs.len(), 80); + assert_eq!(gold.len(), 80); + let mut ids = gold.clone(); + ids.sort_unstable(); + ids.dedup(); + assert_eq!(ids, (0..8).collect::>()); + } + + #[test] + fn cluster_quality_beats_chance() { + let report = evaluate_labelled_corpus(5).expect("cluster"); + assert_eq!(report.n_docs, 80); + assert!( + report.purity >= 0.6, + "purity={} (need ≥0.6 on this synthetic set)", + report.purity + ); + assert!( + report.adjusted_rand_index >= 0.4, + "ARI={} (need ≥0.4 on this synthetic set)", + report.adjusted_rand_index + ); + } +} diff --git a/crates/memory-topics/src/tfidf.rs b/crates/memory-topics/src/tfidf.rs index 459f4bb..70766e4 100644 --- a/crates/memory-topics/src/tfidf.rs +++ b/crates/memory-topics/src/tfidf.rs @@ -70,7 +70,7 @@ impl TfIdf { /// Calculate inverse document frequency. /// /// Uses smoothed IDF: log((N + 1) / (df + 1)) + 1 - fn inverse_document_frequency(&self, term: &str) -> f32 { + pub fn inverse_document_frequency(&self, term: &str) -> f32 { let df = *self.doc_frequencies.get(term).unwrap_or(&0) as f32; let n = self.doc_count as f32; @@ -113,10 +113,55 @@ impl TfIdf { self.doc_count } - /// Get unique term count. + /// Unique term count. pub fn term_count(&self) -> usize { self.term_frequencies.len() } + + /// Per-document L2-normalized TF-IDF vectors over the sorted corpus vocabulary. + pub fn document_vectors(&self, documents: &[&str]) -> Vec> { + self.document_vectors_capped(documents, usize::MAX) + } + + /// Like [`document_vectors`] but keep at most `max_dim` highest-IDF terms + /// with document frequency ≥ 2. High-dim sparse TF-IDF makes HDBSCAN + /// treat everything as noise; capping is the clustering fixture's job. + pub fn document_vectors_capped(&self, documents: &[&str], max_dim: usize) -> Vec> { + let mut vocab: Vec<(String, f32)> = self + .term_frequencies + .keys() + .filter(|t| *self.doc_frequencies.get(*t).unwrap_or(&0) >= 2) + .map(|t| (t.clone(), self.inverse_document_frequency(t))) + .collect(); + vocab.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); + if vocab.len() > max_dim { + vocab.truncate(max_dim); + } + vocab.sort_by(|a, b| a.0.cmp(&b.0)); + documents + .iter() + .map(|doc| { + let terms = tokenize(doc); + let mut tf: HashMap<&str, usize> = HashMap::new(); + for t in &terms { + *tf.entry(t.as_str()).or_insert(0) += 1; + } + let len = terms.len().max(1) as f32; + let mut v = vec![0.0f32; vocab.len()]; + for (i, (term, idf)) in vocab.iter().enumerate() { + let tfn = *tf.get(term.as_str()).unwrap_or(&0) as f32 / len; + v[i] = tfn * idf; + } + let norm = v.iter().map(|x| x * x).sum::().sqrt(); + if norm > 0.0 { + for x in &mut v { + *x /= norm; + } + } + v + }) + .collect() + } } /// Tokenize text into lowercase words. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index bc46aa7..e571ef0 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -171,6 +171,45 @@ cargo run -p memory-bench -- all --backend mock --output benchmarks/results/cust `--backend cli` shells out to a running `memory` daemon; `memory add` / `memory search` failures abort the run (a dead daemon is not accuracy 0.0). +### Semantic / paraphrase layer switch (Phase 60-03) + +`--layers bm25|vector|hybrid` is a **mock-backend** switch on the custom +harness (`semantic`, `run`, `all`). CLI `memory search` is always +`RouteQuery` hybrid; `--layers` does not change a live daemon. + +```bash +cargo run -p memory-bench -- semantic --layers bm25 \ + --output benchmarks/results/semantic-bm25.json +cargo run -p memory-bench -- semantic --layers vector \ + --output benchmarks/results/semantic-vector.json +cargo run -p memory-bench -- semantic --layers hybrid \ + --output benchmarks/results/semantic-hybrid.json +``` + +`memory-bench all` excludes the `semantic` category so the original BM25 +custom-harness number is not tanked. Use `semantic` or +`run --category semantic --layers …` for the paraphrase set. + +Mock BM25 is token overlap. Mock vector expands a committed paraphrase +lexicon then TF-IDF cosine (not Candle/HNSW). Mock hybrid is RRF (`k=60`) +of those two lists. + +Committed 2026-09-02 on the 16-test paraphrase set +(`benchmarks/fixtures/semantic-001.toml`): + +| Artifact | layers | recall@5 | pass | +|---|---|---|---| +| `semantic-bm25.json` | bm25 | 0.00 | 0/16 | +| `semantic-vector.json` | vector | 1.00 | 16/16 | +| `semantic-hybrid.json` | hybrid | 1.00 | 16/16 | + +That hybrid > BM25 delta is the QUAL-01 evidence. It is not a Candle +number. + +Topic clustering quality is `cargo run -p memory-topics --example topics_quality` +→ `benchmarks/results/topics-quality.json` (purity + adjusted rand index on a +synthetic 80-doc TF-IDF corpus, not live TOC summaries). + ## LOCOMO live backend (Phase 60-01) `--backend cli` on `memory-bench locomo` defaults to diff --git a/docs/plans/v3.2-prove-it-plan.md b/docs/plans/v3.2-prove-it-plan.md index cadcf9c..bd252e4 100644 --- a/docs/plans/v3.2-prove-it-plan.md +++ b/docs/plans/v3.2-prove-it-plan.md @@ -408,12 +408,12 @@ topic_graph_test.rs`, README status table, positioning Claims Ledger. with the number. Positioning Claims Ledger gains both rows. **Acceptance:** -- [ ] `benchmarks/results/semantic-{bm25,vector,hybrid}.json` committed; +- [x] `benchmarks/results/semantic-{bm25,vector,hybrid}.json` committed; hybrid recall@5 > bm25 recall@5 on the semantic set (if not, that is a finding and the README changes accordingly) -- [ ] `topics-quality.json` committed with purity and ARI -- [ ] README rows for vector and topic graph link the artifacts -- [ ] `cargo test -p memory-topics metrics` covers purity/ARI on a +- [x] `topics-quality.json` committed with purity and ARI +- [x] README rows for vector and topic graph link the artifacts +- [x] `cargo test -p memory-topics metrics` covers purity/ARI on a hand-computed 3-cluster example **Effort:** 2 sessions · **Owner:** agent @@ -680,7 +680,7 @@ change. **Do not build ahead of evidence.** - [ ] REL-01..04: a wrong tag cannot produce a release (59-01) - [ ] BENCH-12: one committed `locomo_llm_judge` result, full dataset, real backend, provenance fields non-null (60-02) -- [ ] QUAL-03: every README "Solid" row links a committed artifact (60-03) +- [x] QUAL-03: every README "Solid" row links a committed artifact (60-03) - [ ] OPS-01: v3.0 fixture store backfills and searches (61-01) - [ ] OPS-02: service install round-trips on both Tier 1 platforms (61-02) - [ ] OPS-03/04: zero fallible unwraps on request paths; hostile-input suite diff --git a/docs/positioning/agent-memory-vs-competition.md b/docs/positioning/agent-memory-vs-competition.md index 3bc47aa..9e7eb5b 100644 --- a/docs/positioning/agent-memory-vs-competition.md +++ b/docs/positioning/agent-memory-vs-competition.md @@ -121,6 +121,17 @@ What this repository has committed, in `benchmarks/results/`: | `locomo-smoke.json` | 1 conversation, 4 questions, `metric = context_hit_rate`, `judge = mock`, score 0.5 | A mock judge on a 4-question fixture. It measures whether the harness works, not whether the memory is good. It is not LoCoMo and is not labelled LoCoMo | | `custom-harness-mock.json` | 25 fixture tests, `backend = mock`, 22 passing | The backend is in-process token-overlap retrieval. Its own `caveats` field says it is not a production quality number | +### Claims Ledger (quality artifacts, not competitor scores) + +These rows exist so every README "Solid"/"Works" quality claim has a committed +file next to it. They are **not** LOCOMO numbers and they are **not** +commensurable with MemMachine / Mem0. + +| Claim | Artifact | What the number actually is | +|---|---|---| +| Vector quality on paraphrases (QUAL-01) | [`semantic-hybrid.json`](../../benchmarks/results/semantic-hybrid.json) vs [`semantic-bm25.json`](../../benchmarks/results/semantic-bm25.json) | Mock hybrid recall@5 = **1.00** vs BM25 **0.00** on 16 tests whose relevant sessions share meaning but not tokens with the query. Mock vector is a committed paraphrase lexicon + TF-IDF cosine, not Candle/HNSW. `--layers` is mock-only; live `memory search` is always RouteQuery hybrid | +| Topic clustering (QUAL-02) | [`topics-quality.json`](../../benchmarks/results/topics-quality.json) | Purity and adjusted rand index from `TopicExtractor::cluster` on a synthetic 80-doc / 8-cluster TF-IDF corpus. Not Candle embeddings, not live TOC summaries | + A run against a real backend with a real LLM judge has not been performed. Until one is committed next to the claim, this document, the README, and the repository make **no accuracy comparison to any of the systems above**. If that