research: witnessed evolution — hash-chained provenance for ANN parameter search

[ruvnet]

Hypothesis

Given a fixed, seeded ruvector-coherence-hnsw workload and a fixed
(1+1)-ES over its [coherence_threshold, ef] genome,

when every generation's genome, fitness, and accept/reject decision is
committed to a ruvector-proof-gate HashChainGate as it is produced,

then the witnessed run's final genome and fitness are bit-identical to an
unwitnessed run of the same algorithm and seed, an independent replayer
verifies 100% of honest lineages, and a single forged fitness value is
caught at the exact generation it was forged, 100% of the time,

subject to witnessing wall-clock overhead staying under 15%, the witnessed
search beating a fixed-default baseline, and build/tests remaining green.

Why this topic: ruvector-sona already runs an unwitnessed (1+1)-ES
(darwin_autotuner.rs). ruvector-proof-gate (ADR-227) and
ruvector-retrieval-receipt (ADR-304) give writes and reads tamper-evident
hash chains. Nothing combines an evolutionary search with a witness chain
over its own mutation/fitness/promotion history — the gap this repo's own
nightly-harness Darwin promotion gate (witness_valid, "a failed candidate
must remain part of the lineage") assumes is already closed but isn't.

Architecture

New crate crates/ruvector-witnessed-evolution:

• genome.rs — two-parameter genome (coherence threshold, beam width ef) over ruvector-coherence-hnsw's CoherenceGatedSearch.
• fitness.rs — deterministic evaluation (recall@10 + expansion count; not wall-clock latency, to keep replay exact-match-comparable) against a fixed seeded workload.
• witness.rs — WitnessedLineage wraps ruvector_proof_gate::HashChainGate unmodified (genome → WritePayload.vector, fitness/decision → metadata). replay_verify() independently recomputes hashes, chain, fitness, and every accept/reject decision. tamper_composite() is the adversarial test hook.
• evolve.rs — run_unwitnessed / run_witnessed, identical (1+1)-ES loop and seed, so trajectories are provably comparable.
• src/bin/benchmark.rs — baseline (fixed default) / candidate_A (unwitnessed ES) / candidate_B (witnessed ES) + honest/tampered replay checks + acceptance gate.

Files changed

• crates/ruvector-witnessed-evolution/ (new crate: Cargo.toml, src/{lib,genome,fitness,witness,evolve}.rs, src/bin/benchmark.rs)
• Cargo.toml, Cargo.lock (workspace member registration)
• docs/adr/ADR-305-witnessed-evolution.md
• docs/research/nightly/2026-08-19-witnessed-evolution-ann-tuning/{README,gist}.md

Benchmark command

cargo run --release -p ruvector-witnessed-evolution --bin benchmark

Real benchmark results (run 1 of 3, all independently reproduced)

[baseline]      threshold=0.500 ef= 80  recall=0.9007  avg_expansions=12.3  composite=0.8988  p50=79.2us
[candidate_A]    threshold=0.101 ef= 41  recall=0.9293  avg_expansions=13.0  composite=0.9274  p50=45.6us  wall=332.80ms  (40 generations, unwitnessed)
[candidate_B]    threshold=0.101 ef= 41  recall=0.9293  avg_expansions=13.0  composite=0.9274  p50=46.9us  wall=315.13ms  (40 generations, witnessed, chain_len=41)

witnessing overhead: -5.31% wall-clock
chain root: 7a711211a356d300cf43d6f67df14e948ca6fae267c4abb65c735b81dca34a89

replay_verify(honest lineage)   -> verified=true chain_integrity=true first_divergence=None (41 generations checked)
replay_verify(tampered gen 20) -> verified=false first_divergence=Some(20)

ACCEPTANCE RESULT: ACCEPT

Final genome, fitness, and chain root were bit-identical across all 3 independent release runs (full determinism, including every rejected intermediate generation). Measured wall-clock "overhead" was negative in all 3 runs (-5.3%, -30.1%, -3.3%) — read honestly as noise, not a speedup: HashChainGate::admit costs ~200ns/call, so 41 commits cost ~8µs against a ~300-460ms search budget, far below what Instant-based wall-clock measurement can resolve at this scale. See the README's "Honest Reading of the Overhead Number" section.

Acceptance result

ACCEPT. All 5 mandatory criteria held on all 3 independent runs: bit-identical witnessed/unwitnessed trajectories, ES beats fixed baseline (+3.2% composite fitness), overhead within the 15% budget (measured negative), honest-lineage replay verifies, tampered-lineage is caught at the exact tampered generation.

Darwin result

Bounded (1+1)-ES, 40 generations / 41 candidates evaluated, 1 promotion (the final accepted genome). Parent (records()[0], the untouched default genome) and every rejected intermediate mutation retained in the lineage, not discarded.

Flywheel result

WitnessedLineage::records() is a directly reusable evidence format for future nightly Darwin runs tuning ANN parameters elsewhere in the ecosystem — genome, fitness, and decision for every generation including rejections, ~154 bytes/generation.

Security review

Reuses ruvector-proof-gate's existing SHA-256 HashChainGate unmodified — no new cryptographic primitive, no new attack surface. No secrets/PII pass through the crate. The witness chain is unsigned: it detects post-issuance mutation of evidence (verified in tests: tampered_fitness_is_detected), it does not prove the search process itself ran honestly in the first place — same threat model ruvector-retrieval-receipt already documents for its own receipts. tamper_composite is a public test-only hook; a tampered record cannot be made to pass replay_verify because it doesn't (and structurally can't) touch the receipt's payload_hash captured at admission time.

Main limitations

• Single-machine, 3-run wall-clock measurement — not a statistically rigorous latency study.
• Two-parameter genome only; graph-topology parameters (m, m_longjump) would need an O(N²) rebuild per generation this design doesn't attempt.
• Unsigned hash chain, not a signature — no non-repudiation against the search process itself.
• No named competitor (Milvus, Qdrant, Weaviate, etc.) documents an equivalent feature, so this is a novelty claim, not a demonstrated win over a specific product.

Production recommendation

Keep experimental (not wired into any production tuning path). Adopt WitnessedLineage as the standard evidence format for future nightly Darwin runs that tune ANN parameters — it's a working, tested, negligible-overhead primitive that satisfies this harness's own "retained evidence, not fabricated summaries" requirement. Do not yet claim a production speedup from witnessing itself — the honest claim is "the cost is unmeasurably small," not "free" or "faster."

Research document / ADR / gist

• Full report: docs/research/nightly/2026-08-19-witnessed-evolution-ann-tuning/README.md
• ADR: docs/adr/ADR-305-witnessed-evolution.md
• Public gist: docs/research/nightly/2026-08-19-witnessed-evolution-ann-tuning/gist.md

Validation performed

• cargo test -p ruvector-witnessed-evolution — 11/11 passing
• cargo clippy -p ruvector-witnessed-evolution --all-targets — clean
• cargo fmt -p ruvector-witnessed-evolution -- --check — clean
• cargo metadata --no-deps — workspace resolves cleanly with the new member
• cargo run --release -p ruvector-witnessed-evolution --bin benchmark — run 3× independently, results above

---

Generated by Claude Code

[ruvnet]

The cargo audit (RustSec advisories) / Security audit checks are red on this PR, but not because of this diff: git diff <base>..<head> -- Cargo.lock shows the only change is the new ruvector-witnessed-evolution package entry — no version bumps to h2 or lru, the two crates the audit is flagging (RUSTSEC-2026-0258, RUSTSEC-2026-0253). The supply-chain workflow has been failing on main on the same advisories independently of this PR (confirmed via recent main-branch workflow run history). Not attempting a fix here — upgrading h2/lru transitively across the workspace is out of scope for this research PR. Will re-check once the base branch recovers.

---

Generated by Claude Code