RAG Reliability Lab

A reproducible study of retrieval, evidence freshness, and deterministic answer/defer decisions in retrieval-augmented generation (RAG) pipelines.

The study compares lexical BM25, dense MiniLM, hybrid reciprocal-rank fusion (RRF), and status-aware reranking. It then evaluates a frozen retrieval-consensus gate on a versioned synthetic benchmark with an explicit development/holdout split.

Executive summary

RAG Reliability Study v1 is complete. Its pre-registered top1_consensus_gate v1.0.0 passed all three frozen criteria on the 36-query Atlas Support Operations v2 holdout:

Criterion	Observed	Threshold
Answered-query precision	17/20 = 0.850	>= 0.75
Answerable-query coverage	17/22 = 0.773	>= 0.70
False-answer rate on expected-defer	3/14 = 0.214	<= 0.50

The overall decision accuracy was 28/36 = 0.778. Three expected-defer queries were answered and five answerable queries were deferred.

These results apply only to the committed synthetic benchmark and protocol. They do not establish production readiness, generalization to enterprise data, generated-answer factuality, calibrated confidence, or hallucination reduction. Answer generation is not implemented.

Five-minute tour

1. Read the final report for the research question, chronology, results, error analysis, and claim boundaries.
2. Inspect the pre-registration and the holdout results.
3. Review the frozen benchmark manifest and split.
4. Open the machine-readable study summary and its study configuration.
5. Use the reproduction guide to verify artifacts or rerun the experiments.

Study design

flowchart LR
    A["Synthetic versioned corpus"] --> B["BM25 retrieval"]
    A --> C["Dense MiniLM retrieval"]
    B --> D["Hybrid RRF"]
    C --> D
    D --> E["Status-aware reranking"]
    B --> F["Frozen top-1 consensus gate"]
    C --> F
    E --> F
    F --> G["Answer or defer"]
    G --> H["Versioned reports and integrity checks"]

Loading

The frozen gate predicts answer if and only if:

1. the final status-aware rank-1 evidence document is current;
2. BM25 rank 1 and dense rank 1 contain the same document ID; and
3. that shared document is the final status-aware evidence document.

Otherwise it predicts defer.

Evidence and failures

The v2 benchmark contains 28 documents, 72 queries, and 86 relevance labels. Its development and holdout splits contain 36 queries each. All data represents the fictional Atlas Support Operations organisation and is synthetic.

The one-time holdout evaluation produced:

Outcome	Result
Predicted answer / defer	20 / 16
Decision accuracy	28/36 = 0.778
Answered-query precision	17/20 = 0.850
Answerable-query coverage	17/22 = 0.773
False-answer rate on expected-defer	3/14 = 0.214
Clarify deferral recall	6/6
Abstain deferral recall	5/8

The three false answers were abstain queries where both retrievers agreed on an unsupported rank-1 document. The five false deferrals were answerable queries where BM25 and dense rank 1 disagreed. Query-level IDs and analysis are recorded in the final report.

The holdout is now unblinded and cannot be reused as unseen evidence for future gate selection. New gate versions require a new frozen holdout or must be labelled explicitly as post-holdout exploratory work.

Quick verification

Requirements:

• uv
• Python 3.13.14

Run the quality checks that do not require the embedding model:

uv sync --locked --dev
uv run ruff format --check .
uv run ruff check .
uv run mypy src tests
uv run pytest -m "not model"
uv build

Verify the protected study package directly:

uv run pytest tests/integration/test_reliability_study_v1_package.py

Dense and hybrid reproduction requires the optional dense dependencies and a one-time, explicit download of the pinned MiniLM snapshot. After it is cached, the actual-model tests can run offline:

uv sync --locked --dev --extra dense

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
uv run --extra dense pytest -m model

See the complete reproduction guide for every report-generation command and byte-level comparison procedure.

Repository map

Path	Purpose
src/rag_reliability_lab/	Retrieval, evaluation, corpus, and reporting code
tests/	Unit and integration tests for observable behaviour and evidence integrity
data/synthetic/	Versioned synthetic corpora, queries, relevance labels, and manifests
configs/	Frozen retrieval, gate, benchmark, and study configurations
reports/examples/	Committed machine-readable evaluation artifacts
docs/experiments/	Protocols, pre-registration, analyses, results, and final report
docs/reproduction/	Reproduction and integrity-verification instructions

Reproducibility controls

• deterministic BM25, fusion, reranking, and gate logic;
• pinned dense model ID, revision, and runtime dependencies;
• explicit, opt-in external model download;
• committed benchmark and split fingerprints;
• versioned configurations and machine-readable reports;
• SHA-256 integrity checks across the final evidence package;
• strict typing, deterministic tests, and read-only CI permissions.

The embedding model is sentence-transformers/all-MiniLM-L6-v2, pinned to revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. It is used without fine-tuning. No model weights are committed. See model provenance.

Scope and limitations

• The benchmark is small, curated, synthetic, English-only, and limited to one fictional support-operations domain.
• Relevance labels are manually authored.
• Development/holdout separation was procedural and versioned. Holdout labels existed in the repository before gate freeze, so cryptographic blindness and absence of human peeking are not demonstrated.
• The gate is a deterministic retrieval decision policy, not a calibrated confidence model.
• Answer generation, ingestion, deployment, authentication, monitoring, and production operations are outside the repository's scope.
• No client, employer, or confidential information is included.

License

The repository is available under the MIT License.

Package version is 0.1.0. The completed study release is tagged v1.0.0.