RAGLean

A cost/latency optimization layer for production RAG pipelines: retrieval-aware semantic caching, complexity-based model routing, and context compression, backed by an eval harness that checks answer quality doesn't degrade. See docs/ARCHITECTURE.md for the full design and roadmap.

General LLM gateways (LiteLLM, Portkey, Helicone) optimize at the API-call level. RAGLean is scoped narrower — it understands the retrieval step of a RAG pipeline, not just the final prompt — and ships with a benchmark suite that proves the numbers rather than asserting them.

Quickstart

pip install -r requirements.txt

# Run the test suite
python -m pytest -q

# Run the full benchmark (quality check + 500-request scale check), writes
# data/benchmark_report.json
python -m raglean.benchmark.run_benchmark --n 500

# Start the API proxy (MockProvider by default, no API key needed)
uvicorn raglean.proxy.app:app --reload

# Start the observability dashboard
streamlit run raglean/dashboard/streamlit_app.py

Minimal integration example:

from raglean.pipeline import RAGLeanPipeline
from raglean.providers.mock_provider import MockProvider  # swap for OpenAICompatibleProvider

pipeline = RAGLeanPipeline(provider=MockProvider(), baseline_model="gpt-4o")

result = pipeline.run(
    query="What is the refund window?",
    context_chunks=retrieved_chunks,  # whatever your retriever already returns
)
print(result.answer, result.tier, result.event.cost_saved_usd)

What it does

1. Semantic cache — dedupes at the query-embedding level, not the full assembled prompt, so paraphrases and context-reordering don't cause unnecessary cache misses the way a naive full-prompt cache would.
2. Complexity router — scores each query (length, reasoning keywords, multi-hop structure, retrieval-confidence) and sends simple factual lookups to a cheap model, reserving expensive models for queries that actually need them.
3. Context compressor — extractively trims retrieved chunks to the sentences most relevant to the query before they hit the LLM, cutting input tokens.
4. Budget tracker — per-tenant cost ledger and hard budget caps.
5. Eval harness — runs every optimized answer and a true unoptimized-baseline answer through the same provider, and scores both against gold answers, so cost/latency savings are reported alongside a quality check, not instead of one.

Real numbers (reproducible via python -m raglean.benchmark.run_benchmark)

Everything below comes from data/benchmark_report.json, generated by the command above using MockProvider (deterministic, latency-simulated, illustrative published pricing — see the caveats section). Re-run the command yourself to regenerate it.

Quality check (27 dataset items, real baseline comparison per item):

• Mean answer F1: optimized 0.103 vs. baseline 0.107 — within noise of each other.
• 55.6% of items had zero quality loss (optimized F1 ≥ baseline F1).
• Cost saved: 91.2% · Latency saved: 71.0% · cache hit rate: 29.6% (single pass, mostly unique questions).

Scale check (500 simulated requests, Zipfian repeat-question traffic — realistic for FAQ/support-style RAG traffic):

• Cache hit rate: 96.2%
• Cost saved: 99.6% ($0.4525 → $0.0018 simulated spend)
• Latency saved: 98.0% (974ms → 19ms average)
• Tier breakdown: 481 cache / 14 cheap / 4 mid / 1 premium

The scale-check hit rate is high because the simulated traffic is 500 draws from a small (27-question) FAQ set with a Zipfian popularity skew — a fair model of a support chatbot, not of open-ended RAG over a large unique document set. Don't quote 96% without that context.

Honest caveats (read before quoting numbers publicly)

• MockProvider, not a real LLM. All cost/latency numbers above come from a deterministic, latency-simulated provider so the whole project runs with zero API keys and zero cost. The system (caching, routing, compression, accounting) is real and fully tested; the token counts and per-model behavior are simulated. Swap in OpenAICompatibleProvider (raglean/providers/openai_provider.py) with real API keys to get real numbers — no other code changes needed.
• Pricing table is illustrative and will drift. providers/pricing.py uses approximate published per-token prices; refresh from each provider's live pricing page before citing numbers in a report or interview.
• Default embedder is offline, not semantic-rich. LocalHashEmbedder (bag-of-words hashing) requires zero network/model download, which is why paraphrase-level cache hits (e.g. "refund window" vs. "how do I get my money back") mostly miss in this demo — exact/near-exact repeats hit reliably. Swapping in SentenceTransformerEmbedder (raglean/embeddings.py, needs pip install sentence-transformers) is a one-line config change that closes this gap; not enabled by default to keep the project dependency-light and reproducible offline.
• Answer-quality metric is token-F1 against gold answers, not an LLM judge. It checks whether the right content survived the optimizations, not fluency. A RAGAS-style LLM-judge is a documented upgrade once API budget is available.

Project layout

raglean/
  embeddings.py        shared embedding backend (offline default + prod swap-in)
  pipeline.py           orchestrates cache -> router -> compressor -> provider -> budget
  providers/             provider abstraction, pricing table, mock + OpenAI-compatible
  cache/                 semantic cache + vector index (numpy, FAISS swap-in documented)
  router/                complexity-based model router
  compression/            extractive context compressor
  budget/                 per-tenant cost ledger + budget caps
  proxy/                  FastAPI integration point
  eval/                   QA dataset + F1 metric + eval harness
  benchmark/              synthetic workload + full benchmark report
  dashboard/              Streamlit observability UI
tests/                    pytest suite (20 tests covering every module above)
docs/ARCHITECTURE.md      full design doc + phased roadmap
data/benchmark_report.json   output of the last benchmark run

Roadmap

See docs/ARCHITECTURE.md for the phased plan — next up: a trained (not heuristic) complexity classifier, a real sentence-embedding backend by default, a public QA benchmark (HotpotQA/NaturalQuestions subset) for the eval harness, and a hosted dashboard demo.