I build LLM gateways — and I measure the ones I don't build.
Production routing infra for 100+ models · reproducible benchmarks · evidence over vendor claims
package main
type Engineer struct {
Now, Research, Rule, Before string
Method []string
}
// Solo founder. I run an LLM API gateway in production and publish what I learn
// tearing the rest of them apart.
var cuihuan = Engineer{
Now: "Synthorai — an LLM API gateway (Go + React, multi-cloud, blue-green on AWS)",
Research: "how AI gateways actually behave: cost, protocol fidelity, data retention",
Method: []string{"black-box probes", "reproducible benchmarks", "dated evidence"},
Rule: "Don't trust the claims. Measure the behavior.",
Before: "globalization architecture @ DiDi — i18n · l10n · data governance",
}Three tools, one thesis: every number about an AI gateway should be reproducible by a stranger.
| Project | What it measures | Try it |
|---|---|---|
awesome-ai-gateway awesome list |
A decision tree over 160+ gateways, an 8-chapter handbook on how they work and fail, a reproducible cost benchmark, and dated evidence behind every claim — including what I exclude and why. CC0, no vendor money, no affiliate links. Data ships as an API; CI-gated; refreshed daily. | 🧭 Picker · 💰 Cost calc · 🌐 Live |
| llm-gateway-bench | Black-box probe for any OpenAI-compatible gateway. Point it at a base URL with your own key and it scores you against the best on five axes — trust, price, speed, stability, cache — and fingerprints the behavior underneath. Keys never leave your machine. | 🌐 Live |
modelprobe Go |
Dependency-free availability prober: one deliberately tiny real call per model, 2xx = up, non-zero exit on any outage so CI goes red. Single static binary, stdlib only. Skips models that already served live traffic — the cheapest call is the one you don't make. |
📦 pkg.go.dev |
- 106× cost spread for the identical 100K-token job —
$0.03vs$3.01— same task, different model behind the same API. - Fake streaming: relays that buffer the entire completion, then replay it as SSE. Your TTFT metric is fiction.
- Model substitution: gateways that answer with a model you didn't ask for, while echoing the name you did.
- Inflated
usage: billed token counts that don't reconcile with what actually went upstream. - Who keeps your prompts: a data-retention / ZDR matrix built from terms and probes, not from vendor blog posts.
clients ──▶ gateway (Go) ──┬──▶ Anthropic · OpenAI · Gemini
│ ├──▶ Bedrock · Vertex (SigV4 / OAuth)
│ └──▶ regional & first-party upstreams
│
├─ protocol translation .... OpenAI ⇄ Anthropic ⇄ Gemini, SSE both ways
├─ routing ................. priority, failover, per-key channel binding
├─ accounting .............. per-token billing, prompt-cache ledger (Redis + MySQL)
├─ integrity ............... inbound redaction on the bytes that actually leave
└─ observability ........... synthetic probes, cost ledger, alerting
ship: merge → GitHub Actions → blue-green on AWS → probe → report
The interesting problems live in the seams: billing correctness (a cache-hit accounting bug is a refund), protocol fidelity (reasoning params silently downgraded upstream, streams that die mid-JSON), and honest error attribution (never dress up your own 400 as the upstream's 502).
Open PRs adding a new OpenAI-compatible provider across the agent stack:
BerriAI/litellm · pipecat-ai/pipecat · agno-agi/agno · livekit/agents · langbot-app/LangBot · danielmiessler/Fabric · charmbracelet/catwalk · anomalyco/models.dev · CherryHQ/cherry-studio-docs · yincongcyincong/MuseBot
Earlier work — awesome-privacy-chinese 
2019_nCov 
