Gateway

Single-binary LLM access point in Rust (binary: gw): OpenAI- and Anthropic-compatible APIs in front of pluggable model providers, with key-based auth, quotas, rate limits, failover, and a billing ledger.

Documentation: cocoonstack.github.io/gateway (source in docs/).

Highlights

• OpenAI + Anthropic compatible surface — /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages, /v1/embeddings, /v1/images/{generations,edits}, /v1/videos/generations + /v1/videos/{id}, /v1/audio/{speech,transcriptions,translations}, /v1/moderations, /v1/search, /v1/rerank, /v1/batches + /v1/files, /v1/models, /v1/realtime (WebSocket) — streaming and non-streaming
• Cross-protocol conversion — serve Anthropic-style /v1/messages on OpenAI-protocol models and vice versa, including streaming event mapping
• MCP gateway — /mcp/{server} proxies Model Context Protocol servers (Streamable HTTP) behind the same access keys: per-key server entitlement and tool allowlists, tools/list filtered to the allowlist, tool results reviewed by the tenant's moderator (mask or block), every tool call, denial and intervention audited, sessions bound to the key that opened them, server credentials kept in the gateway's environment as static bearers or OAuth 2.0 client credentials the gateway refreshes itself (Security model)
• Coding agents drop in — Claude Code, Codex CLI, VS Code chat (Copilot), Cursor and opencode work with a base URL and an access key; the anthropic-beta header rides through to Anthropic-wire upstreams and each client's captured wire shape replays in the live matrix (Examples)
• Reasoning on every surface — reasoning_effort / reasoning{} on /v1/chat/completions maps to each family's thinking dialect (Anthropic budget or adaptive by model generation, OpenAI effort, compatible vendors verbatim); reasoning comes back as reasoning_content + signed reasoning_details and replays into tool loops; /v1/responses forwards reasoning items and its native event stream verbatim; /v1/messages preserves signed thinking/redacted_thinking blocks end-to-end, pins reasoning traffic to its requested model, and audits tool-loop continuations against what was actually served (fail-open; tampering is a local 400)
• Staged request pipeline — a 4-layer DAG per request: model resolve / quota / cache lookup → account selection (priority, PTU-first, round-robin or latency-ranked within a tier, failover) → rate limits + engine call (retry on upstream 5xx, then the model's fallback_models chain before any byte is sent) → usage extraction, billing, cache store
• Governance built in — access-key auth, daily token quotas, QPS / QPM / TPM limits at key, product, and model level, request-level TTL cache, account cooldown and recovery, DLP redaction and blocklist plugins. Admission reserves then settles, so concurrent requests can't overshoot a quota
• Multi-tenant — keys carry a tenant; tenants get a pooled QPS bucket, a model entitlement allowlist, per-(key, model) quota defaults with an optional fallback-model degrade, key lifecycle (expiry/ban), and tenant-scoped admin tokens. Billing records charged cost and (optionally) vendor cost per row, so margin is queryable per tenant × model
• Per-user billing & enterprise audit — every ledger row attributes to an effective end user (the key's owner, else the request's x-gw-user / user hint) with a request_id, so cost rolls up per user (/admin/usage/users) and soft per-user daily budgets apply on every surface; daily and calendar-month cost budgets (charged micro-dollars) cap a tenant pool, each key and each end user, with optional month-to-month rollover of the unspent remainder, raising a webhook alert when reached. Per-tenant content policy adds blocklist action tiers (block / flag / shadow), regex recognizers, secret masking, and an external-moderation seam with an AWS Bedrock Guardrails backend (deny on blocked policies, mask anonymized PII); every hit is recorded without prompt text. An admin-operation trail (key CRUD / config / reload, with source IP) and optional at-rest content retention complete the audit surfaces (/admin/audit/*)
• Fleet-ready — run N instances behind a load balancer: Postgres shares config (versioned + a change feed), the access-key table, the ledger/files/batches store, and a distributed batch queue any instance drains; Redis shares rate/quota/TPM counters, monthly cost counters, account health, and optionally the response cache. Single-node stays zero-dependency
• Providers behind traits — engines talk to upstreams through a Transport seam; accounts with a real endpoint go over HTTP (reqwest + rustls), accounts without one are served by a deterministic in-process mock; AWS Bedrock (Claude, Llama, Cohere natively; every model through Converse) via SigV4 or API key with EventStream streaming
• Fast — the whole pipeline (auth, admission, DLP, engine, billing) costs ~25 µs per request in-process; over HTTP one node serves ~90k requests/s at p99 under 10 ms on small bodies and ~40k/s at 256 concurrency on 52 KB / 13k-token prompts, mock upstream (numbers and method)
• Observability built in — Prometheus /metrics (per-route request/status counters, per-pipeline-stage latency, token counters), structured access logs, and one OTLP span per request (route, model, tenant, user, tokens, routing decisions; W3C traceparent joins the caller's trace) as soon as OTEL_EXPORTER_OTLP_ENDPOINT names a collector
• One binary, one YAML — no external services required to start; in-process state by default, SQLite for one-node durability, Postgres + Redis for a shared fleet; graceful shutdown
• Web control plane — a role-aware browser console (control-plane/): members see their own usage and charges, tenant admins manage keys and security events under gateway-enforced tenant scope, system admins get fleet economics, instance health, config publish/rollback and audit. Go BFF + React UI, its own identity store, everything proxied through the gateway admin API

Quick Start

# Run with the embedded demo config (mock upstreams, zero egress)
cargo run -p gw-server

# Chat completion
curl -s localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}]}'

# Anthropic-style messages, streaming SSE
curl -sN localhost:8080/v1/messages \
  -H 'x-api-key: ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"claude-sonnet","stream":true,"max_tokens":128,"messages":[{"role":"user","content":"hi"}]}'

# Your own config
GW_CONFIG=conf/gateway.yaml cargo run -p gw-server
gw --version                       # the built binary takes no other arguments

# Go live: give an account `endpoint` + `api_key_env` in the config — that's it.
# GW_TRANSPORT=mock forces zero egress; GW_TRANSPORT=http disables the mock.

Guides: Examples · API · Providers · Governance · Observability · Deployment · Configuration · Architecture · Development · Performance · Security · Roadmap

Docker

docker build -t gateway .
docker run -p 8080:8080 gateway            # embedded demo config
docker run -p 8080:8080 -v $PWD/conf/gateway.yaml:/etc/gateway.yaml \
  -e GW_CONFIG=/etc/gateway.yaml gateway

The image binds 0.0.0.0 (GW_HOST) and ships a /health HEALTHCHECK. Published multi-arch (amd64 + arm64) to ghcr.io/cocoonstack/gateway on v* tags, alongside ghcr.io/cocoonstack/gateway-control-plane and control-plane binary tarballs (linux/darwin × amd64/arm64) on the GitHub Release.

Development

make all      # fmt + lint + test + build
make test     # cargo test --workspace
make lint     # clippy -D warnings
make fmt      # cargo fmt --all
make deny     # cargo deny check (advisories + licenses)
make release  # optimized `gw` binary (--locked)
make docker   # build the container image

CI runs fmt/clippy/test + cargo deny on every push to main and every PR; tagged v* pushes build multi-arch binaries (release) and a multi-arch image (docker).

License

This project is licensed under the GNU Affero General Public License v3.0. See LICENSE.