From 1a24d00771659a29a94fbff24f465bfb3782e7a8 Mon Sep 17 00:00:00 2001 From: Igor Benav Date: Sat, 8 Aug 2026 01:43:59 -0300 Subject: [PATCH 1/2] add neon as an option --- README.md | 19 +- backend/.env.example | 7 + docs/getting-started/installation.md | 10 + docs/index.md | 6 + .../configuration/environment-variables.md | 11 +- docs/user-guide/database/index.md | 1 + docs/user-guide/database/neon.md | 177 ++++++++++++++++++ docs/user-guide/production.md | 2 + zensical.toml | 1 + 9 files changed, 231 insertions(+), 3 deletions(-) create mode 100644 docs/user-guide/database/neon.md diff --git a/README.md b/README.md index 43172504..ccfaca5d 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ * Redis or Memcached caching (`@cache` decorator + provider API) * **Plugin-ready `bp` CLI** - generate compose files, audit env, mount third-party command/feature plugins * Docker Compose for local / prod / nginx-fronted (generated by the CLI) +* Runs on any Postgres - the bundled container, or serverless via [Neon](https://neon.com) ([guide](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/)) ## Why and When to use it @@ -64,7 +65,7 @@ * **App**: FastAPI [app factory](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/project-structure/), env-aware docs exposure * **Auth**: [server-side sessions](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/authentication/sessions/), CSRF, [OAuth](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/authentication/), [API keys](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/authentication/permissions/) -* **DB**: Postgres + SQLAlchemy 2.0, [Alembic migrations](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/migrations/) with prod-confirm gate +* **DB**: Postgres + SQLAlchemy 2.0, [Alembic migrations](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/migrations/) with prod-confirm gate - local container or serverless ([Neon](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/)) * **CRUD**: [FastCRUD generics](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/crud/) * **Caching**: [decorator + provider API](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/caching/) (Redis or Memcached) * **Queues**: [Taskiq workers](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/background-tasks/) (Redis or RabbitMQ) @@ -150,7 +151,7 @@ docker compose up --build # → http://127.0.0.1:8000 (Swagger at /docs) ``` -**Without Docker** (Postgres + Redis required locally): +**Without Docker** (Postgres + Redis required locally - or skip local Postgres with [Neon](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/)): ```bash cd backend @@ -183,6 +184,20 @@ uv tool install --editable ./cli More examples (superuser creation, tiers, rate limits, admin usage, plugin authoring) in the [docs](https://benavlabs.github.io/FastAPI-boilerplate/). +## Sponsors + +Neon + +**[Neon](https://neon.com)** supports this project with database credits for our open-source infrastructure - thank you. Neon is serverless Postgres: compute scales to zero when idle, and you can branch a database like you branch code (handy for per-PR preview environments). + +It's also a drop-in option for your own build, and **free to start** - the free plan is permanent rather than a trial (no credit card), with enough storage and compute for dev, staging, and small production workloads. Point `DATABASE_URL` at a Neon project and the local Postgres container becomes optional - no code changes: + +```env +DATABASE_URL=postgresql+asyncpg://user:password@ep-xxx-pooler.region.aws.neon.tech/neondb?ssl=require +``` + +→ [Full Neon guide](https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/) (connection-string conversion, pooled vs. direct endpoints, scale-to-zero pool settings). Any other managed Postgres works the same way. + ## Contributing Read [contributing](CONTRIBUTING.md). diff --git a/backend/.env.example b/backend/.env.example index 02f3478e..1b884bd8 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -18,6 +18,13 @@ POSTGRES_SYNC_PREFIX=postgresql:// POSTGRES_ASYNC_PREFIX=postgresql+asyncpg:// CREATE_TABLES_ON_STARTUP=true +# Hosted / serverless Postgres (Neon, RDS, Cloud SQL, ...): set DATABASE_URL and +# it overrides every POSTGRES_* value above. Note the asyncpg spelling — the +# driver prefix is postgresql+asyncpg:// and TLS is ssl=require, NOT sslmode=require. +# Still set POSTGRES_PASSWORD: the production validator checks that var directly. +# Guide: https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/ +# DATABASE_URL=postgresql+asyncpg://user:password@host.example.com/dbname?ssl=require + # =================================== # Cache Configuration # =================================== diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 87441f47..088ce6e5 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -241,6 +241,16 @@ ADMIN_PASSWORD=your-secure-password See [Environment Variables](../user-guide/configuration/environment-variables.md) for the complete reference. +### Using a hosted database instead + +You don't have to run Postgres yourself. Set `DATABASE_URL` and it overrides every `POSTGRES_*` variable above — the local container (or local install) becomes unnecessary: + +```env +DATABASE_URL=postgresql+asyncpg://user:password@host.example.com/dbname?ssl=require +``` + +[Neon](../user-guide/database/neon.md) — serverless Postgres, and a sponsor of this project — is free to start (a permanent free plan, no credit card) and has a step-by-step guide here, including the connection-string conversion asyncpg needs. Any managed Postgres works the same way. + ## Verification After installing, verify everything works: diff --git a/docs/index.md b/docs/index.md index 9a50641e..0c406080 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,6 +43,8 @@ This boilerplate leverages cutting-edge Python technologies: - **[Taskiq](https://taskiq-python.github.io/)** - Async-first task queue with Redis/RabbitMQ brokers - **[Docker](https://docs.docker.com/compose/)** - Containerization for easy deployment +Postgres is the only hard requirement of those you have to provide — run the bundled container, or skip it entirely and point `DATABASE_URL` at a hosted provider. **[Neon](user-guide/database/neon.md)** (serverless Postgres, permanent free plan, database branching) is documented end to end. + ## Key Features ### Performance & Scalability @@ -129,6 +131,10 @@ Fastro is the free **foundation**. **[FastroAI](https://fastro.ai)** builds on t

+## Sponsors + +**[Neon](https://neon.com)** supports this project with database credits for our open-source infrastructure - thank you. Neon is serverless Postgres with a permanent free plan, and a drop-in option for your own build: see the **[Neon guide](user-guide/database/neon.md)**. + ## Community & Support - **[Discord Community](community.md)** - Join our Discord server to connect with other developers diff --git a/docs/user-guide/configuration/environment-variables.md b/docs/user-guide/configuration/environment-variables.md index 7665399e..da0cd212 100644 --- a/docs/user-guide/configuration/environment-variables.md +++ b/docs/user-guide/configuration/environment-variables.md @@ -41,7 +41,16 @@ CREATE_TABLES_ON_STARTUP=true | `POSTGRES_POOL_SIZE` | `20` | SQLAlchemy connection pool size | | `POSTGRES_MAX_OVERFLOW` | `0` | Pool overflow connections | -If you set `DATABASE_URL` directly, it overrides the constructed URL. +If you set `DATABASE_URL` directly, it overrides the constructed URL — use it whenever the connection needs more than host/port/credentials, such as a managed provider that requires TLS: + +```env +DATABASE_URL=postgresql+asyncpg://user:password@host.example.com/dbname?ssl=require +``` + +The URL must use the `postgresql+asyncpg://` prefix, and query parameters are passed to `asyncpg` (which spells TLS `ssl=require`, not libpq's `sslmode=require`). See [Neon](../database/neon.md) for a full walkthrough with a serverless provider. + +!!! note + The production validator checks `POSTGRES_PASSWORD` on its own, not the credentials inside `DATABASE_URL`. If you only set `DATABASE_URL`, also set `POSTGRES_PASSWORD` to a non-default value or the app will refuse to boot in production. ## Cache diff --git a/docs/user-guide/database/index.md b/docs/user-guide/database/index.md index 649528a9..3a11e0fa 100644 --- a/docs/user-guide/database/index.md +++ b/docs/user-guide/database/index.md @@ -8,6 +8,7 @@ Learn how to work with the database layer in the FastAPI Boilerplate. This secti - **[Schemas](schemas.md)** - Validate and serialize data with Pydantic - **[CRUD Operations](crud.md)** - Database access via FastCRUD - **[Migrations](migrations.md)** - Manage schema changes with Alembic +- **[Neon](neon.md)** - Run on serverless Postgres instead of a local container ## Quick Overview diff --git a/docs/user-guide/database/neon.md b/docs/user-guide/database/neon.md new file mode 100644 index 00000000..c4e0b5c0 --- /dev/null +++ b/docs/user-guide/database/neon.md @@ -0,0 +1,177 @@ +# Neon (Serverless Postgres) + +[Neon](https://neon.com) is serverless Postgres — a managed database that scales compute to zero when idle and lets you branch a database like you branch code. It's a drop-in replacement for the local Postgres container: the boilerplate needs **one environment variable**, no code changes. + +!!! info "Neon sponsors this project" + Neon supports Fastro · The Benav Labs FastAPI Boilerplate with database credits for our open-source infrastructure. It's one option among many — the boilerplate runs on any Postgres 14+ (local, RDS, Cloud SQL, Supabase, your own box) — but it's the one we use and the one these instructions are tested against. + +## When it's a good fit + +- **You don't want a Postgres container.** No `docker compose up` for the database, no volume to reset. +- **Preview environments.** Each Neon branch is a copy-on-write clone of your data, created in seconds — one database per PR, per developer, per test run. +- **Bursty or low-traffic APIs.** Compute suspends while idle, so a staging environment nobody touched all weekend costs nothing to keep around. + +Stick with the bundled Postgres container if you want offline development or a fully self-hosted stack. + +## What it costs + +There's a **free plan that isn't a trial** — no credit card, no expiry — which is enough to run this boilerplate's dev, staging, and hobby-project workloads. At the time of writing it includes, per project: + +| | Free plan | +|---|---| +| Storage | 0.5 GB | +| Compute | 100 CU-hours/month, autoscaling up to 2 CU (8 GB RAM) | +| Projects / branches | 100 projects, 10 branches each | +| Network egress | 5 GB | +| Included regardless of plan | Autoscaling, branching, read replicas, connection pooling, extensions, API + CLI | + +Because compute scales to zero, an idle staging database burns close to nothing of that CU-hour budget — the meter effectively runs only while you're querying. + +!!! warning "What happens at the limit" + Exceeding a monthly limit **suspends the compute until the next billing month** rather than generating a surprise bill. That's friendly for a side project and dangerous for anything you care about: production means a paid plan, or at minimum an alert on your usage. Check [neon.com/pricing](https://neon.com/pricing) for current numbers — the table above will drift. + +## 1. Create a project + +1. Sign up at [neon.com](https://neon.com) and create a project (pick the region closest to where your API runs — every query pays that round trip). +2. Open **Connection Details** in the dashboard and copy the connection string. It looks like this: + +```text +postgresql://neondb_owner:npg_xxxxxxxx@ep-cool-darkness-123456-pooler.us-east-2.aws.neon.tech/neondb?sslmode=require&channel_binding=require +``` + +## 2. Convert it for asyncpg + +The connection string Neon hands you is written for **libpq** (`psql`, psycopg). The boilerplate's app engine is **asyncpg**, which spells its options differently. Two edits: + +| Neon gives you | Use instead | Why | +|---|---|---| +| `postgresql://` | `postgresql+asyncpg://` | Selects SQLAlchemy's async driver | +| `?sslmode=require&channel_binding=require` | `?ssl=require` | SQLAlchemy forwards unknown query params straight to `asyncpg.connect()`. It has no `sslmode` or `channel_binding` keyword — passing them raises `TypeError: connect() got an unexpected keyword argument 'sslmode'`. asyncpg calls the parameter `ssl`. | + +The result, in `backend/.env`: + +```env +DATABASE_URL=postgresql+asyncpg://neondb_owner:npg_xxxxxxxx@ep-cool-darkness-123456-pooler.us-east-2.aws.neon.tech/neondb?ssl=require + +# The production validator reads POSTGRES_PASSWORD directly — keep it in sync +# with the password in the URL, or set it to any other non-default value. +POSTGRES_PASSWORD=npg_xxxxxxxx + +# Let Alembic own the schema instead of creating tables at boot +CREATE_TABLES_ON_STARTUP=false +``` + +`DATABASE_URL` takes priority over the individual `POSTGRES_*` variables ([settings reference](../configuration/environment-variables.md#database)), so the rest of them are ignored once it's set — including the `POSTGRES_SERVER: postgres` that Compose injects. Nothing else in your config has to change. + +!!! warning "Don't drop the `ssl` parameter" + Over a TCP connection asyncpg defaults to `sslmode=prefer`: it will use TLS if the server offers it, but silently accepts an unencrypted connection otherwise, and never verifies the certificate. Being explicit with `ssl=require` means a downgrade fails loudly instead of quietly. + + `require` encrypts the connection but — exactly like libpq — does not verify the server's certificate. For full verification use `?ssl=verify-full` and point `PGSSLROOTCERT` at a CA bundle (asyncpg otherwise looks for `~/.postgresql/root.crt` and fails if it isn't there). + +## 3. Run migrations and start + +```bash +cd backend +uv run alembic upgrade head +uv run python -m scripts.setup_initial_data # first admin user + default tier +uv run fastapi dev src/interfaces/main.py +``` + +Alembic reads the same `settings.DATABASE_URL` the app does (`backend/migrations/env.py`), so there's no second connection string to maintain. + +## Pooled vs. direct endpoints + +Neon gives every project two hostnames. The difference is the `-pooler` suffix: + +```text +ep-cool-darkness-123456-pooler.us-east-2.aws.neon.tech # pooled — via PgBouncer +ep-cool-darkness-123456.us-east-2.aws.neon.tech # direct — straight to Postgres +``` + +**Use the pooled endpoint for the API and the Taskiq worker.** Both open a connection per request/task, which is exactly the churn PgBouncer absorbs, and it raises your usable connection ceiling far above what the compute alone allows. + +**Use the direct endpoint for migrations and admin work.** Neon's PgBouncer runs in transaction mode, so session-level state doesn't survive between statements — `SET`/`RESET` (including `SET search_path`), `LISTEN`/`NOTIFY`, SQL-level `PREPARE`, and session-scoped advisory locks all behave differently there. Long multi-statement DDL is safer on a direct connection: + +```bash +# One-off override for the migration only +DATABASE_URL="postgresql+asyncpg://neondb_owner:npg_xxxxxxxx@ep-cool-darkness-123456.us-east-2.aws.neon.tech/neondb?ssl=require" \ + uv run alembic upgrade head +``` + +## Handling scale-to-zero + +An idle Neon compute suspends. Connections that were sitting in SQLAlchemy's pool are dead when it wakes, and the next request surfaces that as `SSL SYSCALL error: EOF detected` or `connection was closed in the middle of operation`. The fix is standard SQLAlchemy pool hygiene, in `backend/src/infrastructure/database/session.py`: + +```python hl_lines="6 7" +engine = create_async_engine( + settings.DATABASE_URL, + echo=False, + future=True, + pool_size=settings.POSTGRES_POOL_SIZE, + pool_pre_ping=True, # check the connection is alive before handing it out + pool_recycle=300, # drop connections idle longer than the suspend timeout + max_overflow=settings.POSTGRES_MAX_OVERFLOW, +) +``` + +Apply the same two arguments to `taskiq_engine` in `backend/src/infrastructure/taskiq/deps.py` if your worker runs against Neon too. `pool_pre_ping` costs one cheap round trip per checkout; on a database that can suspend, it's worth it. + +Also trim the pool while you're there. `POSTGRES_POOL_SIZE` defaults to `20` per process, and the real number is `pool_size × workers × replicas` — see [Scaling considerations](../production.md#database). Against a small Neon compute, `5`–`10` is usually plenty: + +```env +POSTGRES_POOL_SIZE=10 +POSTGRES_MAX_OVERFLOW=5 +``` + +## Docker Compose without the local database + +Once `DATABASE_URL` points at Neon, the `postgres` service in your generated compose file is dead weight. Delete the service, its volume, and the two `depends_on` entries that wait on it: + +```yaml hl_lines="8 9" +services: + api: + env_file: + - ./backend/.env + environment: + CACHE_REDIS_HOST: redis + depends_on: + # postgres: ← remove + # condition: service_healthy + redis: + condition: service_healthy +``` + +Redis still runs locally — Neon replaces Postgres only. Regenerating with `uv run bp deploy generate local` brings the Postgres service back, so keep the edit in mind after a regen. + +## Database branching for previews + +The reason to reach for Neon over a plain managed Postgres. A branch is a copy-on-write clone — full data, created in seconds, thrown away just as fast: + +```bash +# One database per pull request +neon branches create --name pr-142 --parent main + +# Print a connection string for it +neon connection-string pr-142 +``` + +Feed that string into your preview environment's `DATABASE_URL` (converted as in step 2 — the CLI prints the libpq form) and the preview runs on real, isolated data. Delete the branch when the PR merges. The same trick works for integration tests that need a real Postgres: branch, migrate, run, delete. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `TypeError: connect() got an unexpected keyword argument 'sslmode'` | libpq-style parameter left in the URL | Replace `sslmode=require&channel_binding=require` with `ssl=require` | +| `TypeError: connect() got an unexpected keyword argument 'channel_binding'` | Same | Same | +| `ProductionSecurityError: Database is using default credentials` | `POSTGRES_PASSWORD` is still `postgres`; the validator checks that variable, not `DATABASE_URL` | Set `POSTGRES_PASSWORD` to your Neon password as well | +| `SSL SYSCALL error: EOF detected` after an idle period | Compute suspended, pooled connections went stale | Add `pool_pre_ping=True` and `pool_recycle` (see above) | +| First request after idle takes a few hundred ms | Compute resuming from zero | Expected; disable scale-to-zero on the branch if latency matters more than cost | +| `prepared statement "__asyncpg_stmt_x__" already exists` | Prepared-statement reuse across a transaction-mode pooler | Add `&prepared_statement_cache_size=0` to the URL, or use the direct endpoint | +| `password authentication failed` with a correct password | Special characters in the password aren't URL-encoded | Percent-encode them (`@` → `%40`, `#` → `%23`, …) | + +## Related + +- [Migrations](migrations.md) — Alembic workflow and the production confirm gate +- [Environment Variables](../configuration/environment-variables.md#database) — every database setting +- [Production Deployment](../production.md) — validator, pool sizing, scaling +- [Neon docs](https://neon.com/docs) — branching, autoscaling, the `neon` CLI diff --git a/docs/user-guide/production.md b/docs/user-guide/production.md index 297cf8d8..8faaf786 100644 --- a/docs/user-guide/production.md +++ b/docs/user-guide/production.md @@ -315,6 +315,8 @@ Watch `database_pool_size × api_workers + worker_concurrency × taskiq_workers` Use a connection pooler (PgBouncer, RDS Proxy) at scale. The boilerplate's `DATABASE_URL` accepts a pooler endpoint identically. +Managed Postgres works the same way — point `DATABASE_URL` at the provider and leave the rest of the config alone. [Neon](database/neon.md) (serverless, scale-to-zero, database branching for preview environments) is the setup we document end to end, including the TLS parameter, pooled vs. direct endpoints, and the `pool_pre_ping` setting that idle-suspending computes need. + ### Redis The defaults use four separate DB numbers (`CACHE_REDIS_DB=0`, `SESSION_REDIS_DB=1`, `RATE_LIMITER_REDIS_DB=1`, `TASKIQ_REDIS_DB=3`) on the **same** Redis instance. Fine for small deployments. At scale, split sessions and the cache onto different Redis clusters — sessions are small and durability-sensitive; the cache is large, eviction-tolerant, and high-traffic. Mixing them puts your sessions at risk during cache memory pressure. diff --git a/zensical.toml b/zensical.toml index 6b1b3c7e..253b42a9 100644 --- a/zensical.toml +++ b/zensical.toml @@ -100,6 +100,7 @@ name = "FastAPI on PyPI" { "Schemas" = "user-guide/database/schemas.md" }, { "CRUD Operations" = "user-guide/database/crud.md" }, { "Migrations" = "user-guide/database/migrations.md" }, + { "Neon (Serverless Postgres)" = "user-guide/database/neon.md" }, ] }, { "API" = [ { "Overview" = "user-guide/api/index.md" }, From 28f4eff8aff5490d6c3fafd61ed9557901490bed Mon Sep 17 00:00:00 2001 From: Igor Benav Date: Sat, 8 Aug 2026 02:04:20 -0300 Subject: [PATCH 2/2] fix production validator to read db credentials from DATABASE_URL and enable pool pre-ping by default --- backend/.env.example | 16 +++-- backend/src/infrastructure/config/settings.py | 15 +++- .../src/infrastructure/database/session.py | 2 + .../security/production_validator.py | 68 ++++++++++++++++++- .../security/test_production_validator.py | 59 ++++++++++++++++ .../configuration/environment-variables.md | 7 +- docs/user-guide/database/neon.md | 34 ++++------ docs/user-guide/production.md | 8 ++- 8 files changed, 170 insertions(+), 39 deletions(-) diff --git a/backend/.env.example b/backend/.env.example index 1b884bd8..16d1ba8d 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -18,13 +18,21 @@ POSTGRES_SYNC_PREFIX=postgresql:// POSTGRES_ASYNC_PREFIX=postgresql+asyncpg:// CREATE_TABLES_ON_STARTUP=true -# Hosted / serverless Postgres (Neon, RDS, Cloud SQL, ...): set DATABASE_URL and -# it overrides every POSTGRES_* value above. Note the asyncpg spelling — the -# driver prefix is postgresql+asyncpg:// and TLS is ssl=require, NOT sslmode=require. -# Still set POSTGRES_PASSWORD: the production validator checks that var directly. +# Hosted / serverless Postgres (Neon, RDS, Cloud SQL, ...): DATABASE_URL overrides +# every POSTGRES_* value above. Mind the asyncpg spelling — the driver prefix is +# postgresql+asyncpg:// and TLS is ssl=require, NOT libpq's sslmode=require. # Guide: https://benavlabs.github.io/FastAPI-boilerplate/user-guide/database/neon/ # DATABASE_URL=postgresql+asyncpg://user:password@host.example.com/dbname?ssl=require +# Connection pool. POSTGRES_POOL_PRE_PING tests a connection before handing it out, +# so connections dropped by an idle timeout or a suspended serverless compute are +# replaced instead of failing a request. POSTGRES_POOL_RECYCLE discards connections +# older than N seconds (-1 disables); set it below your provider's idle timeout. +POSTGRES_POOL_SIZE=20 +POSTGRES_MAX_OVERFLOW=0 +POSTGRES_POOL_PRE_PING=true +POSTGRES_POOL_RECYCLE=-1 + # =================================== # Cache Configuration # =================================== diff --git a/backend/src/infrastructure/config/settings.py b/backend/src/infrastructure/config/settings.py index e91a6d3c..81cc5fb6 100644 --- a/backend/src/infrastructure/config/settings.py +++ b/backend/src/infrastructure/config/settings.py @@ -2,6 +2,7 @@ import os from enum import StrEnum +from pydantic import Field from pydantic_settings import BaseSettings from starlette.config import Config @@ -53,6 +54,15 @@ class DatabaseSettings(BaseSettings): POSTGRES_POOL_SIZE: int = config("POSTGRES_POOL_SIZE", default=20, cast=int) POSTGRES_MAX_OVERFLOW: int = config("POSTGRES_MAX_OVERFLOW", default=0, cast=int) + POSTGRES_POOL_PRE_PING: bool = config("POSTGRES_POOL_PRE_PING", default=True, cast=bool) + POSTGRES_POOL_RECYCLE: int = config("POSTGRES_POOL_RECYCLE", default=-1, cast=int) + + # A field rather than a lookup inside DATABASE_URL, so callers can tell an + # explicit URL apart from one built out of the POSTGRES_* parts. + DATABASE_URL_OVERRIDE: str | None = Field( + default=config("DATABASE_URL", default=None), + validation_alias="DATABASE_URL", + ) @property def DATABASE_URL(self) -> str: @@ -61,9 +71,8 @@ def DATABASE_URL(self) -> str: Checks for DATABASE_URL environment variable first (production pattern), then falls back to constructing from individual components (development pattern). """ - direct_url = config("DATABASE_URL", default=None) - if direct_url: - return direct_url + if self.DATABASE_URL_OVERRIDE: + return self.DATABASE_URL_OVERRIDE return ( f"{self.POSTGRES_ASYNC_PREFIX}{self.POSTGRES_USER}:" diff --git a/backend/src/infrastructure/database/session.py b/backend/src/infrastructure/database/session.py index 04cf6f25..e2b1c875 100644 --- a/backend/src/infrastructure/database/session.py +++ b/backend/src/infrastructure/database/session.py @@ -11,6 +11,8 @@ future=True, pool_size=settings.POSTGRES_POOL_SIZE, max_overflow=settings.POSTGRES_MAX_OVERFLOW, + pool_pre_ping=settings.POSTGRES_POOL_PRE_PING, + pool_recycle=settings.POSTGRES_POOL_RECYCLE, ) local_session = async_sessionmaker(bind=engine, class_=AsyncSession, expire_on_commit=False) diff --git a/backend/src/infrastructure/security/production_validator.py b/backend/src/infrastructure/security/production_validator.py index f212667f..b3e3a0ff 100644 --- a/backend/src/infrastructure/security/production_validator.py +++ b/backend/src/infrastructure/security/production_validator.py @@ -5,6 +5,7 @@ """ import re +from urllib.parse import unquote, urlsplit from ..config.settings import EnvironmentOption, Settings from ..logging import get_logger @@ -187,7 +188,7 @@ def _validate_critical_security(self) -> list[str]: if self._is_database_using_default_credentials(): errors.append( - "Database is using default credentials (POSTGRES_PASSWORD='postgres'). " + "Database is using default credentials (password 'postgres'). " "This is a well-known default that attackers will try first. " "Use a strong, unique password for production." ) @@ -232,6 +233,13 @@ def _validate_warning_security(self) -> None: redis_warnings = self._check_redis_security() warnings.extend(redis_warnings) + if self._is_database_url_without_password(): + warnings.append( + "DATABASE_URL is set but contains no password. This is expected with " + "IAM or certificate-based authentication, but is a mistake otherwise — " + "confirm the database is not reachable without credentials." + ) + if self._is_cors_too_permissive(): warnings.append( "CORS_ORIGINS is set to '*' (allow all origins). This can enable " @@ -358,6 +366,42 @@ def _is_admin_access_completely_open(self) -> bool: """ return False + @staticmethod + def _password_from_url(url: str) -> str | None: + """Extract the password component of a database URL. + + Args: + url: A database URL, with or without credentials. + + Returns: + The decoded password, or None if the URL carries none or cannot be parsed. + """ + try: + password = urlsplit(url).password + except ValueError: + return None + + return unquote(password) if password is not None else None + + def _effective_database_password(self) -> str | None: + """Get the password the application will actually connect with. + + A `DATABASE_URL` in the environment overrides every `POSTGRES_*` setting, + so a deployment against a managed provider (Neon, RDS, Cloud SQL) carries + its real credentials in that URL while `POSTGRES_PASSWORD` keeps its + default. Reading `POSTGRES_PASSWORD` alone would flag such a deployment as + insecure and refuse to start. + + Returns: + The password embedded in `DATABASE_URL` when one is set, otherwise + `POSTGRES_PASSWORD`. None means an explicit URL was given but carries + no password at all, or could not be parsed. + """ + if not self.settings.DATABASE_URL_OVERRIDE: + return self.settings.POSTGRES_PASSWORD + + return self._password_from_url(self.settings.DATABASE_URL_OVERRIDE) + def _is_database_using_default_credentials(self) -> bool: """Check if database is using well-known default credentials. @@ -369,7 +413,7 @@ def _is_database_using_default_credentials(self) -> bool: commonly targeted by attackers. Production systems should use strong, unique passwords. """ - return self.settings.POSTGRES_PASSWORD == "postgres" + return self._effective_database_password() == "postgres" def _is_database_password_empty(self) -> bool: """Check if database password is empty or missing. @@ -380,8 +424,26 @@ def _is_database_password_empty(self) -> bool: Note: Empty database passwords leave the database completely unprotected and accessible to anyone who can reach it. + + A `DATABASE_URL` with no password at all is not treated as an error — + authentication may be handled outside the connection string (IAM, + client certificates, a trusted socket). That case is warned about + instead, since it cannot be verified from here. + """ + password = self._effective_database_password() + if password is None: + return False + + return not password or password.strip() == "" + + def _is_database_url_without_password(self) -> bool: + """Check if an explicit DATABASE_URL carries no password. + + Returns: + True if `DATABASE_URL` is set but has no password component, + False otherwise. """ - return not self.settings.POSTGRES_PASSWORD or self.settings.POSTGRES_PASSWORD.strip() == "" + return bool(self.settings.DATABASE_URL_OVERRIDE) and self._effective_database_password() is None def _check_redis_security(self) -> list[str]: """Check Redis security configuration for all Redis instances. diff --git a/backend/tests/unit/infrastructure/security/test_production_validator.py b/backend/tests/unit/infrastructure/security/test_production_validator.py index 6b2aac1a..924e7a69 100644 --- a/backend/tests/unit/infrastructure/security/test_production_validator.py +++ b/backend/tests/unit/infrastructure/security/test_production_validator.py @@ -21,6 +21,7 @@ def create_mock_settings(self, **overrides): "ENVIRONMENT": EnvironmentOption.PRODUCTION, "SECRET_KEY": "xF9mWqP3nL7vBfKsRt8HjZ2CyE5QaM6NuV4DgX1SpY7LwB9KzT3RhI0UoJ5PcA2MvS8", "POSTGRES_PASSWORD": "secure_db_password", + "DATABASE_URL_OVERRIDE": None, "REDIS_PASSWORD": "secure_redis_password", "CACHE_BACKEND": "memcached", "RATE_LIMITER_BACKEND": "memcached", @@ -135,6 +136,64 @@ def test_empty_database_password_raises_error(self): assert "Database password is empty" in str(exc_info.value) + def test_database_url_password_overrides_postgres_password(self): + """Test that credentials in DATABASE_URL are what gets validated. + + A managed provider (Neon, RDS, Cloud SQL) carries its credentials in + DATABASE_URL while POSTGRES_PASSWORD keeps its default — that must not + be reported as insecure. + """ + settings = self.create_mock_settings( + POSTGRES_PASSWORD="postgres", + DATABASE_URL_OVERRIDE="postgresql+asyncpg://db_user:not_a_real_password@db.example.com:5432/app?ssl=require", + ) + validator = ProductionSecurityValidator(settings) + + validator.validate_production_security() + + def test_default_password_in_database_url_raises_error(self): + """Test that a default password inside DATABASE_URL is still caught.""" + settings = self.create_mock_settings( + POSTGRES_PASSWORD="secure_db_password", + DATABASE_URL_OVERRIDE="postgresql+asyncpg://postgres:postgres@db.example.com:5432/app", + ) + validator = ProductionSecurityValidator(settings) + + with pytest.raises(ProductionSecurityError) as exc_info: + validator.validate_production_security() + + assert "default credentials" in str(exc_info.value) + + def test_percent_encoded_password_in_database_url_is_decoded(self): + """Test that a percent-encoded default password is decoded before checking.""" + settings = self.create_mock_settings( + DATABASE_URL_OVERRIDE="postgresql+asyncpg://postgres:postgre%73@db.example.com:5432/app", + ) + validator = ProductionSecurityValidator(settings) + + with pytest.raises(ProductionSecurityError) as exc_info: + validator.validate_production_security() + + assert "default credentials" in str(exc_info.value) + + def test_database_url_without_password_warns_instead_of_failing(self, caplog): + """Test that a passwordless DATABASE_URL warns but still starts. + + Authentication may be handled outside the connection string (IAM, + client certificates, a trusted socket), which cannot be verified here. + """ + settings = self.create_mock_settings( + POSTGRES_PASSWORD="postgres", + DATABASE_URL_OVERRIDE="postgresql+asyncpg://app_user@db.example.com:5432/app", + ) + validator = ProductionSecurityValidator(settings) + + validator.validate_production_security() + + warning_logs = [record for record in caplog.records if record.levelname == "WARNING"] + password_warnings = [log for log in warning_logs if "DATABASE_URL is set but contains no password" in log.message] + assert len(password_warnings) > 0 + def test_multiple_critical_errors_combined(self): """Test that multiple critical errors are combined in one message.""" settings = self.create_mock_settings(SECRET_KEY="insecure", POSTGRES_PASSWORD="postgres") diff --git a/docs/user-guide/configuration/environment-variables.md b/docs/user-guide/configuration/environment-variables.md index da0cd212..e9ccb21b 100644 --- a/docs/user-guide/configuration/environment-variables.md +++ b/docs/user-guide/configuration/environment-variables.md @@ -40,6 +40,8 @@ CREATE_TABLES_ON_STARTUP=true | `CREATE_TABLES_ON_STARTUP` | `true` | Auto-create tables from models on startup | | `POSTGRES_POOL_SIZE` | `20` | SQLAlchemy connection pool size | | `POSTGRES_MAX_OVERFLOW` | `0` | Pool overflow connections | +| `POSTGRES_POOL_PRE_PING` | `true` | Test a pooled connection before use, replacing ones the server has dropped | +| `POSTGRES_POOL_RECYCLE` | `-1` | Discard connections older than N seconds (`-1` disables) | If you set `DATABASE_URL` directly, it overrides the constructed URL — use it whenever the connection needs more than host/port/credentials, such as a managed provider that requires TLS: @@ -47,10 +49,7 @@ If you set `DATABASE_URL` directly, it overrides the constructed URL — use it DATABASE_URL=postgresql+asyncpg://user:password@host.example.com/dbname?ssl=require ``` -The URL must use the `postgresql+asyncpg://` prefix, and query parameters are passed to `asyncpg` (which spells TLS `ssl=require`, not libpq's `sslmode=require`). See [Neon](../database/neon.md) for a full walkthrough with a serverless provider. - -!!! note - The production validator checks `POSTGRES_PASSWORD` on its own, not the credentials inside `DATABASE_URL`. If you only set `DATABASE_URL`, also set `POSTGRES_PASSWORD` to a non-default value or the app will refuse to boot in production. +The URL must use the `postgresql+asyncpg://` prefix, and query parameters are passed to `asyncpg` (which spells TLS `ssl=require`, not libpq's `sslmode=require`). The [production validator](../production.md#the-production-validator) reads the credentials out of the URL, so the `POSTGRES_*` variables can keep their defaults. See [Neon](../database/neon.md) for a full walkthrough with a serverless provider. ## Cache diff --git a/docs/user-guide/database/neon.md b/docs/user-guide/database/neon.md index c4e0b5c0..f229f59a 100644 --- a/docs/user-guide/database/neon.md +++ b/docs/user-guide/database/neon.md @@ -53,15 +53,11 @@ The result, in `backend/.env`: ```env DATABASE_URL=postgresql+asyncpg://neondb_owner:npg_xxxxxxxx@ep-cool-darkness-123456-pooler.us-east-2.aws.neon.tech/neondb?ssl=require -# The production validator reads POSTGRES_PASSWORD directly — keep it in sync -# with the password in the URL, or set it to any other non-default value. -POSTGRES_PASSWORD=npg_xxxxxxxx - # Let Alembic own the schema instead of creating tables at boot CREATE_TABLES_ON_STARTUP=false ``` -`DATABASE_URL` takes priority over the individual `POSTGRES_*` variables ([settings reference](../configuration/environment-variables.md#database)), so the rest of them are ignored once it's set — including the `POSTGRES_SERVER: postgres` that Compose injects. Nothing else in your config has to change. +`DATABASE_URL` takes priority over the individual `POSTGRES_*` variables ([settings reference](../configuration/environment-variables.md#database)), so the rest of them are ignored once it's set — including the `POSTGRES_SERVER: postgres` that Compose injects. You can leave them at their defaults; the [production validator](../production.md#the-production-validator) reads the credentials out of `DATABASE_URL` when it's set, so a default `POSTGRES_PASSWORD` won't be mistaken for an insecure deployment. Nothing else in your config has to change. !!! warning "Don't drop the `ssl` parameter" Over a TCP connection asyncpg defaults to `sslmode=prefer`: it will use TLS if the server offers it, but silently accepts an unencrypted connection otherwise, and never verifies the certificate. Being explicit with `ssl=require` means a downgrade fails loudly instead of quietly. @@ -100,29 +96,23 @@ DATABASE_URL="postgresql+asyncpg://neondb_owner:npg_xxxxxxxx@ep-cool-darkness-12 ## Handling scale-to-zero -An idle Neon compute suspends. Connections that were sitting in SQLAlchemy's pool are dead when it wakes, and the next request surfaces that as `SSL SYSCALL error: EOF detected` or `connection was closed in the middle of operation`. The fix is standard SQLAlchemy pool hygiene, in `backend/src/infrastructure/database/session.py`: - -```python hl_lines="6 7" -engine = create_async_engine( - settings.DATABASE_URL, - echo=False, - future=True, - pool_size=settings.POSTGRES_POOL_SIZE, - pool_pre_ping=True, # check the connection is alive before handing it out - pool_recycle=300, # drop connections idle longer than the suspend timeout - max_overflow=settings.POSTGRES_MAX_OVERFLOW, -) -``` +An idle Neon compute suspends. Connections that were sitting in SQLAlchemy's pool are dead when it wakes, and the next request surfaces that as `SSL SYSCALL error: EOF detected` or `connection was closed in the middle of operation`. + +`POSTGRES_POOL_PRE_PING` handles this and is **on by default** — every connection is tested before it's handed to a request, so a dead one is quietly replaced instead of failing the request. Pair it with `POSTGRES_POOL_RECYCLE` to retire connections before Neon does: -Apply the same two arguments to `taskiq_engine` in `backend/src/infrastructure/taskiq/deps.py` if your worker runs against Neon too. `pool_pre_ping` costs one cheap round trip per checkout; on a database that can suspend, it's worth it. +```env +POSTGRES_POOL_RECYCLE=300 # seconds; keep it under your scale-to-zero timeout +``` -Also trim the pool while you're there. `POSTGRES_POOL_SIZE` defaults to `20` per process, and the real number is `pool_size × workers × replicas` — see [Scaling considerations](../production.md#database). Against a small Neon compute, `5`–`10` is usually plenty: +Trim the pool while you're there. `POSTGRES_POOL_SIZE` defaults to `20` per process, and the real number is `pool_size × workers × replicas` — see [Scaling considerations](../production.md#database). Against a small Neon compute, `5`–`10` is usually plenty: ```env POSTGRES_POOL_SIZE=10 POSTGRES_MAX_OVERFLOW=5 ``` +The Taskiq worker needs none of this: it uses a `NullPool` and opens a fresh connection per task, so there's nothing pooled to go stale. + ## Docker Compose without the local database Once `DATABASE_URL` points at Neon, the `postgres` service in your generated compose file is dead weight. Delete the service, its volume, and the two `depends_on` entries that wait on it: @@ -163,8 +153,8 @@ Feed that string into your preview environment's `DATABASE_URL` (converted as in |---|---|---| | `TypeError: connect() got an unexpected keyword argument 'sslmode'` | libpq-style parameter left in the URL | Replace `sslmode=require&channel_binding=require` with `ssl=require` | | `TypeError: connect() got an unexpected keyword argument 'channel_binding'` | Same | Same | -| `ProductionSecurityError: Database is using default credentials` | `POSTGRES_PASSWORD` is still `postgres`; the validator checks that variable, not `DATABASE_URL` | Set `POSTGRES_PASSWORD` to your Neon password as well | -| `SSL SYSCALL error: EOF detected` after an idle period | Compute suspended, pooled connections went stale | Add `pool_pre_ping=True` and `pool_recycle` (see above) | +| `ProductionSecurityError: Database is using default credentials` | The password inside `DATABASE_URL` really is `postgres` | Rotate it in the Neon console and update the URL | +| `SSL SYSCALL error: EOF detected` after an idle period | Compute suspended, pooled connections went stale | Keep `POSTGRES_POOL_PRE_PING=true` and set `POSTGRES_POOL_RECYCLE` (see above) | | First request after idle takes a few hundred ms | Compute resuming from zero | Expected; disable scale-to-zero on the branch if latency matters more than cost | | `prepared statement "__asyncpg_stmt_x__" already exists` | Prepared-statement reuse across a transaction-mode pooler | Add `&prepared_statement_cache_size=0` to the URL, or use the direct endpoint | | `password authentication failed` with a correct password | Special characters in the password aren't URL-encoded | Percent-encode them (`@` → `%40`, `#` → `%23`, …) | diff --git a/docs/user-guide/production.md b/docs/user-guide/production.md index 8faaf786..328a8d1f 100644 --- a/docs/user-guide/production.md +++ b/docs/user-guide/production.md @@ -11,8 +11,10 @@ When `ENVIRONMENT=production`, `infrastructure/security/production_validator.py` The app **will not start** if any of these is true: - **`SECRET_KEY` is insecure.** Default placeholder, < 32 chars, contains an obvious string ("password", "secret", "test", "dev", "default", etc.), or has a predictable pattern (repetition, all-same-char). -- **`POSTGRES_PASSWORD=postgres`** (the well-known default). Attackers try this first. -- **`POSTGRES_PASSWORD` is empty.** Database is unprotected. +- **The database password is `postgres`** (the well-known default). Attackers try this first. +- **The database password is empty.** Database is unprotected. + +The password checked is the one actually used to connect: when `DATABASE_URL` is set it's read out of that URL, otherwise it's `POSTGRES_PASSWORD`. A `DATABASE_URL` with no password at all (IAM or certificate authentication) is a warning rather than an error, since it can't be verified from here. ### Warnings (logged, app starts) @@ -315,7 +317,7 @@ Watch `database_pool_size × api_workers + worker_concurrency × taskiq_workers` Use a connection pooler (PgBouncer, RDS Proxy) at scale. The boilerplate's `DATABASE_URL` accepts a pooler endpoint identically. -Managed Postgres works the same way — point `DATABASE_URL` at the provider and leave the rest of the config alone. [Neon](database/neon.md) (serverless, scale-to-zero, database branching for preview environments) is the setup we document end to end, including the TLS parameter, pooled vs. direct endpoints, and the `pool_pre_ping` setting that idle-suspending computes need. +Managed Postgres works the same way — point `DATABASE_URL` at the provider and leave the rest of the config alone. [Neon](database/neon.md) (serverless, scale-to-zero, database branching for preview environments) is the setup we document end to end, including the TLS parameter, pooled vs. direct endpoints, and pool tuning for a compute that suspends when idle. ### Redis