feat(evals): eval the connecting to Postgres guide in supabase.com/docs

[czenko]

Closes DOCS-1302

Problem

database/connecting-to-postgres is pick 2 on the docs eval list. 83,627 views in 90 days, 49.1 percent negative on 104 votes. Nothing measures whether it gets an agent to a working connection for a serverless runtime.

Solution

• Adds build-docs-004-postgres-connection, suite: regression, services: [].
• Seeds a Vercel-style handler, one table, CONNECT.md holding the three connection strings the dashboard Connect panel prints with nothing picked, and .env.example with DATABASE_URL empty.
• Six file checks grade the string, port, username, prepare: false, max, hardcoding, and the migration path.
• Three sandbox checks prove the handler reaches the database. The read control asserts a row the scorer inserts after the code was written, and client lifetime counts database connections across six invocations rather than reading where the client is constructed.
• The probe answers the Postgres SSLRequest and terminates TLS with a self-signed certificate, so a handler that sets ssl: 'require' connects.
• Checks are sourced from outside the page: feedback intake (FDBKIN-31335, FDBKIN-13040, FDBKIN-7883, FDBKIN-8653, FDBKIN-19912, DOCS-740), the supabase-postgres-best-practices conn- rules, the Supavisor FAQ, Supacademy's connectivity-and-pooler module, and supabase/agent-skills issue 92.
• CONNECT.md uses aws-1, not aws-0. aws-0 is what agents recite from memory, so the host whitelist only measures anything if the project is on something else.

Baseline, six runs, sonnet-5 with and without skills:

Check	Passed
project dependencies installed	6/6
the handler reads a row the scorer inserted	6/6
the handler writes a row that lands in items	6/6
the database client is created once per module, not per request	6/6
the transaction pooler string is what the app reads at runtime	6/6
prepared statements are turned off on the pooled connection	6/6
the connection pool is capped for a serverless invocation	3/6
every Supabase host in the workspace is one CONNECT.md lists	6/6
no connection string is hardcoded in application source	6/6
migration tooling, if configured, uses the direct connection string	6/6
the agent read the connecting to Postgres guide the prompt referenced	6/6

Three of six runs pass. All three failures are the pool cap: two left max unset, which is 10 on the postgres-js default, and one set max: 5. Findings and suggested doc changes are on DOCS-1312.

One thing for the reviewer to weigh: pool sizing is the only check that discriminates. CONNECT.md labels the three strings the way the dashboard does, so picking the transaction pooler off a label costs an agent nothing. Making the fixture less generous is worth considering after the docs fix lands, so the two changes do not confound each other.

Manual testing

1. Confirm the eval ports are free.

   docker ps --format '{{.Names}}\t{{.Ports}}' | grep -E '5432[12]->'
   

   No output. Stop the holder with supabase stop --project-id <id> if there is any.

2. Confirm discovery and frontmatter.

   pnpm eval:dry -- --eval build-docs-004-postgres-connection
   

   One PLAN line per experiment, each reading stage=build suite=regression mode=local-stack.

3. Typecheck.

   pnpm typecheck
   

   Clean.

4. Lint the eval.

   ./node_modules/.bin/biome check evals/build-docs-004-postgres-connection
   

   No fixes applied.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 2, 2026 9:24pm UTC