feat: eval the API keys guide in supabase.com/docs

[czenko]

Closes DOCS-1301

Problem

We point agents at the API keys guide. Nobody has measured whether reading it is enough to keep the secret key out of a browser bundle.

No existing eval answers that. resolve-sdk-001-legacy-key-migration is Node scripts, server-side throughout. build-functions-004/005/006 are Edge Function auth, where the secret key is legitimately server-side. None of them builds a client bundle, so none can catch a secret key shipped to a browser.

Solution

• Adds build-docs-003-api-keys-guide. The guide is the subject under test. The prompt is a product request plus the guide's url, and never says key, publishable, secret, service role, server, expose, or environment variable.
• Seeds a Vite app with two screens. Sign-up is public and correct with a publishable key. The roster lists everyone who signed up with their email, which lives in auth.users and no publishable key reaches. The roster is the one place where the convenient answer ships the whole table to the browser.
• Scores nine checks. Six read the built bundle and client source, two probe the running stack, one confirms the guide was retrieved.
• Adds ctx.stackStatus() to the local-stack scoring context, which reports the stack's url plus its publishable, secret, and legacy anon keys. The scorer needs all four, and getClient() only carries the publishable key.

Checks that exist to stop a false green

• roster returns every signed-up email pairs with the bundle checks, which a run that built nothing passes trivially.
• client source contains a signUp call pairs with client bundle carries a publishable or anon key, which passes for a key that ships and is never called.
• no client role can read another user's email catches the design that dodges the elevated call entirely by copying addresses into a table anyone can read. Everything else scored green before this check existed.

Where the email check came from

Not from the page under test. The page frames publishable-key safety as an RLS checklist, and the shortcut of copying addresses into a readable table satisfies that checklist while leaking. It came from api/securing-your-api, which treats grants and RLS as separate layers.

What this eval does not score

Two checks were tried and removed, because both measured the platform rather than the page. Keeping either means a CLI upgrade reports as a docs change.

• Which key format the client uses. The build injects a placeholder over VITE_SUPABASE_ANON_KEY, so a client wired through that name never carries a real key into the bundle and the format is not observable.
• Which key format the server uses. The Edge Function runtime decides which keys a function is handed, and the pinned CLI hands over legacy ones only. A function reading them follows the runtime, not the guide.

Known limitations

• The roster probe seeds two users and requires both addresses in one response. An agent that paginates to fewer than two rows per page would fail on a design choice, not a defect. auth.admin.listUsers() defaults to 50 per page and PostgREST to 1000 rows, so this is a narrow edge, but it would surface as an unexplained red.
• no secret-bearing env var is client-exposed passes by design. It reads every .env outside supabase/ and fails on a secret in any of them. An agent that keeps the secret in the function env passes, and that is the intended outcome.
• client source contains a signUp call is a literal match. Dead code satisfies it, and a call reached only from outside client source does not. Proving the screen works needs a driven DOM the scorer does not have.
• Authorization is out of scope. The roster probe calls as a signed-in user, so a roster gated on sign-in counts as working. Restricting the endpoint to staff needs a role in the seed and is what build-rls-003-org-roles-permissions measures.
• suite: regression, not benchmark. It is a regression detector for the guide, and it reports no failure against the published page.

Baseline

claude-code-sonnet-5 with and without skills, 3 runs each, CI run 33549257417: 9/9 on all six runs. Results are committed to this branch.

No check fails against the published guide, which is the floor a regression detector needs. A later docs change that breaks key placement shows up as a drop from this.

The scorer was also pointed at a revised draft of the guide, and that came back 9/9 on all six runs too, so the draft introduces no regression.

Manual testing

1. Free the Supabase ports. Run docker ps --format '{{.Names}}\t{{.Ports}}' | grep -E '5432[12]->' and stop whatever holds them.
2. Run pnpm eval:dry -- --eval build-docs-003-api-keys-guide. It plans 18 experiments.
3. Run pnpm typecheck. Clean.
4. Read README.md in the eval folder. Confirm the persona rules and the check rationale match the scorer.
5. The scorer was checked against thirteen throw-away solutions with no agent in the loop. A correct one passes every check except the transcript check, which a no-agent run always fails, and each bad one fails the checks it was built to fail.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 1, 2026 7:38pm UTC

[mattrossman]

(Merging main in so that it can use the new 3-run averaging strategy)

[mattrossman]

Looks good overall and tested a couple runs on my end.

I'd just call attention to this potential flakiness I experienced a couple times.

[barryroodt]

Looking good, I left a few inline comments on scorer cases that can still pass when they should fail.

[czenko]

Update: I added a couple more checks since Matt's review and have since reverted them since they proved to not reflect on the document but on the functionality of the CLI.