Hardening B/2 (stacked on #290): doc cache correctness, 429 instead of 500, health check, CI runs tests again

[gregv]

Stacked on #290 (hardening/a-security). Base is that branch, so this shows only the reliability changes; merge #290 first, then this. Second half of the backend hardening plan (frontend-ohack.dev/docs/plans/hardening-security-seo-reliability-2026-09.md, sections 3.1–3.4). Test-first throughout; each new test was seen failing for the documented reason before the fix.

What changed

Problem (verified on develop)	Fix
doc_to_json cache served stale, shared, mutable dicts. Keyed by doc id only (1h, all collections) and returned the SAME object to every caller. get_team passed a fresh snapshot but got the cached dict → team pages stale up to 1h in the other gunicorn worker. get_single_hackathon_event popped project_story and replaced users[] on the cached team dicts → GET /team/<id> in that worker lost the story. _enrich_teams_users_batch zeroed users[] on already-enriched teams. (The merged CLAUDE.md sentence saying the cache was keyed by update_time was wrong.)	Snapshots convert directly and are never cached (the read already happened); references cache for 10 min; every return is a fresh copy (_copy_json_like, never deepcopy); doc_to_json.cache_clear() API kept. get_single_hackathon_id / get_single_npo / get_npo_by_hackathon_id pass snapshots. _enrich_teams_users_batch preserves dict users. CLAUDE.md corrected.
Judges' team pages showed no members. judging_service iterated enriched dict users as ids → get_user_by_id(dict) threw → members=[].	Resolve u["id"] for dict users and still re-fetch (the member shape needs email). Judging DATA only — rubric, rounds, scoring untouched.
A tripped rate limit was a 500 for everyone. ratelimit counters are per-process, all clients combined; RateLimitException had no handler. get_npo_list had @limits(20/min) ABOVE @cached, so cache HITS counted — 20 requests/min to /api/messages/npos (client-fetched by /nonprofits) 500'd every user.	App-wide handler → 429 {"error":"rate_limited"} + Retry-After; @cached outermost on get_npo_list; get_problem_statement_list_old 100 → 600/min. Views with a blanket except Exception (volunteers/planning/store) still convert it to their own 500 — noted follow-up.
No health check; Fly could not tell a wedged machine from a healthy one	GET /api/health (zero imports beyond Flask) + [[http_service.checks]] in fly.toml
CI ran no tests. The tests job copied .env.example and stopped (pytest commented out), so deploy: needs [lint, tests] gated on nothing. .env.example's placeholder FIREBASE_CERT_CONFIG also blocks collection of nearly every suite (the Certificate is built at import).	Job generates a throwaway service-account JSON with openssl at runtime (never committed), runs with EMPTY Slack/Resend/OpenAI/PropelAuth vars, and loops over the verified green set — 554 tests — with api/messages/tests one file at a time. Excluded with reasons in the workflow: contact (real signature bug + recaptcha env), github/slack (init_auth at import), leaderboard (2 known), certificates (network hang). Python 3.9 → 3.10 (matches the Dockerfile).

New tests: test/common/utils/test_firestore_helpers_cache.py (copy isolation, snapshot-beats-cache, reference leaf preserved, cache_clear contract), api/messages/tests/test_enrich_teams_batch.py, api/judging/tests/test_team_members_shape.py, api/messages/tests/test_ratelimit_handling.py (429 + Retry-After; 30 get_npo_list calls → one read, no exception), api/health/tests/test_health.py.

Deploy notes

• Merge order: Hardening A/2: close the open routes — decorator order, newsletter relay, request edit link, upload directories, hacker/team PII, deposit amount #290 → this. Both are backend-only and backwards compatible with the deployed frontend.
• Firestore reads: get_single_npo/get_npo_by_hackathon_id no longer ride the 1h reference cache (they read the doc each call; both are low-volume and nonprofit edits now show immediately). Event-payload nonprofit/team fan-out still uses the 10-min reference cache.
• After merge to main, the Fly health check starts probing /api/health every 30s.

Test plan

1. Edit a team's project story on /hack/<event>/manageteam, then open /hack/<event>/team/<id> and reload ~6 times (both workers): the new story shows every time, and it is still present after loading the event page /hack/<event> (which used to strip it from the shared cache).
2. /judge/<event>/team/<id>: team members are listed (was empty).
3. for i in $(seq 1 31); do curl -s -o /dev/null -w '%{http_code}\n' https://<staging>/api/messages/npos; done → all 200 (was 500 from request 21).
4. Force a limit (e.g. 601 rapid GET /api/messages/problem_statements) → 429 with Retry-After, never 500.
5. curl -s https://<staging>/api/health → {"status":"ok"}; Fly dashboard shows the check passing.
6. Open this PR's Checks tab: the tests job runs ~554 tests and is green; break a test in a scratch commit → the job goes red and deploy is blocked.

🤖 Generated with Claude Code