test: CV and cover-letter rendering, model evaluation, frontend smoke tests

.github/workflows/ci.yml

@@ -23,6 +23,10 @@ jobs:
           python-version: "3.13"
           cache: pip
           cache-dependency-path: api/requirements*.txt
+      # WeasyPrint's libraries and the CV template's font, as in api/Dockerfile: the PDF tests
+      # measure page counts, so they must lay out with the same font as production.
+      - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends
+          libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 fonts-liberation
       - run: pip install -r requirements.txt -r requirements-dev.txt
       - run: pytest
 
@@ -60,7 +64,7 @@ jobs:
         run: docker run --rm -v "$PWD/nginx/nginx.conf:/etc/nginx/nginx.conf:ro" nginx:alpine nginx -t
 
   frontend:
-    name: Frontend type-check + build
+    name: Frontend type-check, build + smoke tests
     runs-on: ubuntu-latest
     defaults:
       run:
@@ -75,3 +79,11 @@ jobs:
       - run: npm ci
       - run: npx tsc --noEmit
       - run: npm run build
+      - run: npx playwright install --with-deps chromium
+      - run: npm run test:e2e
+      - if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: playwright-traces
+          path: frontend/test-results/
+          retention-days: 7

.gitignore

@@ -12,6 +12,8 @@ __pycache__/
 frontend/node_modules/
 frontend/.next/
 frontend/tsconfig.tsbuildinfo
+frontend/test-results/
+frontend/playwright-report/
 
 # IDE
 .idea/
@@ -26,3 +28,6 @@ frontend/tsconfig.tsbuildinfo
 /import_summer_internships.py
 /.claude/settings.local.json
 /test_workable.py
+
+# Model evaluation output (api/evals)
+/api/evals/last_run.json

README.md

@@ -248,7 +248,24 @@ pytest                      # fast suite — no network, LLM or browser
 pytest -m browser           # real headless Chromium (needs: playwright install chromium)
 ```
 
-CI (`.github/workflows/ci.yml`) runs both, plus a frontend type-check and build.
+The CV and cover-letter PDF tests need WeasyPrint's Pango libraries; where those are missing (Windows) they skip, except in CI. To run them locally, use the API image: `docker run --rm -v "$PWD/api:/app" -w /app windrush-api sh -c "pip install -r requirements-dev.txt && pytest tests/test_documents.py"`.
+
+**Model evaluation** (`api/evals/`) scores the LLM's reading of job ads (seniority, sponsorship) against labelled data, so a prompt or model change that makes it worse shows up. It calls the real LLM (about 500 calls, a few minutes), so it only runs when asked for:
+
+```bash
+cd api && set -a && . ../.env && set +a
+WINDRUSH_LLM_EVAL=1 pytest evals -s   # -s prints the scores; per-item answers go to evals/last_run.json
+```
+
+**Frontend smoke tests** (`frontend/e2e/`) drive the built app in Chromium with the API mocked:
+
+```bash
+cd frontend
+npm run build && npx playwright install chromium
+npm run test:e2e
+```
+
+CI (`.github/workflows/ci.yml`) runs the API suite, the browser suite, the frontend type-check, build and smoke tests, and checks the Docker config. The model evaluation is not in CI.
 
 **Switching backend:** set `LLM_BACKEND` and the matching key (see above).
 

api/evals/conftest.py

@@ -0,0 +1,22 @@
+"""
+Model evaluations call the real LLM, so they only run when asked for:
+
+    cd api && set -a && . ../.env && set +a && WINDRUSH_LLM_EVAL=1 pytest evals
+
+They live outside tests/ so the test suite's fake-LLM environment (tests/conftest.py) never applies.
+"""
+import os
+import sys
+
+import pytest
+
+# Reuse labelled cases that the regex tests already hold.
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "tests"))
+
+
+def pytest_collection_modifyitems(config, items):
+    if os.environ.get("WINDRUSH_LLM_EVAL") == "1":
+        return
+    skip = pytest.mark.skip(reason="calls the real LLM: set WINDRUSH_LLM_EVAL=1 with the LLM settings from .env")
+    for item in items:
+        item.add_marker(skip)