Publish local runs to analytics_dev on the real database

[lassebenni]

What I built

A third database role, analytics_dev_user, that owns the analytics_dev schema and can only read everything else. Trainees get that credential and publish their in-progress mart to analytics_dev from a local Airflow run; the scheduled pipeline keeps analytics_user and stays the only writer of production analytics. The publish step also stamps the table it writes with the schema and time the rows came from, so a reader can tell a hand-run mart from a scheduled one.

Why this approach

analytics_dev already existed after #19, but analytics_user owned both analytics schemas. That means handing a trainee the credential to write analytics_dev also hands them ownership of production analytics, so the dev/prod split would have rested on people not typing the wrong schema name. Making it a separate role moves the boundary into Postgres, where it is enforced rather than agreed.

This keeps #19's SCHEMA_OWNERS structure exactly as it is, which is better than the two-list version I originally wrote. The whole change is three lines: a new role constant, adding it to ROLES, and pointing analytics_dev at it. The existing loop grants full access on owned schemas and read-only on the rest, so nothing else needed touching.

Rejected: one role with CREATE on analytics_dev and only USAGE on analytics. It works, but the same login then appears in both the trainee's .env and the scheduled pipeline's Key Vault secret, so rotating it for one breaks the other.

docker-compose.override.yml also loses its BACKEND_PG_HOST: db override. It pointed Airflow tasks at a Postgres container while uv run on the same machine wrote the real database, so the two documented ways of running the pipeline silently disagreed about the destination.

Contract impact

None for the backend. analytics.fct_postings keeps its shape and analytics_user keeps writing it. The new analytics_dev schema is trainee-facing only; app_user gains read access to it but nothing reads it yet.

How to run

# Build the three roles on a throwaway database and see the grants
docker run -d --name pg-probe -e POSTGRES_PASSWORD=probe -e POSTGRES_USER=admin \
  -p 55439:5432 postgres:16
POSTGRES_PASSWORD=probe ./scripts/db-setup.py --host localhost --port 55439 --admin-user admin

# Then, as the printed analytics_dev_user password, the boundary holds:
#   create table analytics_dev.t (id int)   -> succeeds
#   create table analytics.t (id int)       -> permission denied for schema analytics
docker rm -f pg-probe

cd data && uv run pytest -q && uv run ruff check . && uv run black --check .

Self-check

• I ran this and it works
• Tests pass locally
• No secrets, tokens, or connection strings in the diff
• This pull request does one thing

Verified beyond the tests: the same seven-check boundary probe passes both against a database db-setup.py built from scratch and against the live project database, where the role now exists.

[Copilot]

Pull request overview

This PR updates local development publishing to target the real backend Postgres database by default (writing into analytics_dev), and adds a provenance “stamp” onto published tables so teams can identify which warehouse schema last populated the shared dev mart.

Changes:

• Extend backend DB setup to include an analytics_dev schema and a dedicated analytics_dev_user role with constrained privileges.
• Add optional source stamping to the publish swap so the final table gets a COMMENT ON TABLE like from <source> at <timestamp>.
• Update Airflow/local documentation and defaults to publish to analytics_dev with the dev role and TLS required.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/db-setup.py	Adds analytics_dev schema/role and updates the privilege layout to enforce the dev/prod boundary via grants.
data/src/publishing/sync.py	Adds optional source parameter and stamps the published table via COMMENT ON TABLE after the swap.
data/tests/publishing/test_sync.py	Adds tests asserting the stamp is written and occurs after the rename/swap.
data/README.md	Documents new publish schema/user settings and explains shared analytics_dev behavior + stamp visibility.
data/airflow/dags/pipeline_dag.py	Publishes to a configurable schema and supplies the source string for stamping.
data/.env.example	Updates local env guidance to use the real DB, analytics_dev_user, analytics_dev, and TLS required.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.