Repository navigation
Give the observability containers least privilege #443
Description
Summary
Two observability containers can reach the Docker socket, and the Postgres exporter holds the application's full database URL. A bug in any of them would give much more than monitoring needs: the Docker socket means root on the host, and the database URL means full read and write on application data.
- Severity: Low (needs a bug in a monitoring container first)
- Area:
docker-compose.observability.yml - Source: security review of 2026-10-06, finding L17
Where
- Alloy mounts the socket:
docker-compose.observability.yml:26(/var/run/docker.sock:/var/run/docker.sock:ro).:rodoes not limit the Docker API. - cAdvisor mounts all of
/var/run:docker-compose.observability.yml:67. postgres-exporteruses the application URL:docker-compose.observability.yml:248(DATA_SOURCE_NAME: ${DATABASE_URL}).- The compose comment near
:62-64calls these mounts read-only and safe, and citesdocs/observability.md, which does not exist.
Fix plan
- Add a Docker socket proxy (for example
tecnativa/docker-socket-proxy, pinned by digest like the other images) that allows only the read endpoints Alloy and cAdvisor use (containers, info). Point both at it, and remove their socket mounts. - Create a
pg_monitorlogin role for the exporter in a migration or the deploy setup, and give the exporter its ownPOSTGRES_EXPORTER_DSNenv var. - Correct the compose comment and remove the dead
docs/observability.mdreference.
Acceptance criteria
- No observability container mounts
/var/run/docker.sockor/var/run. - Container metrics and logs still arrive in Grafana.
-
postgres-exporterconnects as apg_monitorrole, not the application role, and its metrics still arrive.
Verify
docker compose -f docker-compose.observability.yml configshows no socket mounts and the new DSN.- On the server: Grafana dashboards for containers and Postgres still update after deploy.
Activity
fix(ops): run prod jobs only from main and narrow monitoring access
docs(hocuspocus): sync server docs with the 9 October fixes
docs(repo): record the 9 Oct security and email batch in the changelo…
HMarzban commentedon Oct 9, 2026
Code is live (
73e5bd972): Alloy reads the Docker API throughdocker-socket-proxy(GET only), and node-exporter no longer sees the host socket.postgres-exporterprefersPOSTGRES_EXPORTER_DSN.Left before closing (maintainer): create a
pg_monitorlogin role on the DigitalOcean cluster and setPOSTGRES_EXPORTER_DSNin.env.observability. KeepDATABASE_URLthere until a follow-up moves the deploy check.HMarzban commentedon Oct 19, 2026
Maintainer ruling, 2026-10-19
The cAdvisor
/var/runmount is accepted as a documented exception. Without the containerd socket, container names vanish and the container alerts go NoData.HMarzban commentedon Oct 21, 2026
Status 2026-10-21. The exporter login exists on production:
metrics_reader, withPOSTGRES_EXPORTER_DSNset in.env.observability.One change to the plan: on DigitalOcean managed PostgreSQL,
doadmincannot grantpg_monitor(must have admin option on role "pg_monitor"). It can grantpg_read_all_stats, which covers the stats viewspostgres-exporterreads. Sometrics_readerholdspg_read_all_stats. The self-hosting guide and the backend rules now say this.Owed: the next
(build): observabilitydeploy, then check that the Postgres panels in Grafana still fill.DATABASE_URLstays in.env.observabilityuntil the deploy check moves.fix(hocuspocus): serve a Private document's media only to its owner
HMarzban commentedon Oct 21, 2026
Done on production on 2026-10-21.
redis-exportersendsREDIS_PASSWORD, read from.env.observability, andredis_upis1.postgres-exporterlogs in as the read-onlymetrics_reader(pg_read_all_stats; DigitalOcean refusespg_monitor).pg_upis1, and the Grafana Postgres panels have data again.Two problems came up, and both are fixed:
sslmode=verify-full&sslrootcert=/path/to/ca-certificate.crt. That path is a placeholder, so every scrape failed and "Postgres down" fired. The DSN now usessslmode=require, likeDATABASE_URL. The self-host docs now warn about it.walcollector callspg_ls_waldir, which needspg_monitor.17a67c262turns it off with--no-collector.wal; no panel readspg_wal_*. It goes live with the next observability deploy. Until then, it logs one error per scrape and changes nothing else.Follow-up, not required: use
sslmode=verify-fullwith DigitalOcean's real CA file mounted into the exporter.fix(observability): turn off the exporter wal collector
fix(mermaid): lower the default diagram limits to the phone-measured …
docs(ops): add the Postgres down runbook and link the alert to it