[FAQ] HW4 Q6: Grafana alert fires but the webhook never reaches the incident responder on localhost:8001 #441

Course

ai-dev-tools-zoomcamp

Question

HW4 Q6: My Grafana alert is Firing, but the webhook never reaches my incident responder on http://localhost:8001/alerts. Grafana's contact point shows connection refused or dial tcp: lookup host.docker.internal: no such host. How do I connect Grafana in Docker Compose to a responder running on my machine?

Answer

Grafana runs inside a container, so localhost there means the Grafana container itself, not your machine. Point the webhook at the host and make sure the host is reachable:

1. Use host.docker.internal in the contact point URL, not localhost:

   # observability/grafana/provisioning/alerting/alerts.yaml
   contactPoints:
     - orgId: 1
       name: incident-responder
       receivers:
         - uid: incident-responder-webhook
           type: webhook
           settings:
             url: http://host.docker.internal:8001/alerts
             httpMethod: POST

2. Map that name in Compose. Docker Desktop on Mac and Windows provides it automatically. On Linux, Colima, and some other engines you have to add it yourself (Docker 20.10+):

   grafana:
     image: grafana/grafana:11.4.0
     extra_hosts:
       - "host.docker.internal:host-gateway"

3. Bind the responder to all interfaces, not only loopback. A server bound to 127.0.0.1 refuses connections from containers. Use 0.0.0.0, for example ThreadingHTTPServer(("0.0.0.0", 8001), Handler) or uvicorn ... --host 0.0.0.0 --port 8001.

4. Test from inside the Grafana container before waiting for a real alert:

   docker compose exec grafana wget -qO- --post-data='{"alerts":[]}' \
     --header='Content-Type: application/json' http://host.docker.internal:8001/alerts

   You can also use Alerting → Contact points → Test in Grafana.

5. If the alert fired before the webhook worked, Grafana won't resend it until repeat_interval has passed. Use a short group_wait (e.g. 5s) in the notification policy, then let the alert go back to Normal and trigger the failing request again.

Also, if you provision the URL from an environment variable (url: ${RESPONDER_WEBHOOK_URL}), set that variable on the grafana service. Grafana expands $VAR in provisioning files, so escape literal template dollars as $$labels.

Docs: https://docs.docker.com/reference/cli/docker/container/run/#add-host and https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links