Observability Platform

Hands-on observability demo built around metrics, logs, traces, alerting, runbooks, and chaos validation. The repository now has one coherent primary path: bring the stack up locally with Docker Compose, exercise the app, and inspect the resulting signals across Prometheus, Grafana, Jaeger, Kibana, and Alertmanager.

Why this repo matters

A lot of monitoring demos stop at dashboards. This project goes further by tying together the three pillars of observability, SLI/SLO thinking, alert routing, runbooks, and fault injection.

What is included

• instrumented application source under app/
• Docker Compose stack for Prometheus, Grafana, Alertmanager, OpenTelemetry, ELK, Jaeger, and Filebeat
• Prometheus alerting rules and auto-provisioned Grafana dashboard
• chaos scripts for latency and error scenarios
• runbooks and postmortem templates for operational response
• starter Kubernetes manifests for the demo app and Prometheus configuration

Observability scope

• Metrics: Prometheus and Grafana for collection, dashboards, and alerting
• Logs: ELK stack for centralized log aggregation and search
• Traces: OpenTelemetry and Jaeger-style tracing pipeline
• Response: Alertmanager, runbooks, and postmortem templates
• Validation: chaos scripts to test the system under failure conditions

Quick start (recommended)

# Start the full local stack
docker compose up -d --build

# Inject faults and observe behavior
./chaos-scripts/chaos-runner.sh latency
./chaos-scripts/chaos-runner.sh errors
./chaos-scripts/chaos-runner.sh reset

Once the stack is up:

• app: http://localhost:8000/health
• metrics: http://localhost:8000/metrics
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin / admin)
• Alertmanager: http://localhost:9093
• Jaeger: http://localhost:16686
• Kibana: http://localhost:5601

Kubernetes status

The k8s/ directory is a starter set of manifests, not a production-ready deployment. It includes the demo app deployment/service/namespace plus Prometheus, Alertmanager, OpenTelemetry, and ELK configuration snippets, but it does not yet include the full set of Services, RBAC, and environment-specific manifests needed for a one-command cluster install.

Repository layout

.
|-- app/                  # instrumented application
|-- chaos-scripts/        # fault injection scripts
|-- dashboards/           # Grafana dashboard assets
|-- docker/               # local container assets
|-- k8s/                  # Kubernetes deployment manifests
|-- postmortem-templates/ # incident review templates
|-- runbooks/             # operational runbooks
|-- docs/                 # diagrams and supporting docs
`-- .github/              # validation workflows

What this demonstrates

• an end-to-end local observability flow across metrics, logs, and traces
• operational maturity through alerting, runbooks, and postmortems
• SLI/SLO-oriented monitoring instead of dashboard sprawl
• validation of reliability assumptions through controlled chaos testing