Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ It covers the full lifecycle from infrastructure provisioning to GitOps-managed
| **Thanos** | Long-term metric storage - Prometheus sidecar ships data to S3; Query, Compactor, and StoreGateway provide durable retention |
| **HPA + metrics-server** | Horizontal pod autoscaling based on CPU utilization |
| **Loki + Fluent Bit** | Centralized log aggregation backed by S3 object storage, queryable in Grafana |
| **OTel Collector (agent + gateway)** | Two-tier telemetry pipeline: a DaemonSet agent on every node receives spans from local pods; a central Deployment gateway batches and forwards to Tempo |
| **Grafana Tempo** | Distributed trace storage - receives spans from the OTel gateway via OTLP gRPC, queryable in Grafana with trace-to-log correlation through Loki; backed by S3 |
| **Velero** | Cluster backup and restore - backs up Kubernetes resources and EBS volume snapshots to S3 |
| **Kyverno** | Admission controller - evaluates `ClusterPolicy` rules on every pod admission and writes `PolicyReport` objects; all policies run in Audit mode by default |
| **Policy Reporter** | Web UI for Kyverno `PolicyReport` and `ClusterPolicyReport` objects - accessible via `kubectl port-forward` |
Expand Down Expand Up @@ -72,6 +74,7 @@ See [docs/cost.md](docs/cost.md) for a full per-component breakdown, cost-contro
| [gitops/monitoring/README.md](gitops/monitoring/README.md) | Prometheus, Grafana, Thanos, Prometheus Adapter, ServiceMonitor, EBS persistence |
| [gitops/alerts/README.md](gitops/alerts/README.md) | PrometheusRules, Slack alerting, silencing, grouping |
| [gitops/logs/README.md](gitops/logs/README.md) | Loki S3 backend, Fluent Bit, log querying in Grafana |
| [gitops/tracing/README.md](gitops/tracing/README.md) | OTel Collector agent/gateway pipeline, Tempo S3 backend, trace querying and log correlation in Grafana |
| [gitops/security/README.md](gitops/security/README.md) | Kyverno policies, Policy Reporter UI, inspecting policy reports |
| **Design** | |
| [docs/design-decisions.md](docs/design-decisions.md) | Why each architectural choice was made - tool selection, configuration defaults, and demo trade-offs |
Expand Down Expand Up @@ -194,7 +197,7 @@ bash terraform/scripts/upgrade.sh
2. Generates or accepts ArgoCD bcrypt hash and Grafana password
3. Runs `terraform init` → `terraform validate` → `terraform apply`
4. Updates kubeconfig
5. Re-applies the ArgoCD cluster secret with the latest Terraform outputs (account ID, VPC ID, domain, Karpenter instance profile, Thanos/Loki/Velero bucket names)
5. Re-applies the ArgoCD cluster secret with the latest Terraform outputs (account ID, VPC ID, domain, Karpenter instance profile, Thanos/Loki/Velero/Tempo bucket names)
6. Force-annotates `argocd-admin`, `grafana-admin`, and `alertmanager-webhook` ExternalSecrets to trigger an immediate re-sync from Secrets Manager

---
Expand All @@ -211,7 +214,9 @@ bash terraform/scripts/upgrade.sh
| 6 | HPA reacts to load | `kubectl get hpa -n simple-time-service -w` while running `python3 scripts/load_test.py` |
| 7 | Slack alert fires | Fire test alert (see [gitops/alerts/README.md](gitops/alerts/README.md#testing-the-slack-receiver)) - appears in `#alerts-test` within 30s |
| 8 | Loki API reachable | `kubectl port-forward svc/loki-gateway -n logging 3100:80` → `curl 'http://localhost:3100/loki/api/v1/labels'` |
| 9 | Logs in Grafana | Explore → Loki datasource → `{namespace="simple-time-service"}` |
| 9 | Logs in Grafana | Explore → Loki datasource → `{namespace="simple-time-service"}` - log lines include `trace_id` / `span_id` fields in JSON |
| 9a | OTel pipeline healthy | `kubectl get pods -n tracing` - `otel-gateway` (Deployment) and `otel-agent` (DaemonSet) all `Running` |
| 9b | Traces in Grafana | Explore → Tempo datasource → **Search** → service `simple-time-service` - spans visible; clicking a trace shows correlated Loki logs |
| 10 | Thanos Query healthy | `kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-query` - `Running`; add Thanos datasource in Grafana pointing to `thanos-query.monitoring.svc:9090` |
| 11 | EBS volumes provisioned | `kubectl get pv` - PVs for Prometheus (20Gi), Alertmanager (2Gi), Thanos Compactor (10Gi), StoreGateway (5Gi) all `Bound` |
| 12 | Velero running | `kubectl get pods -n velero` - `Running`; `kubectl get backupstoragelocation -n velero` - `Available` |
Expand Down Expand Up @@ -345,6 +350,14 @@ bash terraform/scripts/upgrade.sh
│ │ │ └── grafana-loki-datasource.yaml # ApplicationSet - Loki datasource ConfigMap (sync-wave 4)
│ │ └── fluent-bit/
│ │ └── fluent-bit.yaml # ApplicationSet - Fluent Bit DaemonSet (sync-wave 4)
│ ├── tracing/
│ │ ├── README.md # OTel Collector pipeline, Tempo S3 backend, trace querying and log correlation
│ │ ├── tempo/
│ │ │ ├── tempo.yaml # ApplicationSet - Tempo trace store S3 backend (sync-wave 6)
│ │ │ └── grafana-tempo-datasource.yaml # ApplicationSet - Tempo datasource ConfigMap + Loki correlation (sync-wave 8)
│ │ └── otel-collector/
│ │ ├── otel-collector-gateway.yaml # ApplicationSet - OTel Collector Deployment gateway (sync-wave 6)
│ │ └── otel-collector-agent.yaml # ApplicationSet - OTel Collector DaemonSet agent (sync-wave 7)
│ └── security/
│ └── kyverno/
│ ├── kyverno.yaml # ApplicationSet - Kyverno admission controller (sync-wave 3)
Expand Down Expand Up @@ -376,6 +389,7 @@ bash terraform/scripts/upgrade.sh
├── thanos.tf # S3 bucket + IRSA roles for Thanos Prometheus sidecar, Compactor, StoreGateway
├── loki.tf # S3 bucket + IRSA role for Loki object storage
├── velero.tf # S3 bucket + IRSA role for Velero backups
├── tempo.tf # S3 bucket + IRSA role for Tempo trace storage
├── secrets-manager.tf # AWS Secrets Manager secrets (ArgoCD, Grafana, Slack)
├── .tflint.hcl # TFLint AWS ruleset plugin config
├── .checkov.yaml # Checkov global skip rules (documented false positives)
Expand Down Expand Up @@ -426,7 +440,7 @@ The cleanup script runs six steps in order:
1. **Kubernetes resources** - deletes the root app, all ArgoCD Applications and ApplicationSets, Ingresses, LoadBalancer Services, Karpenter NodePools and NodeClaims, PersistentVolumes, and all namespaces labelled `app.kubernetes.io/part-of=eks-gitops-platform` (including `security`) (every namespace created by the `cluster-namespaces` chart)
2. **AWS Load Balancers** - deletes any leftover ALB/NLB/Classic ELBs tagged for the cluster that were not removed by step 1
3. **EBS volumes and snapshots** - deletes EBS volumes tagged `kubernetes.io/cluster/<cluster>=owned` (CSI Driver PVCs with Retain policy), then deletes EBS snapshots tagged for the cluster or with a `velero.io/backup` tag
4. **S3 buckets** - empties all versioned S3 buckets in the same region whose name contains the cluster name (Thanos, Loki, Velero), draining all object versions and delete markers in batches so that Terraform can delete the buckets cleanly
4. **S3 buckets** - empties all versioned S3 buckets in the same region whose name contains the cluster name (Thanos, Loki, Velero, Tempo), draining all object versions and delete markers in batches so that Terraform can delete the buckets cleanly
5. **Kyverno CRDs** - deletes all CRDs in the `kyverno.io` and `wgpolicyk8s.io` API groups; this is handled automatically by `cleanup.sh` using a label selector
6. **Terraform destroy** - removes all AWS resources provisioned by Terraform

Expand Down
59 changes: 52 additions & 7 deletions app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ A minimal Python microservice that returns the current UTC timestamp and the cal
- **Framework**: FastAPI + Uvicorn
- **Container**: single-stage build based on `python:3.12-slim`, kept small by clearing pip cache.
- **ASGI server**: Uvicorn (production-grade async Python server)
- **Metrics**: `prometheus-fastapi-instrumentator` — `/metrics` endpoint scraped by Prometheus via ServiceMonitor
- **Tracing**: OpenTelemetry SDK with OTLP HTTP exporter — a `TracerProvider` is always configured so every request gets a real span; spans are exported to Grafana Tempo only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set
- **Logging**: JSON-structured logs written to stdout; `trace_id` and `span_id` are injected into every log line for every HTTP request

---

Expand All @@ -36,7 +39,7 @@ A minimal Python microservice that returns the current UTC timestamp and the cal
docker compose up --build
```

The service is available at `http://localhost:8080`.
The service is available at `http://localhost:8080`. Logs include `trace_id` and `span_id` on every request even without a collector.

### Docker only

Expand Down Expand Up @@ -66,13 +69,12 @@ Expected response:

The image is published on Docker Hub under two distinct tags:

| Tag | Metrics endpoint | Use when |
|-----|-----------------|----------|
| `v1` | No | You don't need Prometheus metrics |
| `latest` | Yes (`/metrics`) | You want Prometheus scraping via ServiceMonitor |
| Tag | Metrics | Tracing | Use when |
|-----|---------|---------|----------|
| `v1` | No | No | Pinned baseline — original service without instrumentation |
| `latest` | Yes (`/metrics`) | Yes | Tracks `main` — full instrumentation; deploy with the Helm chart |

> **`v1`** is a pinned baseline - the original service with no instrumentation.
> **`latest`** tracks `main` and includes the Prometheus `/metrics` endpoint exposed by `prometheus-fastapi-instrumentator`. Use this tag when deploying with the Helm chart and `serviceMonitor.enabled=true`.
> **`latest`** includes the Prometheus `/metrics` endpoint and OpenTelemetry tracing. Tracing is off by default and activates only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set in the pod's environment. Use this tag when deploying with the Helm chart and `serviceMonitor.enabled=true`.

```bash
# No metrics (baseline)
Expand Down Expand Up @@ -166,6 +168,49 @@ To enforce that the scan must pass before a PR can be merged, configure a branch

---

## Observability

### Structured logging

All logs are written to stdout as JSON. Every log line includes:

| Field | Example | Notes |
|-------|---------|-------|
| `timestamp` | `2026-04-07T12:00:00.000000+00:00` | ISO-8601 UTC |
| `level` | `INFO` | |
| `logger` | `simple-time-service` | |
| `message` | `Request completed: ...` | |
| `trace_id` | `4bf92f3577b34da6...` | Present on every HTTP request |
| `span_id` | `00f067aa0ba902b7` | Present on every HTTP request |

`trace_id` and `span_id` are injected from the active OpenTelemetry span on every request. When `OTEL_EXPORTER_OTLP_ENDPOINT` is set, the same span is also exported to Tempo, enabling a direct Loki → Tempo link from any log line.

### Distributed tracing

The platform uses a two-tier OpenTelemetry Collector topology:

```
Pod → OTel Agent (DaemonSet, node-local, :4318) → OTel Gateway (Deployment) → Tempo
```

The Helm chart injects the node IP at runtime via the downward API so the pod always reaches its local agent:

| Variable | Set by | Value |
|----------|--------|-------|
| `NODE_IP` | Downward API (`status.hostIP`) | Node's internal IP |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Helm template | `http://$(NODE_IP):4318` |
| `OTEL_SERVICE_NAME` | Helm values | `simple-time-service` |

All three are injected automatically when `tracing.enabled: true` in `charts/simple-time-service/values.yaml`. The `/health` and `/metrics` endpoints are excluded from tracing to avoid noise.

Locally, `trace_id` appears in logs without any collector. To also export spans, point the endpoint at a local OTel Collector or Tempo:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 docker compose up --build
```

---

## Prometheus metrics exposed

Metrics are emitted by [`prometheus-fastapi-instrumentator`](https://github.com/trallnag/prometheus-fastapi-instrumentator) and the standard Python `prometheus_client` collectors.
Expand Down
6 changes: 5 additions & 1 deletion app/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
fastapi==0.136.3
uvicorn[standard]==0.34.0
prometheus-fastapi-instrumentator==7.0.0
starlette==0.52.1
starlette==0.52.1
opentelemetry-api==1.28.0
opentelemetry-sdk==1.28.0
opentelemetry-instrumentation-fastapi==0.49b0
opentelemetry-exporter-otlp-proto-http==1.28.0
64 changes: 43 additions & 21 deletions app/src/app.py
Original file line number Diff line number Diff line change
@@ -1,37 +1,68 @@
import json
import logging
import os
from datetime import datetime, timezone

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI(title="simple-time-service")

Instrumentator().instrument(app).expose(app, include_in_schema=False)


class JSONFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
span = trace.get_current_span()
ctx = span.get_span_context()
log_record = {
"timestamp": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
}
if ctx.is_valid:
log_record["trace_id"] = format(ctx.trace_id, "032x")
log_record["span_id"] = format(ctx.span_id, "016x")
return json.dumps(log_record)


def configure_logging() -> logging.Logger:
"""
Configure and return the application logger.

Logs are written to stdout.
"""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
)
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
root = logging.getLogger()
root.handlers.clear()
root.addHandler(handler)
root.setLevel(logging.INFO)
logger = logging.getLogger("simple-time-service")
logger.setLevel(logging.INFO)
return logger


def configure_tracing() -> None:
service_name = os.environ.get("OTEL_SERVICE_NAME", "simple-time-service")
provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
if endpoint:
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces"))
)
trace.set_tracer_provider(provider)


configure_tracing()
logger = configure_logging()
FastAPIInstrumentor.instrument_app(app, excluded_urls="health,metrics")


def get_client_ip(request: Request) -> str:
"""
Return the originating client IP address from the incoming request.
"""
x_forwarded_for = request.headers.get("x-forwarded-for", "")
if x_forwarded_for:
return x_forwarded_for.split(",")[0].strip()
Expand All @@ -48,9 +79,6 @@ def get_client_ip(request: Request) -> str:

@app.middleware("http")
async def log_requests(request: Request, call_next):
"""
Log incoming requests and response status codes.
"""
client_ip = get_client_ip(request)
logger.info(
"Incoming request: method=%s path=%s remote_addr=%s user_agent=%s",
Expand All @@ -74,9 +102,6 @@ async def log_requests(request: Request, call_next):

@app.get("/")
async def get_service_metadata(request: Request):
"""
Return the current UTC timestamp and resolved client IP address.
"""
response = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"ip": get_client_ip(request),
Expand All @@ -88,8 +113,5 @@ async def get_service_metadata(request: Request):

@app.get("/health")
async def health():
"""
Return a simple health response.
"""
logger.info("Health check successful")
return JSONResponse(content={"status": "ok"}, status_code=200)
17 changes: 17 additions & 0 deletions charts/simple-time-service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,10 @@ kubectl delete namespace simple-time-service
| serviceMonitor.labels | object | `{}` | Extra labels added to the ServiceMonitor (used to target a specific Prometheus instance) |
| serviceMonitor.path | string | `"/metrics"` | Metrics endpoint path |
| tolerations | list | `[]` | Pod tolerations |
| tracing.agentPort | int | `4318` | OTLP HTTP port on the node-local OTel Collector agent; used only when `endpoint` is empty |
| tracing.enabled | bool | `false` | Enable OpenTelemetry tracing |
| tracing.endpoint | string | `""` | Direct OTLP HTTP endpoint (e.g. `http://otel-collector.tracing.svc.cluster.local:4318`). When empty, uses the node-local DaemonSet agent via `status.hostIP:agentPort` |
| tracing.serviceName | string | `"simple-time-service"` | Service name reported in traces and correlated in Grafana Tempo |

---

Expand All @@ -139,6 +143,19 @@ helm install simple-time-service charts/simple-time-service \
--set serviceMonitor.enabled=true
```

### Deploy with tracing enabled

Requires the OTel Collector DaemonSet to be running on each node (deployed via `gitops/tracing/`).

```bash
helm install simple-time-service charts/simple-time-service \
--set image.tag=latest \
--set serviceMonitor.enabled=true \
--set tracing.enabled=true
```

The chart injects `NODE_IP` (via the downward API) and constructs `OTEL_EXPORTER_OTLP_ENDPOINT=http://$(NODE_IP):4318` automatically so pods always reach the node-local agent without any manual endpoint configuration.

---

## Updating this README
Expand Down
13 changes: 13 additions & 0 deletions charts/simple-time-service/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,19 @@ helm install simple-time-service charts/simple-time-service \
--set serviceMonitor.enabled=true
```

### Deploy with tracing enabled

Requires the OTel Collector DaemonSet to be running on each node (deployed via `gitops/tracing/`).

```bash
helm install simple-time-service charts/simple-time-service \
--set image.tag=latest \
--set serviceMonitor.enabled=true \
--set tracing.enabled=true
```

The chart injects `NODE_IP` (via the downward API) and constructs `OTEL_EXPORTER_OTLP_ENDPOINT=http://$(NODE_IP):4318` automatically so pods always reach the node-local agent without any manual endpoint configuration.

---

## Updating this README
Expand Down
13 changes: 13 additions & 0 deletions charts/simple-time-service/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@ spec:
- name: simple-time-service
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
{{- if .Values.tracing.enabled }}
env:
{{- if not .Values.tracing.endpoint }}
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
{{- end }}
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: {{ if .Values.tracing.endpoint }}{{ .Values.tracing.endpoint | quote }}{{ else }}"http://$(NODE_IP):{{ .Values.tracing.agentPort }}"{{ end }}
- name: OTEL_SERVICE_NAME
value: {{ .Values.tracing.serviceName | quote }}
{{- end }}
ports:
- name: http
containerPort: {{ .Values.service.targetPort }}
Expand Down
10 changes: 10 additions & 0 deletions charts/simple-time-service/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,16 @@ serviceMonitor:
# -- Extra labels added to the ServiceMonitor (used to target a specific Prometheus instance)
labels: {}

tracing:
# -- Enable OpenTelemetry tracing
enabled: false
# -- Direct OTLP HTTP endpoint (e.g. `http://otel-collector.tracing.svc.cluster.local:4318`). When empty, uses the node-local DaemonSet agent via `status.hostIP:agentPort`
endpoint: ""
# -- OTLP HTTP port on the node-local OTel Collector agent; used only when `endpoint` is empty
agentPort: 4318
# -- Service name reported in traces and correlated in Grafana Tempo
serviceName: simple-time-service

# -- Extra annotations added to every pod
podAnnotations: {}

Expand Down
Loading
Loading