Skip to content

logging.md

Latest commit

 

History

History
81 lines (60 loc) · 2.77 KB

logging.md

File metadata and controls

81 lines (60 loc) · 2.77 KB
Raw

Logging for devops.local (central Loki + Docker agent)

Apps write logs to stdout (or a file). A Docker agent (loki-agent = Grafana Alloy) on that host ships them to Loki.

┌─ app server ─────────────────────┐         HTTPS + Basic auth
│  note-keeper (Docker)            │      ┌──────────────────────┐
│       └─ stdout                  │      │ devops.local         │
│  loki-agent (Docker) ────────────┼─────►│ nginx /loki/ → Loki  │
│       reads docker.sock          │      │ Grafana Explore      │
└──────────────────────────────────┘      └──────────────────────┘

Package to copy to every app host: stacks/monitoring/agent/
(see agent/README.md)

1. One-time on devops.local

make setup          # LOKI_PUSH_USERNAME / LOKI_PUSH_PASSWORD
make monitoring
make middleware     # https://devops.local/loki/api/v1/push

2. Install the Docker agent on the app server

scp -r stacks/monitoring/agent user@appserver:/opt/loki-agent
ssh user@appserver
cd /opt/loki-agent
curl -fk https://devops.local/ca.crt -o ca.crt
cp .env.example .env          # set LOKI_PUSH_PASSWORD, AGENT_HOST=appserver-01
docker compose up -d

The agent container:

  • mounts /var/run/docker.sock
  • tails all container logs on that host
  • pushes to https://devops.local/loki/api/v1/push

3. Application Dockerfile / Compose

Do not put Fluent Bit / Loki inside the app image. Only console logging:

# Spring example — logs to stdout
ENV JAVA_TOOL_OPTIONS="-Dlogging.pattern.console=%d{ISO8601} %-5level %logger{36} - %msg%n"
ENTRYPOINT ["java", "-jar", "app.jar"]
services:
  note-keeper:
    image: note-keeper:latest
    container_name: note-keeper
    # agent on the same host will pick this up automatically

Full app+agent example: stacks/monitoring/agent/examples/spring-boot/.

Bare-metal (no Docker for the app)

Write to /var/log/apps/*.log (or set APP_LOG_DIR). The same loki-agent container already tails that path.

4. Verify in Grafana

Explore → Loki:

Query Meaning
{host="appserver-01"} everything from that agent
{container="note-keeper"} one container
{job="docker"} all Docker logs from agents

5. Flags

Variable Role
LOKI_ENABLED Loki + Alloy on devops.local
LOKI_PUSH_* auth for remote loki-agent → /loki/