Skip to content

feat: added capability to extract metrics from the logs - #101

Open
CptSchnitz wants to merge 3 commits into
masterfrom
logs-to-metrics
Open

feat: added capability to extract metrics from the logs#101
CptSchnitz wants to merge 3 commits into
masterfrom
logs-to-metrics

Conversation

@CptSchnitz

Copy link
Copy Markdown

Main feature: derive Prometheus metrics from nginx access logs via Fluent Bit, instead of just shipping raw logs.

  • OGC request parsing (Lua) — new custom.lua script hooked into Fluent Bit that parses both KVP (?service=WMS&request=...) and RESTful (/wmts/.../{z}/{x}/{y}.{fmt}) URLs to extract ogc_service, ogc_operation, ogc_layer, ogc_zoom (bucketed into ogc_zoom_group to control cardinality), and ogc_format, plus a normalized result (ok/error) and request_time_seconds.
  • Metrics generation (log_to_metrics filters) — new Fluent Bit filters that turn the parsed log fields into:
    • mapproxy_wms_requests_total / mapproxy_wmts_requests_total counters (by operation, layer, format, result — WMTS also labeled by client name)
    • nginx_http_errors_total counter for all 4xx/5xx responses
    • mapproxy_{wms,wmts}_request_duration_seconds latency histograms (labeled by result)
  • Simplified log format — dropped the large, verbose log_format.conf (nginx log_format directive with ~50 fields) since raw stdout logs are no longer needed now that metrics are extracted (chore: remove stdout).
  • Dependency bump — nginx subchart upgraded 2.3.1 → 2.4.0 to pick up the fluentbit/log_to_metrics/Lua filter support this feature relies on.

Net effect: mapproxy nginx now exports structured request/latency metrics (by OGC service, operation, layer, zoom bucket, format, and error status) directly from access logs, rather than relying on log aggregation alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant