From 1065eda160dc67353c0efa2043607103f9278a95 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Mon, 2 Nov 2026 17:54:59 +0100 Subject: [PATCH] docs: pipeline: inputs: opentelemetry: document worker log processing and busy responses - Add HTTP workers section describing the deferred ingress queue - Document which log processors run inside HTTP workers and the fallback - Document 503 / gRPC status 14 responses when the ingress queue is full - clarify retry guidance for ingress queue rejections - Explain that mem_buf_limit can lower the effective ingress queue byte limit - Separate the temporary queue-full rejection from the oversized-request rejection - Recommend retrying only when the queue is full; oversized requests need smaller requests or a higher limit - Compare the effective byte limit against each queued event instead of the whole request - Note that one request can add several events to the ingress queue - Align the retry guidance with the per-event limit - Note that clients can't distinguish a full queue from an oversized event in the response - Recommend retrying with backoff and explain that repeated failures point to an event exceeding the effective byte limit Signed-off-by: Eric D. Schabell --- pipeline/inputs/opentelemetry.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/pipeline/inputs/opentelemetry.md b/pipeline/inputs/opentelemetry.md index 757fe9e8d..b6f53825d 100644 --- a/pipeline/inputs/opentelemetry.md +++ b/pipeline/inputs/opentelemetry.md @@ -87,6 +87,35 @@ The `http_server.ingress_queue_event_limit` and When `raw_traces` is set to `false` (default), the traces endpoint (`/v1/traces`) processes incoming trace data using the unified JSON parser with strict validation. The endpoint accepts both `protobuf` and `JSON` encoded payloads. When `raw_traces` is set to `true`, any data forwarded to the traces endpoint will be packed and forwarded as a log message without processing, validation, or conversion to the Fluent Bit internal trace format. +### HTTP workers + +When `http_server.workers` is greater than `1`, each worker decodes incoming requests and places the data in a deferred ingress queue, which the input then consumes. + +#### Log processing in HTTP workers + +If every processor in the input's `logs` processor chain can run in a worker, the chain runs inside the HTTP worker that received the request, and the processed logs are added to the ingress queue. The following processors can run in HTTP workers: + +- [Content modifier](../processors/content-modifier.md) +- [SQL](../processors/sql.md) + +If the chain includes any other processor, or a filter used as a processor, the whole chain runs after the logs leave the ingress queue, the same as with a single worker. You don't need to change any configuration to use either mode. + +#### Full ingress queue + +The deferred ingress queue is limited by `http_server.ingress_queue_event_limit` and an effective byte limit. The effective byte limit is `http_server.ingress_queue_byte_limit`, or `mem_buf_limit` when `mem_buf_limit` is set to a lower value. + +The plugin rejects a request in either of these cases: + +- **The queue is full**: The queue has reached one of its limits and doesn't free enough room within a short wait. +- **An event is too large**: A single event that the plugin adds to the queue exceeds the effective byte limit. Each event holds the data from one OTLP message, so one request can add several events to the queue. + +In both cases, the plugin returns the following responses: + +- **HTTP/1.1 and HTTP/2**: Returns `503 Service Unavailable` with the message `server overloaded: deferred ingress queue is full`. +- **gRPC**: Returns gRPC status `14 (UNAVAILABLE)` with the message `server overloaded`. + +Clients can't tell the two cases apart from the response. Retry rejected requests with backoff, because a full queue is temporary. If a request keeps failing after several retries, an event it produces probably exceeds the effective byte limit, and retrying won't succeed. Send smaller requests, or raise the effective byte limit by increasing `http_server.ingress_queue_byte_limit` and, if it's set, `mem_buf_limit`. + ### OpenTelemetry transport protocol endpoints Fluent Bit exposes the following endpoints for data ingestion based on the OpenTelemetry protocol: