From 6414300c0718c51fc9f851a2da75b381a2666123 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Sat, 7 Nov 2026 15:12:04 -0700 Subject: [PATCH] docs: pipeline: inputs: opentelemetry: document v5.2 request limits - buffer_max_size: default is now 64 MiB instead of 4M - add http_server.max_decoded_body_size and protobuf_decode_max_size - add request size limits section: protocol overhead, 413 and gRPC status responses, decode budget checked before ingestion, logs conversion limit, decimal units - gRPC message size limit now follows buffer_max_size instead of a fixed 16 MiB - documents fluent/fluent-bit#12705 - buffer_max_size: note it's the alias for http_server.buffer_max_size - raw_traces: applies to JSON only; Protocol Buffers and gRPC traces are size-checked and decoded as traces - request size limits: the 413 message is only sent when the request reaches the plugin; the listener rejects oversized requests with an empty 413 (HTTP/1.1) or fails them (HTTP/2) Signed-off-by: Eric D. Schabell --- pipeline/inputs/opentelemetry.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/pipeline/inputs/opentelemetry.md b/pipeline/inputs/opentelemetry.md index 757fe9e8d..4b91b68c6 100644 --- a/pipeline/inputs/opentelemetry.md +++ b/pipeline/inputs/opentelemetry.md @@ -26,7 +26,7 @@ For a cross-plugin explanation of the shared listener settings, see |-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------| | `alias` | Sets an alias for multiple instances of the same input plugin. If no alias is specified, a default name will be assigned using the plugin name followed by a dot and a sequence number. | _none_ | | `buffer_chunk_size` | Size of each buffer chunk allocated for HTTP requests (advanced users only). | `512K` | -| `buffer_max_size` | Maximum size of the HTTP request buffer in `KB`, `MB`, or `GB`. | `4M` | +| `buffer_max_size` | Maximum size of an OTLP request payload, as received and after decompression. Compatibility alias for `http_server.buffer_max_size`. See [Request size limits](#request-size-limits). | 64 MiB | | `encode_profiles_as_log` | Encode profiles received as text and ingest them in the logging pipeline. | `true` | | `host` | The hostname. | `localhost` | | `http2` | Enable HTTP/2 protocol support for the OpenTelemetry receiver. | `true` | @@ -34,6 +34,7 @@ For a cross-plugin explanation of the shared listener settings, see | `http_server.workers` | Number of HTTP listener worker threads. | `1` | | `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | | `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | +| `http_server.max_decoded_body_size` | Maximum size of an `OTLP/HTTP` body after decompression. Defaults to `buffer_max_size`. Supported in v5.2 or later. See [Request size limits](#request-size-limits). | 64 MiB | | `listen` | The network address to listen on. | `0.0.0.0` | | `log_level` | Specifies the log level for this plugin. If not set here, the plugin uses the global log level specified in the `service` section of your configuration file. | `info` | | `log_suppress_interval` | Suppresses log messages from this plugin that appear similar within a specified time interval. `0` no suppression. | `0` | @@ -55,7 +56,8 @@ For a cross-plugin explanation of the shared listener settings, see | `oauth2.validate` | Enable `OAuth 2.0` `JWT` validation for incoming requests. | `false` | | `port` | The port for Fluent Bit to listen for incoming connections. | `4318` | | `profiles_support` | This is an experimental feature, feel free to test it but don't enable this in production environments. | `false` | -| `raw_traces` | Forward traces without processing. When set to `false` (default), traces are processed using the unified JSON parser with strict validation. When set to `true`, trace data is forwarded as raw log messages without validation or processing. | `false` | +| `protobuf_decode_max_size` | Maximum estimated memory to decode the Protocol Buffers payloads in one request. Must be a positive size. Supported in v5.2 or later. See [Request size limits](#request-size-limits). | 128 MiB | +| `raw_traces` | Forward JSON traces without processing. When set to `false` (default), JSON traces are processed using the unified JSON parser with strict validation. When set to `true`, JSON trace data is forwarded as raw log messages without validation or processing. Protocol Buffers and gRPC traces are always decoded as traces. | `false` | | `routable` | If set to `true`, the data generated by the plugin will be routable, meaning that it can be forwarded to other plugins or outputs. If set to `false`, the data will be discarded. | `true` | | `storage.pause_on_chunks_overlimit` | Enable pausing on an input when they reach their chunks limit. | _none_ | | `storage.type` | Sets the storage type for this input, one of: `filesystem`, `memory` or `memrb`. | `memory` | @@ -85,7 +87,7 @@ The `http_server.ingress_queue_event_limit` and `http_server.ingress_queue_byte_limit` settings matter only when `http_server.workers` is greater than `1`. -When `raw_traces` is set to `false` (default), the traces endpoint (`/v1/traces`) processes incoming trace data using the unified JSON parser with strict validation. The endpoint accepts both `protobuf` and `JSON` encoded payloads. When `raw_traces` is set to `true`, any data forwarded to the traces endpoint will be packed and forwarded as a log message without processing, validation, or conversion to the Fluent Bit internal trace format. +When `raw_traces` is set to `false` (default), the traces endpoint (`/v1/traces`) processes incoming trace data using the unified JSON parser with strict validation. The endpoint accepts both `protobuf` and `JSON` encoded payloads. When `raw_traces` is set to `true`, JSON payloads sent to the traces endpoint are packed and forwarded as a log message without processing, validation, or conversion to the Fluent Bit internal trace format. `raw_traces` doesn't apply to Protocol Buffers or gRPC payloads: they still go through the checks in [Request size limits](#request-size-limits), including the decode budget, and are decoded as traces. ### OpenTelemetry transport protocol endpoints @@ -112,11 +114,21 @@ For `OTLP/GRPC`: - `/opentelemetry.proto.collector.trace.v1.TraceService/Export` - `/opentelemetry.proto.collector.traces.v1.TracesService/Export` +### Request size limits + +In v5.2 or later, the plugin applies the following limits to each request: + +- `buffer_max_size` limits the request payload. It defaults to 64 MiB (67,108,864 bytes), the size recommended by the OTLP specification. Earlier versions used the shared `4M` default. Fluent Bit adds 64 KiB to the HTTP request buffer for headers and gRPC framing. An `OTLP/HTTP` request whose payload is larger than `buffer_max_size`, before or after decompression, is rejected with `413`. When an uncompressed payload reaches the plugin, the response includes the message `OTLP request size exceeds limit`. A request larger than the HTTP request buffer (`buffer_max_size` plus 64 KiB) is rejected by the listener before it reaches the plugin. On HTTP/1.1, the listener sends a `413` response with an empty body and closes the connection. On HTTP/2, the request fails without a `413` response. A compressed body that exceeds the limit during decompression gets a `413` response with an empty body. +- `http_server.max_decoded_body_size` limits an `OTLP/HTTP` body while it's decompressed. If it isn't set, it equals the `buffer_max_size` value. Setting it to `0` selects `100M`. Because the decompressed payload is also checked against `buffer_max_size`, a higher value doesn't allow larger payloads. +- `protobuf_decode_max_size` limits the estimated memory needed to decode the Protocol Buffers payload of a request, summed across all of its gRPC messages. The default is 128 MiB (134,217,728 bytes). Fluent Bit checks this budget before it ingests any data from the request. A request over the budget is rejected with `413` or gRPC status `8 (RESOURCE_EXHAUSTED)`, and a malformed payload with `400` or gRPC status `3 (INVALID_ARGUMENT)`. Both use the message `protobuf decode budget exceeded or invalid payload`. The same value also limits the memory used to convert each Protocol Buffers logs message of 8 KiB or more. A logs message that exceeds it during conversion fails with `400` or a gRPC error. JSON payloads aren't affected. + +Size values such as `64M` use decimal units. To set exactly 64 MiB, use `67108864`. If you raise `buffer_max_size`, you might also need to raise `protobuf_decode_max_size`, because larger payloads need more memory to decode. + ### gRPC message size limits In v5.1.4 or later, the plugin enforces the following limits on `OTLP/GRPC` requests: -- Each gRPC message can't exceed 16 MiB, both as received and after decompression. +- Each gRPC message can't exceed `buffer_max_size` in v5.2 or later, or 16 MiB in earlier versions, both as received and after decompression. - The combined size of all messages in a request, after decompression, can't exceed `buffer_max_size`. Messages compressed with `gzip`, `zstd`, or `snappy` (set by the `grpc-encoding` header) are decompressed in bounded steps, so a highly compressed payload is rejected as soon as its expanded size reaches the limit. Requests that exceed a limit are rejected with gRPC status `8 (RESOURCE_EXHAUSTED)` and the message `gRPC message size exceeds limit` or `expanded gRPC payload size exceeds limit`. Compressed messages that use any other `grpc-encoding` value are rejected with gRPC status `12 (UNIMPLEMENTED)`. Uncompressed messages aren't affected by the `grpc-encoding` value.