diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index f0712db73..e65a77747 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -167,7 +167,8 @@ plugin-specific behavior. | Key | Description | Default | | --- | ----------- | ------- | | `http_server.http2` | Enable HTTP/2 support for the input listener. | `true` | -| `http_server.buffer_max_size` | Set the maximum size of the HTTP request buffer. | `4M` | +| `http_server.buffer_max_size` | Set the maximum size of the HTTP request buffer. In v5.2 or later, the `opentelemetry` input defaults to 64 MiB instead. | `4M` | +| `http_server.max_decoded_body_size` | Set the maximum size of a request body after automatic decompression, independently of `http_server.buffer_max_size`. `0` selects `100M`, for every input. Introduced in v5.2. When this key isn't set, the `opentelemetry` input uses a different default; see [Request body decompression](#request-body-decompression). | `100M` | | `http_server.buffer_chunk_size` | Set the allocation chunk size used for the HTTP request buffer. | `512K` | | `http_server.max_connections` | Set the maximum number of concurrent active HTTP connections. `0` means unlimited. | `0` | | `http_server.workers` | Set the number of HTTP listener worker threads. | `1` | @@ -183,6 +184,22 @@ For backward compatibility, some plugins also accept the legacy aliases `http2`, These plugins also pause and resume the shared listener in response to [backpressure](../../backpressure.md#pause-behavior-for-http-based-inputs). While an input is paused, Fluent Bit accepts and immediately closes incoming connections instead of buffering the requests, so clients must retry. +#### Request body decompression + +In v5.0 or later, these inputs automatically decompress request bodies sent with the `Content-Encoding` header set to `gzip`, `zstd`, or `snappy`, whatever the `http2` setting. In earlier versions, the supported encodings depended on the input and, for some inputs, on the `http2` setting. Requests without a `Content-Encoding` header, or with `identity`, aren't decompressed. `http_server.buffer_max_size` limits the request as received, and `http_server.max_decoded_body_size` limits the body after decompression. + +The `opentelemetry` input applies a different default. When `http_server.max_decoded_body_size` isn't set, the decompressed body is limited to the input's request size limit, which is its `buffer_max_size` value, or 64 MiB if that isn't set. Setting `http_server.max_decoded_body_size` to `0` selects `100M`. The `opentelemetry` input also checks the decompressed body against its `buffer_max_size` value, so it rejects a larger body even when `http_server.max_decoded_body_size` is higher. + +In v5.2 or later, Fluent Bit rejects a request that it can't decompress, and responds with one of the following statuses: + +- `413`: the decompressed body is larger than `http_server.max_decoded_body_size`. +- `503`: Fluent Bit couldn't allocate memory to decompress the body. Clients can retry the request. +- `400`: the body is invalid, or uses an unsupported or stacked encoding, such as `deflate` or `gzip, zstd`. + +Earlier versions passed bodies with `deflate` or other unsupported encodings to the input without decompressing them. If your clients send `deflate`, switch them to `gzip`, `zstd`, or `snappy` before you upgrade. + +When `oauth2.validate` is enabled on the `http` or `opentelemetry` input, Fluent Bit v5.2 or later validates the token before it decompresses the body, so a request without a valid token is rejected with `401` before decompression. In v5.0 and v5.1, Fluent Bit decompressed the body first and validated the token afterward, so unauthenticated requests could still use decompression resources. On the `http` input with `enable_health_endpoint` enabled, `GET /health` requests don't require a token, so a compressed body sent with one is still decompressed, up to the `http_server.max_decoded_body_size` limit. Decompression within that limit still uses CPU and memory. + ### Incoming `OAuth 2.0` `JWT` validation settings The HTTP-based input plugins that support bearer token validation share the following `oauth2.*` settings: @@ -196,7 +213,7 @@ The HTTP-based input plugins that support bearer token validation share the foll | `oauth2.allowed_clients` | Authorized `client_id` or `azp` claim values. This key can be specified multiple times. | _none_ | | `oauth2.jwks_refresh_interval` | How often in seconds to refresh cached `JWKS` keys. | `300` | -When validation is enabled, requests without a valid `Authorization: Bearer ` header are rejected. +When validation is enabled, requests without a valid `Authorization: Bearer ` header are rejected. On the `http` input, `GET /health` requests are exempt when `enable_health_endpoint` is enabled. ### Example input configuration diff --git a/pipeline/inputs/elasticsearch.md b/pipeline/inputs/elasticsearch.md index 40aab8cc3..f1d7b9504 100644 --- a/pipeline/inputs/elasticsearch.md +++ b/pipeline/inputs/elasticsearch.md @@ -28,6 +28,7 @@ For a cross-plugin explanation of the shared listener settings, see | `http_server.workers` | Number of HTTP listener worker threads. | `1` | | `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | | `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | +| `http_server.max_decoded_body_size` | Maximum size of a request body after automatic decompression, independently of `buffer_max_size`. `0` uses the default. Introduced in v5.2. See [Request body decompression](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#request-body-decompression). | `100M` | | `listen` | The address to listen on. | `0.0.0.0` | | `meta_key` | Specify a key name for meta information. | `@meta` | | `port` | The port for Fluent Bit to listen on. | `9200` | diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md index 1929447a7..6ca5f1061 100644 --- a/pipeline/inputs/http.md +++ b/pipeline/inputs/http.md @@ -28,6 +28,7 @@ For a cross-plugin explanation of the shared listener settings, see | `http_server.workers` | Number of HTTP listener worker threads. | `1` | | `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | | `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | +| `http_server.max_decoded_body_size` | Maximum size of a request body after automatic decompression, independently of `buffer_max_size`. `0` uses the default. Introduced in v5.2. See [Request body decompression](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#request-body-decompression). | `100M` | | `listen` | The address to listen on. | `0.0.0.0` | | `oauth2.allowed_audience` | Audience claim to enforce when validating incoming `OAuth 2.0` `JWT` tokens. | _none_ | | `oauth2.allowed_clients` | Authorized `client_id` or `azp` claim values. Can be specified multiple times. | _none_ | @@ -52,11 +53,11 @@ HTTP input plugin supports TLS/SSL. For more details about the properties availa ### gzipped content -The HTTP input plugin will accept and automatically handle gzipped content in version 2.2.1 or later if the header `Content-Encoding: gzip` is set on the received data. +The HTTP input plugin will accept and automatically handle gzipped content in version 2.2.1 or later if the header `Content-Encoding: gzip` is set on the received data. It also decompresses `zstd` and `snappy` content: in v3.x only when `http2` is `true`, which is the default, and in v4.0 or later whatever the `http2` value. In v5.2 or later, requests with any other encoding except `identity`, such as `deflate`, are rejected with `400`, and decompressed bodies are limited by `http_server.max_decoded_body_size`. See [Request body decompression](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#request-body-decompression). ### `OAuth 2.0 JWT` validation -When `oauth2.validate` is set to `true`, the HTTP input plugin validates the `Authorization: Bearer ` header on every incoming request. Requests with a missing, expired, or invalid token are rejected with a `401` response. +When `oauth2.validate` is set to `true`, the HTTP input plugin validates the `Authorization: Bearer ` header on every incoming request, except `GET /health` requests when `enable_health_endpoint` is enabled. In v5.2 or later, the token is validated before the request body is decompressed. Requests with a missing, expired, or invalid token are rejected with a `401` response. `oauth2.issuer` and `oauth2.jwks_url` are both required when validation is enabled. `JWKS` keys are fetched lazily: the first request that requires validation triggers the initial retrieval from `oauth2.jwks_url`. Keys are then cached and refreshed every `oauth2.jwks_refresh_interval` seconds. diff --git a/pipeline/inputs/prometheus-remote-write.md b/pipeline/inputs/prometheus-remote-write.md index 7f97db714..b8670e55f 100644 --- a/pipeline/inputs/prometheus-remote-write.md +++ b/pipeline/inputs/prometheus-remote-write.md @@ -29,6 +29,7 @@ For a cross-plugin explanation of the shared listener settings, see | `http_server.workers` | Number of HTTP listener worker threads. | `1` | | `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | | `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | +| `http_server.max_decoded_body_size` | Maximum size of a request body after automatic decompression, independently of `buffer_max_size`. `0` uses the default. Introduced in v5.2. See [Request body decompression](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#request-body-decompression). | `100M` | | `listen` | The address to listen on. | `0.0.0.0` | | `port` | The port to listen on. | `8080` | | `successful_response_code` | Specifies the success response code. Supported values are `200`, `201`, and `204`. | `201` | diff --git a/pipeline/inputs/splunk.md b/pipeline/inputs/splunk.md index 7820e67cc..05949b026 100644 --- a/pipeline/inputs/splunk.md +++ b/pipeline/inputs/splunk.md @@ -28,6 +28,7 @@ For a cross-plugin explanation of the shared listener settings, see | `http_server.workers` | Number of HTTP listener worker threads. | `1` | | `http_server.ingress_queue_event_limit` | Maximum number of deferred ingress queue entries. Applies only when `http_server.workers` is greater than `1`. | `8192` | | `http_server.ingress_queue_byte_limit` | Maximum size of the deferred ingress queue. Applies only when `http_server.workers` is greater than `1`. | `256M` | +| `http_server.max_decoded_body_size` | Maximum size of a request body after automatic decompression, independently of `buffer_max_size`. `0` uses the default. Introduced in v5.2. See [Request body decompression](../../administration/configuring-fluent-bit/yaml/pipeline-section.md#request-body-decompression). | `100M` | | `listen` | The address to listen on. | `0.0.0.0` | | `port` | The port for Fluent Bit to listen on. | `8088` | | `remote_addr_key` | Record key name used to store the remote address when `add_remote_addr` is enabled. | `remote_addr` |