From f4aa5f131e47a500178c30a57329cf0d7dee8e3e Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 6 Nov 2026 10:27:06 -0700 Subject: [PATCH 1/2] docs: administration: document processor hold metrics and reload validation - add fluentbit_processor_accounting_errors_total metric - add fluentbit_processor_items_held_current and items_held_total metrics - add hot reload processor validation section with halt log messages - note that filters used as processors skip startup checks when inactive This is the missing docs PR for fluent/fluent-bit#12731. Signed-off-by: Eric D. Schabell --- administration/hot-reload.md | 11 +++++++++++ administration/monitoring.md | 3 +++ 2 files changed, 14 insertions(+) diff --git a/administration/hot-reload.md b/administration/hot-reload.md index 889e13af0..4f39ed837 100644 --- a/administration/hot-reload.md +++ b/administration/hot-reload.md @@ -51,6 +51,17 @@ If Fluent Bit can't preserve the directory while reloading, it stops the reload, [reload] copying configuration path failed. Reloading is halted ``` +## Processor validation + +Fluent Bit version 5.2 and greater validates the processors attached to every input and output before it replaces the running pipeline. If a processor has an invalid configuration, Fluent Bit stops the reload, keeps the previous configuration running, and logs: + +```text +[reload] processor configuration preflight failed +[reload] reloaded config is invalid. Reloading is halted +``` + +Validation checks the properties of every processor, including [filters used as processors](../pipeline/processors/filters.md). For filters used as processors, it also runs the filter's startup checks, unless the filter sets `active: false`. + ## How to reload After updating the configuration, use one of the following methods to perform a hot reload: diff --git a/administration/monitoring.md b/administration/monitoring.md index 15789fdb6..c06e86ed8 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -236,10 +236,13 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_output_retried_records_total` | name: the name or alias for the output instance | The number of log records that experienced a retry. This metric is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin might perform multiple actions that generate many error messages when uploading a single chunk. | counter | records | | `fluentbit_output_retries_failed_total` | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a `Retry_Limit`, which applies to chunks. When the `Retry_Limit` is exceeded, the chunk is discarded and this metric is incremented. | counter | chunks | | `fluentbit_output_retries_total` | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks | +| `fluentbit_processor_accounting_errors_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of invalid record retention accounting operations by a processor unit that holds log records. Fluent Bit also logs `invalid retention accounting operation` the first time this happens for each processor unit. | counter | errors | | `fluentbit_processor_errors_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of errors that a processor unit encountered. | counter | errors | | `fluentbit_processor_invocations_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of processor unit invocations. | counter | invocations | | `fluentbit_processor_items_add_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of items that a processor unit added. Incremented when `items_out_total` exceeds `items_in_total`. | counter | records | | `fluentbit_processor_items_drop_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of items that a processor unit dropped. Incremented when `items_in_total` exceeds `items_out_total`. | counter | records | +| `fluentbit_processor_items_held_current` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The number of log records currently held by a processor unit, such as records retained by the `fts` processor. Only reported for log processors attached to inputs. | gauge | records | +| `fluentbit_processor_items_held_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of log records that a processor unit admitted for retention. Only reported for log processors attached to inputs. | counter | records | | `fluentbit_processor_items_in_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of items that a processor unit received. | counter | records | | `fluentbit_processor_items_out_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of items that a processor unit produced. | counter | records | | `fluentbit_uptime` | hostname: the hostname on running Fluent Bit | The number of seconds that Fluent Bit has been running. | counter | seconds | From dedc3a78768bb85a57f36ac8d6cd4de096d2bc91 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 6 Nov 2026 20:36:36 -0700 Subject: [PATCH 2/2] docs: administration: address review feedback on processor validation and metrics - hot-reload: limit the reload guarantee to preflight failures; startup-time processor errors abort the reload and Fluent Bit exits - hot-reload: correct preflight scope (property checks; deeper checks only for fts, Lua, and Wasm); note --dry-run runs the same validation - monitoring: accounting_errors_total is only reported for log processors attached to inputs Signed-off-by: Eric D. Schabell --- administration/hot-reload.md | 6 ++++-- administration/monitoring.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/administration/hot-reload.md b/administration/hot-reload.md index 4f39ed837..5fbd3b1ac 100644 --- a/administration/hot-reload.md +++ b/administration/hot-reload.md @@ -53,14 +53,16 @@ If Fluent Bit can't preserve the directory while reloading, it stops the reload, ## Processor validation -Fluent Bit version 5.2 and greater validates the processors attached to every input and output before it replaces the running pipeline. If a processor has an invalid configuration, Fluent Bit stops the reload, keeps the previous configuration running, and logs: +Fluent Bit version 5.2 and greater validates the processors attached to every input and output before it replaces the running pipeline. If this validation fails, Fluent Bit stops the reload, keeps the previous configuration running, and logs: ```text [reload] processor configuration preflight failed [reload] reloaded config is invalid. Reloading is halted ``` -Validation checks the properties of every processor, including [filters used as processors](../pipeline/processors/filters.md). For filters used as processors, it also runs the filter's startup checks, unless the filter sets `active: false`. +Validation checks the property names and values of every processor, including [filters used as processors](../pipeline/processors/filters.md). Deeper checks run only for processors that support them: the `fts` processor, and the Lua and Wasm filters used as processors, unless the filter sets `active: false`. + +Validation doesn't catch every error. Errors that only appear when a processor starts, such as an invalid regular expression in a `content_modifier` processor, are found after Fluent Bit has stopped the previous pipeline. In that case, the reload is aborted and Fluent Bit exits instead of keeping the previous configuration. Test configuration changes in a test environment before you reload a production instance. The `--dry-run` option runs the same validation, so it doesn't catch these errors either. ## How to reload diff --git a/administration/monitoring.md b/administration/monitoring.md index c06e86ed8..f406a16f1 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -236,7 +236,7 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_output_retried_records_total` | name: the name or alias for the output instance | The number of log records that experienced a retry. This metric is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin might perform multiple actions that generate many error messages when uploading a single chunk. | counter | records | | `fluentbit_output_retries_failed_total` | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a `Retry_Limit`, which applies to chunks. When the `Retry_Limit` is exceeded, the chunk is discarded and this metric is incremented. | counter | chunks | | `fluentbit_output_retries_total` | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks | -| `fluentbit_processor_accounting_errors_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of invalid record retention accounting operations by a processor unit that holds log records. Fluent Bit also logs `invalid retention accounting operation` the first time this happens for each processor unit. | counter | errors | +| `fluentbit_processor_accounting_errors_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of invalid record retention accounting operations by a processor unit that holds log records. Fluent Bit also logs `invalid retention accounting operation` the first time this happens for each processor unit. Only reported for log processors attached to inputs. | counter | errors | | `fluentbit_processor_errors_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of errors that a processor unit encountered. | counter | errors | | `fluentbit_processor_invocations_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of processor unit invocations. | counter | invocations | | `fluentbit_processor_items_add_total` | scope: the plugin type that owns the processor (`input` or `output`), owner: the name or alias of the input or output instance that owns the processor chain, processor: the name of the processor unit, stage: the zero-based position of this processor in the chain, signal: the data type being processed (`logs`, `metrics`, `traces`, or `profiles`) | The total number of items that a processor unit added. Incremented when `items_out_total` exceeds `items_in_total`. | counter | records |