diff --git a/SUMMARY.md b/SUMMARY.md index d8acb8769..c0e768451 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -139,6 +139,18 @@ * [Content modifier](pipeline/processors/content-modifier.md) * [Cumulative to delta](pipeline/processors/cumulative-to-delta.md) * [Filters as processors](pipeline/processors/filters.md) + * [Fluent Typed Scripting (FTS)](pipeline/processors/fluent-typed-scripting.md) + * [Quickstart](pipeline/processors/fluent-typed-scripting/quickstart.md) + * [Configuration](pipeline/processors/fluent-typed-scripting/configuration.md) + * [Typed values](pipeline/processors/fluent-typed-scripting/typed-values.md) + * [Logs](pipeline/processors/fluent-typed-scripting/logs.md) + * [Metrics](pipeline/processors/fluent-typed-scripting/metrics.md) + * [Traces](pipeline/processors/fluent-typed-scripting/traces.md) + * [Common helpers](pipeline/processors/fluent-typed-scripting/common-helpers.md) + * [Errors and limits](pipeline/processors/fluent-typed-scripting/errors-and-limits.md) + * [Fluent Typed Scripting (FTS) and the Lua filter](pipeline/processors/fluent-typed-scripting/lua-filter.md) + * [Native recipe blocks](pipeline/processors/fluent-typed-scripting/native-recipes.md) + * [Recipes](pipeline/processors/fluent-typed-scripting/recipes.md) * [Journald OpenTelemetry](pipeline/processors/journald-otel.md) * [Labels](pipeline/processors/labels.md) * [Metrics selector](pipeline/processors/metrics-selector.md) diff --git a/installation/downloads/source/build-and-install.md b/installation/downloads/source/build-and-install.md index dc9287e44..37acc109f 100644 --- a/installation/downloads/source/build-and-install.md +++ b/installation/downloads/source/build-and-install.md @@ -137,7 +137,7 @@ Fluent Bit provides configurable options to CMake that can be enabled or disable | `FLB_UTF8_ENCODER` | Build with UTF8 encoding support | `Yes` | | `FLB_WASM` | Build with Wasm runtime support | `Yes` | | `FLB_WASM_STACK_PROTECT` | Build with WASM runtime with strong stack protector flags | `No` | -| `FLB_WAMRC` | Build with Wasm AOT compiler executable | `No` | +| `FLB_WAMRC` | Build with Wasm ahead-of-time compiler executable | `No` | | `FLB_WINDOWS_DEFAULTS` | Build with predefined Windows settings | `Yes` | | `FLB_ZIG` | Enable zig integration | `Yes` | @@ -238,6 +238,7 @@ The following table describes the processors available: | Option | Description | Default | |:-------------------------------------------------------------------------------------------------|:--------------------------------------------|:--------| | [`FLB_PROCESSOR_CONTENT_MODIFIER`](../../../pipeline/processors/content-modifier.md) | Enable content modifier processor | `On` | +| [`FLB_PROCESSOR_FTS`](../../../pipeline/processors/fluent-typed-scripting.md) | Enable Fluent Typed Scripting processor | `On` | | [`FLB_PROCESSOR_JOURNALD_OTEL`](../../../pipeline/processors/journald-otel.md) | Enable journald OpenTelemetry processor | `On` | | [`FLB_PROCESSOR_LABELS`](../../../pipeline/processors/labels.md) | Enable metrics label manipulation processor | `On` | | [`FLB_PROCESSOR_METRICS_SELECTOR`](../../../pipeline/processors/metrics-selector.md) | Enable metrics selector processor | `On` | diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 8f93e74ef..7b02eb345 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -8,6 +8,8 @@ The _Lua_ filter lets you modify incoming records (or split one record into multiple records) using custom [Lua](https://www.lua.org/) scripts. +For policies that need typed telemetry values, metric or span transformations, or per-item rollback, see [Fluent Typed Scripting (FTS)](../processors/fluent-typed-scripting.md) and the [FTS and Lua filter comparison](../processors/fluent-typed-scripting/lua-filter.md). + A Lua-based filter requires two steps: 1. Configure the filter in the main configuration. diff --git a/pipeline/processors.md b/pipeline/processors.md index ca274ba87..a9dc60c71 100644 --- a/pipeline/processors.md +++ b/pipeline/processors.md @@ -15,6 +15,7 @@ Only [YAML configuration files](../administration/configuring-fluent-bit/yaml.md Fluent Bit offers the following processors: - [Content modifier](./processors/content-modifier.md): Manipulate the content, metadata, and attributes of logs and traces. +- [Fluent Typed Scripting (FTS)](./processors/fluent-typed-scripting.md): Write policies in Lua with typed access to logs, metrics, and traces. - [Journald OpenTelemetry](./processors/journald-otel.md): Map `systemd` journal fields to the OpenTelemetry logs data model. - [Labels](./processors/labels.md): Add, update, or delete metric labels. - [Metrics selector](./processors/metrics-selector.md): Choose which metrics to keep or discard. diff --git a/pipeline/processors/fluent-typed-scripting.md b/pipeline/processors/fluent-typed-scripting.md new file mode 100644 index 000000000..9aba08711 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting.md @@ -0,0 +1,35 @@ +# Fluent Typed Scripting (FTS) + +{% hint style="info" %} +**Supported event types:** `logs`, `metrics`, `traces` +{% endhint %} + +Fluent Typed Scripting (FTS) lets you write policies in Lua with typed access to telemetry. Its configuration name is `fts`. Use it to redact structured fields, normalize log content and OpenTelemetry context, transform metric families, or edit spans. + +_Typed_ refers to the telemetry values exposed to the policy, including maps, arrays, binary data, exact 64-bit integers, and nanosecond timestamps. FTS doesn't add static typing to Lua. Type checks happen at runtime. + +## Availability + +FTS is included in the development source for Fluent Bit 5.2.0 and hasn't shipped in a stable release yet. It isn't included in the 5.1 release series. Use a development build with FTS enabled; don't assume a container tagged `latest` includes it. These pages describe the current implementation on `master`. Check your binary with `fluent-bit --version` and run the [quickstart](fluent-typed-scripting/quickstart.md) to confirm that it contains `fts`. + +For a native source build, enable `FLB_PROCESSOR_FTS`, `FLB_LUAJIT`, and `FLB_YYJSON`. FTS defaults to interpreted execution even when built with LuaJIT. See [Build and install](../../installation/downloads/source/build-and-install.md) for general build instructions. + +The experimental browser WASM profile also includes FTS when `FLB_WASM_LUA=ON`, the profile default. It uses portable Lua 5.4 and rejects `jit: true`. Setting `FLB_WASM_LUA=OFF` disables both FTS and the Lua filter in that profile. This experimental profile is separate from native support and from the [Wasm filter](../filters/wasm.md). + +## Choose FTS + +Choose FTS when a transformation needs typed values, per-item rollback, or programmable access to logs, metrics, and traces. Prefer a dedicated processor when its existing operations meet your needs. For an existing Lua filter policy, review [FTS and the Lua filter](fluent-typed-scripting/lua-filter.md) before migrating: their callback and value models differ. + +Each signal uses its own processor entry and script under `processors.logs`, `processors.metrics`, or `processors.traces`. A callback receives one log record, one metric family, or one span, respectively. Each processor instance has its own Lua state. FTS doesn't join signals or share globals between those instances. + +Processors require [YAML configuration](../../administration/configuring-fluent-bit/yaml.md). Attach FTS to an input or output and place it in the order in which transformations should run. See [Processors](../processors.md) for pipeline placement. + +## Get started and reference + +1. Run your first transformation with the [Quickstart](fluent-typed-scripting/quickstart.md). +2. Configure the policy and execution limits with [Configuration](fluent-typed-scripting/configuration.md). +3. Learn the value model in [Typed values](fluent-typed-scripting/typed-values.md). +4. Use the signal references for [Logs](fluent-typed-scripting/logs.md), [Metrics](fluent-typed-scripting/metrics.md), and [Traces](fluent-typed-scripting/traces.md). +5. Look up [Common helpers](fluent-typed-scripting/common-helpers.md) and [Errors and limits](fluent-typed-scripting/errors-and-limits.md). +6. Adapt the complete [Recipes](fluent-typed-scripting/recipes.md). +7. Configure built-in blocks without Lua by using [Native recipe blocks](fluent-typed-scripting/native-recipes.md). diff --git a/pipeline/processors/fluent-typed-scripting/common-helpers.md b/pipeline/processors/fluent-typed-scripting/common-helpers.md new file mode 100644 index 000000000..4c173ce01 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/common-helpers.md @@ -0,0 +1,121 @@ +# Common helpers + +Load the Fluent Typed Scripting (FTS) module with `local fb = require("fluentbit")`. The following helpers operate on typed values. For signal-specific helpers, see [Logs](logs.md#log-helpers) and [Metrics](metrics.md#metric-helpers). See [Typed values](typed-values.md#constructors-and-conversions) for constructors. + +## Redact structured fields + +`fb.redact(value, keys, replacement)` recursively replaces values of matching keys in nested maps and arrays. `keys` is a Lua table of key names. Matching is case-insensitive. The default replacement is `[REDACTED]`. + +```lua +fb.redact(log.body, {"authorization", "password", "access_token"}) +``` + +For a custom replacement: + +```lua +fb.redact(log.body, {"password"}, "removed") +``` + +This helper doesn't search inside strings. It won't remove a password embedded in `message: "password=example-secret"`. Write and test a separate string policy for known text formats, or remove the text field when appropriate. Keep `error_mode: drop` for redaction policies when forwarding an unmodified failed item would expose sensitive data. + +## Keep selected keys + +`fb.keep_keys(map, keys)` removes every key not listed in the Lua table `keys`. Matching is case-sensitive. This helper acts on the supplied map only; it isn't recursive. + +```lua +fb.keep_keys(log.body, {"message", "level", "service.name"}) +``` + +## Parse JSON + +`fb.parse_json(text)` returns a typed value, preserving integer widths. Input is limited to 1 MiB and nesting to 64 levels. Duplicate keys and invalid JSON raise errors. A valid JSON document can produce a scalar or array, so check the result before assigning a log body. + +```lua +local message = log.body:get("message") +if message and message:type() == "string" then + local ok, parsed = pcall(fb.parse_json, message:value()) + if ok and parsed:type() == "map" then + log.body = parsed + else + return fb.DROP + end +end +``` + +This fragment belongs in `process(log)` with `fb` loaded. Catching a parse error with `pcall` lets the policy choose a result; it doesn't roll back earlier edits automatically. An uncaught error invokes the processor's rollback and `error_mode` behavior. + +`fb.try_parse_json` and `fb.decode_json` require a build that includes Fluent Bit commit `94ef261ba11bf51b2c5ec57dd486a87db0d2e03c`, such as development revision `e090fbd054c82ae67f12ebe926600827b6cf5f48`. Revision `7ea6759f2` doesn't define these helpers, so the following examples fail there. + +`fb.try_parse_json(text)` uses the same parser and limits, but returns `nil, error` instead of raising an error when the text is invalid JSON or isn't a string, including a missing field. JSON `null` and `false` return typed values, so check for `nil` rather than for a false value: + +```lua +local message = log.body:get("message") +if message and message:type() == "string" then + local parsed = fb.try_parse_json(message:value()) + if parsed and parsed:type() == "map" then + log.body = parsed + end +end +``` + +The example reads `message` with `get` and checks its type first, because `get_string` raises an error when the field isn't a string. A number, boolean, or `null` in `message` skips the parsing step instead of failing the callback, and the rest of the policy still runs. + +## Decode a JSON field in place + +`fb.decode_json(container, key)` replaces one string field with its decoded JSON value. It works with body maps, nested maps, OpenTelemetry attributes, and array elements, and accepts any valid JSON value, including arrays, `null`, and booleans. It returns `true` on success. For a missing or non-string field, or text that isn't valid JSON, it returns `false, error` and leaves the field unchanged. + +```lua +fb.decode_json(log.body, "message") +if fb.is_otel(log) then + fb.decode_json(log.attributes, "payload") +end +``` + +For example, `{"message":"{\"level\":\"info\"}","host":"web"}` becomes `{"message":{"level":"info"},"host":"web"}`. Strings inside the decoded value aren't decoded again; each one needs its own call. To replace the whole log body with a decoded object instead, use [`fb.unpack_json`](logs.md#log-helpers). + +For OpenTelemetry log bodies and attributes, the JSON field helpers reject unsigned integers greater than `9223372036854775807`, including nested values, because OpenTelemetry integers are signed 64-bit values. They return `false, error` without changing the field. `fb.parse_json` and `fb.try_parse_json` keep full unsigned 64-bit support. + +The `true` or `false` status of a helper isn't a callback result. To remove a record when decoding fails, return `fb.DROP` explicitly. If a JSON helper exhausts the instruction limit, the callback fails instead of returning `false`, and `error_mode` applies. + +## Parse time + +`fb.parse_time(text)` parses RFC 3339 text and returns a Lua table `{sec, nsec}` for `log.timestamp`. It accepts `Z` or numeric timezone offsets and up to nine fractional digits. Invalid dates, leap seconds, excess fractional precision, and values outside `0..4294967293` epoch seconds raise errors. The supported calendar-year input range is 1969 through 2107; the resulting epoch seconds must still fit the event range and the platform's timestamp range. + +```lua +log.timestamp = fb.parse_time("2026-01-02T03:04:05.123456789Z") +``` + +## Hash and sign values + +`fb.hash64(bytes)` returns a 16-character lowercase XXH3-64 fingerprint of a byte string up to 1 MiB. `fb.hash64_mod(bytes, modulus)` returns the remainder of that 64-bit hash for an integer modulus from `1` to `1000000000`. Use it for stable sampling. `fb.hash64_algorithm` names the fixed algorithm, `xxh3-64-v1-seed0`. Hashes provide no secrecy. + +```lua +local user_id = log.body:get("user_id") +if user_id and user_id:type() == "string" and fb.hash64_mod(user_id:value(), 100) >= 10 then + return fb.DROP +end +``` + +`fb.hmac_sha256(key_ref, bytes)` returns a 64-character lowercase hexadecimal HMAC-SHA256 of a message up to 1 MiB. `key_ref` names an entry in the processor's `secret_refs` setting, which maps references to environment variable names: + +```yaml +processors: + logs: + - name: fts + script: pseudonymize.lua + secret_refs: + email-v1: FTS_PSEUDONYM_KEY_V1 +``` + +```lua +local email = log.body:get("email") +if email and email:type() == "string" then + log.body:set("email", "email-v1:" .. fb.hmac_sha256("email-v1", email:value())) +end +``` + +Fluent Bit reads each environment variable once during initialization and uses its raw bytes as the key; hexadecimal or Base64 text isn't decoded. Lua code never sees the key. A missing variable or an empty key fails initialization. `secret_refs` supports up to 16 references, and keys up to 4096 bytes. Each reference name must contain 1 to 128 ASCII letters, digits, underscores (`_`), periods (`.`), or hyphens (`-`). Each environment variable name must contain 1 to 128 ASCII letters, digits, or underscores, and can't start with a digit. Set the key as a process environment variable, for example through your orchestrator. Don't put the key in the YAML `env` section, which stores it as plain text in the configuration file. Include the key version in emitted values so you can rotate keys with a new reference. + +## Callback constants + +`fb.KEEP` commits edits and `fb.DROP` removes the item. Returning nothing also commits edits. These constants aren't the numeric return codes of the Lua filter. See [Errors and limits](errors-and-limits.md#callback-results). diff --git a/pipeline/processors/fluent-typed-scripting/configuration.md b/pipeline/processors/fluent-typed-scripting/configuration.md new file mode 100644 index 000000000..649d6aea0 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/configuration.md @@ -0,0 +1,83 @@ +# Configuration + +Fluent Typed Scripting (FTS) uses the configuration name `fts`. Attach a processor entry under the signal it processes. See the [quickstart](quickstart.md) for a complete pipeline. + +## Configuration parameters + +| Key | Description | Default | +| --- | ----------- | ------- | +| `call` | Name of the Lua callback. The script must define this function before initialization finishes. | `process` | +| `code` | Inline Lua policy. Set exactly one of `script`, `code`, or `recipe`. | _none_ | +| `context` | Log context that `recipe` steps accept: `native`, `otel`, or `any`. Any other value fails validation. | `any` | +| `error_mode` | Action after a policy failure: `drop` removes the item; `keep` forwards the original item without edits. Values are case-sensitive. | `drop` | +| `instruction_limit` | Lua instruction budget for each callback and script initialization. With `jit: false`, the range is `1000` through `2147483647`. With `jit: true`, must be `0`. | `1000000` | +| `jit` | Enable LuaJIT compilation. Requires a LuaJIT runtime and `instruction_limit: 0`. | `false` | +| `memory_limit` | Maximum policy memory in bytes, including the Lua heap and retained typed values. Minimum: `262144` bytes. | `32M` | +| `output_limit` | Maximum serialized size of a processed batch. Minimum: `65536` bytes. Records held by stateful recipe steps count toward `state_limit` instead. See the signal-specific failure behavior in [Errors and limits](errors-and-limits.md#output-limit). | `64M` | +| `packages_dir` | Directory that contains trusted local Lua packs. Required when `packs` is set. Relative paths resolve against the Fluent Bit process working directory, not the configuration file directory. | _none_ | +| `packs` | List of local pack manifest paths, relative to `packages_dir`. Requires `recipe` and `packages_dir`. Packs are loaded only from local files. | _none_ | +| `recipe` | Ordered list of built-in [recipe blocks](native-recipes.md). Set exactly one of `script`, `code`, or `recipe`. Requires `recipe_version: 1` and `jit: false`. | _none_ | +| `recipe_version` | Recipe grammar version. Only `1` is supported. | `1` | +| `script` | Path to a Lua policy file. Set exactly one of `script`, `code`, or `recipe`. | _none_ | +| `secret_refs` | Map of HMAC key references to environment variable names, resolved once during initialization. Used by `fb.hmac_sha256`. See [Hash and sign values](common-helpers.md#hash-and-sign-values). | _none_ | +| `state_commit` | Commit mode for stateful recipe steps. The only accepted value is `local`, and it requires `recipe`. The stateful blocks `logs.dedupe`, `logs.throttle`, and `window.hold_until` require `state_commit: local`, a processor attached to an input, and the stateful block as the last recipe step; otherwise initialization fails. State is volatile and local to the processor instance. | _none_ | +| `state_limit` | Maximum native state memory for stateful recipe steps, including provisional entries and bookkeeping. | `1M` | + +Size suffixes use decimal multipliers: `64K` is 64,000 bytes, less than the minimum `output_limit`. Use `65536` or a larger value. `32M` is 32,000,000 bytes and `64M` is 64,000,000 bytes. + +FTS has no `workers` parameter. Its execution follows the input or output that it's attached to. + +## Check recipe support + +`recipe` and the related `context`, `packages_dir`, `packs`, `recipe_version`, `secret_refs`, `state_commit`, and `state_limit` parameters require a build that includes Fluent Bit commit `6032b237d2275cb2883ed8942d7b269a5a8169a3`, such as development revision `e090fbd054c82ae67f12ebe926600827b6cf5f48`. Revision `7ea6759f2` doesn't support them. A build that runs `script` or `code` policies, or that has FTS enabled, doesn't necessarily support `recipe`. + +To check a binary, start Fluent Bit with a minimal recipe, such as the example in [Native recipe blocks](native-recipes.md#recipe-structure). A build without recipe support fails at startup and reports an unknown configuration property `recipe`. Start the pipeline for this check instead of using `--dry-run`, because `--dry-run` on older development builds doesn't validate processor parameters. + +## Inline policies + +Use a YAML block scalar for an inline policy: + +```yaml +pipeline: + inputs: + - name: dummy + samples: 1 + processors: + logs: + - name: fts + code: | + function process(log) + log.body:set("environment", "production") + end + outputs: + - name: stdout + match: '*' + format: json_lines +``` + +## Separate signals + +For an input that receives multiple signals, use separate entries and scripts. The following fragment belongs inside an input or output configuration: + +```yaml +processors: + logs: + - name: fts + script: logs.lua + metrics: + - name: fts + script: metrics.lua + traces: + - name: fts + script: traces.lua +``` + +Each script can define `process`, because each instance has its own Lua state. A logs script receives a log, a metrics script receives a family, and a traces script receives a span. There is no shared callback that receives all three signals. + +## Invalid combinations + +Startup fails when more than one of `script`, `code`, and `recipe` is set, none is set, the callback is missing, a limit is invalid, or the script fails during initialization. `error_mode` handles processing failures; it doesn't make an invalid configuration usable. + +Recipe and pack settings have their own requirements. Startup fails when `packs` is set without both `recipe` and `packages_dir`, when `recipe` is combined with a `recipe_version` other than `1` or with `jit: true`, when `context` isn't `native`, `otel`, or `any`, or when `state_commit` is set without `recipe` or to a value other than `local`. + +To enable LuaJIT compilation on a native LuaJIT build, set both `jit: true` and `instruction_limit: 0`. Setting only one fails validation. Portable Lua, including the experimental browser WASM runtime, rejects `jit: true`. See [Runtime restrictions](errors-and-limits.md#runtime-restrictions) before choosing these settings. diff --git a/pipeline/processors/fluent-typed-scripting/errors-and-limits.md b/pipeline/processors/fluent-typed-scripting/errors-and-limits.md new file mode 100644 index 000000000..137ec5b77 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/errors-and-limits.md @@ -0,0 +1,62 @@ +# Errors and limits + +Fluent Typed Scripting (FTS) policies are trusted configuration. The restricted runtime reduces available operations, but its sandbox isn't a security boundary. Review policies before deploying them. + +## Callback results + +| Result | Effect | +| --- | --- | +| No return, `nil`, or `fb.KEEP` | Commit the item's edits. | +| `fb.DROP` | Remove the item. Metric emissions remain. | +| Uncaught Lua error, invalid return, or invalid telemetry edit | Roll back edits and apply `error_mode`. | + +Transactions cover one log record, metric family, or span, including its resource and scope edits. For logs, body, metadata, context, and timestamp edits commit or roll back together. Metric emissions are discarded on an error. A transaction doesn't include other callbacks or restore Lua globals modified by the policy. + +With `error_mode: drop`, the failed item is removed. With `error_mode: keep`, the original item is forwarded without the policy's edits. For a redaction policy, `keep` can forward secrets. Leave the default `drop` when that outcome is unacceptable. + +A caught error, such as a failed `pcall(fb.parse_json, text)`, lets the callback continue. Earlier edits remain unless the callback ultimately fails. Instruction budget exhaustion can't be caught to continue an endless loop. + +## Memory and instruction limits + +The [configuration reference](configuration.md#configuration-parameters) lists defaults and valid ranges. + +`memory_limit` accounts for the Lua heap and typed values held by the policy. It isn't a limit on the whole Fluent Bit process, input chunks, or output buffers. Retained globals, copies, and metric/span snapshots consume this budget. + +`instruction_limit` counts Lua instructions during script initialization and each callback, using a hook in steps of 1,000 instructions. It isn't an elapsed-time deadline. Lua string patterns execute in C and aren't interrupted by the instruction hook. Avoid patterns whose runtime can grow without a defined bound on untrusted input. + +`jit: true` requires `instruction_limit: 0`, removing the instruction budget. Use it only when you control policy execution time. LuaJIT compilation isn't available in the portable Lua runtime. No throughput or latency improvement is guaranteed by these settings. + +## Output limit + +`output_limit` bounds serialized output from a processed batch. Failure behavior depends on the signal: + +| Signal | Behavior when transformed output doesn't fit | +| --- | --- | +| Logs | The whole batch fails. `drop` discards the batch; `keep` forwards the original batch. | +| Metrics and traces | The overflowing item follows `error_mode`. The whole batch fails if retained original items don't fit either. | + +Internal failures that FTS detects after a batch is decoded, such as failing to build the policy output, apply `error_mode` to the whole batch. A record that can't be decoded is handled earlier, by the processor layer: the records decoded before it are kept and processed, and the rest of the batch is discarded. In that case, `error_mode: keep` can't restore the original batch. + +A processing batch isn't the same as a callback invocation: one batch can invoke the callback many times. Small chunks are processed as one batch. Larger log buffers, including chunks passed to output processors, are processed in batches of about 64 KiB or 1,024 records. A group received within one input append isn't split across batches, so a group can exceed those targets. A group whose start and end markers arrive in separate appends can remain open across batches. Records received inside a group opened in an earlier batch don't receive a context view. See [Logs](logs.md#opentelemetry-context) for details. + +## Value and format limits + +- Typed values support at most 64 levels of nesting. +- `parse_json` input is limited to 1 MiB and rejects duplicate keys. +- OpenTelemetry strings and keys must be valid UTF-8 without null (`\0`) characters. Use typed bytes for binary data. +- OpenTelemetry integer fields have their own signed or unsigned ranges. A successful typed constructor doesn't guarantee that an arbitrary signal field accepts that value. +- Invalid metric layouts, duplicate series/windows introduced by edits, invalid IDs, and invalid timestamps can fail a policy. See [Metrics](metrics.md) and [Traces](traces.md). + +## Runtime restrictions + +Only `require("fluentbit")` is available. Filesystem, process, foreign function interface, debug, coroutine, and general module-loading libraries aren't available. The runtime also removes `print`, `xpcall`, `getfenv`, `setfenv`, `module`, and dynamic code-loading functions such as `load`, `loadfile`, `loadstring`, and `dofile`. + +Policies can use the restricted base, `math`, `string`, and `table` functions. `collectgarbage` accepts only `collect`, `count`, and `step`. Each processor instance has its own Lua state; globals persist across callbacks within that instance. + +The experimental browser WASM profile uses the portable Lua 5.4 interpreter with the same typed-value model and policy limits. It rejects `jit: true`. See [Availability](../fluent-typed-scripting.md#availability) for the build distinction. + +## Diagnostics + +Each batch logs at most five policy errors plus a summary count. A low number of error messages doesn't imply that only five records failed. The processor error metric counts batches with handled policy failures, and drop metrics count removed records. + +If output disappears, check policy errors, expected field types, `error_mode`, and all three budgets. A startup failure instead points to configuration, script loading, initialization limits, or an unavailable runtime feature. diff --git a/pipeline/processors/fluent-typed-scripting/logs.md b/pipeline/processors/fluent-typed-scripting/logs.md new file mode 100644 index 000000000..5652e2295 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/logs.md @@ -0,0 +1,99 @@ +# Logs + +The callback `process(log)` runs once per log record. An implicit return or `fb.KEEP` commits edits; `fb.DROP` removes the record. See [Errors and limits](errors-and-limits.md) for rollback behavior. + +## Log properties + +| Property | Description | +| --- | --- | +| `log.body` | Typed body map. Can be replaced with another typed map. | +| `log.metadata` | Typed record metadata map. Mutate its entries; don't replace the root. | +| `log.timestamp` | Lua table with `sec` and `nsec`. Assign a new table to change event time. | +| `log.tag` | Read-only routing tag. | +| `log.attributes` | Typed OpenTelemetry log attribute map, when OpenTelemetry context is present. | +| `log.resource.attributes`, `log.resource.schema_url` | OpenTelemetry resource attributes and schema URL. | +| `log.scope.attributes`, `log.scope.name`, `log.scope.version`, `log.scope.schema_url` | OpenTelemetry instrumentation scope. | +| `log.context.attributes`, `log.context.metadata` | Typed group maps for grouped logs, including non-OpenTelemetry groups. | + +Use `set` and `remove` on attributes. Don't replace context roots or OpenTelemetry aliases, change the transport schema, or edit transport resource and scope identifiers. Use semantic properties to change resource and scope data. + +## Event time and metadata + +Change event time through `log.timestamp`, not the raw OpenTelemetry timestamp metadata. Seconds must be in the log event range, `0` through `4294967293` on platforms with 64-bit `time_t` (`2147483647` with 32-bit `time_t`); nanoseconds must be `0` through `999999999`. For example: + +```lua +local fb = require("fluentbit") + +function process(log) + local time = log.body:get("time") + if time and time:type() == "string" then + local ok = pcall(function() + log.timestamp = fb.parse_time(time:value()) + end) + if not ok then + log.metadata:set("invalid_time", true) + end + end + log.metadata:set("policy", "normalize-time") +end +``` + +A record whose `time` field is missing, isn't a string, or can't be parsed keeps its original event time. The policy flags unparsable values in metadata instead of failing the callback, because an uncaught error from `fb.parse_time` would apply `error_mode` and, with the default `drop`, remove the record. + +For OpenTelemetry logs, changing `log.timestamp` also updates the event timestamp metadata. `observed_timestamp` is a separate field and isn't changed by this assignment. + +## OpenTelemetry context + +Use `fb.is_otel(log)` before reading or editing OpenTelemetry properties. Logs from the [OpenTelemetry input](../../inputs/opentelemetry.md) can carry this context. For ordinary logs, place the [OpenTelemetry envelope](../opentelemetry-envelope.md) processor before Fluent Typed Scripting (FTS) under `processors.logs`. + +```lua +local fb = require("fluentbit") + +function process(log) + if fb.is_otel(log) then + log.resource.attributes:set("deployment.environment.name", "production") + log.scope.name = "application-policy" + log.attributes:set("sanitized", true) + fb.set_severity(log, "info") + end +end +``` + +Resource, scope, and group context edits apply only to the current record. Record order is preserved. A group that spans input appends can remain open across batches. Records received inside a group opened in an earlier batch have no context view and can't set one; guard access instead of assuming every grouped record has editable context. + +## Log helpers + +| Helper | Description | +| --- | --- | +| `fb.is_otel(log)` | Whether the log carries OpenTelemetry context. | +| `fb.attributes(log)` | The OpenTelemetry log attribute map; requires OpenTelemetry context. Equivalent to `log.attributes`. | +| `fb.set_severity(log, level)` | Set severity text and number. Accepts case-insensitive `trace` (1), `debug` (5), `info` (9), `warn` (13), `error` (17), and `fatal` (21). Also accepts `information`, `informational`, `warning`, and `critical`. Preserves the supplied text. | +| `fb.set_trace_context(log, trace_hex, span_hex)` | Set correlation IDs. Trace ID must have 32 hexadecimal characters and span ID 16; neither can be all zero. | +| `fb.log_field(log, name)` | Read an OpenTelemetry field as a typed value, or `nil` if absent. Requires OpenTelemetry context. | +| `fb.unpack_json(log, field)` | Decode a JSON object from a body string field and replace `log.body` with it. `field` defaults to `message`. Returns `true` on success, or `false, error` when the field is missing or not a string, the text isn't valid JSON, or the JSON isn't an object. On failure the body is unchanged. | + +Use `log_field` for fields such as `severity_number`, `severity_text`, `trace_id`, `span_id`, `trace_flags`, `timestamp`, `observed_timestamp`, and `event_name`. IDs read this way are typed bytes; `value()` returns binary data, not hexadecimal text. + +```lua +fb.set_trace_context(log, + "0102030405060708090a0b0c0d0e0f10", "0102030405060708") +local severity = fb.log_field(log, "severity_number") +if severity and severity:number() >= 17 then + log.attributes:set("needs_review", true) +end +``` + +`fb.unpack_json` requires a build that includes Fluent Bit commit `6c52cc70ce1969f7a9e9748adfa52d52fd4b0e88`, such as development revision `e090fbd054c82ae67f12ebe926600827b6cf5f48`. Revision `7ea6759f2` doesn't define the helper, so the following example fails there. To parse logs whose `message` field contains JSON, while keeping plain-text messages as they're: + +```lua +local fb = require("fluentbit") + +function process(log) + fb.unpack_json(log, "message") + fb.redact(log.body, {"authorization", "password"}) +end +``` + +Records whose `message` doesn't decode keep their original body, and `fb.redact` still runs on it. Because `fb.redact` matches keys, not text, a message that fails to decode keeps any secret it contains. To forward only decoded records, use `if not fb.unpack_json(log, "message") then return fb.DROP end`. + +This fragment belongs in an OpenTelemetry log callback. See [Common helpers](common-helpers.md) for JSON parsing, redaction, and time parsing, and [Recipes](recipes.md#redact-logs) for a complete pipeline. diff --git a/pipeline/processors/fluent-typed-scripting/lua-filter.md b/pipeline/processors/fluent-typed-scripting/lua-filter.md new file mode 100644 index 000000000..9dd428755 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/lua-filter.md @@ -0,0 +1,30 @@ +# Fluent Typed Scripting (FTS) and the Lua filter + +Fluent Typed Scripting (FTS) and the [Lua filter](../../filters/lua.md) both run Lua policies, but expose different APIs. FTS means typed access to telemetry; it doesn't add static typing to Lua. + +| Requirement | FTS | Lua filter | +| --- | --- | --- | +| Signals | Logs, metrics, and traces, with a separate processor entry and script for each signal. | Logs. | +| Placement | Processor attached to an input or output, configured in YAML. | Filter with tag matching, or a [filter used as a processor](../filters.md). Classic and YAML filter configuration are supported. | +| Value model | Typed maps, arrays, null, bytes, and exact integer values. | Lua tables and values, with options such as `type_int_key`, `type_array_key`, and `enable_flb_null`. | +| Timestamp precision | Log `{sec, nsec}` table; exact typed nanoseconds in metric and span fields. | `time_as_table` option for log `{sec, nsec}` precision. | +| OpenTelemetry log context | Semantic resource, scope, and log attribute properties. | Extended callback supports group metadata and record metadata. | +| Callback result | Mutate an item; return nothing, `fb.KEEP`, or `fb.DROP`. | Return status code, timestamp, and record; extended callback also returns metadata. | +| Splitting a log into multiple records | No log emission API. | Supported by returning an array of records. | +| Metric transformations | Family/datapoint edits, aggregation, copies, and derived metrics. | Not supported. | +| Span transformations | Span, event, link, resource, and scope edits. | Not supported. | +| Failure handling | Per-item rollback and `error_mode`, plus batch failure rules. | `protected_mode` catches script exceptions. | +| Runtime access | Restricted libraries; instruction, policy memory, and serialized output budgets. | Existing Lua runtime and filter configuration model; doesn't expose the FTS budgets. | + +Choose FTS when preserving telemetry types, working with metric families or spans, or using its rollback and limits is central to your policy. Keep the Lua filter when you need log splitting, classic configuration, or compatibility with an established filter script. Both can work with OpenTelemetry log metadata; this capability alone isn't a reason to migrate. + +## Migrate a policy + +1. Replace the filter callback signature with `process(log)`, `process(metric)`, or `process(span)`. +2. Replace Lua-table field access with typed methods, for example `log.body:get_string("message")` and `log.body:set("message", text)`. +3. Replace numeric filter status codes with the FTS callback results. An implicit return commits edits. +4. Use integer, bytes, null, map, and array constructors where needed. +5. Review module dependencies against [Runtime restrictions](errors-and-limits.md#runtime-restrictions). +6. Test the success and failure paths, especially `error_mode: keep` for policies that remove sensitive data. + +There is no automatic script compatibility. Start with the [quickstart](quickstart.md) and use the [signal references](../fluent-typed-scripting.md#get-started-and-reference) for the destination API. diff --git a/pipeline/processors/fluent-typed-scripting/metrics.md b/pipeline/processors/fluent-typed-scripting/metrics.md new file mode 100644 index 000000000..b71652fd6 --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/metrics.md @@ -0,0 +1,88 @@ +# Metrics + +The callback `process(metric)` runs once per metric family, including all its data points. An implicit return or `fb.KEEP` commits edits; `fb.DROP` removes the original family. Resource and scope edits apply only to that family. + +## Metric properties + +| Property | Description | +| --- | --- | +| `metric.name`, `metric.description`, `metric.unit` | Family name, description, and unit strings. Changing the unit alone doesn't scale values. | +| `metric.type` | `gauge`, `sum`, `histogram`, `exponential_histogram`, or `summary`. Only gauge and sum can be converted into each other by assignment. | +| `metric.datapoints` | Typed array of datapoint maps. Can be replaced with another typed array. | +| `metric.aggregation_temporality` | `unspecified`, `delta`, or `cumulative`. Not applicable to gauge or summary. | +| `metric.is_monotonic` | Boolean, applicable only to sums. | +| `metric.resource.attributes`, `metric.resource.schema_url` | Resource context. | +| `metric.scope.attributes`, `metric.scope.name`, `metric.scope.version`, `metric.scope.schema_url` | Instrumentation scope context. | +| `metric.tag` | Read-only routing tag. | + +Converting a gauge to a sum initializes cumulative temporality and `is_monotonic: false`. Choose temporality and monotonicity deliberately; a type change doesn't calculate deltas or reconstruct counter history. Negative edited values aren't valid for a monotonic sum. + +## Data points + +Datapoint maps use OpenTelemetry field names in snake case. Use typed methods to read and edit them. + +| Fields | Meaning | +| --- | --- | +| `attributes` | Typed map identifying the series. | +| `time_unix_nano`, `start_time_unix_nano` | Exact nonnegative integer timestamps. An edited end time must not precede the start time. | +| `as_int`, `as_double` | Scalar gauge or sum value; exactly one must be present. | +| `count`, `sum`, `min`, `max` | Distribution fields where supported. Counts use exact integers. | +| `bucket_counts`, `explicit_bounds` | Explicit histogram arrays. Counts are per bucket, not cumulative; there must be one more count than bound, and counts must total `count`. | +| `scale`, `zero_count`, `zero_threshold`, `positive`, `negative` | Exponential histogram fields. Positive and negative bucket maps contain `offset` and `bucket_counts`. | +| `quantile_values` | Summary array of maps with `quantile` and `value`. | +| `exemplars`, `flags` | Exemplars and datapoint flags where supported by the metric type. | + +Explicit histogram data points in a family must have matching layouts. Summary data points must have matching quantile layouts. Edited histogram bounds must increase; edited summary quantiles must increase within `0..1`. Exponential histograms allow at most 65,535 buckets on each side and edited scale values from −10 through 20. Invalid edits fail the policy. + +Scalar metric values use OpenTelemetry int64 and double types. Native unsigned values up to 9223372036854775807 are exposed as int64. A family containing larger native unsigned values isn't passed to the callback: `error_mode` determines whether to retain or drop it. + +## Units and scaling + +Use `fb.scale_metric` to change both values and units. For example, convert milliseconds to seconds: + +```lua +local fb = require("fluentbit") + +function process(metric) + if metric.name == "http.server.duration" then + metric.name = "http.server.request.duration" + fb.scale_metric(metric, 0.001, "s") + end +end +``` + +Scaling uses Lua numbers and can fail when an integer is outside the safe conversion range. It supports gauge, sum, explicit histogram, and summary families, including bounds, quantile values, and exemplars where applicable. It rejects exponential histograms because scaling requires bucket remapping. + +## Series collisions and aggregation + +Removing an attribute can make previously distinct series identical within a time window. Fluent Typed Scripting (FTS) rejects edits that create duplicate series/windows. Don't rely on implicit overwrite or combination. + +Use `fb.aggregate_metric` to keep selected attributes and deliberately combine matching data points: + +```lua +local fb = require("fluentbit") + +function process(metric) + if metric.name == "http.requests" and metric.type == "sum" then + fb.aggregate_metric(metric, {"service.name", "http.request.method"}, "sum") + end +end +``` + +Aggregation happens within the current family and callback. It doesn't combine families, batches, resources, scopes, or different start/end timestamp pairs. Within a group, flags must match. Histogram layouts must match. Integer addition checks for overflow. This is dimension aggregation, not a time-window store. + +## Metric helpers + +| Helper | Description | +| --- | --- | +| `fb.metric_value(point)` | Typed `as_int` or `as_double` value, or `nil` if neither exists. | +| `fb.set_metric_value(point, value)` | Set the scalar value and remove the other scalar representation. Accepts a finite Lua number or typed numeric value; OpenTelemetry range validation still applies. | +| `fb.scale_metric(metric, factor, unit)` | Multiply supported values by a finite positive factor; optionally set the unit. | +| `fb.aggregate_metric(metric, keys, op, max_exemplars)` | Keep only the keys in the Lua table and combine equal series/windows. Default `op`: `sum`. Supports gauge, sum, and explicit histogram. Gauge also supports `min`, `max`, `mean`, and `count`; `count` changes the unit to `1`. Optional `max_exemplars`, an integer from `0` through `128`, keeps at most that many exemplars per combined point, in input order. When omitted, all exemplars are kept. Kept exemplars retain their attributes, even for dropped keys. `max_exemplars` requires a build that includes Fluent Bit commit `6032b237d2275cb2883ed8942d7b269a5a8169a3`; older builds ignore the argument and keep all exemplars. | +| `fb.merge_histogram_buckets(metric, bounds)` | Merge explicit histogram buckets. Bounds must be an increasing subset of existing bounds; accepts a Lua table or typed array. | +| `fb.copy_metric(metric, name, description, unit)` | Emit an independent copy. Optional arguments replace identity fields. Returns the emitted family for further edits. | +| `fb.extract_metric(metric, field, name, monotonic)` | Emit a derived family from a histogram, exponential histogram, or summary. `field` is `count`, `sum`, or `avg`; optional `name` defaults to the original name plus `.` and the field. Returns the family, or `nil` if no data points qualify. | + +`extract_metric` emits a sum for `count` or `sum`, and a gauge for `avg`. Count uses unit `1` and defaults to monotonic; extracted sum defaults to non-monotonic. The optional `monotonic` argument applies to count and sum. Average uses sum/count and omits zero-count points. It doesn't average summary quantiles. Numeric conversions can fail for large integers. + +Emitted families follow the original through subsequent pipeline stages and aren't passed to this callback again. Returning `fb.DROP` removes the original but retains emissions. An error rolls back the original and discards emissions. See [Errors and limits](errors-and-limits.md) and the complete [metric recipe](recipes.md#normalize-and-aggregate-metrics). diff --git a/pipeline/processors/fluent-typed-scripting/native-recipes.md b/pipeline/processors/fluent-typed-scripting/native-recipes.md new file mode 100644 index 000000000..1ac123ecb --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/native-recipes.md @@ -0,0 +1,201 @@ +# Native recipe blocks + +A _native recipe_ configures Fluent Typed Scripting (FTS) with an ordered list of built-in blocks instead of a Lua policy. Set `recipe` in place of `script` or `code`. Each step runs a reviewed block with validated parameters, and every step for an item runs in the same transaction: a failure in any step rolls back the item's earlier recipe edits and applies `error_mode`. + +Native recipe blocks are different from the [Recipes](recipes.md) page, which contains complete Lua policy examples. + +## Recipe structure + +```yaml +pipeline: + inputs: + - name: dummy + dummy: '{"message": "login", "password": "example-secret"}' + processors: + logs: + - name: fts + context: native + error_mode: drop + recipe: + - id: privacy + use: shared.redact_keys + with: + targets: [{root: body, path: []}] + keys: [password, authorization] + outputs: + - name: stdout + match: '*' +``` + +Each step accepts the following keys: + +| Key | Description | +| --- | ----------- | +| `id` | Unique step name. Starts with a letter and contains letters, digits, `_`, or `-`, up to 64 bytes. | +| `use` | Block ID, such as `shared.redact_keys`, or `/` for a block from a [local pack](#local-packs). | +| `with` | Block parameters. | +| `when` | Optional condition for log and trace steps, using the same grammar as `logs.drop_matching`. When the condition is false, the step is skipped. Metric steps don't support `when`, because metrics have no selector roots; setting it on a metric step fails initialization. | + +A recipe can contain up to 64 steps. Unknown blocks, unknown parameters, duplicate step IDs, and blocks that don't support the processor's signal fail initialization. Set `recipe_version: 1`, the default. Recipes don't support `jit: true`. + +On a build that supports recipes, run `fluent-bit --dry-run -c fluent-bit.yaml` to validate recipes without starting the pipeline. Older development builds don't validate processor parameters with `--dry-run`, so a recipe configuration can pass the dry run and still fail at startup. See [Check recipe support](configuration.md#check-recipe-support) to confirm that a build supports recipes. [Hot reload](../../../administration/hot-reload.md) also validates recipes before it replaces the running pipeline. + +## Selectors + +Blocks address fields with a selector: a `root` and a `path` array of literal keys. An empty path selects the root itself. + +```yaml +source: {root: body, path: [request, headers]} +``` + +| Signal | Roots | +| ------ | ----- | +| Logs | `body`, `metadata`, `attributes`, `resource.attributes`, `scope.attributes` | +| Traces | `attributes`, `events`, `links`, `resource.attributes`, `scope.attributes` | + +`metadata` selectors require `context: native`, and their first path element can't be `otlp`. Paths can contain up to 32 keys of up to 256 bytes each. + +The processor's `context` setting declares which logs the recipe expects: `native` for Fluent Bit logs, `otel` for logs with OpenTelemetry context, or `any`, the default. + +Constant values can be strings, booleans, plain YAML numbers, or explicit typed values such as `{type: uint64, value: "18446744073709551615"}`, `{type: double, value: 1.5}`, `{type: "null"}`, or `{type: bytes, value: "0001ff"}`. Use a quoted decimal string in a typed value when a large integer must stay exact. + +## Stateless blocks + +| Block | Signals | Parameters | +| ----- | ------- | ---------- | +| `logs.parse_json` | Logs | `source` and `destination` selectors (required); `remove_source` (default `false`); `on_missing`: `error` (default) or `skip`; `collision`: `error` (default), `overwrite`, or `preserve_destination`. The `source` and `destination` paths must be non-empty and must not overlap, and `remove_source: true` can't be combined with `collision: preserve_destination`; otherwise initialization fails. The source must be a string that contains a JSON object; arrays and scalar JSON values are rejected and apply `error_mode`. | +| `logs.normalize_severity` | Logs | `source` selector and `destination` (required). `destination` is a selector, or `otel` to set the OpenTelemetry severity, which requires `context: otel`; `profile`: `common` (default); `overwrite` (default `false`); `on_missing`: `skip` (default) or `error`; `on_unknown`: `preserve` (default) or `error`. A selector `destination` needs a non-empty path; otherwise initialization fails. When `source` and `destination` are the same selector, the field is normalized in place and `overwrite` is always `true`, even if set to `false`. A non-string source, or an `otel` destination on a log without OpenTelemetry context, fails and applies `error_mode`. | +| `logs.parse_timestamp` | Logs | `source` selector of RFC 3339 text (required); `on_missing`: `error` (default) or `skip`. Sets event time and preserves observed time. A non-string source fails and applies `error_mode`. | +| `logs.drop_matching` | Logs | `condition` (required). Drops the log when the condition matches. | +| `logs.validate_fields` | Logs | `rules` (required): a non-empty list of rules, each with a `selector`, a `type` (`integer`, `int64`, `uint64`, `double`, `string`, `bool`, `bytes`, `null`, `array`, or `map`), a boolean `required` (default `true`), and an optional `enum` of 1 to 128 values of that type, for scalar types only. Invalid rules fail initialization. A missing field fails a rule unless the rule sets `required: false`; a field that's present must still match `type` and `enum`. A failed rule applies `error_mode`, which by default drops the log. | +| `fields.edit` | Logs, traces | `operations` (required): a non-empty, ordered list using `set`, `set_default`, `copy`, `rename`, `remove`, and `keep`. Targets can't use the `events` or `links` roots. `keep`, and `remove` on a whole map, need `names`, `prefixes`, or both; other operations need a non-empty target path. `copy` and `rename` sources can't overlap their targets, and `rename` needs a non-empty source path. Invalid operations fail initialization. | +| `shared.redact_keys` | Logs, traces | `targets` selectors and `keys` (required); `replacement` (default `[REDACTED]`). Requires `error_mode: drop`. A target that isn't a map or array fails and applies `error_mode`. | +| `fields.truncate` | Logs, traces | `target` and `marker` selectors and `max_bytes` from `0` to `1048576` (required); `collision`: `error` (default) or `overwrite`; `on_missing`: `skip` (default) or `error`. The `target` and `marker` paths must be non-empty, can't use the `events` or `links` roots, and can't overlap; otherwise initialization fails. When the target is truncated, Fluent Bit sets `marker` to boolean `true`. If `marker` already exists and `collision` is `error`, the item fails and applies `error_mode`, which by default drops it. A non-string target or invalid UTF-8 fails and applies `error_mode`. | +| `traces.name_from_route` | Traces | `method_attribute` (default `http.request.method`); `route_attribute` (default `http.route`); `on_missing`: `preserve` (default) or `error`. Renames server spans only. A non-string route or method fails and applies `error_mode`. | +| `metrics.select` | Metrics | Exactly one of `keep` or `drop`, each with `names`, literal `prefixes`, or both. Selects whole metric families. | +| `metrics.rename` | Metrics | `mapping` of 1 to 128 old names to new names, each 1 to 256 bytes (required). Two names can't map to the same target, and mappings can't form chains or cycles; otherwise initialization fails. | +| `metrics.scale` | Metrics | `names`, `input_unit`, `output_unit`, and a positive `factor` (required); `numeric_policy`: `preserve_integer` (default) or `allow_double`. With `preserve_integer`, a fractional `factor` fails on any integer datapoint or exemplar, and the failure applies `error_mode`; use `allow_double` for fractional scaling of metrics that can contain integers. A selected metric whose unit isn't `input_unit` also fails and applies `error_mode`. | +| `metrics.merge_histogram_buckets` | Metrics | `names` and `bounds` (required). Bounds must be an increasing subset of the existing bounds. | +| `metrics.aggregate_dimensions` | Metrics | `names` and `operation` (required); `keep_attributes` (default empty); `max_exemplars` from `0` to `128` (default `0`). Gauges accept `sum`, `min`, `max`, `mean`, or `count`; sums and explicit histograms accept `sum` only. Each `keep_attributes` entry must be 1 to 256 bytes; otherwise initialization fails. A selected family that isn't a gauge, sum, or explicit histogram, or a non-gauge with an operation other than `sum`, fails and applies `error_mode`. | + +`shared.redact_keys` matches key names in maps and arrays. It doesn't inspect text, including JSON embedded in a string, so parse or remove raw text first. For `events` and `links` targets, it redacts keys only within each entry's `attributes` map; other fields, such as an event's name, IDs, and timestamps, are unchanged. These targets require an empty `path`. `fields.edit` and `shared.redact_keys` update the OpenTelemetry dropped attribute counts when they remove attributes. + +Merge histogram buckets before scaling the same metric; the loader rejects the opposite order. `metrics.rename` checks for name collisions only within the current resource and scope batch. + +### Conditions + +`logs.drop_matching` and a step's `when` key use the same condition grammar. A condition is either a leaf with `selector`, `op`, and `value` (`values` for `in` and `not_in`), or an `all` or `any` list of conditions: + +```yaml +- id: drop_health_checks + use: logs.drop_matching + with: + condition: + all: + - {selector: {root: body, path: [path]}, op: eq, value: /health} + - {selector: {root: body, path: [status]}, op: eq, value: 200} +``` + +Operators are `eq`, `neq`, `in`, `not_in`, `gt`, `gte`, `lt`, and `lte`. A missing field or a value of the wrong type never matches, including for negative operators such as `neq`. Integer comparisons are exact across signed and unsigned 64-bit values. Conditions evaluate the item as already edited by earlier steps. + +The following rules apply when the recipe loads; a condition that breaks them fails initialization. Condition selectors can't use the `events` or `links` roots. `all` and `any` lists, and `values` for `in` and `not_in`, must be non-empty, and all `values` must have the same type. `gt`, `gte`, `lt`, and `lte` require a numeric `value`. + +## Stateful log blocks + +`logs.dedupe` and `logs.throttle` keep per-key state. They must be the last step of a recipe in a processor attached to an input, and they require `state_commit: local`. `state_limit` bounds their native memory. + +```yaml +processors: + logs: + - name: fts + context: native + state_commit: local + state_limit: 1M + recipe: + - id: privacy + use: shared.redact_keys + with: + targets: [{root: body, path: []}] + keys: [password, authorization] + - id: duplicates + use: logs.dedupe + with: + tenant: {root: body, path: [trusted_tenant]} + keys: [{root: body, path: [event_id]}] + ttl_ms: 60000 +``` + +Both blocks accept the following parameters: + +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `keys` | List of 1 to 16 selectors whose values form the key. Required. | _none_ | +| `tenant` | Optional selector for a trusted tenant identity added to the key. | _none_ | +| `ttl_ms` | Key lifetime in milliseconds, from `1` to `86400000`. Required. | _none_ | +| `max_keys` | Maximum number of tracked keys, up to `65536`. | `1024` | +| `max_key_bytes` | Maximum encoded key size in bytes, up to `65536`. | `1024` | +| `on_missing` | `error` fails the item when a key field is missing; `keep` passes it through without tracking. | `error` | +| `overflow` | `keep` passes an item through untracked when capacity is full or its encoded key exceeds `max_key_bytes`. No other value is supported. | `keep` | + +`logs.dedupe` keeps the first item for a key and drops repeats until the key expires after `ttl_ms`. A repeat doesn't extend the expiry time. + +`logs.throttle` is a token bucket per key. It also requires `rate`, in tokens per second, and `burst`, the initial and maximum number of tokens, each from `1` to `1000000`. Each admitted item consumes one token. `ttl_ms` measures inactivity and must be at least `1000 * burst / rate` milliseconds. + +Key values are typed: the string `"1"`, the integer `1`, and the double `1.0` are different keys. A key value that's a map, an array, or a non-finite double fails and applies `error_mode`. Apply parsing, normalization, and privacy steps before the stateful step. + +State is volatile and local to one processor instance. It's lost on restart, reload, and shutdown, and separate inputs or processes don't share it. State is committed when FTS finishes a batch, before later processing or delivery, so a later failure can suppress a duplicate whose original was lost, or consume throttle capacity without delivery. These blocks provide best-effort local suppression, not end-to-end deduplication. When capacity is full, or an item's encoded key exceeds `max_key_bytes`, the item passes through untracked instead of evicting existing keys. + +### Request log windows + +`window.hold_until` holds log records under a key until a fixed processing-time deadline, which starts with the first record and isn't extended by later records. If any record in the window matches `retain_when` before the deadline, every record in the window is released. Otherwise, only records that match `keep_when` are released, and the rest are discarded. For example, hold a request's logs, keep all of them when an `ERROR` occurs, and otherwise keep everything except `DEBUG`. Like the stateful blocks, it must be the last step of an input processor recipe and requires `state_commit: local`. It supports only the `dummy`, `tail`, `forward`, `http`, and `opentelemetry` inputs; attaching it to any other input fails initialization. Held records count toward `state_limit`. + +Retention isn't guaranteed for every window. When the window store is full, or when a record continues a group that started in an earlier chunk, the record passes through without entering the window. A `retain_when` match on such a record doesn't apply to the window, so records already held for that key can still be discarded. + +| Parameter | Description | Default | +| --------- | ----------- | ------- | +| `keys` | Selectors that identify a window. Required. | _none_ | +| `tenant` | Optional trusted tenant selector. | _none_ | +| `hold_ms` | How long to hold records, from `1` to `86400000` milliseconds. Required. | _none_ | +| `retain_when` | Condition that marks a window for retention. Required. | _none_ | +| `keep_when` | Optional condition for records to release when no record matches `retain_when`. When omitted and no record matches `retain_when`, every held record in the window is discarded. | _none_ | +| `late_ms` | How long to recognize late records for a closed window, from `0` to `86400000` milliseconds. | `60000` | +| `ready_timeout_ms` | How long to retry releasing records, from `1` to `86400000` milliseconds. | `60000` | +| `max_keys` | Maximum open windows, up to `65536`. | `1024` | +| `max_key_bytes` | Maximum encoded key size in bytes, up to `65536`. | `1024` | +| `on_missing` | `error` or `keep`. | `error` | +| `overflow` | `keep`. | `keep` | + +Decisions use processing time, not record timestamps. Records that arrive for a closed window pass through and are counted as late. Records still held at shutdown, or when the release retry expires, are discarded and counted. The block processes logs only. + +## Local packs + +A recipe can use blocks from trusted local Lua packs. Set `packages_dir` to the directory that holds the packs, and `packs` to a list of pack manifest paths relative to that directory. Reference a pack block in `use` as `/`. Packs are loaded only from local files. + +```yaml +processors: + logs: + - name: fts + packages_dir: /etc/fluent-bit/packs + packs: [my-pack/pack.json] + recipe: + - id: custom_step + use: / + with: + : +``` + +Replace `` with the `id` field of the pack manifest and `` with the `name` of one of its blocks. List the parameters that the pack block declares under `with`. When a block has no parameters, omit `with` or set it to an empty mapping, `with: {}`. For example, a manifest with the `id` `org.fluentbit.policy.pseudonyms` and a block named `apply` is used as `org.fluentbit.policy.pseudonyms/apply`. Relative `packages_dir` paths resolve against the Fluent Bit process working directory, not the configuration file directory, so prefer an absolute path. Pack code runs with the same trust as other configuration; review packs before deploying them. + +## Metrics + +FTS recipes report the following metrics. Each has the labels `name` (the owning input or output), `scope`, `processor`, `stage`, `signal`, and `outcome`. These metrics count processing decisions, not delivery. + +| Metric | Description | +| ------ | ----------- | +| `fluentbit_fts_recipe_callbacks_total` | Recipe callback outcomes: `keep`, `drop`, `hold`, or `error`. A failure handled by `error_mode: keep` counts as `error`. | +| `fluentbit_fts_state_keys` | Committed keys currently tracked by a stateful block. | +| `fluentbit_fts_evictions_total` | Keys removed after `ttl_ms` expired. | +| `fluentbit_fts_overflow_untracked_total` | Items passed through untracked because capacity was full or their encoded key exceeded `max_key_bytes`. | +| `fluentbit_fts_local_state_commits_total` | Committed state batches. | +| `fluentbit_fts_dropped_by_policy_total` | Items dropped by a committed dedupe or throttle decision. | +| `fluentbit_fts_window_events_total` | Request window events, with outcomes including `held`, `decision`, `released`, `release_retry`, `ready_expired`, `late_passthrough`, `context_passthrough`, `overflow_keep`, and `shutdown`. | diff --git a/pipeline/processors/fluent-typed-scripting/quickstart.md b/pipeline/processors/fluent-typed-scripting/quickstart.md new file mode 100644 index 000000000..9eb1d011e --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/quickstart.md @@ -0,0 +1,73 @@ +# Quickstart + +This example uses the `dummy` input to generate one log. Fluent Typed Scripting (FTS) redacts a password and adds an environment field. You need a native Fluent Bit build containing FTS; see [Availability](../fluent-typed-scripting.md#availability). + +## Create the policy + +In an empty directory, save this script as `policy.lua`: + +```lua +local fb = require("fluentbit") + +function process(log) + fb.redact(log.body, {"password"}) + log.body:set("environment", "production") +end +``` + +## Configure the pipeline + +Save the following as `fluent-bit.yaml` in the same directory. The `dummy` value is the input record. `samples: 1` generates it once. + +```yaml +service: + flush: 1 + grace: 1 + +pipeline: + inputs: + - name: dummy + tag: fts.quickstart + samples: 1 + dummy: '{"message":"login","password":"example-secret"}' + processors: + logs: + - name: fts + script: policy.lua + outputs: + - name: stdout + match: '*' + format: json_lines + json_date_key: false +``` + +Processors require YAML; there is no classic configuration equivalent. Run from the directory containing both files: + +```shell +fluent-bit -c fluent-bit.yaml +``` + +After the next flush, standard output contains the following JSON body. Field order can differ, and Fluent Bit also prints startup messages. + +```json +{"message":"login","password":"[REDACTED]","environment":"production"} +``` + +Stop Fluent Bit with Ctrl+C. The process continues running after the single sample is emitted. + +If startup reports that `fts` isn't registered, check the build options in [Availability](../fluent-typed-scripting.md#availability). If the script can't be loaded, check the path and run directory. If no record appears, inspect policy errors: `error_mode` defaults to `drop`. + +## Change the transformation + +To remove a record, return `fb.DROP`. To commit edits, return nothing or `fb.KEEP`. For example, add this at the beginning of `process` to discard health checks: + +```lua +local path = log.body:get("path") +if path and path:type() == "string" and path:value() == "/health" then + return fb.DROP +end +``` + +The example checks the type of `path` before comparing it, so records with a missing or non-string `path` aren't dropped. They continue through the rest of `process`, which still redacts `password` and sets `environment`. Reading the field with `get_string` instead would raise an error for a non-string value. See [Typed values](typed-values.md) and [Errors and limits](errors-and-limits.md) before applying a policy to mixed input. + +Key-based redaction doesn't inspect free-form strings. A password inside the `message` text requires an explicit string policy. Continue with the complete [log redaction recipe](recipes.md#redact-logs). diff --git a/pipeline/processors/fluent-typed-scripting/recipes.md b/pipeline/processors/fluent-typed-scripting/recipes.md new file mode 100644 index 000000000..80453a50f --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/recipes.md @@ -0,0 +1,260 @@ +# Recipes + +These recipes use complete policies and sample inputs. Run commands from the directory containing the named files, using a native Fluent Bit binary with Fluent Typed Scripting (FTS) enabled. All configurations use YAML because processors don't support classic configuration. + +## Redact logs + +This policy redacts selected structured keys, removes a free-form field, drops health checks, and adds OpenTelemetry context. Save it as `logs.lua`: + +```lua +local fb = require("fluentbit") + +function process(log) + local path = log.body:get("path") + if path and path:type() == "string" and path:value() == "/health" then + return fb.DROP + end + + fb.redact(log.body, {"authorization", "password", "access_token"}) + log.body:remove("debug_text") + + if fb.is_otel(log) then + log.resource.attributes:set("deployment.environment.name", "production") + log.attributes:set("sanitized", true) + fb.set_severity(log, "info") + end +end +``` + +Save this configuration as `logs.yaml`. The envelope runs before FTS so that the dummy log has OpenTelemetry context. + +```yaml +service: + flush: 1 + grace: 1 + +pipeline: + inputs: + - name: dummy + tag: fts.logs + samples: 1 + dummy: '{"event":"login","password":"example-secret","nested":{"Authorization":"Bearer example"},"debug_text":"password=example-secret"}' + processors: + logs: + - name: opentelemetry_envelope + - name: fts + script: logs.lua + error_mode: drop + outputs: + - name: stdout + match: '*' + format: otlp_json +``` + +Run: + +```shell +fluent-bit -c logs.yaml +``` + +Expected output is an OTLP JSON log with this body content, redacted at both nesting levels: + +```json +{"event":"login","password":"[REDACTED]","nested":{"Authorization":"[REDACTED]"}} +``` + +The sample uses `event` instead of `message` because the OTLP JSON encoder of the `stdout` output selects `log` or `message` as the body when present. With this sample, the full structured map appears in the OTLP body. + +The enclosing OTLP data also contains resource attribute `deployment.environment.name=production`, log attribute `sanitized=true`, severity text `info`, and severity number `9`. `debug_text` is absent because the policy removes that field explicitly. Key redaction alone wouldn't remove its embedded secret. This policy doesn't inspect secrets inside `message` or other text fields; adapt it to your data before using it as a privacy policy. Stop the process with Ctrl+C. + +## Normalize and aggregate metrics + +This policy converts a duration gauge from milliseconds to seconds and combines data points after retaining only the `route` attribute. Save it as `metrics.lua`: + +```lua +local fb = require("fluentbit") + +function process(metric) + if metric.name ~= "request.duration" or metric.type ~= "gauge" then + return + end + + metric.name = "request.duration.mean" + fb.scale_metric(metric, 0.001, "s") + fb.aggregate_metric(metric, {"route"}, "mean") +end +``` + +The mean is over data points with equal retained attributes and equal start/end timestamps in this family. It isn't weighted by request count and doesn't aggregate across callbacks. For counters, choose a sum policy with the appropriate temporality instead. + +Save this configuration as `metrics.yaml`: + +```yaml +service: + flush: 1 + grace: 1 + +pipeline: + inputs: + - name: opentelemetry + listen: 127.0.0.1 + port: 4318 + processors: + metrics: + - name: fts + script: metrics.lua + error_mode: drop + outputs: + - name: stdout + match: '*' + format: otlp_json +``` + +Save this OTLP JSON payload as `metrics.json`: + +```json +{ + "resourceMetrics": [{ + "scopeMetrics": [{ + "metrics": [{ + "name": "request.duration", + "unit": "ms", + "gauge": { + "dataPoints": [ + { + "attributes": [ + {"key": "route", "value": {"stringValue": "/login"}}, + {"key": "instance", "value": {"stringValue": "a"}} + ], + "startTimeUnixNano": "1790812800000000000", + "timeUnixNano": "1790812801000000000", + "asDouble": 100 + }, + { + "attributes": [ + {"key": "route", "value": {"stringValue": "/login"}}, + {"key": "instance", "value": {"stringValue": "b"}} + ], + "startTimeUnixNano": "1790812800000000000", + "timeUnixNano": "1790812801000000000", + "asDouble": 300 + } + ] + } + }] + }] + }] +} +``` + +Start Fluent Bit: + +```shell +fluent-bit -c metrics.yaml +``` + +In another terminal in the same directory, send the payload: + +```shell +curl --fail-with-body -H 'Content-Type: application/json' \ + --data-binary @metrics.json http://127.0.0.1:4318/v1/metrics +``` + +After a flush, expect one family named `request.duration.mean`, unit `s`, with one gauge datapoint: attribute `route=/login` and `asDouble` approximately `0.2`. The `instance` attribute is absent. Both timestamps are preserved. Removing `instance` without aggregation would create a series collision and fail the policy. Stop Fluent Bit before starting the next recipe, which uses the same port. + +## Transform spans + +This policy normalizes a span name from the route, redacts structured attributes on spans, events, and links, and adds a resource attribute. It preserves the trace structure. Save it as `traces.lua`: + +```lua +local fb = require("fluentbit") + +function process(span) + local route = span.attributes:get("http.route") + if route and route:type() == "string" then + span.name = "HTTP " .. route:value() + end + + fb.redact(span.attributes, {"authorization", "password"}) + fb.redact(span.events, {"authorization", "password"}) + fb.redact(span.links, {"authorization", "password"}) + span.resource.attributes:set("deployment.environment.name", "production") +end +``` + +Save this configuration as `traces.yaml`: + +```yaml +service: + flush: 1 + grace: 1 + +pipeline: + inputs: + - name: opentelemetry + listen: 127.0.0.1 + port: 4318 + processors: + traces: + - name: fts + script: traces.lua + error_mode: drop + outputs: + - name: stdout + match: '*' + format: otlp_json +``` + +Save this OTLP JSON payload as `traces.json`. IDs in OTLP JSON use hexadecimal text; inside FTS link maps they're typed bytes. + +```json +{ + "resourceSpans": [{ + "scopeSpans": [{ + "spans": [{ + "traceId": "0102030405060708090a0b0c0d0e0f10", + "spanId": "0102030405060708", + "name": "GET /login?user=example", + "kind": 2, + "startTimeUnixNano": "1790812800000000000", + "endTimeUnixNano": "1790812801000000000", + "attributes": [ + {"key": "http.route", "value": {"stringValue": "/login"}}, + {"key": "authorization", "value": {"stringValue": "Bearer example"}} + ], + "events": [{ + "name": "authenticated", + "timeUnixNano": "1790812800500000000", + "attributes": [ + {"key": "password", "value": {"stringValue": "example-secret"}} + ] + }], + "links": [{ + "traceId": "1112131415161718191a1b1c1d1e1f20", + "spanId": "1112131415161718", + "attributes": [ + {"key": "authorization", "value": {"stringValue": "Bearer example"}} + ] + }] + }] + }] + }] +} +``` + +Start Fluent Bit: + +```shell +fluent-bit -c traces.yaml +``` + +In another terminal, send the payload: + +```shell +curl --fail-with-body -H 'Content-Type: application/json' \ + --data-binary @traces.json http://127.0.0.1:4318/v1/traces +``` + +After a flush, expect a span named `HTTP /login`. The authorization attribute on the span and link and the password attribute on the event contain `[REDACTED]`. The resource contains `deployment.environment.name=production`. Trace and span IDs, timestamps, event name, and link targets are preserved. Stop Fluent Bit with Ctrl+C. + +The policy doesn't inspect event names or other free-form text. Dropping spans or changing IDs would require a separate decision about trace structure: FTS doesn't repair references in other spans. See [Traces](traces.md#trace-topology). diff --git a/pipeline/processors/fluent-typed-scripting/traces.md b/pipeline/processors/fluent-typed-scripting/traces.md new file mode 100644 index 000000000..644e3a08c --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/traces.md @@ -0,0 +1,66 @@ +# Traces + +The callback `process(span)` runs once per span. An implicit return or `fb.KEEP` commits edits; `fb.DROP` removes the span. Resource and scope edits apply only to the current span. + +## Span properties + +| Property | Description | +| --- | --- | +| `span.name`, `span.trace_state` | Span name and trace state strings. | +| `span.kind` | Numeric OpenTelemetry kind: 0 unspecified, 1 internal, 2 server, 3 client, 4 producer, 5 consumer. | +| `span.flags` | Numeric span flags. | +| `span.trace_id`, `span.span_id`, `span.parent_span_id` | Hexadecimal ID strings. Trace ID has 32 characters, span and parent IDs have 16. An empty parent ID marks a root span. Nonempty IDs must be nonzero. | +| `span.start_time_unix_nano`, `span.end_time_unix_nano` | Typed exact integer timestamps. Assign integer typed values; an edited end time must not precede the start. | +| `span.attributes` | Typed attribute map. | +| `span.status` | Typed map containing integer `code` and string `message`. Codes: 0 unset, 1 OK, 2 error. | +| `span.events`, `span.links` | Typed arrays of event and link maps. | +| `span.dropped_attributes_count`, `span.dropped_events_count`, `span.dropped_links_count` | Numeric dropped counts. | +| `span.resource.attributes`, `span.resource.schema_url` | Resource context. | +| `span.scope.attributes`, `span.scope.name`, `span.scope.version`, `span.scope.schema_url` | Instrumentation scope context. | +| `span.tag` | Read-only routing tag. | + +Update attributes through `set` and `remove`. For status codes, use an integer constructor: + +```lua +local fb = require("fluentbit") + +function process(span) + if span.attributes:get_bool("application.failed") == true then + span.status:set("code", fb.int64(2)) + span.status:set("message", "application failure") + end +end +``` + +## Events and links + +Event maps contain `name`, `time_unix_nano`, `attributes`, and `dropped_attributes_count`. Link maps contain `trace_id`, `span_id`, `trace_state`, `attributes`, `dropped_attributes_count`, and `flags`. + +Unlike the top-level span ID properties, IDs inside typed link maps are bytes. Use `fb.bytes` containing 16 bytes for a trace ID and 8 bytes for a span ID, not hexadecimal text. Each event and link must have an attribute map. Event times must be nonnegative integers. + +The following callback redacts nested structured attributes and adds an event: + +```lua +local fb = require("fluentbit") + +function process(span) + fb.redact(span.attributes, {"authorization", "password"}) + fb.redact(span.events, {"authorization", "password"}) + fb.redact(span.links, {"authorization", "password"}) + + local event = fb.map() + event:set("name", "policy.applied") + event:set("time_unix_nano", span.end_time_unix_nano) + event:set("attributes", fb.map()) + event:set("dropped_attributes_count", fb.uint64(0)) + span.events:append(event) +end +``` + +Redaction matches keys; it doesn't inspect event names, status messages, or other free-form text. Removing an event or link requires building a replacement array. Set dropped counts deliberately if your policy needs to account for removals; don't assume an array edit updates them. + +## Trace topology + +Dropping a span doesn't drop its children or the whole trace. Changing a trace ID, span ID, or parent ID doesn't update other spans or links. These changes can create orphaned spans or split a trace. Fluent Typed Scripting (FTS) doesn't repair parent references or perform trace-wide transactions. + +Prefer attribute and name normalization when that meets your goal. If you change identifiers or drop spans, define how downstream consumers should interpret the resulting trace structure. See [Recipes](recipes.md#transform-spans) for a complete example that preserves identifiers. diff --git a/pipeline/processors/fluent-typed-scripting/typed-values.md b/pipeline/processors/fluent-typed-scripting/typed-values.md new file mode 100644 index 000000000..c962bee4d --- /dev/null +++ b/pipeline/processors/fluent-typed-scripting/typed-values.md @@ -0,0 +1,85 @@ +# Typed values + +Fluent Typed Scripting (FTS) exposes telemetry as typed values, not ordinary Lua tables. Checks happen at runtime; Lua itself remains dynamically typed. Load constructors and helpers with `local fb = require("fluentbit")`. + +## Maps and arrays + +Maps have string keys. Key matching uses exact bytes and is case-sensitive unless a helper documents otherwise. A dot is part of the key: `attributes:get("service.name")` doesn't traverse nested maps. Arrays use indexes starting at 1. + +| Method | Applies to | Result or effect | +| --- | --- | --- | +| `v:get(key)` | Map, array | Typed child value. A missing map key returns `nil`; an array index outside `1..v:len()` raises an error. | +| `v:get_string(key)` | Map, array | Lua string; `nil` for a missing map key. A different type raises an error. | +| `v:get_number(key)` | Map, array | Lua number; `nil` for a missing map key. Uses the same precision checks as `number()`. | +| `v:get_bool(key)` | Map, array | Lua boolean; `nil` for a missing map key. A different type raises an error. | +| `v:set(key, value)` | Map, array | Set a value. For maps, replace all entries with that key. For arrays, replace an existing index. | +| `v:remove(key)` | Map | Remove all entries with the key. | +| `v:ensure_map(key)` | Map | Return a child map or create it if missing. An existing non-map value raises an error. | +| `v:append(value)` | Array | Append a value. | +| `v:keys()` | Map | Lua table of keys in entry order. | +| `v:len()` | Map, array | Number of entries. | +| `v:type()` | Any typed value | `map`, `array`, `string`, `bytes`, `bool`, `int64`, `uint64`, `double`, or `null`. | +| `v:value()` | Scalar | Lua scalar; integers are exact decimal strings, bytes are Lua strings, and null is `nil`. Containers raise an error. | +| `v:number()` | Numeric value | Lua number. Rejects integers outside the safe range −(2^53 − 1) through 2^53 − 1. | +| `v:copy()` | Any typed value | Independent writable copy. | + +Use `append` to grow arrays. To remove or reorder elements, build a new array and assign it to the containing field. + +## Constructors and conversions + +| Value | Creates | +| --- | --- | +| `fb.map()` | Empty typed map. | +| `fb.array()` | Empty typed array. | +| `fb.int64(value)` | Signed 64-bit integer from a decimal string or safe integral Lua number. Range: −9223372036854775808 through 9223372036854775807. | +| `fb.uint64(value)` | Unsigned 64-bit integer from a decimal string or safe integral Lua number. Range: 0 through 18446744073709551615. | +| `fb.bytes(text)` | Binary value from a Lua string, including null (`\0`) bytes. | +| `fb.null` | Explicit null value. | + +Pass Lua strings, booleans, and finite numbers directly to `set` or `append`. Lua numbers are stored as doubles, including integral numbers. Use the integer constructors to request integer storage. Ordinary Lua tables and `nil` aren't telemetry values. Use typed constructors and `fb.null`; use `remove` to delete a map key. + +```lua +local fb = require("fluentbit") + +function process(log) + local body = log.body + body:set("request_id", fb.uint64("18446744073709551615")) + body:set("attempts", fb.int64(2)) + body:set("optional", fb.null) + body:set("payload", fb.bytes("a\0b")) + local labels = fb.array() + labels:append("production") + body:set("labels", labels) + + assert(body:get("request_id"):value() == "18446744073709551615") + assert(body:get_number("attempts") == 2) + assert(body:get("optional"):type() == "null") + assert(body:get("missing") == nil) +end +``` + +Don't convert an exact integer to a Lua number and then expect to recover its precision. Use decimal strings for large integer constructors and `value()` for exact reads. Numeric strings aren't converted by `get_number`. + +OpenTelemetry scalar metric values and attribute integers use signed int64. A typed `uint64` doesn't make every OpenTelemetry field capable of representing the full unsigned range. See [Metrics](metrics.md) and the limits of your output format. JSON output also can't convey a distinction between bytes and strings or between every numeric storage type. + +## Value lifetime + +Log values and views obtained from them are valid only during that callback. Retaining a log body or child view in a global and using it in a later callback raises an error. + +Metric and span values are snapshots made for the callback. They remain readable afterward, but subsequent mutations don't affect committed output or later items. Retaining snapshots also retains their memory. + +To retain data deliberately, store `v:copy()`. A copy is independent; editing it doesn't change a committed record. Globals persist for the lifetime of a processor instance, so retained data counts against `memory_limit`. Bound any cache you keep in a policy. + +```lua +local previous + +function process(log) + if previous then + log.body:set("previous", previous) + end + local message = log.body:get("message") + previous = message and message:copy() or nil +end +``` + +This example retains only the previous message value, keeping the number of retained values bounded. diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md index d12233241..4ecc8d427 100644 --- a/pipeline/processors/metrics-selector.md +++ b/pipeline/processors/metrics-selector.md @@ -96,4 +96,6 @@ pipeline: outputs: - name: stdout match: '*' -``` \ No newline at end of file +``` + +For programmable metric normalization and explicit aggregation after attribute removal, see [Fluent Typed Scripting (FTS)](fluent-typed-scripting/metrics.md). diff --git a/pipeline/processors/opentelemetry-envelope.md b/pipeline/processors/opentelemetry-envelope.md index 941b2317a..f5efe3df7 100644 --- a/pipeline/processors/opentelemetry-envelope.md +++ b/pipeline/processors/opentelemetry-envelope.md @@ -180,3 +180,7 @@ The collector JSON output will resemble the following: For more details about further processing, read the [content modifier](../processors/content-modifier.md) processor documentation. + +## Transform the envelope with Fluent Typed Scripting + +Place [Fluent Typed Scripting (FTS)](fluent-typed-scripting.md) after this processor to access resource, scope, and log attributes through typed properties. See the complete [log redaction recipe](fluent-typed-scripting/recipes.md#redact-logs). diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 4dbdbfa66..3d82d4789 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -267,6 +267,7 @@ VMs Wasm Wavefront winevtlog +winstat winlog Winlogbeat Worldmap