Repository navigation
Conversation
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Configure and process batches include/fluent-bit/flb_input.h, plugins/filter_rewrite_tag/*, tests/integration/scenarios/filter_rewrite_tag/tests/* |
rewrite_tag adds a batch_size option, validates input-processor placement, and groups eligible records by destination tag. Unsupported or failed batch attempts use record-at-a-time processing. Integration tests cover batch sizes, keep behavior, record order and retention, output-processor rejection, and group context. |
Emitter and processor lifecycle
| Layer / File(s) | Summary |
|---|---|
Emitter shutdown and processor lifecycle plugins/in_emitter/emitter.c, src/flb_input.c, src/flb_processor.c, tests/integration/scenarios/filter_rewrite_tag/tests/test_filter_rewrite_tag_batch.py |
The emitter adds an ingestion guard and queues pending chunks during shutdown. Input initialization skips repeated initialization, and processor_run frees a zero-sized modified-filter buffer. An EOF test checks delivery across filter arrangements. |
Benchmark tooling and results
| Layer / File(s) | Summary |
|---|---|
Benchmark tooling and results benchmarks/rewrite-tag/* |
The benchmark driver runs four rewrite_tag configurations and records delivery and resource measurements. The README and JSON files document the benchmark workloads, results, validation, and reproduction commands. |
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant InputProcessor
participant cb_rewrite_tag_filter
participant rewrite_tag_batch
participant Emitter
InputProcessor->>cb_rewrite_tag_filter: invoke processor callback
cb_rewrite_tag_filter->>rewrite_tag_batch: attempt eligible batch
rewrite_tag_batch->>Emitter: ingest grouped destination records
Emitter-->>rewrite_tag_batch: return admission result
rewrite_tag_batch-->>InputProcessor: return retained records and result
Possibly related issues
- rewrite_tag loses records when emitter buffer limit is reached, even with filesystem buffering #12667: The changes cover emitter re-emission and record preservation paths, but the issue’s filesystem retry requirement is not shown as implemented.
- Records flushed by in_tail's multiline at shutdown skip all filters (or are lost with memory storage) #12628: Emitter shutdown handling and EOF delivery tests address related shutdown behavior in the rewrite_tag paths.
Possibly related PRs
- fluent/fluent-bit#12668: It changes the same rewrite_tag lifecycle paths, including placement validation, emitter naming, and shutdown handling.
- fluent/fluent-bit#12589: It changes processor handling of empty results, which overlaps with the zero-sized output cleanup here.
Suggested reviewers: cosmo0920
Merge Risk: 
3f22a
-
No confirmed issue in the supplied evidence blocks merging; proceed with normal checks.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 21.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely identifies the main change: opt-in batching for the rewrite_taginput processor. It also accurately notes the supporting fixes.Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 21.05% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 19 functions across 9 files. (3 skipped: 3 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Using
rewrite_tagas an input processor appends rewritten records individually, repeatedly invoking downstream stages even when records share a destination tag. This adds opt-in batching by destination tag so each batch continues at the next processor stage through the original input append/storage path.For example,
batch_size: 4096batches eligible callbacks of up to 4096 records; the default0retains the existing path. Callbacks above the count limit or 256 KiB payload bound fall back to per-record processing. Regular filter placement retains its emitter path. Ordering is preserved within each destination tag and among retained originals; cross-tag interleaving can change, which matters for stateful downstream processors.The series also fixes issues exposed by validation:
Regression coverage exercises processor stage continuation, KEEP modes, invalid destinations, per-tag order, two-output fan-out, group context, multiple inputs, and EOF shutdown chains. Components, integration tests, and benchmark evidence are kept in separate DCO-signed commits.
The included benchmark report records 36–45% lower CPU cost for repeated destination tags compared with the baseline input processor. High-cardinality tags can regress; batching remains disabled by default. Measurements are from the recorded baseline and environment in
benchmarks/rewrite-tag/, not a new benchmark run for this PR.Validation from the branch checkout:
Results: 39 integration cases passed normally (117.99 s), and 39 passed under strict Valgrind; all 39 Valgrind logs report zero errors and zero definitely, indirectly, or possibly lost bytes. All 11 focused CTest executables passed (107.32 s). Full PR-range commit-prefix lint and whitespace checks passed.
Summary by CodeRabbit
rewrite_tagcan now batch records in input processors. Configure a batch size up to 4096; batching is disabled by default, and unsupported batch sizes fall back to individual processing.