Fix infinite loop when an error event fails in @ERROR

[tarun9715m]

What this fixes

Fixes #5462.

A record larger than chunk_limit_size raises BufferChunkOverflowError at emit time and the event is routed to @ERROR. If the output inside @ERROR raises the same error, the event was routed to @ERROR again, and this repeated without bound. Nothing marked the event as having already been through the error path, so there was no termination condition.

The routing is recursive rather than a flat loop:

EventRouter#emit_stream rescues → RootAgent#handle_emits_error → @error_collector.emit_stream → the plugin in @ERROR raises → that router's rescue → handle_emits_error → …

So each iteration adds stack frames and the worker eventually dies with SystemStackError (~7000 levels in my reproduction). None of the buffer's failure-handling parameters apply, because they govern flush-time failures: retry_max_times / retry_timeout need a flush that never happened, overflow_action governs BufferOverflowError (buffer full) which is a different error class, and <secondary> receives a chunk after flush retries are exhausted when no chunk was ever created.

How

@ERROR is the last-resort route, so it should be bounded by construction. An error event that fails while being routed to @ERROR is now dumped with a distinct error-level log and dropped, instead of being routed back. This uses a thread-local re-entrancy guard in RootAgent, applied to both handle_emits_error (event streams) and emit_error_event (single records — reached when a plugin inside @ERROR, such as a filter, calls router.emit_error_event itself).

The distinct message also addresses the second half of the issue: a genuine loop is now distinguishable from repeated single failures.

[error]: #0 dump an error event stream because it failed in @ERROR. It is dropped to prevent an infinite loop: error_class=Fluent::Plugin::Buffer::BufferChunkOverflowError ...

Behaviour when no @ERROR label is configured is unchanged, including raise error and the emit_error_log_interval suppression. The guard resets in an ensure, so the next error event still gets its normal single attempt at @ERROR.

Reproduction

Config from the issue (chunk_limit_size 7 in both the primary output and @ERROR), 3 × 2 MB-line input file, 25 second window:

	before	after
send an error event stream to @ERROR	2329, still climbing	1
dropped-with-reason log	0	1
SystemStackError / worker death	yes (~7020 frames)	none, worker stays healthy

Known limitation

The guard catches synchronous re-entry, which covers every emit-time error path including the reported one. An error re-emitted from a different thread inside @ERROR would not be caught — bounding that would require a marker on the event itself. Happy to explore that here if a reviewer would prefer it.

Tests

4 regression tests added to test/test_root_agent.rb, covering both entry points, the dropped-and-not-rerouted behaviour, and that the guard resets so a subsequent error event is still routed normally.

bundle exec rake test on macOS / Ruby 4.0.6: 4345 tests, 15851 assertions, 0 failures, 0 errors, 3 pendings, 36 omissions.

🤖 Generated with Claude Code

[Watson1978]

Thank you for working on this.

I cannot take this change as it is. However well-intentioned, discarding events without the user asking for it is not acceptable behaviour for Fluentd. @ERROR is a recovery path, so dropping the events routed to it defeats its own purpose. Data should be discarded only where the user has explicitly configured it to be (e.g. overflow_action drop_oldest_chunk), never as a built-in fallback.

Note also that in the event stream path the discarded records are not preserved anywhere: the new log line carries only record_count, so their contents are lost entirely.

[tarun9715m]

Thank you for working on this.

I cannot take this change as it is. However well-intentioned, discarding events without the user asking for it is not acceptable behaviour for Fluentd. @ERROR is a recovery path, so dropping the events routed to it defeats its own purpose. Data should be discarded only where the user has explicitly configured it to be (e.g. overflow_action drop_oldest_chunk), never as a built-in fallback.

Note also that in the event stream path the discarded records are not preserved anywhere: the new log line carries only record_count, so their contents are lost entirely.

@Watson1978 Thanks for the review, the objection makes sense, @error shouldn't silently discard, and you're right that record_count alone loses the contents.

Worth flagging that the current behaviour isn't lossless either: the recursion kills the worker with SystemStackError, so everything in flight is lost, not just the offending record. We have faced the same in the production where the records are getting missed.

Each iteration adds stack frames, so the worker eventually dies with SystemStackError (~7000 levels in my reproduction) rather than looping indefinitely.
So the outcome today isn't just noisy logs, the worker goes down and everything in flight is lost, not only the oversized record.