Skip to content

Make stdio logging fail-soft on write errors#231

Open
dineshadhi wants to merge 1 commit into
cloudflare:mainfrom
dineshadhi:dineshadhi/logging-fail-soft
Open

Make stdio logging fail-soft on write errors#231
dineshadhi wants to merge 1 commit into
cloudflare:mainfrom
dineshadhi:dineshadhi/logging-fail-soft

Conversation

@dineshadhi

Copy link
Copy Markdown

Foundations currently wraps terminal/stderr logging drains with Fuse before passing them to slog_async. If the underlying stdout/stderr stream returns an I/O error such as BrokenPipe, Fuse turns that error into a panic in the async logging worker.

This is unsafe for services whose stdout/stderr is captured by systemd and journald: if the journald stream is interrupted, normal logging can crash the service.

Handle terminal and stderr output errors with ignore_res() instead, so stdio logging failures do not panic the async logging worker. File logging keeps the existing fused behavior.

Add a regression test proving a failing async stdio drain keeps processing subsequent records instead of terminating after the first write error.

Foundations currently wraps terminal/stderr logging drains with `Fuse`
before passing them to `slog_async`. If the underlying stdout/stderr stream
returns an I/O error such as `BrokenPipe`, `Fuse` turns that error into a
panic in the async logging worker.

This is unsafe for services whose stdout/stderr is captured by systemd and
journald: if the journald stream is interrupted, normal logging can crash
the service.

Handle terminal and stderr output errors with `ignore_res()` instead, so
stdio logging failures do not panic the async logging worker. File logging
keeps the existing fused behavior.

Add a regression test proving a failing async stdio drain keeps processing
subsequent records instead of terminating after the first write error.
@TheJokr TheJokr self-requested a review July 14, 2026 13:10
@TheJokr

TheJokr commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

The use of fuse() for our loggers is intentional - keeping a service running without log output is bad for observability. That being said, I understand there is a use case for this. I'm happy to merge a PR that makes this configurable:

  • Add a field ignore_io_errors: bool to LoggingSettings
  • Wrap the AsyncDrain construction in a new function that takes drain: impl Drain<...> and settings: &LoggingSettings
  • Apply fuse() or ignore_res() depending on the setting there
  • Use that function for the output types that are currently fused (terminal/stderr/file)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants