Skip to content

fix: keep handler errors from crashing the app - #33

Open
bengl wants to merge 1 commit into
nodejs:mainfrom
bengl:bengl/no-crash
Open

bengl wants to merge 1 commit into
nodejs:mainfrom
bengl:bengl/no-crash

Conversation

@bengl

@bengl bengl commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Node rethrows an error from a diagnostics_channel subscriber or a timer as an uncaught exception, and an error from an event listener comes out of emit() and skips the listeners after it. So a bug in microtel could take down the app it traces, or keep the app's own 'end' listener from running and leave its request hanging.

What changed. guard(name, fn) in lib/guard.js wraps every function microtel hands to Node:

Where What
diagnostics_channel All 13 subscriptions
Response listeners Server 'close', client 'end' and 'close'
Flushes The interval timer, the buffer-full flush, and beforeExit
Exports The response callback, its 'end' and 'error' listeners, the deadline timer, and the request's 'error' listener

A caught error becomes an OTelInstrumentationWarning, throttled the way OTelExportWarning is: one per 30 seconds, with a running count. The warning's detail carries the original stack, since that's what fixing the bug takes. The request it happened on carries on untraced or partly traced. microtel doesn't try to repair its own state.

Exports always settle. Each export handler settles its export before anything that can throw, and the export deadline settles it too. So a failure a guard catches never leaves an awaited flush() waiting.

Cost. Each wrapper is made once, when it's subscribed or attached, so a guarded call allocates nothing. Its extra call still costs something. CPU profiles of 50k traced HTTP requests, 8 alternating rounds against main:

Median main this PR
microtel's own time, without the wrappers 128.2 ms 126.2 ms
The wrappers 6.2 ms
The whole process 1,138 ms 1,155 ms

That's about 120 ns per traced request, or half a percent of the process's CPU. A microbenchmark puts a guarded diagnostics_channel call at 2.9 ns more than a plain one. In the real path the one shared wrapper can't inline the many handlers it calls, so it costs more. A wrapper per call shape, calling handlers directly instead of through apply, measured the same 6.2 ms.

Raw samples

Wrapper time, ms: 5.3, 8.2, 5.4, 6.1, 7.2, 5.0, 6.4, 7.2

microtel's own time without the wrappers, ms:

  • main: 131.9, 118.9, 127.1, 139.9, 132.9, 116.5, 127.3, 129.2
  • this PR: 132.5, 132.0, 138.3, 121.6, 125.5, 126.8, 123.3, 119.9

A throughput A/B of the node:bench suite, 5 interleaved rounds, was within this machine's noise: traced HTTP moved −3.4% and traced fetch +3.9%, while the untraced controls, which never load microtel, moved +0.2% and −4.1%.

Not taken.

  • A try/catch inside each handler instead of a wrapper. It would avoid the wrapper's cost, at the price of the same few lines in every handler.
  • Guarding the public flush(). A direct call throws to its caller, the way Node's own APIs do.
  • Turning a handler off after it fails repeatedly. A warning per window is enough to notice, and a bug that only hits some requests would lose the rest of their traces.

Tests. Two child-process fixtures make microtel's own code throw:

  • instrumentation-failure-child.js patches span methods and poisons spans. It makes HTTP/1, HTTP/2, and fetch requests, including aborted ones, and drives the timer, buffer-full, and beforeExit flushes. Every request must complete, nothing may reach uncaughtException, and the last warning's running count must equal the number of failures injected.
  • export-failure-child.js replaces http.request with a fake, so each export guard fires on demand, each in a fresh warning window, and every export's flush() must still resolve.

Removing any single guard, breaking the throttle either way, dropping the stack, or settling an export after a step that can throw fails a test. That's 32 mutations in all.

@bengl
bengl marked this pull request as ready for review November 2, 2026 21:54
@bengl
bengl requested a review from a team November 2, 2026 21:55

@AbhiPrasad AbhiPrasad left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd clean up some of the code comments, but otherwise lgtm

@bengl

bengl commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

@AbhiPrasad how's this for a comment cleanup?

Node rethrows an error from a diagnostics_channel subscriber or a
timer as an uncaught exception, and an error from an event listener
comes out of emit() and skips the listeners after it. Previously, any
bug in a microtel handler could take down the app it was tracing, or
keep the app's own listeners from running.

Now every function microtel hands to Node is wrapped once, when it is
subscribed or attached, and reports its errors as an
OTelInstrumentationWarning. The warnings are throttled the way export
failures are, one per 30 seconds with a running count, and each
carries the original stack. The request it happened on carries on
untraced or partly traced.

Each export handler settles its export before anything that can
throw, and the export deadline settles it too, so a failure a guard
catches never leaves an awaited flush() waiting.

Signed-off-by: Bryan English <bryan@bryanenglish.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants