Skip to content

fix(core): clear the buffer active bit when the snapshot supplier throws - #2446

Open
apc999 wants to merge 1 commit into
prometheus:mainfrom
apc999:fix/buffer-active-bit-leak-on-throw
Open

fix(core): clear the buffer active bit when the snapshot supplier throws#2446
apc999 wants to merge 1 commit into
prometheus:mainfrom
apc999:fix/buffer-active-bit-leak-on-throw

Conversation

@apc999

@apc999 apc999 commented Sep 3, 2026

Copy link
Copy Markdown

Problem

Buffer.run() sets the buffer-active bit on every striped observation counter, waits for
in-flight observations to land, calls createResult.get(), and then clears the bit again.
The surrounding finally only releases runLock — the bit-clearing block sits inside the
try, so when createResult.get() throws, the bit is left set permanently.

https://github.com/prometheus/client_java/blob/main/prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java#L90-L139

Two things follow from a bit that stays set:

  • append() keeps returning true, so observations go into the buffer instead of being
    recorded, and are replayed at an arbitrary later point.
  • The next run() derives expectedCount from counters that still carry the sign bit, so
    complete.apply(expectedCount) may never be satisfied. The wait loop then spins on
    Thread.yield() indefinitely while holding runLock, which blocks every subsequent
    scrape of that registry.

Whether the second case spins or instead produces a corrupt snapshot depends on the number
of stripes, because 2 * Long.MIN_VALUE == 0: the sign-bit contributions cancel out for an
even Runtime.getRuntime().availableProcessors(). With an even stripe count you get a run
whose buffer semantics are inverted (observations recorded directly rather than buffered,
i.e. a torn snapshot) rather than a hang. With an odd count — including the single
AtomicLong this code used before striping — the spin is deterministic.

Nothing clears the bit in-process, so the only recovery is a process restart.

Fix

Move the wait-and-create block into its own try, with the "signal that the buffer is
inactive" step and the buffer drain in the matching finally.

The drain has to move with it rather than just the bit flip: leaving bufferPos set makes
the next run() skip its own wait and replay stale values on top of the new ones. The
arithmetic is unaffected on the exception path — expectedCount is computed before the
throw, so expectedBufferSize -= expectedCount still holds.

Test

BufferTest.bufferIsDeactivatedWhenCreateResultThrows asserts the invariant rather than the
symptom: after run() returns or throws, the buffer must be inactive, probed directly via
append(). Using count -> true for complete keeps the test independent of the stripe
count, so it fails deterministically on any machine.

Verified both ways:

Result
With the fix, BufferTest 2/2 pass
With the fix reverted fails — Expecting value to be false but was true
With the fix, full prometheus-metrics-core suite 166/166 pass

How we hit this

Found in production in Alluxio on prometheus-metrics-core 1.0.0. An OutOfMemoryError
raised inside createResult.get() — snapshot creation allocates several arrays per data
point, so it is a likely place for an allocation to fail — left the bit set and wedged the
/metrics endpoint of several workers for 4.5 days, until they were restarted. Three
separate data points were poisoned within 12 seconds of each other, consistent with a single
JVM-wide allocation failure rather than a race.

Note that Buffer.run() catches nothing, so an Error leaks the bit exactly as an exception
does.

Relation to #2287

#2287 reports two things about this class: unbounded growth of observationBuffer in
doAppend(), and the Thread.yield() wait loop having no timeout. This PR fixes neither of
those directly — it removes the error path that makes the second one permanent. Without a
leaked active bit, the wait loop is a bounded spin over in-flight observations; with one, it
never terminates. So the two changes are complementary rather than overlapping, and I have
deliberately left the loop itself alone here to keep this diff reviewable.

Notes

  • The unbounded Thread.yield() wait loop is left as-is; this change only ensures a thrown
    exception cannot turn it into an infinite spin.
  • Built and tested locally with JDK 24 (-Dtest.java.version=24), since test.java.version
    defaults to 25. Main sources are unaffected (java.version=8). I was not able to run
    mise run lint locally, so please let CI arbitrate formatting.

Buffer.run() sets the buffer-active bit on every striped observation
counter, waits for in-flight observations to land, calls
createResult.get(), and then clears the bit again. The surrounding
finally only releases runLock, so when createResult.get() throws, the
bit is left set permanently.

Two things follow from a bit that stays set:

  * append() keeps returning true, so observations go into the buffer
    instead of being recorded, and are replayed at an arbitrary later
    point.

  * The next run() derives expectedCount from counters that still carry
    the sign bit, so complete.apply(expectedCount) may never be
    satisfied. The wait loop then spins on Thread.yield() indefinitely
    while holding runLock, which blocks every subsequent scrape of that
    registry. Whether it spins or instead produces a corrupt snapshot
    depends on the number of stripes, because the sign-bit contributions
    cancel out for an even Runtime.availableProcessors():
    2 * Long.MIN_VALUE == 0.

Nothing clears the bit in-process, so the only recovery is a restart.

Move the wait-and-create block into its own try, with the "signal that
the buffer is inactive" step and the buffer drain in the matching
finally. The drain has to move with it: leaving bufferPos set makes the
next run() skip its own wait and replay stale values on top of the new
ones.

Found in production in Alluxio on prometheus-metrics-core 1.0.0, where
an OutOfMemoryError raised inside createResult.get() wedged the /metrics
endpoint of several workers for 4.5 days until they were restarted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bin Fan <fanbin103@gmail.com>

@Karthik-Chowdary Karthik-Chowdary left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I walked the exception path through both striped-counter modes. Pairing active-bit clearing and buffer draining in the inner finally restores the invariant even when snapshot construction throws; moving only the bit flip would indeed leave stale bufferPos state for the next run. The regression test is stripe-count independent and directly probes the externally relevant append behavior after the throw. This is a focused fix that does not alter the existing bounded in-flight wait on the success path. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants