fix(core): clear the buffer active bit when the snapshot supplier throws - #2446
Open
apc999 wants to merge 1 commit into
Open
fix(core): clear the buffer active bit when the snapshot supplier throws#2446apc999 wants to merge 1 commit into
apc999 wants to merge 1 commit into
Conversation
Buffer.run() sets the buffer-active bit on every striped observation
counter, waits for in-flight observations to land, calls
createResult.get(), and then clears the bit again. The surrounding
finally only releases runLock, so when createResult.get() throws, the
bit is left set permanently.
Two things follow from a bit that stays set:
* append() keeps returning true, so observations go into the buffer
instead of being recorded, and are replayed at an arbitrary later
point.
* The next run() derives expectedCount from counters that still carry
the sign bit, so complete.apply(expectedCount) may never be
satisfied. The wait loop then spins on Thread.yield() indefinitely
while holding runLock, which blocks every subsequent scrape of that
registry. Whether it spins or instead produces a corrupt snapshot
depends on the number of stripes, because the sign-bit contributions
cancel out for an even Runtime.availableProcessors():
2 * Long.MIN_VALUE == 0.
Nothing clears the bit in-process, so the only recovery is a restart.
Move the wait-and-create block into its own try, with the "signal that
the buffer is inactive" step and the buffer drain in the matching
finally. The drain has to move with it: leaving bufferPos set makes the
next run() skip its own wait and replay stale values on top of the new
ones.
Found in production in Alluxio on prometheus-metrics-core 1.0.0, where
an OutOfMemoryError raised inside createResult.get() wedged the /metrics
endpoint of several workers for 4.5 days until they were restarted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Bin Fan <fanbin103@gmail.com>
apc999
requested review from
dhoard,
fstab,
jaydeluca and
zeitlinger
as code owners
September 3, 2026 17:31
Karthik-Chowdary
approved these changes
Sep 4, 2026
Karthik-Chowdary
left a comment
There was a problem hiding this comment.
I walked the exception path through both striped-counter modes. Pairing active-bit clearing and buffer draining in the inner finally restores the invariant even when snapshot construction throws; moving only the bit flip would indeed leave stale bufferPos state for the next run. The regression test is stripe-count independent and directly probes the externally relevant append behavior after the throw. This is a focused fix that does not alter the existing bounded in-flight wait on the success path. LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Buffer.run()sets the buffer-active bit on every striped observation counter, waits forin-flight observations to land, calls
createResult.get(), and then clears the bit again.The surrounding
finallyonly releasesrunLock— the bit-clearing block sits inside thetry, so whencreateResult.get()throws, the bit is left set permanently.https://github.com/prometheus/client_java/blob/main/prometheus-metrics-core/src/main/java/io/prometheus/metrics/core/metrics/Buffer.java#L90-L139
Two things follow from a bit that stays set:
append()keeps returningtrue, so observations go into the buffer instead of beingrecorded, and are replayed at an arbitrary later point.
run()derivesexpectedCountfrom counters that still carry the sign bit, socomplete.apply(expectedCount)may never be satisfied. The wait loop then spins onThread.yield()indefinitely while holdingrunLock, which blocks every subsequentscrape of that registry.
Whether the second case spins or instead produces a corrupt snapshot depends on the number
of stripes, because
2 * Long.MIN_VALUE == 0: the sign-bit contributions cancel out for aneven
Runtime.getRuntime().availableProcessors(). With an even stripe count you get a runwhose buffer semantics are inverted (observations recorded directly rather than buffered,
i.e. a torn snapshot) rather than a hang. With an odd count — including the single
AtomicLongthis code used before striping — the spin is deterministic.Nothing clears the bit in-process, so the only recovery is a process restart.
Fix
Move the wait-and-create block into its own
try, with the "signal that the buffer isinactive" step and the buffer drain in the matching
finally.The drain has to move with it rather than just the bit flip: leaving
bufferPosset makesthe next
run()skip its own wait and replay stale values on top of the new ones. Thearithmetic is unaffected on the exception path —
expectedCountis computed before thethrow, so
expectedBufferSize -= expectedCountstill holds.Test
BufferTest.bufferIsDeactivatedWhenCreateResultThrowsasserts the invariant rather than thesymptom: after
run()returns or throws, the buffer must be inactive, probed directly viaappend(). Usingcount -> trueforcompletekeeps the test independent of the stripecount, so it fails deterministically on any machine.
Verified both ways:
BufferTestExpecting value to be false but was trueprometheus-metrics-coresuiteHow we hit this
Found in production in Alluxio on
prometheus-metrics-core1.0.0. AnOutOfMemoryErrorraised inside
createResult.get()— snapshot creation allocates several arrays per datapoint, so it is a likely place for an allocation to fail — left the bit set and wedged the
/metricsendpoint of several workers for 4.5 days, until they were restarted. Threeseparate data points were poisoned within 12 seconds of each other, consistent with a single
JVM-wide allocation failure rather than a race.
Note that
Buffer.run()catches nothing, so anErrorleaks the bit exactly as an exceptiondoes.
Relation to #2287
#2287 reports two things about this class: unbounded growth of
observationBufferindoAppend(), and theThread.yield()wait loop having no timeout. This PR fixes neither ofthose directly — it removes the error path that makes the second one permanent. Without a
leaked active bit, the wait loop is a bounded spin over in-flight observations; with one, it
never terminates. So the two changes are complementary rather than overlapping, and I have
deliberately left the loop itself alone here to keep this diff reviewable.
Notes
Thread.yield()wait loop is left as-is; this change only ensures a thrownexception cannot turn it into an infinite spin.
-Dtest.java.version=24), sincetest.java.versiondefaults to 25. Main sources are unaffected (
java.version=8). I was not able to runmise run lintlocally, so please let CI arbitrate formatting.