feat(nativemem): measure allocator overhead instead of assuming a blanket factor by rkennke · Pull Request #759 · DataDog/java-profiler

rkennke · 2026-08-28T11:58:56Z

What

Two new families of counters that replace estimated numbers with measured ones in memory-overhead accounting:

native_mem_chunk_overhead_bytes.<category> — the rounding-plus-header cost the allocator adds on top of the logical bytes the NM_* gauges record, measured per allocation via malloc_usable_size().
malloc_{arena,in_use,free_held,trimmable,mmap}_bytes — process-wide glibc arena state from mallinfo2(), sampled on the JFR flush path.

Why

Reconciling the profiler's counters against process RSS previously required two correction factors borrowed from a different workload: a ×1.17 chunk-overhead multiplier and an unmeasured allowance for free-but-held arena pages. Both are properties of the allocator, not of the profiler — under tcmalloc or jemalloc they behave differently — so they belong in reported counters rather than folded into a "profiler cost" figure.

Overhead is a function of per-allocation size, so no single multiplier can serve categories with different size mixes. Measured:

request	overhead	%
96 B (`MethodMap` node)	16 B	16.67 % ← `×1.17` is right here
6 KB (`SBTable`)	16 B	0.26 %
512 KB (string arena)	16 B	0.003 %
2 MB (large buffer)	4088 B	0.19 %

On a real profiled JVM the difference is material:

category	live MiB	measured overhead	%
`calltrace`	24.534	0.0000	0.00 (mmap-backed)
`native_symbols`	11.882	0.9796	8.24
`dictionary`	4.553	0.0350	0.77
`thread_info`	0.003	0.0004	13.97

Across the malloc-backed categories the blanket factor charges ~3.0 MiB where 1.0 MiB is measured. native_symbols — the largest category — pays 8.24 %, about half what was assumed.

Design notes

Overhead is a separate gauge, not folded into native_mem_live_bytes. Logical bytes stay directly comparable to sizeof() arithmetic; the extra that RSS pays is visible alongside.

The per-chunk header is probed at first use, not hardcoded. The allocator in force is not knowable at compile time: an LD_PRELOAD'd tcmalloc or jemalloc adds no per-object header while __GLIBC__ stays defined, so assuming glibc's 8 bytes would invent overhead per allocation. The probe allocates same-size blocks and takes the smallest positive address stride (stride = usable + header); implausible results fall back to 0, reporting only observable rounding rather than guessing. It finds 8 on glibc and independently reproduces a known case (96 B request → 112 B footprint).

Arena counters are Counters, not NM_* categories. nativeMem.h documents that the categories partition the profiler's own allocations (sum(category) == total); arena slack is mostly other subsystems' chunks stranded by interleaving and is not attributable to any profiler allocation, so adding it as a category would break that invariant. Reported separately, the two bracket the profiler's contribution: lower bound = per-category counters, upper bound = those plus process-wide free-but-held. keepcost is reported apart from fordblks because it separates glibc's trim-threshold retention (reclaimable) from genuine fragmentation (not).

NM_NATIVE_SYMBOLS needed a different mechanism. It is a gauge — setLive(memoryUsage()) over a sum of logical sizes — so no pointer exists at record time. CodeCache::memoryUsage() already walks every symbol name, so it accumulates overhead in that same pass via an out-param. The measurement lives in NativeFunc::nameOverhead() because only NativeFunc knows the real allocation base: name points into the block at offset sizeof(NativeFunc), so querying the allocator with name itself would be undefined.

Cost: mallinfo2() walks every arena taking each arena lock — flush path only, never reachable from the sampling signal handler. malloc_usable_size() is a size-word read on the alloc/free paths that already call into the allocator.

Known limits

Coverage is partial, so the measured overhead is a lower bound. jfr_buffers, liveness and line_tables still record via plain record() and contribute zero. calltrace's zero is correct rather than missing — it is mmap-backed and pays no malloc chunk overhead.
The _blobs array is excluded from native_symbols overhead: it comes from new CodeBlob[], whose returned pointer is not guaranteed to be the allocator's block base. One allocation per library, so excluding it understates negligibly rather than risking an unsound reading.
Arena counters read zero off glibc 2.33+ (musl, macOS). The older int-based mallinfo() silently truncates past 2 GiB, so it is not a usable fallback.

Testing

mallocFootprint_ut.cpp and mallocArenaStats_ut.cpp validate the premises, not just the plumbing — and two of them corrected assumptions made while writing them:

Large-allocation overhead is not a function of request size. The identical 512 KB request measures 4088 bytes of overhead in one allocator state and 16 in another, because glibc's mmap threshold is dynamic and rises as large blocks are freed. This rules out any arithmetic formula over the requested size — including align16(S + 8) — and settles that it must be measured.
Churn alone does not strand arena memory. 26,450 blocks of 6,144 B (the shape of a real JFR chunk flush) allocated then all freed are fully returned to the OS — arena 163 MB back down to 905 KB. Stranding requires live allocations interleaved among the freed ones to pin the region: 117.5 MiB retained, of which malloc_trim reclaims 0.03 MiB.

Sanitizer builds substitute an allocator that reports no usable-size slack and defeats the header probe, so overhead reads 0 there. That is the honest answer for such an allocator, so magnitude assertions skip rather than fail.

Release, debug, ASan and TSan all pass, including codeCache_ut and libraries_ut.

🤖 Generated with Claude Code

…accounting Adds five counters sampled from mallinfo2() on the JFR flush path: malloc_arena_bytes, malloc_in_use_bytes, malloc_free_held_bytes, malloc_trimmable_bytes, malloc_mmap_bytes. Motivation: memory reconciliation against process RSS currently applies a x1.17 "chunk overhead" factor and treats free-but-held arena pages as an unmeasured residual term, both borrowed from a different workload. Both are properties of the ALLOCATOR, not of the profiler -- under tcmalloc or jemalloc they behave differently -- so they belong in reported counters rather than folded into a "profiler cost" figure or estimated by analogy. These are deliberately NOT NM_* categories. nativeMem.h documents that the per-category gauges partition the profiler's own allocations (sum(category) == total, no double counting); free-but-held arena pages are mostly other subsystems' chunks stranded by interleaving and cannot be attributed to any profiler allocation, so adding them as a category would break that invariant. Reported separately, they bracket the profiler's contribution: lower bound = the per-category counters, upper bound = those plus process-wide free-but-held. keepcost is reported separately from fordblks because it distinguishes glibc's trim-threshold retention (reclaimable by malloc_trim) from genuine fragmentation (not reclaimable). Flush path only: mallinfo2() walks every arena taking each arena lock, so it is neither cheap nor async-signal-safe and must never be reached from the sampling signal handler. Once per JFR chunk is negligible. Guarded to glibc 2.33+; the older int-based mallinfo() silently truncates past 2 GiB so it is not a usable fallback, and the counters simply read zero on musl and macOS. The tests validate the premise rather than the plumbing, and correct an assumption I held while writing them: - Contiguous churn of the shape a real flush produces (26,450 blocks of 6,144 B = sizeof(SBTable), from StringDictionaryBuffer::insert_with_id) allocated and then all freed is FULLY RETURNED to the OS -- arena 163 MB back down to 905 KB. Churn alone strands nothing. - Stranding requires live allocations INTERLEAVED among the freed ones, which pin the region so free chunks cannot coalesce to the arena top. Measured: 117.5 MiB retained, of which malloc_trim reclaims 0.03 MiB. That distinction matters for interpreting flush-time memory steps: a burst is not self-evidently a stranding mechanism, and whether it strands depends on what else is live at the time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… assuming it Adds native_mem_chunk_overhead_bytes.<category>: the rounding-plus-header cost that RSS pays on top of the logical bytes the NM_* gauges record. Measured from malloc_usable_size() at each allocation, kept in a separate gauge so native_mem_live_bytes stays directly comparable to sizeof() arithmetic. This replaces a blanket x1.17 multiplier applied during RSS reconciliation. That factor was measured on one workload (~650,000 live allocations averaging ~83 B) and then applied to categories with completely different size mixes. Overhead is a function of PER-ALLOCATION size, so no single factor can serve them: 96 B (MethodMap node) overhead 16 B 16.67 % <- x1.17 is right here 6 KB (SBTable) overhead 16 B 0.26 % 512 KB (string arena) overhead 16 B 0.003 % (arena-served) 2 MB (large buffer) overhead 4088 B 0.19 % Charging the 512 KB string-arena chunks and the JFR buffers 17 % over-credits the explained total by over 1 MiB against a residual of only a few MiB. The header size is probed at first use rather than hardcoded, because the allocator in force is not knowable at compile time: an LD_PRELOAD'd tcmalloc or jemalloc adds no per-object header while __GLIBC__ stays defined, so assuming glibc's 8 bytes would invent overhead that does not exist. The probe allocates same-size blocks and takes the smallest positive address stride (stride = usable + header); implausible results fall back to 0, reporting only the rounding we can see rather than guessing. Wired into CountingAllocator (all STL-node categories) and StringArena's chunk alloc/free. Call sites still using plain record() contribute zero here, which is why the gauge is additive rather than a correction to the live figure -- notably NM_NATIVE_SYMBOLS, which is a computed sum of logical sizes with no pointer available at record time and needs a different approach. Two findings from the tests, both correcting assumptions I had made: - The 512 KB chunk does NOT pay a negligible fixed 16 bytes as predicted. The identical request measures 4088 bytes of overhead in one allocator state and 16 in another, because glibc's mmap threshold is DYNAMIC and rises as large blocks are freed. Large-allocation overhead is therefore not a function of request size at all, which rules out any arithmetic formula -- including the align16(S + 8) approach originally proposed -- and settles that it must be measured. - Sanitizer builds substitute an allocator that reports no usable-size slack and defeats the header probe, so overhead reads 0. That is the honest answer for such an allocator, so the magnitude assertions skip rather than fail. Release, debug, ASan and TSan all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Extends the measured chunk-overhead accounting to the largest category, which the previous change could not cover. NM_NATIVE_SYMBOLS is a gauge -- setLive(native_libs.memoryUsage()) -- computed as a sum of logical sizes, so there is no pointer in hand at record time and recordAlloc() does not apply. CodeCache::memoryUsage() already walks every symbol name, so it now optionally accumulates the measured overhead in that same pass via an out-param, and Profiler publishes it with a new gauge-style NativeMem::setOverhead(). Single pass, no extra walk. NativeFunc::nameOverhead() is where the measurement lives because only NativeFunc knows the real allocation base: `name` points *into* the block at offset sizeof(NativeFunc), so querying the allocator with `name` itself would be undefined. It sits next to allocSize(), which the header already documents as the single source of truth for that allocation's size. The _blobs array is deliberately excluded: one allocation per library, large enough that its overhead is a rounding error, and it comes from new CodeBlob[] whose returned pointer is not guaranteed to be the allocator's block base. Excluding it understates by a negligible amount rather than risking an unsound reading. Measured on a real profiled JVM (classes sweep, 60k classes): category live MiB overhead MiB overhead % calltrace 24.534 0.0000 0.00 (mmap-backed) native_symbols 11.882 0.9796 8.24 dictionary 4.553 0.0350 0.77 thread_info 0.003 0.0004 13.97 native_symbols measures 8.24 %, roughly half the 17 % the blanket factor assumed, and it is the largest category -- so this is where that factor did the most damage. Across the malloc-backed categories the blanket x1.17 would charge about 3.0 MiB where 1.0 MiB is measured, over-crediting the explained total by ~2 MiB against a residual of only a few MiB. Correcting it therefore WIDENS the reconciliation gap, which is the direction predicted. Coverage is still partial and the measured figure is a lower bound: jfr_buffers, liveness and line_tables record through plain record() and contribute zero here. calltrace's zero is correct rather than missing -- it is mmap-backed and pays no malloc chunk overhead at all. Release, debug, ASan and TSan pass, including codeCache_ut and libraries_ut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

dd-octo-sts · 2026-08-28T12:01:45Z

Scan-Build Report

User:	runner@runnervmgx7h7
Working Directory:	/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:	make -j4 all
Clang Version:	Ubuntu clang version 18.1.3 (1ubuntu1)
Date:	Fri Aug 28 12:00:45 2026

Bug Summary

Logic error
Bug Type	Quantity	Display?
All Bugs	1
Dereference of null pointer	1

Reports

Bug Group	Bug Type ▾	File	Function/Method	Line	Path Length
Logic error	Dereference of null pointer	faultInjection.cpp	crashNow	24	2

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec7bfbd2b9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-28T12:03:53Z

+void Recording::updateMallocArenaStats() {
+#ifdef DD_HAVE_MALLINFO2
+  struct mallinfo2 mi = mallinfo2();


Avoid glibc stats when a replacement allocator is active

When the process uses LD_PRELOAD with tcmalloc or jemalloc—as the repository's reliability jobs do—__GLIBC__ remains defined, so this branch still calls glibc's mallinfo2(). That function reports glibc's internal arenas rather than allocations redirected to the replacement allocator, causing the five new process-wide counters to contain zero or unrelated glibc state precisely during allocator-comparison experiments. Detect the active allocator and use its statistics API, or mark these counters unavailable outside glibc malloc.

Useful? React with 👍 / 👎.

datadog-datadog-prod-us1 · 2026-08-28T12:06:09Z

Tests

🔄 Datadog auto-retried 1 job - 1 passed on retry

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ec7bfbd | Docs | View more details | Give us feedback!}

dd-octo-sts · 2026-08-28T12:17:40Z

CI Test Results

Run: #33169229844 | Commit: 3d5691b | Duration: 15m 26s (longest job)

❌ 13 of 32 test jobs failed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	❌	-	-
8-ibm	-	❌	-	-
8-j9	✅	❌	-	-
8-librca	-	-	✅	✅
8-orcl	-	❌	-	-
11	-	❌	-	-
11-j9	✅	❌	-	-
11-librca	-	-	✅	✅
17	✅	❌	-	-
17-graal	✅	❌	-	-
17-j9	✅	❌	-	-
17-librca	-	-	✅	✅
21	✅	❌	-	-
21-graal	✅	❌	-	-
21-librca	-	-	✅	✅
25	✅	❌	-	-
25-graal	✅	❌	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled