memory: most live Rust heap has no resident owner; swap excluded

[ScriptedAlchemy]

Summary

Most of the daemon's live Rust heap has no resident owner, so the resident-memory authority charges a fraction of what it measures. Separately, the admission sample excludes swapped anonymous pages.

Evidence

Operator daemon, beta.59, 12:40 to 13:10 UTC, read-only. Sources: /proc/<pid>/smaps and tracedecay tool status --format json.

Region	Resident	Swapped
mimalloc arenas (Rust heap)	6.60 GiB	3.64 GiB
glibc arena heaps (C libraries; fixed separately, see the linked PR)	5.54 GiB	3.76 GiB

• memory.retained_bytes (sum of charged owners): 3,550,304,770
• memory.resident_bytes: 17,516,769,280
• VmSwap: 8,387,836 kB. It is not counted in the admission sample (unreclaimable_bytes = RssAnon + RssShmem).
• 4 registered projects, 1 worktree each. No linked-worktree duplication.

Isolated reproduction. Debug production,hotpath build, MemoryMax=6G, MemorySwapMax=1G. Corpus: depth-1 clone of this repository (5,747 files). Cold index, then settle.

• Charged owners: graph_catalog 400,257,698 plus graph_engine 665,139,632 (1,065,397,330 total).
• mimalloc arenas: 2,530 MiB resident plus 478 MiB swapped.
• mi_collect(true) on all 113 threads (gdb, scratch daemon only) released nothing, so this is live or fragmented heap, not freed-but-retained pages.
• Symptom: every refresh after a commit is refused: code-index worker resident-memory admission refused: ... observed_rss=3234574336 ... high_watermark=5798205849 requested=4429185024. The index stays warming indefinitely. That matches the operator's tracedecay project, which sat at rebuild_in_flight=true / code_graph_serving=pending for hours.

Expected

Retained Rust state that the daemon holds after a cold index (about 1.5 GB uncharged on this corpus) is registered as a resident owner, so it can be shed or at least attributed. Admission accounting also decides explicitly whether swapped anonymous pages count.

Not done

Attribution of the uncharged Rust heap. The Hotpath alloc lane reports allocated bytes per function, not live retention, so this needs a live-heap profile (for example dhat on a system-allocator build).

[ScriptedAlchemy]

#2592 (0e4a5ab) removed about 486 MB of uncharged live heap on this issue's journey: retained parse trees, per-thread credential regex caches, and duplicated catalog copies. Numbers are in #2525. The production journey gap above charged owners fell from 1,601,212,999 B to 1,076,560,455 B. Leaving this open: the fixed runtime allowance and the RSS-within-owners-plus-allowance test are not done, because the remaining gap is mostly allocator retention and in-flight graph activation (#2505), not fixed live state. See #2525 for the breakdown.

[ScriptedAlchemy]

Measurement for the fixed-allowance / RSS-invariant step (also #2525). Not closing either: the invariant does not hold yet, for the reasons below.

Setup: master 0e4a5ab, release production,hotpath with mimalloc v3, plus a measurement-only rebuild of the same tree with CFLAGS_x86_64_unknown_linux_gnu=-DMI_STAT=1 so mimalloc tracks live blocks. systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=1G, isolated profile, full clone of this repository (5,845 indexed files). Cold index, then settle. Stats were read from heap_main.stats via gdb after forcing mi_collect(true) on every thread (offsets from the vendored v3 headers).

Allocator retention vs live state (graph serving, settled):

• live blocks: malloc_normal 1,973,118,704 + malloc_huge 461,373,440 = 2,434,492,144
• arena footprint: 2,493 MiB resident + 271 MiB swapped. So fragmentation plus retention is only ~0.34 GB.
• mi_collect(true) on all 109 threads released 9.8 MB. 22,963 of 28,002 in-use pages were abandoned by exited threads.
• charged owners: graph_catalog 420,598,318 + graph_engine 688,456,608 = 1,109,054,926

Attribution by releasing and re-warming owners one at a time (idle release, then find_exact_symbol warms the engine and then the catalog):

component	charged	live	uncharged
baseline, no owners resident	0	636,748,848	636.7 MB
graph_engine	688,456,608	~1,086,979,000	~398.5 MB (1.58×)
graph_catalog	420,598,318	~712,723,976	~292.1 MB (1.69×)

The re-warmed total, 2,436,451,824, matches the settled 2,434,492,144.

What this means for the allowance:

1. The largest uncharged heap is the two graph owners under-reporting themselves by ~690 MB, not per-worktree scratch. graph_engine is charged from grafeo's memory_usage().total_bytes plus the layered store's memory_bytes(), and graph_catalog from retained_bytes(). That gap scales with the corpus, so a fixed allowance cannot absorb it. Truthful owner bytes come first.
2. The 636.7 MB no-owner baseline was measured on a hotpath build (Hotpath's own state is in it). It needs a production-build breakdown before a constant is derived. perf(memory): stop retaining scan caches, parse trees, catalogs #2592's heaptrack categories (catalogs 67.8 MB, serializer 33.1 MB, SQLite 22.6 MB, text generation 12.8 MB, rules 7.0 MB, source pool 102 MB) account for ~248 MB of it.

The refresh starvation this issue first reported had a separate cause. Each build's own worker slab was admitted before its parent decode and counted against it. That is fixed in #2617 (98c1eba), with the journey showing a refresh publish generation 2 and serve callers from its graph under the 6 GiB cap.

[ScriptedAlchemy]

Graph owner charges are now truthful (#2643, 6bc1138bab, with ScriptedAlchemy/grafeo#6). Leaving this open: the runtime allowance and the RSS-within-owners-plus-allowance test are not done, because what remains is not fixed.

Owner charges (store-runtime contract fixture, per-thread counting allocator; release-freed bytes are the reference):

• master: graph_catalog 3,803,041 charged / 6,104,442 freed; graph_engine 5,404,974 / 8,809,711; 742 KB left live after both releases (store-epoch caches never freed on hibernation).
• fix(memory): charge the serving graph owners their real heap #2643: catalog 4,961,474 / 4,961,474; engine 8,850,428 / 9,551,951; the releases return everything warming left live.

Full clone, release production build, 6 GiB scope, settled:

	engine	catalog	RssAnon + swap	charged
master cold	688,195,387	424,266,955	2,827,080 kB	1,112,462,342
#2643 cold	1,120,573,432	548,046,966	3,096,424 kB	1,668,620,398
#2643 refresh	1,120,573,432	548,046,966	2,575,328 kB	1,668,620,398

What remains is proportional, so it is not an allowance:

• The unattributed gap (anonymous minus charged) at a settled cold index on the same fix(memory): charge the serving graph owners their real heap #2643 build was 775,735,767 B on a 1,942-file corpus and 1,069,108,949 B on the 5,846-file clone. That is about 630 MB fixed plus about 75 KB per indexed file.
• smaps at the settled refresh: [anon:mimalloc] held 2,493,992 kB, and all other anonymous memory (glibc arenas for C libraries, stacks) was 81,336 kB. So the gap is Rust heap: live state without an owner plus mimalloc fragmentation.
• No-owner baseline after both graph owners idle-released: RssAnon 996,916 kB + swap 279,244 kB, with retained_bytes 0.

Next step: get a code-level live-heap attribution of that Rust heap, then charge the proportional owners it names. heaptrack on a system-allocator build was OOM-killed at 6 GiB on the full clone. A 2,422-file run produced 4.9 GB of raw trace whose interpretation exceeded 70 GB. A smaller corpus, or an MI_STAT build with a per-thread mi_stats_merge while releasing candidate state, is the practical path.

[ScriptedAlchemy]

#2661 (154a0eb) removes the largest uncharged parts of the gap. I'm leaving this open because what remains still grows with the corpus, so it isn't a fixed allowance yet.

Attribution. heaptrack was run in --raw mode on a system-allocator release build. heaptrack_interpret was OOM-killed at 16 GB on the 980 MB binary, so the raw event stream was read by a streaming reader instead (peak 1.9 GB). Separately, gdb called mi_collect(true) on each thread of the production build in turn and attributed the RssAnon it returned to that thread. Corpora were every 20th and every 5th tracked file of this repo (296 and 1,168 indexed files), plus the full clone (5,846 files).

• Fixed part, removed. Idle tracedecay-index-* workers held cross-thread-freed heap that mimalloc returns only when the owner allocates or collects. At 296 files with 36 workers that was 233,021,440 B, about 6.5 MB per worker. The idle pool now collects each worker once after a fan-out.
• Per-file part, removed. retained_snapshot_bytes kept each file's sanitized source after sealing, and nothing read it: 16,084 B per file.
• Graph owner charges are truthful. Live bytes are within 1% of graph_engine and graph_catalog.

Gap: anon + swap minus charged owners, settled cold index. Release production builds, 6 GiB scope.

indexed files	master	#2661
296	525,962,184	327,744,456
1,168	744,725,920	392,179,104
5,846	1,019,875,065	751,152,889

After a refresh, the gap at 296 / 1,168 files was 431,581,443 / 623,558,096 on master and 400,877,827 / 531,533,264 on #2661.

What remains is about 306 MB + 75 KB per file on this 96-core host:

• Per file:
  • about 43 KB of mimalloc pages with no live data. Partially used pages under the owners' data survive a full per-thread collect: at 296 files an MI_STAT build had 290 MB of arena resident against 197 MB live.
  • about 16.5 KB of glibc heap (SQLite, tree-sitter).
  • about 6 KB of thread-local canonical-serializer pools.
• Fixed:
  • tool/MCP/SDK catalogs, 67 MB live
  • SQLite, about 21 MB
  • tokio-thread retention, 21 to 28 MB
• After a refresh, the increment's retained parse pool is uncharged. For 38 re-extracted files it held 12.2 MB of tree-sitter parser stacks plus about 6 MB of extraction artifacts.

Next: the fragmentation under owner data and the retained parse pool are proportional. They need an allocation-pattern fix or a charged owner with a structural bound before a fixed allowance and the RSS-within-owners-plus-allowance test can be added.

[ScriptedAlchemy]

#2727 (f5b938b) gives back or charges most of what was left after #2661. Leaving this open: the fixed allowance and the RSS-within-owners-plus-allowance test are still not added, because the invariant does not hold after a refresh. Details below and in #2706.

Changes: idle tokio and SQLite worker threads now return their heap. Threads drop their canonical-serializer pools when they idle. Retained documents no longer keep a tree-sitter parser. The graph engine and catalog are each built in their own mimalloc heap, charged by that heap's pages and deleted whole on release.

Gap = anon + swap - charged owners (release production builds, 6 GiB scope, isolated profile, 96-core host):

indexed files	master cold	#2727 cold	master refresh	#2727 refresh
293	306,152,533	212,824,510	405,674,469	279,094,351
1,164	396,689,019	266,709,683	525,367,467	318,949,300
5,855	716,847,568	298,635,422	refused, index stale	published

With the owners resident, the gap now levels off: only about 7 KB per file is added between 1,164 and 5,855 files. On the full clone, master refused the refresh for memory and never retried; that bug is #2707.

Why this stays open: a refresh leaves the increment's retained parse pool and build residue uncharged. On 300 heavy synthetic files that was 197 MB (gap 178.8 MB cold, 375.9 MB after refresh), so no fixed allowance holds across corpora. With both owners idle-released, 227 / 295 / 435 MB stayed resident at 293 / 1,164 / 5,855 files. #2706 tracks both, with the numbers.

[ScriptedAlchemy]

Update after #2771, #2773, #2775 and #2778. Leaving this and #2525 open, and the allowance test is not added: the gap is now flat across corpus size with the owners resident or released, but a heavy-file refresh still adds about 210 MB (details in #2706).

Gap = anon + swap − charged owners. Debug production (mimalloc) build, 6 GiB scope, isolated profile, settled 120 s:

corpus	owners resident	owners idle-released	after refresh
336 files (every 20th)	174.5 MB	167.1 MB
1,345 files (every 5th)	154.1 MB	177.9 MB
300 heavy Rust files × 160 fns	156.4 MB		366.0 MB (retained_parses 660.6 MB charged)

Changes

• fix(memory): allocate SQLite and tree-sitter from mimalloc #2773 (c36fa64): SQLite and tree-sitter allocate from mimalloc. glibc in-use bytes went from 25.3 MB cold / 52.1 MB after a refresh to 0.44 MB. The 30 s malloc_trim from fix(memory): trim glibc arenas that the mimalloc release skipped #2491 is deleted.
• fix(memory): charge and release the retained parse pool #2775 (aa1e2d5): the retained parse pool is a measured, first-shed resident owner.
• fix(code-index): free captured sources once a pass lets them go #2778 (4e3e1d6): dead weak cache entries no longer pin captured sources and artifact headers, removing about 26 KB per file of no-owner residue.
• fix(memory): wake refused work on the release it waits on #2771 (7b364c8, fixes code-index: a memory-refused reconcile never retries after an idle release #2707): every memory refusal registers with the admission authority, and owner releases, ledger releases and fitting samples all wake it.

#2527's measured-RSS worker slab sizing. Not changed. The ledger holds only in-flight reservations. Owners are charged in the separate inventory, and 155–180 MB stays uncharged at rest, plus the refresh residue. Sizing from the ledger alone would plan slabs that measured admission then refuses. Removing the measured path needs that remainder charged, or a fixed allowance, which is this issue's remaining step.

[ScriptedAlchemy]

Progress from #2784 (merged a29f123) and #2785 (merged 21351ba). This issue stays open: the acceptance (anon + swap within charged owners plus one fixed allowance) is not met yet.

Fixed: refresh residue. Pages held after a refresh came from indexing workers, pinned by a few long-lived blocks. The causes were:

• the retained parse pool allocating its extraction, identity and bookkeeping on the calling worker, outside its heap;
• per-thread glob lazy-DFA caches;
• workers that collected only once per fan-out, while the build, the serving swap and the owner release free rows later.

All three are fixed. Isolated daemon, 6 GiB scope, debug mimalloc build. Gap = anon + swap − charged owners.

corpus	cold gap	refresh gap (20 edits; heavy: all 300)	no owners (released)
heavy 300×160 fns, master	155–162 MB	361–532 MB	282–434 MB
heavy, now	146 MB	181 MB	186 MB
335 files	180 MB	220 MB	198 MB
1,340 files	213 MB	208 MB	207 MB
full clone, 6,703 files	135 MB	134 MB	248 MB

An increment now adds 0–40 MB over cold; before, it added 50–200 MB. The no-owner residual fell from 227 / 295 / 435 MB to 198 / 207 / 248 MB, with growth down from about 34–45 KB per file to about 8 KB per file.

One sizing authority (#2785). Slab, decode and text-build sizing all read ProcessResidentMemoryV1::headroom_below: the ceiling minus the larger of the ledger and the measured process. The separate measured-RSS sizing path is deleted. Measured RSS remains that view's single input for the uncharged runtime, because that part is not yet fixed.

What remains before the allowance test:

• The no-owner residual still grows with the corpus (about 8 KB per file).
• The cold gap swings 135–213 MB between corpora: on the full clone the graph owners' charges come close to the measured anon, while on 1,340 files they fall short.

A fixed allowance would have to cover about 250 MB and still grow, so I have not added the #2490 allowance test.

[ScriptedAlchemy]

This issue's ask, that most of the live heap has a resident owner, now holds: after the graph owners idle-release, uncharged memory is 84–106 MiB against a 6 GiB limit, and every owner's charge is its resident pages.

Closing on the measured evidence. The fixes are #2784, #2785, #2790 (962897f012) and #2803 (2b0c4d9b7a).

What changed

• Owner heaps are charged their resident pages (mincore on Unix, QueryWorkingSetEx on Windows), not their block capacity.
• Text-artifact admission no longer counts the unmodeled live set twice. That double count caused 2–6 refusals per heavy run on master.
• Leaving an owner-heap scope collects the calling thread's theap. Pages emptied by cross-thread frees are no longer left charged to the pool.

12-refresh heavy-corpus run (300 files × 160 functions, every function edited each refresh, 6 GiB cgroup, MemorySwapMax=1G). Each refresh is measured after the superseded generation has retired, as the minimum over a 60 s window, in MiB:

refresh	3	4	5	6	7	8	9	10	11	12
retained-pool charge, master	508	500	500	501	509	503	498	498	504	508
retained-pool charge, fixed	497	498	512	427*	502	487	507	504	505	508
uncharged, fixed (RssAnon − charged)	139	140	193	131	95	84	86	92	106	92

* The kernel reclaimed part of the pool's pages, and the live residency measurement followed it.

Pool heap internals (fixed run)

• Live bytes are flat at 422 MiB (429 MiB on master, where blocks freed cross-thread were never collected).
• Page capacity is 477–482 MiB.
• Reserved is 713–719 MiB.

Graph owners: charges measured live match their build-time charge (catalog 142 vs 142 MiB, engine 95–101 vs 95–99 MiB). Idle release of the graph owners lowers uncharged memory rather than raising it. Nothing is refused, and nothing crosses the 6 GiB limit or VmHWM. refresh_memory_journey asserts the no-refusal and cap properties and fails on the pre-#2790 code.

Why there is no per-refresh residue assertion

• The "creep" seen earlier (+48 → +65 MiB) came from sampling at a fixed delay. At that point the superseded or decoded generation, or the graph owners, were still resident: refresh-2 samples taken that way read about +100 MiB.
• Measured once those transients retire, master and fixed both plateau by refresh 3.
• An assertion that master fails therefore doesn't exist: there is no growth left to detect.
• The remaining run-to-run spread of uncharged memory (84–106 MiB after the graph owners released, single samples up to +50 MiB) is main-heap residency noise, not growth.