Report the sampled peak memory of the whole process tree - #6348
Conversation
The per-process figures from phpstan#6297 are honest but they do not add up: a forked worker shares its inherited pages with the main process and every sibling copy-on-write, so a sum of per-process peaks counts the same physical page once per process. The only number that adds up is PSS, where the kernel divides each physical page among its sharers. The main process now samples /proc/<pid>/smaps_rollup of itself and every worker twice per second on its event loop and keeps the largest sum. The kernel keeps no PSS high-water mark, so the peak must be sampled; both gaps of sampling understate, never exaggerate. The main process's own late peak - collecting results, saving the result cache, after the loop is gone - is covered by its VmHWM, which is itself a lower bound of the tree's peak. Verified against an external observer polling the same process tree during a cold self-analysis with 20 forked workers: printed 3.67 GB, observed max sum(PSS) 3.66 GB. A sum of RSS would have said 4.83 GB. On platforms without /proc (Windows, macOS) nothing is sampled and the line is not printed. Co-Authored-By: Claude Code
Co-Authored-By: Claude Code
|
This pull request has been marked as ready for review. |
ondrejmirtes
left a comment
There was a problem hiding this comment.
We could skip this altogether for non-Linux systems. Otherwise it repeats to read a non-existent file every .5 seconds.
|
Thanks for the review! I believe the repeated reads are already avoided, but I may have hidden it too well.
I went with a probe instead of If you prefer the intent to be explicit, I can add the Claude |
#6297 replaced the summed fiction with two per-process figures that the OS can confirm. This adds the third number people keep asking for — how much memory the whole run took — without bringing the fiction back:
Why the per-process figures cannot just be added up
A forked worker shares its inherited pages with the main process and every sibling copy-on-write. Each process's RSS counts a shared physical page again, so a sum across the tree multiplies it by the process count — that multiplication is exactly what made the old
Used memoryline explode under forking.The kernel already has the number that adds up: PSS divides every physical page among the processes that share it, so the PSS of the tree summed across its processes is the tree's actual footprint at that instant. That covers the turbo arena too — a
/dev/shmmapping shared by all workers is split among them the same way./proc/<pid>/smaps_rollupserves the per-process total without walking the full smaps listing, and reading it needs no privileges for direct children — spawned workers included.How the peak is taken
The kernel keeps no PSS high-water mark, so the peak has to be sampled: the main process reads every process's rollup twice per second on its existing event loop and keeps the largest sum. Sampling only starts at
-vand above — the only place the line prints — so a quiet run pays nothing.Both gaps of sampling understate, never exaggerate:
getPeakBytes()closes that gap with the main process'sVmHWM, its lifetime RSS high-water mark: a process's RSS never exceeds the tree's physical footprint at the same instant, soVmHWMis itself a lower bound of the tree's peak, and taking the larger of the two bounds only tightens the estimate.On platforms without
/proc(Windows, macOS) nothing is sampled and the line is not printed, same as the cgroup-based CPU detection behaves there.Checked against the OS
Cold self-analysis of this repository, 20 forked workers, with an external observer polling
smaps_rollupof the whole process tree at 5 Hz:Total peak memorySpawn mode (pcntl disabled) prints and validates the same way.
The tracker follows the
SystemResourcespattern — a$filesystemRootconstructor argument lets the tests run against fixture/proctrees; five of them cover the sum, the peak-keeps-the-largest-sample behavior, an exited worker mid-sample, theVmHWMlift, and the no-/procplatform.Co-Authored-By: Claude Code