Skip to content

Add resctrl collector - #3797

Draft
rtreffer wants to merge 1 commit into
prometheus:masterfrom
rtreffer:resctrl-collector
Draft

Add resctrl collector#3797
rtreffer wants to merge 1 commit into
prometheus:masterfrom
rtreffer:resctrl-collector

Conversation

@rtreffer

@rtreffer rtreffer commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Expose the Linux resctrl monitoring counters (Intel RDT, AMD PQoS, ARM MPAM): per L3 domain memory bandwidth (mbm_total_bytes, mbm_local_bytes) and last level cache occupancy (llc_occupancy). Parsing is done by the procfs resctrlfs package.

The collector is disabled by default because the resctrl filesystem must be mounted first (mount -t resctrl resctrl /sys/fs/resctrl). The mountpoint is configurable via --collector.resctrl.path.

Counters a CPU does not support (for example mbm_local_bytes on Arm MPAM) or that read "Unavailable" are skipped per sample. The hardware counters wrap; Prometheus treats the wrap as a counter reset, so no correction is applied.

Depends on a procfs release that contains the resctrlfs package (prometheus/procfs#862).

Signed-off-by: René Treffer treffer@measite.de

Reviewers

AI disclosure: This code was written with the help of AI.
I have read through the code and verified that it works

Here is a snippet showing the metrics as exported by node_exporter built against prometheus/procfs#862

# HELP node_resctrl_llc_occupancy_bytes Last level cache bytes occupied in this domain.
# TYPE node_resctrl_llc_occupancy_bytes gauge
node_resctrl_llc_occupancy_bytes{domain="0"} 3.2014336e+07
node_resctrl_llc_occupancy_bytes{domain="1"} 2.3003136e+07
# HELP node_resctrl_memory_bandwidth_bytes_total Bytes moved between the last level cache and memory. scope=total includes remote sockets, scope=local is memory attached to this domain.
# TYPE node_resctrl_memory_bandwidth_bytes_total counter
node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="local"} 9.866260884132983e+18
node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="total"} 1.8399210738624e+13
node_resctrl_memory_bandwidth_bytes_total{domain="1",scope="local"} 5.010254585456994e+18
node_resctrl_memory_bandwidth_bytes_total{domain="1",scope="total"} 4.003923930496e+12

You can use sum by (instance,node) (rate(node_resctrl_memory_bandwidth_bytes_total{scope="total"}[1m])) to get an estimate of the data transferred between memory and CPU.

This is particularly interesting for workloads like CPU inference, in-memory databases or vector databases.

Caveats: The total value is only available if memory access happened. The metric won't be available on every scrape. We could cache the latest value if this is an issue.

Expose the Linux resctrl monitoring counters (Intel RDT, AMD PQoS,
ARM MPAM): per L3 domain memory bandwidth (mbm_total_bytes,
mbm_local_bytes) and last level cache occupancy (llc_occupancy).
Parsing is done by the procfs resctrlfs package.

The collector is disabled by default because the resctrl filesystem
must be mounted first (mount -t resctrl resctrl /sys/fs/resctrl). The
mountpoint is configurable via --collector.resctrl.path.

Counters a CPU does not support (for example mbm_local_bytes on Arm
MPAM) or that read "Unavailable" are skipped per sample. The hardware
counters wrap; Prometheus treats the wrap as a counter reset, so no
correction is applied.

Depends on a procfs release that contains the resctrlfs package.

Signed-off-by: René Treffer <treffer@measite.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant