Skip to content

fix: alert on RDMA hard receive budget - #365

Merged
ketor merged 1 commit into
mainfrom
fix/rdma-receive-hard-budget-alert
Aug 31, 2026
Merged

fix: alert on RDMA hard receive budget#365
ketor merged 1 commit into
mainfrom
fix/rdma-receive-hard-budget-alert

Conversation

@ketor

@ketor ketor commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Problem

DfkvRdmaReceiveSegmentLow divides unleased bytes by currently committed lazy chunks. Healthy pools therefore fire whenever a committed chunk is >90% occupied, even with most of the hard budget still available.

INC #C5ECD7 observed 6.39%-9.38% committed-chunk free while the affected xb01 ring retained 89.78%-92.54% hard-budget headroom, with zero growth/allocation failures.

Change

  • alert on (max_bytes - used_bytes) / max_bytes
  • align the Grafana panel with hard-budget headroom
  • add a regression contract for the alert and dashboard expressions

Verification

  • python3 test/python/test_observability_contract.py (5 tests, OK)
  • promtool check rules against the hotfixed xb01 rule file (26 rules, SUCCESS)
  • live rule state changed from firing/8 active to inactive/0 active

@ketor
ketor added this pull request to the merge queue Aug 31, 2026
Merged via the queue into main with commit ba7b2ec Aug 31, 2026
8 checks passed

@yuhaijun999 yuhaijun999 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants