Skip to content

refactor: report cache usage directly - #4798

Open
lvhan028 wants to merge 1 commit into
InternLM:mainfrom
lvhan028:refactor/cache-usage-metrics
Open

refactor: report cache usage directly#4798
lvhan028 wants to merge 1 commit into
InternLM:mainfrom
lvhan028:refactor/cache-usage-metrics

Conversation

@lvhan028

@lvhan028 lvhan028 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Motivation

ScheduleMetrics.total_blocks and free_blocks expose allocator-specific details to the upper metrics layer.

Having the upper layer derive cache utilization from these fields couples it to a particular allocation strategy. Each engine should instead calculate cache utilization according to its own memory model and report a backend-independent value.

Modification

  • Removed total_blocks and free_blocks from ScheduleMetrics.
  • Changed cache_usage to a concrete float with a default of 0.0.
  • Updated the PyTorch scheduler to calculate and report cache usage directly, including handling zero GPU-cache capacity.

Copilot AI review requested due to automatic review settings July 29, 2026 03:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors scheduler cache reporting to expose cache_usage directly (as a normalized 0.0–1.0 fraction) and removes the older total_blocks/free_blocks fields from schedule-metrics across TurboMind (C++/pybind) and Python.

Changes:

  • Remove total_blocks/free_blocks from TurboMind ScheduleMetrics (struct, logging, and pybind exposure).
  • Compute and expose cache_usage directly in the PyTorch paging scheduler, with added unit coverage including a zero-GPU-blocks edge case.
  • Simplify metrics aggregation to always consume ScheduleMetrics.cache_usage (no fallback derivation).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/pytorch/paging/test_scheduler.py Adds assertions/tests validating schedule_metrics.cache_usage, including num_gpu_blocks=0.
src/turbomind/utils/metrics.h Removes block-count fields from ScheduleMetrics and stops printing them.
src/turbomind/python/bind.cpp Stops exporting total_blocks/free_blocks to Python bindings.
src/turbomind/engine/engine.cc Stops populating removed block-count fields; retains cache_usage.
lmdeploy/turbomind/turbomind.py Stops reading removed TurboMind fields when constructing Python ScheduleMetrics.
lmdeploy/pytorch/paging/scheduler.py Computes cache_usage directly from block-manager utilization.
lmdeploy/metrics/stats.py Uses scheduled_metrics.cache_usage directly for gpu_cache_usage.
lmdeploy/messages.py Updates the shared ScheduleMetrics dataclass to drop block-count fields and make cache_usage non-optional.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@lvhan028
lvhan028 requested review from CUHKSZzxy and lzhangzz July 29, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants