v13 gave the AVERAGED kv-cache series a real per-engine identity. The summed
series never got the same treatment: prefill/decode/prefix-hit rates, queue
depth, host KV usage and the prompt-token source breakdown still grouped on an
exact `start_ns`, so each output tick carried whichever components happened to
share that nanosecond rather than the cluster total.
Components of one metric are not scraped in lockstep. Two shapes produce it:
1. Disaggregated runs put every worker on its own /metrics endpoint and its own
sub-second offset, so an N-worker run emitted N partial-sum points per tick.
2. SGLang splits its token counters by `is_streaming`. On a fully streaming
benchmark the "false" series is a full-length run of zeros on a grid ~16 ms
off the real one.
Downstream this reads as a comb. `rollingAverage` is a sample-count mean, so it
averaged real samples together with the other components' structural gaps and
drew a fraction of the actual throughput — measured 2.2x low on the 8-rank
SGLang DP row (point 439817) and ~7x low on the 7-worker disaggregated rows.
223 of 599 stored rows carry a fragmented prefill series.
Every summed series now goes through `sumOntoGrid`: one canonical grid at the
blob's native cadence (`canonicalTickNs`), each component holding its last
sample between its own scrapes and contributing only inside its observed
window. The grid must be uniform rather than the union of scrape instants the
way `averageAcrossEngines` does it — a mean is scale-free but a sum is not, and
`cumulativeUniqueInputTokens` turns these rates into token totals with
`sum += value`, which only stays correct at one point per scrape bucket.
The lattice is anchored at t=0 and shared across metrics so the pairs that are
divided or added downstream (hits/queries, used/total, running+waiting) land on
identical `t`. Anchoring per-metric instead emptied the prefix-cache-hit-rate
chart outright, because sglang:cached_tokens starts ~0.18 s off the grid
sglang:prompt_tokens starts on.
Mirrored API-server endpoints are collapsed on the summed path too, using a
relative tolerance — a throughput mean is O(10^5), so v13's absolute gauge
threshold would never fire. That is a correction as well as a de-duplication:
the two-API-server vLLM rows were double-counting queue depth and token totals
exactly 2x.
Measured on real blobs (stored v12 -> v14):
439817 sglang/b200 DP8 decode mean 393 -> 859 tok/s; token totals byte-identical
439201 vllm/b300 DP8 queue depth sum 25,680 -> 12,831 (mirror double-count)
439263 vllm/b200 DP4 queue depth sum 155,011 -> 77,507 (mirror double-count)
`aggregateByStart` is gone; nothing groups on an exact `start_ns` any more.
CHART_SERIES_VERSION 13 -> 14. Run `bun run --cwd packages/db
db:backfill-chart-series` after merge — until then the API serves these points
through the slower recompute path.
中文：v13 只为「取平均」的 KV 缓存序列建立了真正的按引擎身份，「求和」类序列
（prefill/decode/前缀缓存命中速率、队列深度、host KV 使用率、prompt token 来源
拆分）仍按精确的 start_ns 分组。同一指标的各个组件并非同步采样：分离式部署下每
个 worker 有各自的 /metrics 端点与亚秒级偏移；SGLang 还会按 is_streaming 把 token
计数器拆成两条序列，全流式压测时 "false" 那条是一整串零、且与真实网格相差约 16 ms。
结果是每个输出点只携带恰好落在该纳秒上的那部分组件，前端的 rollingAverage 又是按
样本计数取平均，于是把真实样本和其它组件的结构性空档一起平均，画出来只有实际吞吐量
的一小部分——在 8 路 DP 的 SGLang 行（439817）上偏低 2.2 倍，在 7 worker 的分离式
行上偏低约 7 倍；599 行已存数据中有 223 行的 prefill 序列存在此问题。
现在所有求和序列都经由 sumOntoGrid：以 blob 原生采样周期构建统一网格
（canonicalTickNs），每个组件在自身两次采样之间保持上一个取值，且只在自己的观测
窗口内参与求和。网格必须是均匀的，不能像 averageAcrossEngines 那样取采样时刻的并
集——平均值与尺度无关，求和则不然，而 cumulativeUniqueInputTokens 是用
sum += value 把速率累加成 token 总量的，只有「每个采样桶一个点」时才成立。网格锚定
在 t=0 并被所有指标共享，这样下游需要相除或相加的成对指标才能落在相同的 t 上；否则
前缀缓存命中率图会直接变空。求和路径同样合并镜像端点，但改用相对容差（吞吐量均值量
级为 10^5，v13 的绝对阈值永远不会触发）——这同时修正了双 API server 的 vLLM 行此前
队列深度与 token 总量正好翻倍的问题。aggregateByStart 已删除。
CHART_SERIES_VERSION 13 → 14，合并后需运行 db:backfill-chart-series。
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
v13 (#710) gave the averaged KV series a real per-engine identity. The summed series never got the same treatment — prefill/decode/prefix-hit rates, queue depth, host KV usage and the prompt-token source breakdown still grouped on an exact
start_ns, so each output tick carried whichever components happened to share that nanosecond instead of the cluster total.223 of 599 stored rows carry a fragmented prefill series, at dilution ratios up to 6×.
Root cause
Components of one metric are not scraped in lockstep. Two shapes produce it:
/metricsendpoint and its own sub-second offset, so an N-worker run emitted N partial-sum points per tick.is_streaminglabel splits. SGLang splits its token counters intois_streaming="true"/"false". On a fully streaming benchmark the"false"series is a full-length run of zeros sitting on a grid ~16 ms off the real one — on point 439817 that is 4,487 structural zeros interleaved with 3,642 real samples.Downstream this reads as a comb.
rollingAverageis a sample-count mean, so it averaged the real samples together with the other components' structural gaps and drew a fraction of the actual throughput.Fix
Every summed series now goes through
sumOntoGrid: one canonical grid at the blob's native cadence (canonicalTickNs), with each component holding its last sample between its own scrapes and contributing only inside its own observed window.The grid has to be uniform, not the union of scrape instants the way
averageAcrossEnginesdoes it. A mean is scale-free; a sum is not.cumulativeUniqueInputTokensturns these rates into token totals withsum += value, so the emitted series only stays correct at one point per scrape bucket — evaluating on the union would have doubled every cumulative token count.The lattice is anchored at
t=0and shared across metrics, so the pairs that get divided or added downstream (hits/queries, used/total, running+waiting) land on identicaltvalues. Anchoring each metric at its own first sample instead emptied the prefix-cache-hit-rate chart outright, becausesglang:cached_tokensstarts ~0.18 s off the gridsglang:prompt_tokensstarts on — there is a regression test for exactly that.Mirrored API-server endpoints are collapsed on the summed path too, using a relative tolerance: a throughput mean is O(10⁵), so v13's absolute gauge threshold would never fire and every mirror would be counted twice. That turned out to be a correction as well as a de-duplication — the two-API-server vLLM rows were double-counting queue depth and token totals exactly 2×.
aggregateByStartis deleted. Nothing groups on an exactstart_nsany more.Verified against real blobs (stored v12 → v14)
DP 0 DP 0 DP 1 DP 1 …)prefixCacheHitRateThe two mirror rows were confirmed by inspecting the raw blob:
localhost:8888andlocalhost:8889both report the identicalengine=0..7label set with matching per-engine means, 19 ms apart. The halving is de-duplication, not data loss.Token totals are preserved exactly on aligned components; an off-grid component can differ by at most its one trailing bucket past the last whole tick, which is invisible on a 4,000-tick row (439817 came out byte-identical).
Tests
Eight new cases in
compute-chart-series.test.tscovering offset scrape grids, the all-zerois_streamingsplit, token-total preservation (aligned and off-grid), mirror de-duplication, disagreeing endpoints that must not fuse, the shared-lattice hit-rate regression, multi-worker queue depth, and the single-component passthrough. Full suite green (477 + 25).No frontend change and no user-visible strings, so no
/zhsibling applies. The agentic detail page is addressed by benchmark id and has no?unofficialrun=overlay path.Deploy note
CHART_SERIES_VERSION13 → 14. Runbun run --cwd packages/db db:backfill-chart-seriesafter merge. Until then the API serves affected points through the slower recompute path, so the fix is live immediately either way.中文说明
v13（#710）为「取平均」的 KV 序列建立了真正的按引擎身份，但「求和」类序列——prefill/decode/前缀缓存命中速率、队列深度、host KV 使用率、prompt token 来源拆分——仍按精确的
start_ns分组，因此每个输出点只携带恰好落在该纳秒上的那部分组件，而不是整个集群的合计值。599 行已存数据中有 223 行 的 prefill 序列存在此问题，稀释比例最高达 6 倍。根因：同一指标的各组件并非同步采样。其一，分离式（disaggregated）部署下每个 worker 都有各自的
/metrics端点和亚秒级偏移，N 个 worker 每个时间点就会产生 N 个「部分和」数据点；其二，SGLang 按is_streaming把 token 计数器拆成两条序列，在全流式压测中"false"那条是一整串零，且所在网格与真实网格相差约 16 ms（在 439817 上即 4,487 个结构性零与 3,642 个真实样本交错）。前端的rollingAverage是按样本计数取平均，于是把真实样本和其它组件的结构性空档一并平均，画出的吞吐量只有实际值的一小部分。修复：所有求和序列改为经由
sumOntoGrid处理——以 blob 原生采样周期构建统一网格（canonicalTickNs），每个组件在自身两次采样之间保持上一个取值，且仅在自己的观测窗口内参与求和。网格必须均匀，不能像averageAcrossEngines那样取采样时刻的并集：平均值与尺度无关，求和则不然，而cumulativeUniqueInputTokens是用sum += value把速率累加成 token 总量的，只有「每个采样桶一个点」时才成立，若按并集求值会使所有累计 token 数翻倍。网格锚定在
t=0并由所有指标共享，这样下游需要相除或相加的成对指标（命中数/查询数、已用/总量、running+waiting）才能落在相同的t上；若按各指标自身的首个样本锚定，前缀缓存命中率图会直接变空（sglang:cached_tokens的起点与sglang:prompt_tokens相差约 0.18 s），已针对该情况补充回归测试。求和路径同样合并镜像端点，但改用相对容差：吞吐量均值量级为 10⁵，v13 的绝对阈值永远不会触发，导致每个镜像都被重复计入。这同时也是一处修正——双 API server 的 vLLM 行此前队列深度与 token 总量正好翻倍。
aggregateByStart已删除，代码中不再有任何按精确start_ns分组的逻辑。验证：已针对真实 blob 验证（见上方表格）。439817 的 decode 均值由 393 提升至 859 tok/s，而 token 总量与修复前逐字节一致；两行镜像数据的队列深度减半，经查看原始 blob 确认
localhost:8888与localhost:8889上报的是完全相同的engine=0..7标签集、各引擎均值一致、采样相差 19 ms，属于去重而非数据丢失。测试：在
compute-chart-series.test.ts中新增 8 个用例，覆盖偏移采样网格、全零is_streaming拆分、token 总量保持（对齐与非对齐两种）、镜像去重、指标不一致的端点不得被合并、共享网格的命中率回归、多 worker 队列深度、单组件直通。完整测试套件通过（477 + 25）。本次改动不涉及前端与任何用户可见文案，因此无需新增/zh页面。部署说明：
CHART_SERIES_VERSION13 → 14，合并后请运行bun run --cwd packages/db db:backfill-chart-series；在此之前 API 会走较慢的重算路径，因此修复会立即生效。🤖 Generated with Claude Code
Note
Medium Risk
Changes core ETL chart-series math and versioned stored rows (223/599 affected prefill series); validated on real blobs with preserved token totals, but incorrect grid logic would skew all agentic detail charts until backfill.
Overview
Summed agentic server chart series (throughput, queue depth, prefix-cache rates, host KV, prompt-by-source) no longer aggregate on an exact
start_ns, which produced comb-shaped partial sums when workers or label splits scraped on offset grids and maderollingAverageand cumulative token math under-report by up to ~6×.The ETL now evaluates those metrics via
sumOntoGridon a shared lattice atcanonicalTickNs, with each component step-held between its own scrapes and only contributing inside its observed window.CHART_SERIES_VERSIONbumps 13 → 14;aggregateByStartis removed. Mirrored API-server endpoints on the sum path are collapsed using a relative mirror tolerance for counter rates (gauge logic unchanged vialooksMirrored/resolveComponents).Downstream joins (hits/queries, running+waiting, used/total) share the same
t=0-anchored grid so divided metrics still align. Docs add a “Summed Series and the Canonical Grid” section;compute-chart-series.test.tsadds eight regression cases.Reviewed by Cursor Bugbot for commit 5b89855. Bugbot is set up for automated code reviews on this repo. Configure here.