Skip to content

Add telemetry-function experiment results (inline cost, exact baseline, accuracy vs size) - #33

Draft
zzylol wants to merge 1 commit into
mainfrom
feat/experiment-results
Draft

zzylol wants to merge 1 commit into
mainfrom
feat/experiment-results

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Being rerun. The stack below now runs the sketch, exact and raw pipelines side by side (#25), and experiments.py has pipelines instead of inline-cost. The results below are from the earlier setup; this PR will be updated with the pipelines and accuracy-size reruns.

Part 10 of the telemetry-functions stack: the measured results of experiments 1 (inline cost) and 2 (accuracy versus size), generated by report_experiments.py from experiments.py output (#30). Every stage is a pinned asap-collector container on 56-core x86 servers, with 10-second windows. Full tables and CSVs are in docs/results/experiments.md.

Inline cost: branch CPU with and without each function

Branch CPU with and without each function

Function Added branch CPU per point Branch CPU at 200,000 points/s per source
none (forwarding only) (about 0.3 µs in total) 0.06 cores
KLL latency quantiles 2.7–4.1 µs 0.60 cores
Count-Min + heap, top-k 3.7–5.3 µs 0.81 cores
HLL over 4 attribute keys 9–11 µs (about 2.3 µs per key) one core reached at about 100,000 points/s
  • The global stages (merge, detector) receive a few KB to 180 KB per window and use under 0.01 cores.
  • The exact baseline, which receives every raw point, needs a full core at about 95,000 points/s for cardinality and about 400,000 points/s for latency quantiles.
  • Answers arrive 2–3 s after a window ends (these runs used a 2-second merge grace).
  • These per-point costs put a branch below the ~1M points/s per core reference. The next step is a profile of the per-point path; decoding OTAP into observations and the per-attribute split are the likely costs, not yet measured.

Accuracy versus size (one window per setting)

Setting Error Merge bytes per window
KLL k = 50 / 100 / 200 / 400 p99 median error 13.3% / 7.0% / 7.5% / 2.0% (routes ≥ 20,000 points) 39 / 58 / 96 / 181 KB
Count-Min 256 / 512 / 1,024 / 2,048 columns, 100,000 keys recall@10 1.0 at every width; top-10 count error 6.0% / 2.8% / 1.3% / 0.46% 7 / 11 / 12 / 19 KB

Single windows, so neighbouring settings can cross (KLL k = 100 vs 200).

@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from 0965b48 to 17186b1 Compare October 12, 2026 22:09
@zzylol
zzylol force-pushed the feat/experiment-results branch from 67930ea to 4cf35de Compare October 12, 2026 22:09
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from 17186b1 to f263a6c Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the feat/experiment-results branch from 4cf35de to be08a0a Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from f263a6c to f19756b Compare October 19, 2026 02:31
@zzylol
zzylol force-pushed the feat/experiment-results branch from be08a0a to 19c8143 Compare October 19, 2026 02:31
@zzylol
zzylol force-pushed the docs/governance-and-rfc branch from f19756b to 330dfc9 Compare October 19, 2026 14:57
@zzylol
zzylol force-pushed the feat/experiment-results branch from 19c8143 to 016efad Compare October 19, 2026 14:57
@zzylol
zzylol changed the base branch from docs/governance-and-rfc to feat/ci-tiers October 19, 2026 17:32
@zzylol
zzylol force-pushed the feat/experiment-results branch from 016efad to 82a6d7e Compare October 19, 2026 17:33
@zzylol
zzylol force-pushed the feat/experiment-results branch 2 times, most recently from 26ac19d to 123d82f Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/experiment-results branch from 123d82f to af506b2 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/ci-tiers branch 2 times, most recently from 2911c44 to b56fea2 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/experiment-results branch from af506b2 to 034fe62 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/experiment-results branch 2 times, most recently from 21f5da0 to f8e6762 Compare October 19, 2026 20:17
@zzylol
zzylol force-pushed the feat/experiment-results branch from f8e6762 to 90f57dd Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/experiment-results branch from 90f57dd to f91377d Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/experiment-results branch from f91377d to 3a78cd7 Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/experiment-results branch from 3a78cd7 to 402cbc3 Compare October 19, 2026 20:19
Inline cost (branch CPU with and without each function, 20k-200k points/s
per source), the exact baseline's own cost, and accuracy versus sketch size
(KLL k, Count-Min width over 100,000 keys), measured on 56-core x86 servers
with every stage a pinned asap-collector container. Generated by
report_experiments.py from experiments.py output.
@zzylol
zzylol force-pushed the feat/experiment-results branch from 402cbc3 to 0e430b8 Compare October 19, 2026 20:19
@zzylol
zzylol changed the base branch from feat/ci-tiers to main October 19, 2026 20:20
@zzylol
zzylol marked this pull request as draft October 20, 2026 23:23
@zzylol

zzylol commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Taken out of the stack and converted to draft. Review found the results come from tooling and a setup that the stack replaces: the forwarding-only raw branch is gone (store and query is now raw-sink-a/b -> loki-raw + query job) and merges now emit as soon as both sources have sent a window. The PRs above now sit directly on main. This PR will be rerun on main after they merge, with the inline-cost baseline redefined and the review points (merge/detector CPU columns, delivered rate from full windows, generator vs exact bottleneck, split Count-Min/KLL tables) addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant