[zzylol]

Part 8 of the telemetry-functions stack: the end-to-end acceptance, plus the tooling for experiments 1 and 2. The measured results come in the last PR of the stack.

deploy/telemetry-functions/acceptance.py

Runs all four use cases on the real collector containers: the sketch, exact and raw pipelines (18 stages) with Prometheus, Loki and loki-raw. It checks the sketch answers against the exact pipeline's ground truth through Prometheus and Loki, checks that every pipeline reports its window completion delay, prints PASS or FAIL per check, and exits non-zero on any failure.

To keep it to a few minutes per use case, time is scaled down 6×: 10-second windows, a 2-second grace (raised back to the deployed 5 s in #31, after the first real CI run showed sketches arriving later than 2 s on a shared 4-CPU runner), changes at second 100, and growth 6× faster. Rules counted in windows (baselines, thresholds) are as deployed. Comparisons are pinned to one window answered by both methods, with PromQL's @ modifier.

Result on a 56-core x86 server: 29/29 checks passed.

Use case	Checks (all passed)
all four	18/18 stages up; 400,000 points in every window, the full offered rate; sketch and exact windows hold identical point counts; every pipeline reports its window completion delay
latency	exact p50/p99 inside the sketch's bound: 40/40
cardinality	sketch and exact give every key the same class (user.id identifier-like, k8s.pod.name growing, http.route and http.method stable); distinct error ≤ 1.13%; class events in Loki
drift	route-3 drift detected by both methods (KS 0.33); no other route flagged; drift event in Loki
top-k	/new/hot-route in both top-10s; recall@10 = 1.00; novelty event in Loki

Window completion delay (10-second windows, 5 s merge grace): sketch 5.2–5.6 s and exact 5.2–5.7 s in every use case. The raw pipeline's rate is reported, not required, because whether it keeps up is what it measures: it delivered every point with 4.6 s (latency), 9.6 s (drift) and 14.7 s (top-k), but for cardinality loki-raw stored only about 190,000 of 400,000 points per window, 39 s late, since every user.id is new to it.

Informational, not a pass condition: with 10-second windows, the p99 relative error for routes with ≥ 20,000 points reached 12.8%. That is KLL's rank error in a log-normal tail (see #25).

experiments.py and report_experiments.py

• pipelines: the three pipelines side by side over offered rates, fed identical points: CPU and memory of each stage, points delivered per second and window completion delay per pipeline. The raw branch only forwards points, so the sketch branch's CPU minus the raw branch's is the cost of the function.
• accuracy-size: error against exact as KLL k and Count-Min width change (100,000 keys), with the bytes merge receives per window.
• report_experiments.py turns the CSVs into a Markdown report with tables and a figure.

Also includes PIPELINES/COMMON and the published Loki port in compose, run.sh --build-only, and a README covering every use case.

cargo fmt --check, cargo clippy ... -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head.