Run every stage as a pinned collector container; sketch, exact and raw pipelines side by side

[zzylol]

Part 3 of the telemetry-functions stack. This is the deployment the use cases run on.

Before → after

• Before: the production asap_sketches processor only ran in-process or behind the file-passing demo. The container benchmark used a separate model processor.
• After: every stage is the official OTAP engine (asap-collector: the upstream controller with the ASAP nodes linked in), in its own container pinned to its own CPU, talking OTLP/HTTP over the network. Three pipelines run side by side on the same generated input and share no processor:

sketch: generator-a/b -> branch-a/b (sketch per window) -> merge -> detector -> Prometheus
exact:  exact-generator-a/b -> exact-branch-a/b (exact window) -> exact-merge -> exact-detector -> Prometheus
raw:    raw-generator-a/b -> raw-branch-a/b (forward) -> raw-merge (forward) -> raw-sink -> loki-raw

Each pipeline has its own two generators. They run the same config with the same scenario start, and the generated points depend only on scenario time (#24), so all three pipelines receive identical points. The pipelines answer three questions about the same traffic: what the sketch answers, what the exact answers are (the ground truth), and what it costs just to move every raw point to a store.

New pieces

• asap-collector: the upstream df_engine startup (admin /metrics, /readyz, signals) with the ASAP nodes registered.
• window_merge: merges sources' sketches by the window they describe, not by arrival time, and emits a window a grace period (5 s) after it ends. Later arrivals are counted as late.
• sketch_detector: turns merged windows into answers with bounds. HLL ±1.96 standard errors; KLL quantiles at q ± the normalized rank error; Count-Min lower bound = estimate − e/w·N.
• The exact pipeline, built like the sketch one. exact_branch keeps every value per quantile series and a value → count map per attribute key for each event-time window (an ExactWindow, sent as an Exact envelope). exact_merge adds up the sources' windows 5 s after each window ends. exact_detector computes the same answers from the merged window. Detector and exact detector both emit asap_quantile, asap_distinct, asap_topk and asap_points, labelled method="sketch" or "exact", as plain OTLP gauges with labels on each point.
• The raw pipeline. raw-branch-a/b and raw-merge only forward points (OTLP receiver → exporter). raw_sink writes every point as one log record to a separate Loki (loki-raw) and watches its acknowledgements. Prometheus cannot hold raw points: it rejects several samples of a series at the same timestamp and out-of-order samples.
• Window completion delay, asap_answer_delay{method}: seconds from a window's end until its result is complete at the end of the pipeline. For sketch and exact, that is when the detector emits the window's answers; for raw, when loki-raw acknowledged the window's last point. raw_sink also reports each point's time from event to acknowledgement (asap_raw_transit, p50/p99) and rejected points (asap_raw_failed).
• aggregate_by is exposed in the processor config.
• Fix: tumbling windows close by event time. Windows used to close on a timer started at the first message, so up to a window of later points landed in the previous window. Measured before the fix: the first window held 751,060 points in the sketch against 175,630 exact. A batch whose event time passes the window's end now closes it first, and the idle timer runs every second. After the fix, sketch and exact hold identical point counts in every window.
• deploy/telemetry-functions: Dockerfile, compose.yaml (18 stages on CPUs 0–17), stage pipelines (pipelines/ per use case, common/ for the stages that do not depend on it), Prometheus (OTLP receiver plus a scrape of each engine's CPU, RSS and bytes), Grafana, Loki, loki-raw and run.sh. The dashboard draws the three pipelines and compares their window completion delay, CPU, bytes and memory.

Result (56-core x86 server, 1-minute windows, 20,000 points/s per source)

The screenshot and numbers below were taken before the exact baseline moved into its own pipeline; the answers are the same, since the exact pipeline sees the same points.

For the busiest route, the sketch p99 was 199.5 ms against 201.7 ms exact, inside the sketch's bound of 166–295 ms. For routes with 20,000–100,000 points per window, the p99 can differ from exact by 10–40%. That is KLL's rank guarantee (about 1.3% at k = 200) in a log-normal tail, where a small rank shift is a large value shift; the bound covers it. The accuracy-vs-k experiment comes in a later PR.

Validation

Run on a 56-core x86 server with deploy/telemetry-functions/run.sh. cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head. The end-to-end acceptance checks come in a later PR of this stack (#30); there, every pipeline delivers the full 400,000 points per window and reports its delay (latency use case: sketch 5.4 s, exact 5.2 s, raw 4.6 s, with 10-second windows and a 5 s grace).