Skip to content

Add CI tiers: per-PR smoke with acceptance, nightly regression, on-demand comparison - #31

Merged
zzylol merged 4 commits into
mainfrom
feat/ci-tiers
Oct 19, 2026
Merged

zzylol merged 4 commits into
mainfrom
feat/ci-tiers

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 9 of the telemetry-functions stack: the three CI benchmark tiers from the design.

Tier Workflow When What
Smoke ci.yml every PR and push to main fmt, clippy, tests; then acceptance.py on real collector containers, with all 18 stages sharing the runner's 4 CPUs (a correctness check, not a capacity measurement); acceptance.json uploaded
Nightly asap-sketch-benchmark.yml daily full rate sweep on vars.BENCH_RUNNER (falls back to ubuntu-24.04 with shared infrastructure); a new step compares against the previous night's summary.csv with benchmarks/compare-summaries.py and fails on a throughput drop beyond regression_margin (default 10%)
On demand benchmark-on-demand.yml a /benchmark PR comment by an owner, member or collaborator reduced sweep on main, then on the PR head; posts the comparison table as a PR comment (the benchmark job runs PR code with read-only permissions; a separate job comments)

Also fixes the nightly's verify step, which still expected the 3-scenario row counts (189/63/441) after the top-k scenarios made it 5 (315/105/735).

To enable dedicated-CPU runs: register a self-hosted runner with Docker and gh, and set the repository variable BENCH_RUNNER to its label.

Validated with actionlint 1.7.7 (no findings) and yaml.safe_load. compare-summaries.py was tested on synthetic CSVs (−15% fails at a 10% margin; −5%, +4% and −1.2% pass). This PR's own CI run is the first real run of ci.yml.

Fixes found by this PR's first real CI runs

The shared 4-CPU runner exposed three problems that dedicated cores hid. All three were reproduced on a 56-core x86 server with every stage on CPUs 0–3 and fixed; the full acceptance then passed 25/25 under that contention.

  1. Merge grace too short in scaled runs. A source's sketch sometimes reached window_merge after the scaled 2 s grace and was dropped as late. Acceptance now keeps the deployed 5 s grace, and the check lists any window whose counts differ.
  2. The idle timer closed tumbling windows too early. Under contention it closed a window while its last batches were still queued, so sketch and exact windows differed by a batch (for example 388,736 vs 401,568 points). Data crossing a boundary already closes a window on time; the timer now closes only idle streams' windows, 5 s after their end.
  3. Loki dropped one of two identical events. Loki keeps one entry per stream, timestamp and line. Sketch and exact novelty events had identical text (identical counts), so whichever arrived second was lost; on the runner that was the sketch's. Event lines now start with [sketch] or [exact].

@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from 9a62ef4 to ff1cc27 Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the feat/ci-tiers branch 2 times, most recently from 1206499 to 2682dc5 Compare October 19, 2026 02:31
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch 2 times, most recently from 70b60ab to 2f1564b Compare October 19, 2026 14:57
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from 2f1564b to deaf7ed Compare October 19, 2026 17:33
@zzylol
zzylol force-pushed the feat/ci-tiers branch 2 times, most recently from b1144fc to d350d99 Compare October 19, 2026 20:15
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch 2 times, most recently from 9906188 to 184bf71 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/ci-tiers branch 2 times, most recently from 3d592d9 to 2911c44 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from 184bf71 to 093aa7b Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch 2 times, most recently from 16fa6ad to ae85b5d Compare October 19, 2026 20:17
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from ae85b5d to 3aed20f Compare October 19, 2026 20:17
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from 3aed20f to d284043 Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from d284043 to 5f11337 Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/ci-tiers branch 2 times, most recently from 057a971 to 808c655 Compare October 19, 2026 20:18
@zzylol
zzylol force-pushed the feat/e2e-acceptance branch from 5f11337 to 3f5525f Compare October 19, 2026 20:18
- ci.yml: fmt/clippy/test, then the telemetry-functions acceptance smoke
  on real collector containers (stages share the runner's CPUs).
- asap-sketch-benchmark.yml: run on vars.BENCH_RUNNER when set, compare
  summary.csv with the previous successful main nightly and fail on a
  drop beyond regression_margin (default 10%); fix stale verify counts
  for the five scenarios.
- benchmark-on-demand.yml: `/benchmark` PR comment runs a reduced sweep
  on main and the PR head and posts the comparison.
- benchmarks/compare-summaries.py and README tier docs.
On a shared 4-CPU CI runner a source's sketch sometimes reached window_merge
after the scaled 2 s grace and was dropped as late, so one window's sketch
held one source fewer than exact. Scaled runs now keep the deployed 5 s
grace, and the check lists any window whose counts differ.
Data crossing a boundary already closes a window on time. On a contended
host (the 4-CPU CI runner) the timer could close a window while its last
batches were still queued, moving them into the next window: sketch windows
then differed from exact by a batch (e.g. 388,736 vs 401,568 points). The
timer now only closes idle streams' windows, 5 s after their end.
Loki keeps one entry per stream, timestamp and line, so a sketch and an exact
event with the same text (identical top-k counts) collapsed into whichever
arrived first; on the CI runner that dropped the sketch's novelty event. Each
event line now starts with [sketch] or [exact]. The acceptance novelty check
reports event counts and the asap_topk_new gauge when it fails.
@zzylol
zzylol changed the base branch from feat/e2e-acceptance to main October 19, 2026 20:19
@zzylol
zzylol merged commit 162de3e into main Oct 19, 2026
@zzylol
zzylol deleted the feat/ci-tiers branch October 19, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant