New plugin: performance — measurement-first optimization workflow (baseline/post snapshots, realistic vs ideal goal tiers, drift-immune metrics)

[kyle-sexton]

Summary

Create a performance plugin whose skill drives a measurement-first optimization workflow end to end: pick targets, construct a goal with realistic and ideal tiers, take a baseline snapshot, change something, take a post snapshot, and refuse to claim a win the data does not support.

This issue is written from a real end-to-end run of that workflow, done by hand in one session against the disk-hygiene destructive-guard hook (PR #3523, issues #3527 / #3528). Everything below is a generalization of something that actually happened, including four places where the naive version of the workflow produced a confidently wrong answer.

This issue REQUIRES a /planning:interview session with the user before implementation. The workflow has real design forks (see "Open questions") and the transcript this is drawn from is one domain — a Windows hook launcher. Do not build from this issue alone.

Why a skill, not a checklist

The session it came from had a competent operator and a strong initial prompt, and it still produced these, each caught only by an explicit verification step:

1. A benchmark harness that silently measured itself. Its shim directory was mktemp -d, and the thing under test cached on PATH — so every run was a forced cache miss and the "after" looked identical to the "before". The harness was measuring its own randomization.
2. A probe that proved nothing while reporting success. It tested a hard-link case; os.link failed cross-volume on Windows, the code fell back to shutil.copyfile, and a copy is a different file — so the probe reported "0 gaps" for a case it never exercised. A green result from a check that did not run.
3. A "before/after" comparison invalidated by drift. Bare bash -c true measured 1825 ms and 283 ms in the same hour at ~10% CPU. Any two-pass measurement attributes that 6x to the change. Only interleaving old/new within one run survives it.
4. A root cause asserted from a plausible mechanism instead of from data. A parallel session diagnosed WDAC code-integrity enforcement (real, CodeIntegrityPolicyEnforcementStatus = 2) as the cause of slow spawns, then had to retract it: a 7-sample spread of min 180.5ms / median 1107.7ms / max 2841.3ms, spread ratio 15.74, at 501 concurrent processes, is a contention signature. The bimodality was the diagnosis; the plausible mechanism was a distraction.

None of these are knowledge gaps. They are all "the measurement was wrong in a way that looked right", which is exactly what a skill with enforced gates prevents and a checklist does not.

Proposed workflow

Phase 0 — target identification

Accept targets from any of: the current session's own pain, a named path/component, a telemetry store, or a broad "what is slow here". Rank candidates by evidence quality, not by suspicion. If nothing is measured yet, the top recommendation is "instrument this first" rather than a guess.

Phase 1 — goal construction (interview-gated, user in the loop)

This is where the user is REQUIRED. Produce, together:

• the metric and the exact command that produces it;
• a realistic target and an ideal target, held separately;
• the floor: the irreducible cost the target cannot go below, computed before any work. In the source session this was one process spawn; the goal asked for p50 ≤ 250 ms on a host charging 0.3–0.9 s per spawn, so the goal was unreachable by any code change and knowing that up front would have reframed the whole task;
• what counts as done, including whether merge is in scope.

The skill must be able to say "your target is below the measured floor" and make the user decide, rather than silently failing the goal at the end.

Phase 2 — baseline snapshot

Refuse to proceed without: sample count, p50 AND p95 (never a single sample or a bare mean), the environment's own noise characterization, and a drift-immune counter alongside the wall clock wherever one exists (spawn count, syscall count, query count, allocation count). In the source run the spawn census 4 → 1 was the number that survived review; the milliseconds were not reproducible even by an independent verifier on the same machine an hour later.

Phase 3 — change

Ordinary implementation, with one rule carried over: re-measure the drift-immune counter after every change, because it catches harness bugs immediately. Both harness bugs above surfaced as a counter that did not move when it should have.

Phase 4 — post snapshot

Interleaved A/B by default: alternate arms within one run, flip the order each iteration, report the median of per-pair ratios alongside per-arm percentiles. Under concurrency, report per-arm percentiles ONLY and suppress the paired ratio — the arms are no longer load-matched, and pairing by index compares samples that never shared conditions.

Phase 5 — verification (fresh context)

Dispatch a verifier that does not inherit the implementer's numbers and re-derives them from the trees. In the source session this caught two blocking correctness defects that the implementer, the implementer's own 143-test suite, and a full green CI run had all missed.

Phase 6 — report

State the target as met or not met, with the measurement that explains why. Never round a miss into a win. Separately state any behavior change the optimization introduced.

Non-negotiable gates the skill should enforce

These are the specific things that went wrong, as rules:

• A harness must prove it is not measuring itself. Anything the harness injects into the environment under test (PATH entries, temp dirs, env vars) must be stable across runs, or keyed out of the subject's caching.
• A probe must assert its own precondition. If a test depends on a hard link, a symlink, a specific filesystem, or a permission, it must FAIL when that precondition is unmet, never silently degrade into a weaker test that passes.
• Never compare two separate passes on a drifting host. Characterize drift first (repeat the no-op baseline); if the spread ratio exceeds a threshold, force interleaving.
• A drift-immune counter outranks wall clock in the report whenever one exists.
• p50 and p95 over ≥20 samples, never a single sample, never a bare mean.
• Optimizing a guard requires a differential, not a passing test suite. A suite proves nothing asserted broke; it does not prove behavior is unchanged. And the differential must cover every MODE the subject runs in — the source session's differential ran one of two modes and missed a real deny→ask downgrade in the other.
• Correctness regressions outrank any speedup. If the change alters behavior, that is a first-class finding stated separately, never folded into the performance claim.

Composition with existing plugins

Should reuse rather than reinvent: /planning:interview for Phase 1 (this is the required user-in-the-loop gate), /verification:measure and /verification:confirm if they already cover snapshot mechanics, /discovery:explore for Phase 0 on an unfamiliar codebase, /implementation:implement for Phase 3, review:fanout or a dedicated verifier agent for Phase 5, and /claude-ops:observability / /claude-ops:audit-performance as telemetry sources for Phase 0. The audit-performance engine already reaches bimodal-spawn-latency, which is exactly the Phase 2 noise characterization this skill needs — worth wiring rather than duplicating.

Scope boundary to settle in the interview: /claude-ops:audit-performance diagnoses a slow Claude Code installation. This skill optimizes an arbitrary target and owns the before/after discipline. They overlap at noise characterization and should share it.

Open questions for the interview

1. Scope. Any code, or start narrow (hooks and CLI tools, where spawn-count metrics are well-defined) and widen?
2. Where do baselines live? Committed artifacts, a plugin data dir, or ephemeral? Committed baselines enable regression detection over time but go stale across machines — and given the drift measured here, a baseline is arguably only valid on the machine and hour that produced it.
3. How hard are the gates? Hard-block on a missing baseline, or warn? The source session's value came from gates that actually blocked.
4. Cross-machine claims. Is the skill allowed to report a number from a host whose noise it has characterized as pathological, or must it refuse and say "measure elsewhere"?
5. Metric catalog. Which drift-immune counters ship built-in (process spawns, syscalls, queries, allocations, bytes) and how are per-domain ones declared?
6. Does it own the fix, or only the measurement? Cleanest split is probably measurement + goal + verification, delegating the change to the implementation lane.
7. Autonomy. The user-in-the-loop requirement is firm for Phase 1. Are Phases 2–6 allowed to run unattended?

Source material

The transcript this is drawn from covers the whole arc and is worth reading before the interview: target identification from live session pain, goal construction (including a target that turned out to be below the floor), baseline, root-cause confirmation before any edit, implementation, two self-inflicted harness bugs and how each was caught, interleaved A/B, concurrent measurement, a differential correctness proof, independent verification by two fresh-context agents, and two blocking defects found only at that last step.

Concrete artifacts from that run, usable as reference implementations:

• spawn-census via a PATH shim directory (the drift-immune counter);
• interleaved A/B harness with order flipping and a paired-ratio summarizer that suppresses itself under concurrency;
• a differential harness comparing a pre-change and post-change binary over a harvested corpus, requiring byte-identical output;
• the two verifier agent prompts, which are written to be adversarial and to distrust the implementer's numbers.

Related

• PR perf(disk-hygiene): cut the destructive guard's per-call spawns 4 to 1, and stop its watchdog blocking commands it never judged #3523 — the run this is generalized from.
• perf(hooks): Windows process-creation tax makes the hook set unusable - 1,778 timeouts / 20.6 hours blocked over 9 days #3508 — the host-contention analysis, including the bimodal-spawn-latency signature this skill should reuse for noise characterization.
• disk-hygiene: _engine_gate_relevant samefile-scans every token of every command, not just separator-carrying words as documented #3527, guardrails: hooks exit 1 with no diagnostic when they abort, so a blocking guard silently enforces nothing #3528 — correctness findings surfaced by the performance work, illustrating why Phase 5 is separate from Phase 4.

[kyle-sexton]

Field data: five verification-harness defects from one session

The session this issue was written from finished with a complete catalogue of the failure mode that motivates it. Every one of these produced a confident wrong answer rather than an error, and every one was caught only because something explicitly re-checked it. Recording them because they are the concrete spec for the "a probe must assert its own precondition" gate, and because four of the five are Windows-specific and would not appear in a Linux-authored test suite.

#	Harness	What it reported	What it actually did
1	spawn census via PATH shim	"no improvement"	mktemp -d shim dir put a fresh path on PATH every run; the subject caches keyed on PATH, so every run was a forced cache miss. It measured its own randomization.
2	hard-link identity probe	"0 divergences"	os.link failed cross-volume on Windows, fell back to shutil.copyfile. A copy is a different file, so it probed an already-accepted residual and never exercised the case.
3	discrimination check (shell)	"NOT DISCRIMINATING"	Passed a D:/... path to bash. Under MSYS that resolves nowhere, so bash exited 127 in both arms and the grep found nothing in either.
4	discrimination check (repeat)	"NOT DISCRIMINATING"	Same trap, second harness. Fixed the path form and it immediately showed FAIL-without / PASS-with.
5	discrimination check (python)	"NOT DISCRIMINATING"	Restored via git checkout --, but the fix under test was uncommitted. The restore silently reverted the fix, so the "with fix" arm ran without it — and destroyed the work in the process.

Two patterns worth encoding as gates rather than advice:

A harness that injects into the environment under test must be stable. Cases 1 and 3/4 are the same root cause seen twice: the harness changed PATH in a way that mattered to the subject. The gate should be: anything the harness adds to the environment (PATH entries, temp dirs, env vars) is either fixed across runs or provably irrelevant to the subject's behavior.

A discrimination check must verify its own patch applied, and must restore from saved bytes, not from version control. Case 5 is the sharpest: using git checkout as the restore mechanism is correct only when the code under test is already committed, and is actively destructive otherwise. The fixed form asserts the patch changed something, then restores from an in-memory copy taken before the edit.

Corollary worth stating in the skill: commit before you verify. It makes the restore path safe and makes an accidental clobber recoverable.

The disproportion is the point. Four of these five defects were in checks I wrote specifically to avoid being fooled — the meta-checks were less reliable than the thing they were checking. A "measure, then verify the measurement" workflow that does not itself enforce these rules mostly generates confident numbers, which is worse than generating none.

Two more inputs for the interview

• Windows drive-letter paths deserve a first-class rule. Three of the five involve D:/... or backslash handling. The independent verifier hit the same PATH-parsing trap on its own harness. On a mixed MSYS/native host this is not an edge case, it is the default hazard.
• The p50/p95 gate needs a companion "is this host measurable" gate. This box's spawn cost spread 15.7x across identical no-op spawns. A percentile from such a host is not wrong so much as meaningless in isolation, which is why the durable result in the source PR was a deterministic spawn count (4 → 1) and not a duration. The skill should be able to refuse to report a wall-clock claim from a host whose noise it has characterized as pathological, and say so.

[kyle-sexton]

Interview complete — all seven questions answered, plus two this issue did not ask

The /planning:interview this issue requires ran on 2026-08-31. Twelve questions registered,
nine answered by the user, three deferred to planning with arbiter tags. The brief is committed at
docs/topics/performance-plugin/PLAN.md
on feat/performance-plugin.

Answers

Q	Decision
1. Scope	Narrow metrics, broad targets. Any target reducible to one repeatable command. Process-spawn count is the only built-in drift-immune counter; everything else is user-declared per domain.
2. /verification:measure boundary (not in this issue — added during the interview)	Depend + route. performance owns the discipline and depends on measure for baseline/compare; measure stays and gains a routing line. Never reimplement baseline/compare.
3. Owns the fix?	No. Measurement + goal construction + verification only; the change delegates to /implementation:implement.
4. Gate hardness	Hard-block, with a named per-gate override recorded in the emitted report.
5. Cross-machine / unmeasurable host	Reuse BIMODAL_SPREAD_RATIO rather than inventing a second threshold. The refusal message must name the counter it can still report.
6. Baseline storage	Memory tier, .work/<topic-slug>/baselines/, machine-bound, never committed. Matches /verification:measure.
7. Metric catalog + noise reuse	Promote the algorithm into a shared lib with one home for the threshold, and refactor claude-ops:audit-performance to consume it. No reaching into its private script dir, no copy-and-drift.
8. Autonomy	Phases 2-6 unattended; phase 1 human-gated always. The loop is opt-in, may open PRs, may never merge.
9. Pairing under concurrency (new — see below)	Both modes. Sequential interleaving suppresses the paired ratio under load; simultaneous duet-style arms report it.

Two corrections to this issue

A research pass over the benchmarking literature ran alongside the interview. It contradicted this
issue in two places, and the brief follows the evidence rather than the issue text.

Phase 4's concurrency rule is backwards. This issue says "under concurrency, report per-arm
percentiles ONLY and suppress the paired ratio — the arms are no longer load-matched." Bulej,
Horký, Tůma, Farquet & Prokopec, "Duet Benchmarking: Improving Measurement Accuracy in the
Cloud" (ICPE 2020) found the opposite on shared machines: running
arms in parallel and evaluating their relative performance together improved accuracy 5.03x on
average (ScalaBench/DaCapo) and 37.4x on average (SPEC CPU 2017). Concurrent load is what makes
pairing most valuable, because both arms absorb the same interference.

The reconciliation the brief adopts: duet pairs simultaneously, benchstat interleaves
sequentially, and only the sequential form is vulnerable to bursty load. That reconciliation is
this session's reading, not a sourced claim, and is labelled as such.

The refusal posture is stricter than the field. No benchmarking tool surveyed refuses to report
above a variance threshold — pyperf, Criterion, JMH and benchstat all warn and print anyway. The
refusal ships as a deliberate house rule, not as consensus, and the skill body will say so.

Three claims this issue makes that the research could not ground

Recording these because the plugin's whole premise is refusing to assert what it cannot support.

• "p50 and p95 over ≥20 samples." No benchmarking-community sample count for a meaningful
  percentile exists in any Tier 1/2 source beyond the derivable 1/(1−p) floor. The SRE Book names
  the 99th and 99.9th percentiles, not p95. Whatever ships here is a house choice and is deferred as
  Q12 with arbiter: USER-RESERVED.
• The counts-over-wall-clock rationale. It is grounded in the literature only for instruction
  counts. Syscall, query, and process-spawn counts are an ungrounded generalization of it. Since
  process-spawn count is this plugin's headline metric, that gap is load-bearing and will be labelled
  in the skill body rather than smoothed over. Valgrind's own manual argues both halves: execution
  time "is what users perceive", and its simulations are "unlikely to reflect the behaviour of a
  modern machine".
• Warmup and steady state. Barrett et al. (OOPSLA 2017) found at most 43.5% of ⟨VM, benchmark⟩
  pairs consistently reach a steady state of peak performance. Discarding N warmup iterations is
  fine; claiming it establishes steady state is not.

Two citation traps the skills must avoid, both easy to fall into: benchstat is unpaired (it
recommends interleaved collection but analyzes with Mann-Whitney U, an independent two-sample test),
and coordinated omission is a load-generator problem — citing Tene to justify percentiles in a
synchronous harness that measures every operation miscites the field's most famous source.

Deferred to planning

• Q10 — where the shared noise-characterization lib lives, and whether the
  claude-ops:audit-performance refactor lands in this PR or a follow-up. (arbiter: /planning:plan)
• Q11 — skill decomposition: one workflow skill with six phases, or one skill per phase.
  (arbiter: /planning:plan)
• Q12 — sample count and percentile choice, per the grounding gap above.
  (arbiter: USER-RESERVED)