Extend the publish sweep and attribute its limit to generators or Fanout

README.md

@@ -87,9 +87,9 @@ Nothing is installed system-wide. Existing Fanout services and data directories
 |---|---|---:|---:|---|
 | `quick` | 1 topology per signal and mixed | 5s | 10s | installation smoke test |
 | `standard` | 1, 2, 4, 8 processes per signal | 15s | 60s | engineering comparison |
-| `publish` | 1, 2, 4, 8, 16 processes per signal | 30s | 5m | repeatable public evidence |
+| `publish` | 1, 2, 4, 8, 16, 32, 64 processes per signal | 30s | 5m | repeatable public evidence |
 
-The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity.
+The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity. Both reports record the selected sustained process count per signal. The wider publish sweep adds independent exporters to get past telemetrygen's per-process limits; see [the source analysis](docs/telemetrygen.md).
 
 For publication runs, dedicate the default ccx33 to Fanout and use the separate ccx43 for generators, with three repetitions. Run ingest-only and read-under-write as two groups sharing the same fresh pair of VMs; see [the publication commands](docs/publishing.md#publication-runs). For example:
 
@@ -165,7 +165,7 @@ The terminal output is optimized for decisions:
 ╰─ PASS — report is internally valid ────────────────────────────────╯
 ```
 
-“Valid” means the evidence is internally trustworthy. It does not mean every offered target was sustained. Each phase is separately classified as `target-sustained`, `near-saturation`, `saturated-or-generator-limited`, or `invalid`.
+“Valid” means the evidence is internally trustworthy. It does not mean every offered target was sustained. Each phase is separately classified as `target-sustained`, `generator-limited`, `saturated`, `near-saturation`, `saturated-or-generator-limited`, or `invalid`. Split runs record average generator-host CPU cores and logical CPUs for every trial, from `/proc/stat` reads before and after the phase. After validity and `target-sustained` checks, generator CPU at or above 90% means `generator-limited`; below 90%, an accepted-throughput gain under 10% over the preceding process count for the same signal and repetition means `saturated`. Same-host runs retain the acceptance-ratio fallback. See [metric definitions](docs/metrics.md) for the full rule.
 
 Regenerate Markdown without rerunning load:
 

docs/metrics.md

@@ -32,6 +32,22 @@ CPU is the Fanout process only. On a same-host run it does not include telemetry
 
 The sampled series above, plus `fanout_rows_dropped_total`, `process_start_time_seconds`, `fanout_parquet_publish_waiters`, and `fanout_parquet_publish_timeouts_total`, come from Fanout v2026.10.1-alpha and its registered Go/process collectors.
 
+## Generator resources
+
+Split trials read `/proc/stat` on the generator before and after each phase using the same SSH target, port, identity, known-hosts file, and connection options as telemetrygen. The interval includes startup and final exports. `report.json` records `trials[].generator_cpu.average_cpu_cores` and `logical_cpus`; `report.md` shows both in each phase row. Same-host and older reports omit this measurement and render it as `n/a`.
+
+Average cores = logical CPUs × (Δtotal − Δidle − Δiowait) / Δtotal. Total is the sum of user, nice, system, idle, iowait, irq, softirq, and steal ticks; guest and guest_nice are already included in user/nice and are excluded from the sum. Logical CPUs are counted from the per-CPU lines. This is whole-generator-host utilisation, including other processes, rather than telemetrygen-only CPU. Failed reads, unusable counter deltas, or changed logical CPU counts stop the run with a partial report; they cannot masquerade as idle CPU.
+
+## Limit classification
+
+Invalid trials (generator failures, dropped rows, restarts, or no accepted data) take precedence. Accepting at least 95% of the offered count remains `target-sustained`. For other split trials:
+
+- Generator CPU averaging **at least 90%** of its logical CPUs means `generator-limited`.
+- With generator CPU **below 90%**, a sweep whose accepted items/s rose **less than 10%** over the preceding process count is `saturated`, including a decline. Compare only the same signal and repetition; an invalid or zero-throughput previous point cannot establish saturation. Warmup and sustained phases do not establish sweep plateaus.
+- Otherwise retain the acceptance-ratio fallback: at least 80% is `near-saturation`; below 80% is `saturated-or-generator-limited`. The latter can still occur while split throughput is scaling or no valid prior point is available. Low CPU alone cannot rule out acknowledgement latency or other generator limits; see [telemetrygen's export paths](telemetrygen.md).
+
+Same-host runs keep the existing fallback: at least 95% is `target-sustained`, at least 80% is `near-saturation`, otherwise `saturated-or-generator-limited`. Saturation describes the measured plateau under this workload; it is not proof of a CPU-only server bottleneck.
+
 ## Read workload
 
 Authenticated reads rotate equally across user operations `dashboard-refresh`, `drill-traces` (System overview service-map selection), and `trace-detail`, at 15m, 1h, 6h, 12h, and 24h. A refresh follows `page.tsx`, `use-panel-results.ts`, and `refresh.ts`: one sorted batch of all initially visible panel IDs with the full dashboard spec, range and refresh setting, service All, comparison per spec, and layout widths rounded to 100 pixels for a fixed 1664-pixel viewport. It then sends `POST /api/annotations` with the union of the returned time-panel bounds, when annotations are enabled and those bounds are present.

docs/publishing.md

@@ -9,6 +9,7 @@ A performance number is useful only when another engineer can explain and reprod
 - [ ] exact telemetrygen version;
 - [ ] Fanout and generator Hetzner host descriptions, Fanout CPU model, architecture, logical CPU count, memory, OS, and generator placement;
 - [ ] transport, authentication path, batch size, topology, rate, duration, and repetitions;
+- [ ] average generator-host CPU cores and logical CPUs for each split trial, plus the selected sustained process count per signal;
 - [ ] raw `report.json`, `report.md`, and `samples.jsonl` from the same run;
 - [ ] three publish-preset repetitions with Fanout alone on a fresh default `ccx33` VM and generators on a separate `ccx43`;
 - [ ] signal-specific peak results and a sustained mixed result;
@@ -42,6 +43,8 @@ The second sentence discards the machine, duration, signal mix, generator placem
 
 ## Publication runs
 
+The publish sweep uses 1, 2, 4, 8, 16, 32, and 64 processes per signal, each with 8 workers at 2,500 items/s and 1,000-item batches. Sweep steps last 30 seconds after a 15-second warmup; the selected mixed topology then runs for five minutes. Reports state the sustained process count per signal. Selection still uses highest accepted mixed throughput meeting the read target, with the best read-completion fallback if none qualifies. Generator CPU and sweep scaling determine [limit attribution](metrics.md#limit-classification).
+
 Every publication run uses one invocation of `scripts/hetzner.sh` with Fanout alone on the default Hetzner Cloud `ccx33` (8 dedicated vCPUs, 32 GB RAM) and generators on a separate `ccx43` (16 dedicated vCPUs, 64 GB RAM), both in `fsn1` on `ubuntu-24.04`. The two VMs share a private network (`10.77.0.0/16`, cloud subnet `10.77.0.0/24`) in the location’s network zone. Public IPs serve provisioning; generators send OTLP to Fanout over private IPs. `FANOUT_BENCH_HCLOUD_TYPE` and `FANOUT_BENCH_HCLOUD_GENERATOR_TYPE` override the Fanout and generator types respectively; location, image, and SSH-key settings apply to both. Use the publish preset with three repetitions in each group. Separate the ingest-only and read-under-write groups with a standalone `--` so both groups share the fresh pair of VMs and network while each gets fresh stages and a Fanout data directory:
 
 ```sh
@@ -58,7 +61,7 @@ scripts/hetzner.sh \
 
 The script uses the active `hcloud` context or `HCLOUD_TOKEN`, records both VM types, location, and image alongside the detected Fanout hardware, rebuilds the harness before provisioning, and waits for SSH on both VMs. Groups must be nonempty and use matching SSH port and identity settings; they share a temporary known-hosts file and both host descriptions. Every group runs even if an earlier one fails, downloads its own evidence, and receives a one-line summary at the end; any failed group makes the invocation exit nonzero. The script attempts to delete both servers and then the network after the last group and on early exits, including Ctrl-C. Failed deletion after three attempts prints each resource name for manual cleanup and exits nonzero. Hetzner bills every started hour on both VMs. The hourly VM cost is **ccx33 hourly rate + ccx43 hourly rate** for the chosen location, plus applicable public IPv4 charges and tax; a run crossing an hour boundary incurs another started hour on each VM. Confirm current rates before provisioning; grouping the runs shares the billing time of both machines.
 
-The local provisioner detects both Linux platforms, builds the harness for Fanout and telemetrygen for the generator, and uploads only the binaries needed by each host. It generates an ephemeral ed25519 key, appends the public key to the generator’s root authorized keys with a unique marker, and puts the private key (`0600`) in the Fanout stage (`0700`). `managed` on Fanout owns setup, lifecycle, sampling, reads, storage verification, and reporting. Only telemetrygen launches move to the generator, through private-network SSH with a bounded remote timeout. Cleanup removes the marked key and both stages. Reports record the generator description in the host section and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private addresses.
+The local provisioner detects both Linux platforms, builds the harness for Fanout and telemetrygen for the generator, and uploads only the binaries needed by each host. It generates an ephemeral ed25519 key, appends the public key to the generator’s root authorized keys with a unique marker, and puts the private key (`0600`) in the Fanout stage (`0700`). `managed` on Fanout owns setup, lifecycle, sampling, reads, storage verification, and reporting. Telemetrygen runs on the generator through private-network SSH with a bounded remote timeout; generator CPU reads reuse that SSH setup before and after each phase. Cleanup removes the marked key and both stages. Reports record the generator description in the host section and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private addresses.
 
 A plain `fanout-bench run --host root@<host>` without a generator host remains a same-host run on any clean Linux host. It is useful for diagnostics, but generator CPU contention limits what it says about Fanout’s own ingest ceiling.
 

docs/telemetrygen.md

@@ -0,0 +1,49 @@
+# telemetrygen v0.157.0 throughput limits
+
+The configured 20,000 items/s per process is an approximate generation target, not an achieved send rate. There is no hard-coded 13,000 items/s ceiling. The source has per-item timer pacing, a single export lane for traces, and synchronous export pauses for log and metric workers. Adding processes adds independent pacing loops and exporters, which explains why the split run can scale while Fanout still has spare CPU.
+
+This analysis uses the supplied local module-cache source, including its pinned dependencies: OpenTelemetry Go trace SDK and trace/metric exporters v1.44.0, log exporter v0.20.0, and `golang.org/x/time` v0.15.0 (`telemetrygen/go.mod:12–28`). References below are relative to those module roots.
+
+## Rate and batching
+
+`--workers` creates generator goroutines. Each gets the full `--rate`, so eight workers at 2,500 request 20,000 items/s. Each worker creates a token limiter with **burst 1** and waits for each item. A trace with one child consumes two tokens: one for the parent and one for the child; the rate already counts spans. `--batch-size 1000` batches delivery but does not batch limiter reservations.
+
+At 2,500/s, token spacing is 400 microseconds. `rate.Wait` uses a Go timer when it must wait, and accumulated tokens are capped at one. Timer wakeup and scheduler delays can therefore reduce generation below the requested rate even on a host with spare CPU; a late worker cannot catch up with a large burst. The source establishes this mechanism, but does not quantify how much of the observed shortfall came from pacing versus export latency.
+
+| Source | Evidence |
+|---|---|
+| telemetrygen `internal/config/config.go:264–265,306–307,334` | Per-worker rate, worker count, batch flag/size; batching defaults on |
+| telemetrygen `pkg/traces/traces.go:148–167` | One trace goroutine per worker, each given the configured rate |
+| telemetrygen `pkg/traces/worker.go:100,107,142` | Burst-one limiter and separate parent/child waits |
+| telemetrygen `pkg/logs/worker.go:49,93` | Burst-one limiter and one wait per record |
+| telemetrygen `pkg/metrics/worker.go:100,219` | Burst-one limiter and one wait per metric |
+| x/time `rate/rate.go:241–249,274–289,393–399` | Timer-based waits and burst cap |
+
+## Export concurrency and drops
+
+**Traces:** all workers share one `BatchSpanProcessor` and one OTLP exporter. The processor collects up to 1,000 spans or waits up to one second, then calls the exporter synchronously from its single queue-processing goroutine. The gRPC call waits for the receiver's response before returning. Eight workers do not create eight concurrent trace exports.
+
+The trace SDK queue defaults to **2,048 spans** (unless overridden through the SDK environment). telemetrygen sets the batch size and timeout but never enables `WithBlocking`. When the queue fills, `enqueueDrop` increments its dropped counter and discards spans instead of blocking generation. These losses happen before Fanout and do not increment Fanout's dropped-row counter. A successful telemetrygen exit therefore does not prove every generated span reached Fanout.
+
+**Logs and metrics:** each worker constructs its own exporter and buffer. Reaching the batch size calls `Export` inline; generation in that worker pauses until export completes. Exporters serialize calls within each exporter, but exporters belong to different workers, so a process can have up to eight concurrent log or metric exports. These paths call exporters directly: no SDK batch-processor queue is involved and there is no corresponding SDK queue-full drop. Remaining buffered items are flushed at the end.
+
+| Source | Evidence |
+|---|---|
+| telemetrygen `pkg/traces/traces.go:58,71–98` | One exporter and shared processor; batch timeout and size; no blocking option |
+| trace SDK `trace/batch_span_processor.go:22,98–112,126–129` | Default queue size and one processor goroutine |
+| trace SDK `trace/batch_span_processor.go:285–309,321–355` | Synchronous batch exports, then queue processing resumes |
+| trace SDK `trace/batch_span_processor.go:392–398,417–431` | Default nonblocking queue-full drops |
+| trace exporter `exporter.go:30–40`; trace gRPC exporter `client.go:232–249` | Conversion, upload, synchronous gRPC response and retry handling |
+| telemetrygen `pkg/logs/logs.go:72–106`; `pkg/metrics/metrics.go:70–107` | Exporter created separately for each worker |
+| telemetrygen `pkg/logs/worker.go:115–152`; `pkg/metrics/worker.go:241–280` | Synchronous batch export, buffer reset and final flush |
+| log gRPC exporter `exporter.go:61–73`, `client.go:171–187` | Exporter lock and synchronous RPC/retry |
+| metric gRPC exporter `exporter.go:92–106`, `client.go:129–145` | Exporter lock and synchronous RPC/retry |
+
+## Acknowledgement latency
+
+Let `B` be batch size, `W` workers, `r` requested rate per worker, and `L` time spent exporting a full batch, including encoding, transport, server acknowledgement and any retries.
+
+- **Traces:** generation and export overlap through the queue. Long-run throughput is bounded approximately by `min(actual generation rate, B / L)`. Larger `W` can fill the queue faster without increasing export concurrency. Longer acknowledgements lower throughput and can cause queue drops. At 1,000 spans/batch, 13,000 spans/s would correspond to roughly 77 ms per serial export **if export were the active bottleneck**.
+- **Logs/metrics:** each worker alternates filling a batch and exporting it. Ignoring startup/final partial batches, per-process throughput is approximately `W × B / (actual batch-fill time + L)`. Even with ideal pacing, batch fill is about `B / r`, so the idealized rate is `W × B / (B / r + L)`, rather than a latency-independent `W × r`. Burst-one pacing prevents recovering most tokens missed during export.
+
+The supplied split report's single-process trials accepted approximately 13.5k spans/s, 12.5k records/s and 12.5k points/s. Similar rates across these different export paths cannot be attributed solely to the serialized trace processor. Neither that report nor `samples.jsonl` measures generator pacing or RPC acknowledgement latency, so the exact contribution of each mechanism is unresolved. The wider process sweep and generator CPU measurements provide the next attribution evidence without changing the workload shape.