Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ are reported separately.

## Workload

- Async: one reused client per case, with 10, 50, 100, or 150 closed-loop workers.
- Async: one reused client per case, with 2, 10, 50, or 100 closed-loop workers.
- Blocking: one reused client per logical worker, using a persistent thread
pool. Submit a whole worker batch, not one executor task per request.
- Seven payload and upload-chunk cases matching the Rust benchmark, listed below.
Expand All @@ -16,7 +16,8 @@ are reported separately.
single-threaded and multithreaded runtimes, ry, httpx, aiohttp, niquests, and
curl_cffi.
- Blocking variants: wreq MT/ST/CT, ry, requests, httpx, niquests, curl_cffi, and pycurl.
- Three rounds, each with one warmup and one timed batch of 300 requests.
- Three rounds, each with one warm-up batch of 200 requests and one timed
batch of 300 requests.
Case order is shuffled with a recorded seed.
- Status, negotiated HTTP version, and total response length are checked for
every request. Unsupported protocol or upload combinations are reported as
Expand Down Expand Up @@ -44,7 +45,7 @@ worker clients have separate connection pools.
Both Full and Stream uploads are tested for every payload and concurrency.
The complete matrix contains 1,904 supported cells across 18 client variants;
each cell has three measured rounds. The Rust benchmark uses Criterion and
600 requests per iteration; this suite uses fixed batches of 300 requests.
600 requests per iteration; this suite uses fixed timed batches of 300 requests.
Custom payload sizes use upload chunks of at most 64 KiB.

## Run locally
Expand Down Expand Up @@ -75,11 +76,10 @@ number of warm-up batches. Allow several hours for a full run. The setup command
above prepare the dependencies and native binaries; the runner won't install or
build them for you. Publishing the results to the docs is optional.

For a warm-up experiment, pass `--warmup-requests 150`. This keeps the default
300 timed requests and three rounds, but sends 150 requests per warm-up batch.
The budget must reach every concurrent worker. Compare against the default on
the same machine before using these results; reduced warm-up runs can be
previewed with `docs/build.py --data RUN.json` but cannot replace published data.
The default warm-up budget is 200 requests. Every concurrent worker must
receive a warm-up request. Override it with `--warmup-requests`; compare any
smaller budget against the default on the same machine before ranking clients.
Historical snapshots keep their recorded concurrency and request counts.

Use the same allocator and release flags for both wreq revisions in a comparison,
and record them in the run's build provenance. jemalloc controls Rust allocations
Expand Down Expand Up @@ -170,7 +170,7 @@ The async source and all historical files stay unchanged. The two publication
options cannot be combined.

Smoke runs belong in `bench/data/smoke/` and cannot be published. Publication
requires the complete default or blocking matrix, at least 300 requests per
requires the complete default or blocking matrix, at least 300 timed and 200 warm-up requests per
batch and three rounds, with at least one warm-up and timed sample per round.
These checks confirm coverage; you still need to review the quality of the measurements.

Expand Down Expand Up @@ -199,7 +199,8 @@ Explicit `--blocking-data` overrides may preview a blocking subset or a reduced
warm-up run, provided their workload axes match the async snapshot. Publishing
still requires the complete current matrix and full publication budget.
Combined preview JSON may record `measurement_sources`; each entry must assign
distinct clients, and together they must cover the configured clients. The page
distinct client/concurrency pairs, and together they must cover the configured
matrix. Entries without a concurrency field cover every configured concurrency. The page
shows each run's revision and collection time separately.

## Blocking wreq runtimes
Expand Down
13 changes: 10 additions & 3 deletions bench/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@
from .workloads import (
BODY_CASES,
CONCURRENCY_CASES,
REQUESTS_PER_BATCH,
STREAM_CHUNK_BYTES,
WARMUP_REQUESTS_PER_BATCH,
prepare_chunks,
upload_chunk_bytes,
)
Expand All @@ -53,7 +55,9 @@
from workloads import (
BODY_CASES,
CONCURRENCY_CASES,
REQUESTS_PER_BATCH,
STREAM_CHUNK_BYTES,
WARMUP_REQUESTS_PER_BATCH,
prepare_chunks,
upload_chunk_bytes,
)
Expand Down Expand Up @@ -124,11 +128,14 @@ def parse_args(argv=None):
parser.add_argument("--rounds", type=positive, default=3)
parser.add_argument("--warmup", type=nonnegative, default=1)
parser.add_argument("--samples", type=positive, default=1)
parser.add_argument("--requests", type=positive, default=300)
parser.add_argument("--requests", type=positive, default=REQUESTS_PER_BATCH)
parser.add_argument(
"--warmup-requests",
type=positive,
help="Requests per warm-up batch; defaults to --requests",
help=(
"Requests per warm-up batch; defaults to the smaller of --requests "
f"and {WARMUP_REQUESTS_PER_BATCH}"
),
)
parser.add_argument("--server-workers", type=positive, default=4)
parser.add_argument(
Expand All @@ -144,7 +151,7 @@ def parse_args(argv=None):
parser.add_argument("--worker", choices=CLIENTS, help=argparse.SUPPRESS)
args = parser.parse_args(argv)
if args.warmup_requests is None:
args.warmup_requests = args.requests
args.warmup_requests = min(args.requests, WARMUP_REQUESTS_PER_BATCH)
if args.worker is None:
if args.server is None:
parser.error("--server is required")
Expand Down
Loading
Loading