upload benchmark rewards servers that don't ingest the request body (scoring + validate.sh both miss it)

[enghitalo]

Summary

The upload benchmark currently rewards a server for NOT ingesting the request body. A server can answer the ~60-byte request head with a 200 (reading Content-Length from the header), then close/RST the connection without ever reading the 20 MB body — and it scores as the fastest upload server, because it does the least work. Neither the throughput scoring nor validate.sh detects this.

I found this while optimizing an engine's upload path (vanilla): its epoll backend had a latent bug that made it answer from the head and close, and it topped the upload chart precisely because of that bug. Any engine — accidentally or deliberately — can do the same. This is a harness-robustness gap, not a single-engine problem.

Evidence (local, 8 conns, 16 MiB bodies, default GC)

Server behavior	Req/s	Client socket errors	Body actually ingested?	Server peak RSS
Answer from head + close (the "cheat")	27,691	278,891 writes	❌ no (RST after head)	11 MiB
Correctly stream + ingest the body	615	0	✅ yes	6 MiB

The "cheat" is 45× faster on the leaderboard while transferring almost none of the body — and a load generator records ~278k write failures that the harness never looks at.

Root cause in the harness

1. Throughput = 2xx/sec, with no error accounting.
scripts/lib/tools/gcannon.sh → gcannon_parse() computes rps from the 2xx count only:

rps = 2xx_count / duration

A server that answers every request from the head with a 200 and discards the body maximizes 2xx/sec. Client-side write/connection-reset errors (which a non-ingesting server causes en masse) are never parsed, so they don't penalize the score.

2. The /upload correctness contract is satisfiable without reading the body.
scripts/validate.sh (the upload block, ~L767–805) asserts the response equals the declared Content-Length:

ACTUAL_LARGE=$( { dd if=/dev/urandom bs=1024 count=$upload_bs | \
   curl -s --max-time 60 -X POST --data-binary @- ".../upload"; } || true )
[ "$ACTUAL_LARGE" = "$upload_size" ] && PASS

But the byte count the engine returns comes from the Content-Length header, which is in the head — so a server that never reads the body still echoes the right number and passes. The || true also swallows a curl broken-pipe error (exit 55) caused by the early close, and %{size_upload} is not checked, so a partial/aborted upload still validates.

Net: the cheat is invisible to both the score and the correctness gate.

Suggested improvements (any subset helps)

1. Make the /upload response depend on the body content, not its declared length. Strongest fix: require the handler to return a checksum/digest (e.g. CRC32/FNV/SHA-1) or the last N bytes of the received body. A server that discards the body cannot compute it, so it must actually ingest. validate.sh computes the same digest client-side and compares.

2. Validate full-body transfer in validate.sh. Drop || true, check curl's exit code, and assert curl -w '%{size_upload}' equals the intended body size. An early close → non-zero exit / short size_upload → fail.

3. Parse and penalize load-generator errors. Have gcannon_parse() extract connection-reset / write-error / non-2xx counters and fail (or zero) the result when they exceed a small threshold. A correct upload server produces ~0; a head-and-close server produces ~1 per request.

4. Rank upload on body throughput, not request rate. Report MB/s of accepted request body (bytes the server actually drained), not 2xx/sec. This naturally rewards efficient ingestion and gives a non-ingesting server a near-zero score. gcannon already tracks bytes sent/accepted.

5. (Optional) Slow-body probe. Send the body in timed fragments and assert the response arrives only after the last fragment — catches "respond-before-drain" servers that desync real clients even when they do eventually read the body.

Context / fix on the engine side

The companion fix in the vanilla engine makes both backends drain-then-respond (stream the body off the socket, then answer) so they ingest the body correctly and keep-alive — io_uring upload throughput +58% and peak RSS −41% in the same local test. PR: enghitalo/vanilla#61

Happy to send a PR for any of the harness improvements above (1+2 are small and high-value).

[MDA2AV]

The validation.sh does not prevent frameworks from cheating, that is impossible, it simply performs some basic checks to make sure the implementation at least passes the basic functionality.

For the suggested improvements:
1 - This is computationally too heavy for some frameworks and would become a bottleneck (had this solution in the past).

For the others we can indeed harden the validation script preventing some of the easy cheating, that will however not fully prevent cheating, also note that for all tests in this platform contributors can check our rules https://www.http-arena.com/#doc=test-profiles
which for the upload test states:

Implementation rules
The upload endpoint must actually read and process the request body. The returned byte count must be computed by reading the uploaded data, not inferred from request metadata.

Do not return the value of the Content-Length header - this defeats the purpose of the test, which is to measure how efficiently the framework processes uploaded content.
Frameworks may use small read buffers to process the upload incrementally. Holding the entire payload in memory is allowed but not required.
The goal is to prove that the framework can efficiently compute a result over a large blob being sent to it.

The problem is that most engine implementations are AI generated(and LLMs are not yet capable of producing fully rule compliant implementations) code and it is not feasible to review all the changes commited daily, for this reason most of the tests are unscored in the engine section.

Regardless this is a valid issue and I'll work to improve our validation script.

Thank you

[Kaliumhexacyanoferrat]

Also note that there is #784 which still allows to cheat on requests with Content-Length but makes the situation a little bit more diverse.