fix(http): consume Content-Length bodies exactly + honor Expect: 100-continue#47
Conversation
…continue
The fixed-length body reader looped `read_exact` into a `[0; 1024]` buffer and
added `buffer.len()` (always 1024) per iteration. A Content-Length that is not a
multiple of 1024 therefore:
- over-read the final chunk, consuming bytes of the next pipelined request; and
- against a peer that keeps the connection open, blocked forever inside
`read_exact` waiting for the missing tail of the final 1024-byte chunk —
hanging the upload until the client times out / disconnects.
Clients work around this by padding upload payloads up to a 1024 boundary; this
removes that need. Read with `read` and count the bytes actually returned,
stopping exactly at Content-Length. The larger 64 KiB buffer also cuts the
per-1 KiB read-syscall overhead.
Separately, the server never answered `Expect: 100-continue`, so clients that
send it (curl and many HTTP libraries) waited out their continue-timeout (~1s for
curl) before sending the body on every upload. Send the interim `100 Continue`
before reading the body, per RFC 9110 §10.1.1.
Adds regression tests for both: a non-1024-multiple body is not over-read into a
following pipelined request, and `Expect: 100-continue` receives an interim 100.
|
Have been running some private speedtest servers for my own client, ran into a couple relatively edge-casey issues in production that were easy behavior-preserving fixes under normal circumstances. LMK if questions/concerns. Thanks to the maintainers, we've gotten a lot of utility out of this repo. Once we are out of beta I will open at least one of our servers and add it to the list of public servers.) --- JRJ |
|
Just wanted to add 2w later that this wasn't slop, I was seeing genuine performance degradation and fixed it, I really think these belong in the repo. Not a big deal, I can maintain a fork... but I think these fixes deserve a look. LMK if there's anything I can do to assist, or if I violated contribution guidance, etc. Thanks again! |
Fixes #46
Summary
Two bugs in the HTTP request-body path (
src/http/request.rs), both affecting uploads (POST /empty):read_exactchunks, countingbuffer.len()(always 1024) per iteration. AContent-Lengththat is not a multiple of 1024 therefore (a) over-reads the final chunk, consuming bytes of the next pipelined request, and (b) against a peer that keeps the connection open, blocks forever inread_exactwaiting for the missing tail of the final chunk — hanging the upload until the client times out / disconnects. Clients pad upload payloads to a 1024 boundary to avoid this; the fix removes that need.Expect: 100-continueis never answered. Clients that use it (curl and many HTTP libraries) wait out their continue-timeout (~1s for curl) before sending the body on every upload.Root cause
read_exactcompletes only once the whole 1024-byte buffer is filled, so the last partial chunk of a non-aligned body never completes against a still-open connection. Andlen += buffer.len()counts 1024 regardless of what was actually read, so the loop can stop 1–1023 bytes past the body. TheExpectheader is simply never inspected before the body read.Fix
read(notread_exact) into a 64 KiB buffer, decrementing by the actual bytes returned and stopping exactly atContent-Length. Safe for any length; the larger buffer also cuts per-1 KiB read-syscall overhead.Expect: 100-continue, write the interimHTTP/1.1 100 Continue\r\n\r\n(RFC 9110 §10.1.1).Validation
New regression tests (
src/http/request.rs) that fail onmasterand pass with the fix:Expect: 100-continuereceives an interim 100100 ContinueemittedEnd-to-end (release build, vs unpatched
masteron a second port):Expect: 100-continue(curl default)GET /emptyping · 100 MBgarbagedownloadcargo build --releaseandcargo clippyare clean.