Repository navigation
Hand out chunked body data as it arrives - #32
Conversation
poll-chunked waited until a whole chunk and its CRLF were buffered, and every read on the way re-copied the part of the chunk already buffered: an S-byte chunk cost about S^2/4096 bytes of copying, and nothing was yielded until its last byte arrived. It now tracks the current chunk in a new ResponseStream field, chunk-left, and a poll returns what is buffered of the chunk, or one read's share of it. A poll never crosses a chunk boundary, and the framing errors are unchanged. ResponseStream.init takes one more argument.
There was a problem hiding this comment.
Build & Tests
- CI is green on ubuntu-latest and macos-latest at
b3c8c46. - Local suite, run with
test/run.sh's two servers on the compat core. The local carp'smax-lengthliteral is patched to9223372036854775807LLinmain.c.- head: 227 passed, 0 failed on armhf, and the same on an LP64 build of the same C (
aarch64-linux-gnu-gcc, CI's ABI); - main: 224/0 on both;
- main's
http-client.carpwith the branch's tests: 226/1 on both. The only failure isthe part of a chunk that has arrived is handed out without waiting for the rest(waited for the close), as the body says.
- head: 227 passed, 0 failed on armhf, and the same on an LP64 build of the same C (
- The held-half route adds no wall time to the suite. The head polls once and closes, and the suite takes 3.5 s; with main's library it takes 13.6 s, because main waits.
Findings
1. The new CRLF state's re-read has no test
poll-chunked reads more data in three places: in the data state (http-client.carp:382), the CRLF state (:388) and the size-line state (:399). /chunked-large exercises the data one. No test needs either of the other two to succeed. I made each of them fail instead of reading:
- S1, at
:399: a size line that is only partly buffered fails asunterminated chunk size line. The suite stays at 227/0. - S2, at
:388: fewer than 2 buffered bytes before the CRLF fails astruncated chunk data. The suite stays at 227/0.
S2 sits on the new state's normal path: every read that ends between a chunk's data and its CRLF takes it. /chunked-cr-close can't tell S2 apart from a real close, because both end with the same message.
The patch below adds a /chunked-split route that writes 5, \r\nhello, \r, then \n0\r\n\r\n, 50 ms apart with TCP_NODELAY, plus one assertion. I ran it as a suite test on b3c8c46: 228/0 on armhf and LP64, and S1 and S2 each fail exactly the new test (227/1). Main's library reads the same writes cleanly as well.
2. Chunks under about 64 bytes decode up to 47% slower
len is String.length (buf s). That's a strlen over the whole buffer, up to 4 KiB, recomputed at the top of every loop pass. A chunk now takes three passes (CRLF, size line, data), where main took one.
Timings are for Client.get of a 1 MiB payload, LP64, on a quiet box (load 0.1–0.3). "+ patch" is the head with the patch below:
| framing | main | head | head + patch |
|---|---|---|---|
| 1-byte chunks | 710 ms | 1,041 ms (+47%) | 772 ms |
| 16-byte chunks | 58 ms | 78 ms (+34%) | 62 ms |
| 256-byte chunks | 7 ms | 8 ms | 7 ms |
| one 1 MiB chunk | 40 ms | 2 ms | 2 ms |
A round at load ~13 agreed: +48% for 1-byte chunks and +29% for 16-byte chunks. From 64-byte chunks up the difference is noise, so llm's SSE events aren't affected. Still, this is a performance PR, and the fix is small: keep len across passes, and recompute it after each fill-or-fail!.
The patch for both
It git applys on b3c8c46. Checked with it applied:
- the suite passes 228/0 on armhf and LP64;
- the framing differential below gives 454/454 on both ABIs;
carp-fmt --checkandanglerare clean locally on both.carpfiles, andruff checkis clean onserver.py;- dropping each new
lenupdate has an effect. The data one segfaults the suite, the size-line one fails 29 tests, and the CRLF one fails exactly the new/chunked-splittest. That's why the test and thelenfix belong together.
--- a/http-client.carp
+++ b/http-client.carp
@@ -365,10 +365,10 @@
; chunk-left: -1 before a size line, 0 before a chunk's CRLF, else bytes owed
(defn poll-chunked [s]
(let-do [result (Maybe.Nothing)
- running true]
+ running true
+ len (String.length (buf s))]
(while running
- (let [len (String.length (buf s))
- p @(pos s)
+ (let [p @(pos s)
left @(chunk-left s)]
(cond
(> left 0)
@@ -379,15 +379,19 @@
(set-pos! s (+ p n))
(set-chunk-left! s (- left n))
(set! running false))
- (set! running
- (fill-or-fail! s
- "malformed chunked body: truncated chunk data")))
- (= left 0)
- (cond
- (< (- len p) 2)
+ (do
(set! running
(fill-or-fail! s
"malformed chunked body: truncated chunk data"))
+ (set! len (String.length (buf s)))))
+ (= left 0)
+ (cond
+ (< (- len p) 2)
+ (do
+ (set! running
+ (fill-or-fail! s
+ "malformed chunked body: truncated chunk data"))
+ (set! len (String.length (buf s))))
(crlf-at? (buf s) p)
(do (set-pos! s (+ p 2)) (set-chunk-left! s -1))
(do
@@ -396,11 +400,13 @@
(set! running false)))
(let [eol (String.find-crlf (buf s) p len)]
(if (= eol -1)
- (set! running
- (fill-or-fail! s
- (if (< p len)
- "malformed chunked body: unterminated chunk size line"
- "malformed chunked body: missing terminating zero-size chunk")))
+ (do
+ (set! running
+ (fill-or-fail! s
+ (if (< p len)
+ "malformed chunked body: unterminated chunk size line"
+ "malformed chunked body: missing terminating zero-size chunk")))
+ (set! len (String.length (buf s))))
(let [line &(String.byte-slice (buf s) p eol)]
(match (parse-size-line line)
(Maybe.Nothing)
--- a/test/server.py
+++ b/test/server.py
@@ -229,6 +229,14 @@
return self._chunked_raw(b"5\r\nhelloXX0\r\n\r\n")
if path == "/chunked-cr-close":
return self._chunked_raw(b"5\r\nhello\r")
+ # one chunk whose size line and CRLF each arrive in pieces
+ if path == "/chunked-split":
+ self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+ self._chunked_raw(b"5")
+ for piece in (b"\r\nhello", b"\r", b"\n0\r\n\r\n"):
+ time.sleep(0.05)
+ self.wfile.write(piece)
+ return
if path == "/chunked-leading-space":
return self._chunked_raw(b" 5\r\nhello\r\n0\r\n\r\n")
if path == "/chunked-trailing-space":
--- a/test/http-client.carp
+++ b/test/http-client.carp
@@ -941,6 +941,11 @@
"a close between the CR and the LF after chunk data is an error")
(assert-equal test
+ "200 [hello]"
+ &(status-and-body (Client.get "http://127.0.0.1:8791/chunked-split"))
+ "a size line and a CRLF that arrive in pieces are read whole")
+
+ (assert-equal test
"malformed chunked body: truncated chunk data"
&(status-and-body (Client.get "http://127.0.0.1:8791/chunked-oversize"))
"a chunk size above the old 16 MiB cap is no longer a parse error")3. A read error mid-chunk stays Receive, but nothing tests it (optional, pre-existing)
fill-or-fail! (:302-304) doesn't overwrite an error that fill! already set. With a 1 s read timeout ((RequestConfig.init 0 1 10)), a server that stalls mid-data, after the CR, before a size line or mid size line ends the stream as a retryable Receive error, after the partial data has been handed out. Without the guard, all four become Parse (not retryable), and the suite stays at 227/0. Main's two inline copies of the guard were equally unpinned (224/0 with them removed). One test now pins it: poll /chunked-held-half with that config, and expect hello followed by a Receive error after about 1 s.
Checked and correct
Framing differential. My own server replays 454 framings, each fetched with Client.request-stream and polled to the end, against a Python model of poll-chunked:
- the 51-byte stream
3\r\nabc\r\n2;ext=1\r\nde\r\n7 \r\nfghijkl\r\n0\r\nX-T: 1\r\n\r\n, split into two writes at every byte, sent one byte per write, and cut off at every byte; - CR and LF in separate reads with no close;
- a size line split across three reads;
- the next chunk's size line in the same read as the current chunk's tail;
- sizes
7fffffff,80000000,FFFFFFFF,100000000, 16 hex digits with leading zeros,-1, empty,5,5,5\t;x,5;,0x5,+5; - every bad CRLF after data:
XY,\rX,\n\r, and a loneX,\ror\nfollowed by a close; - a chunk whose data is
0\r\n\r\n12345, split at every byte; - one 256 KiB chunk in 41 writes;
- 3,000 one-byte chunks, in one write and in 7-byte writes;
- 300 seeded random streams, of which a quarter are truncated and a sixth have one corrupted byte:
- chunk sizes up to 5,000;
- data full of CR, LF,
0,;and 0xff; - extensions and trailers;
- random write sizes with 0–2 ms gaps.
Results, the same on armhf and LP64:
- head: 454/454.
- The polled bytes are exactly the data bytes received before the failure point.
- No poll is empty, and no poll spans two chunks.
- The error text matches the model.
- main: 454/454 against a whole-chunk model (one poll per complete chunk). So every error message is unchanged on all 454.
- The models have teeth: the head's output fails the whole-chunk model on 246 framings, and main's output fails the streaming model on 74.
- A
7fffffffchunk streams with no overflow, and anything aboveInt.MAXis still rejected as an invalid size.
Mutants of the new code, in one env-gated build, LP64 suite, baseline 227/0:
- Killed:
- a poll handing out the whole buffered tail (20 tests fail);
- one byte per poll (4 fail);
- both EOF messages, and
(< (- len p) 2)→1(/chunked-cr-closefails); - an unchecked CRLF;
- state 0 instead of -1 after the CRLF (19 fail);
size - 1(20 fail);open-streampassing 0 (31 fail);- no decrement (20 fail);
(<= p len)in the size-line message;Nothingright after a size line (26 fail);- waiting for the whole chunk (
/chunked-held-halffails).
- Survivors:
- S1 and S2 (finding 1);
- the
fill-or-fail!guard (finding 3); skip-trailers!dropped, which is equivalent because the stream is done and never reused.
Large chunks. One 4 MiB chunk takes 2,780–3,807 ms on main and 16–18 ms on the head (at load ~13).
Downstream.
- llm is the only consumer in the org.
llm-stream-extract-line(llm.carp:1826-1842) re-splits lines and holds a trailing CR across polls, so partial chunks are fine. test/llm.carp:41is the only positionalResponseStream.init, as the body says.- llm never reads
ResponseStream.error. On a truncated chunk it now sees that chunk's complete lines, where main dropped the whole chunk. Either way the stream just ends, as it did before this PR.
Poll-count tests. clean after 40 chunks and its siblings count polls, and polls no longer have to match HTTP chunks one to one. The tests stay deterministic, because every server write there is at most 132 bytes, well under one 4 KiB read.
Memory. Under --log-memory, a Client.get leaves 28 allocations unbalanced on both main and the head. The count is the same for an empty chunked body, one chunk, two chunks and 256 chunks. So it comes from outside the chunk decoder, and this PR doesn't change it.
Verdict: revise
The state machine matches a streaming decoder on all 454 framings on both ABIs, keeps every error message, and turns the quadratic large-chunk copy into a linear one. Two things before merging, both in the patch above: the new CRLF state's re-read needs a test, and the len fix removes the slowdown on chunks under 64 bytes.
…sses /chunked-split sends a size line and the CRLF after the data in pieces, so the size-line and CRLF states must read more instead of failing. poll-chunked took a strlen of the read buffer on every pass, and a chunk now takes three passes. The length is now computed once per poll and again after each read: 1-byte chunks cost 10% more client CPU than main, down from 41%, and 16-byte chunks are back at parity. A read timeout mid-chunk must end as a Receive error, which pins fill-or-fail!'s guard against overwriting a read error with Parse.
|
Follow-up to carpentry-reviewer's Findings 1 and 2. I applied the review's patch unchanged: the Finding 3. I added the test: Suite. I built each version once and compiled the C natively (armhf) and with
Mutants of
Timing. Each run is one
Measured inside the process, one 1 MiB chunk takes 205 ms on main, 5.0 ms on
This reply supersedes the PR body on two points. The suite count is now 229/0, not 227/0, and the branch's tests on main's library give 228/1, not 226/1. The body's timing table has no small-chunk rows; the table above covers them. |
There was a problem hiding this comment.
Build & Tests
- CI at
29e4238: ubuntu-latest is green. macos-latest has never run. GitHub cancelled both attempts after 15 minutes, with 0 steps, because no runner was free:The job was not acquired by Runner of type hosted even after multiple attempts, a macOS arm64 capacity notice. No code ran, so this says nothing about the branch, but the check isn't green. The reviewer App can't re-queue it (Resource not accessible by integration). - The follow-up is what it says. I applied round 1's patch to
b3c8c46withgit apply.http-client.carpandtest/server.pythen match29e4238byte for byte.test/http-client.carpdiffers only by the new 7-line timeout test, and every test-file change in the commit is an addition. - Local suite (compat core, the local carp's
max-lengthliteral patched inmain.c, everytest/*.carpport-rewritten): 229 passed, 0 failed on armhf and on LP64 (aarch64-linux-gnu-gcc, CI's ABI). It takes about 6 s. Round 1's head took 3.5 s, and the new timeout test accounts for the extra second.
Prior feedback
All three round-1 points are addressed. I re-ran round 1's mutants as one env-gated build of 29e4238, LP64 suite, baseline 229/0:
| mutant | round 1 (b3c8c46) |
29e4238 |
|---|---|---|
| S1: a partly buffered size line fails instead of reading | survived, 227/0 | killed: only a size line and a CRLF that arrive in pieces are read whole fails |
| S2: the CRLF state fails instead of reading | survived, 227/0 | killed: only that same test fails |
G: fill-or-fail! overwrites a read error with Parse |
survived, 227/0 | killed: only a read timeout mid-chunk stays a Receive error fails, on LP64 and armhf |
data-state len update dropped |
(patch only) | killed: segfault |
CRLF-state len update dropped |
(patch only) | killed: only the /chunked-split test fails |
size-line len update dropped |
(patch only) | killed: 20 failures |
- Finding 1 (no test for the CRLF state's re-read): fixed by
/chunked-splitand its assertion. As the table shows, that test is now the only thing standing between S1 or S2 and a green suite. - Finding 2 (
lenrecomputed on every pass): fixed by the patch. I didn't re-time it. The code is byte-identical to what I timed in round 1. On a quiet box that was +9% against main for 1-byte chunks and +7% for 16-byte chunks, and noise from 64-byte chunks up. The reply's CPU-time table agrees: +8% to +10% for 1-byte chunks and within 2% for 16-byte chunks. - Finding 3 (optional, the guard was unpinned): fixed by the new test. Its only job is to pin the guard, since
hellobeing handed out first is already the held-half test's job. It also passes on main's library, where the two inline copies of the guard behave the same way. - Every test-file change in
29e4238is an addition (+12 lines intest/http-client.carp, +8 intest/server.py). No assertion that killed one of round 1's other mutants was edited or removed. Those mutants ran against code that is unchanged apart from wherelengets recomputed.
Findings
No new findings. The follow-up changes only where len is recomputed and adds two tests, and both behave as intended. Two more things I checked:
- The branch's tests on main's library: 228/1 on armhf and LP64. The held-half test is the only failure, as the reply says. So
/chunked-splitand the timeout test pass on main too: they pin re-reads and a guard that main's inline code already had, not new behaviour. - With #33, the decoder leaks nothing.
git merge-treeof29e4238and #33'sde0dd08is conflict-free (tree929a246), because the two PRs change different functions. I built 17 chunked paths with--log-memoryand recorded the allocation balance per request, on armhf and LP64.- The paths go through
Client.get, and throughrequest-streamwithpolluntilNothingand thenclose. They cover a 1 MiB chunk, 300 small chunks,/chunked-split, an extension, a trailer, five framing errors (truncated, bad hex, missing CRLF, CR then close, no terminator) and the 1 s read timeout mid-chunk. - The merged tree balances to 0 on all 17 rows.
- This branch alone leaks 31 per request (37 on the 300-chunk route). Main leaks exactly the same, row for row.
- So the 28-allocation note from round 1 is the request-loop leak that #33 fixes. This PR's decoder adds none of its own, on the success paths or the error paths.
- The paths go through
Verdict: revise (CI only)
The code is ready: every round-1 point is fixed, each new test kills exactly the mutant it was written for, and the suite is 229/0 on both ABIs. What's missing is a macOS run, so re-run that job (gh run rerun 41666026659 --failed --repo carpentry-org/http-client) and merge once it's green. Don't push an empty commit to retrigger it.
On a chunked body,
ResponseStream.pollwaited until a whole chunk and its CRLF were buffered, and every read on the way re-copied the part of the chunk already buffered. One S-byte chunk cost about S²/4096 bytes of copying, and nothing was handed out until its last byte arrived. Content-Length bodies already stream each read as it comes.poll-chunkedis now a state machine over a newchunk-leftfield: -1 before a size line, 0 before a chunk's CRLF, otherwise the data bytes the chunk still owes. In the data state a poll returns what is buffered of the current chunk, or reads once and returns that read's share. It never crosses into the next chunk and never returns an empty string. Size-line parsing, trailers and every framing error message are unchanged; EOF before a chunk's CRLF is still "truncated chunk data".Measured on a Raspberry Pi over loopback, two runs each while other jobs shared the machine (
Client.get; the held row is the firstpollofrequest-stream):Behaviour change: one
pollmay now return part of an HTTP chunk, and a chunk's data can be handed out before a broken CRLF after it is noticed (the error still lands inerror, anddrain-streamandClient.getstill fail).poll-rawalready returns arbitrary read boundaries, RFC 9112 §7.1 gives chunk boundaries no meaning,drain-streamconcatenates, and llm'sLlmStreambuffers lines itself (llm-stream-extract-line, which also holds a CR split from its LF).API:
ResponseStreamgains a field, soResponseStream.inittakes 10 arguments and the next release is a minor, as with #31. llm's test helper (test/llm.carp:41) builds a stream positionally and needs a-1after thechunkedflag when it bumps past 0.10.0; llm itself never constructs one. Reusingremainingfor the chunk state would avoid the new field, at the cost of giving that field a second meaning.Tests:
/chunked-held-halfsends half a chunk, holds 10 s, then the rest; its first poll must behellowithin 5 s (main: "waited for the close")./chunked-largeis one 1 MiB chunk of numbered lines, compared whole./chunked-cr-closeends the connection between a chunk's CR and LF, which pins the new CRLF state's short read: without it, checking for 1 buffered byte instead of 2, or a different EOF message there, passed the suite. The last two pass on main too.Suite: 227/0 built LP64 (CI's ABI) and armhf, against 224/0 on main; the branch's tests on main's library give 226/1, only the held-half test failing. This Pi's carp compiles
max-lengthto-1l, so those runs patch that one literal in the generated C.Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.