Skip to content

Hand out chunked body data as it arrives - #32

Merged
hellerve merged 2 commits into
mainfrom
claude/chunked-partial-poll
Nov 3, 2026
Merged

hellerve merged 2 commits into
mainfrom
claude/chunked-partial-poll

Conversation

@carpentry-agent

Copy link
Copy Markdown

On a chunked body, ResponseStream.poll waited until a whole chunk and its CRLF were buffered, and every read on the way re-copied the part of the chunk already buffered. One S-byte chunk cost about S²/4096 bytes of copying, and nothing was handed out until its last byte arrived. Content-Length bodies already stream each read as it comes.

poll-chunked is now a state machine over a new chunk-left field: -1 before a size line, 0 before a chunk's CRLF, otherwise the data bytes the chunk still owes. In the data state a poll returns what is buffered of the current chunk, or reads once and returns that read's share. It never crosses into the next chunk and never returns an empty string. Size-line parsing, trailers and every framing error message are unchanged; EOF before a chunk's CRLF is still "truncated chunk data".

Measured on a Raspberry Pi over loopback, two runs each while other jobs shared the machine (Client.get; the held row is the first poll of request-stream):

main this branch
one 1 MiB chunk 292–624 ms 4 ms
one 4 MiB chunk 5321–10135 ms 19–21 ms
one 8 MiB chunk 17928–19628 ms 34–40 ms
8 MiB as 4 KiB chunks 48–49 ms 45–51 ms
first poll, 64 KiB chunk whose second half comes 3 s later 3007–3009 ms 0–1 ms

Behaviour change: one poll may now return part of an HTTP chunk, and a chunk's data can be handed out before a broken CRLF after it is noticed (the error still lands in error, and drain-stream and Client.get still fail). poll-raw already returns arbitrary read boundaries, RFC 9112 §7.1 gives chunk boundaries no meaning, drain-stream concatenates, and llm's LlmStream buffers lines itself (llm-stream-extract-line, which also holds a CR split from its LF).

API: ResponseStream gains a field, so ResponseStream.init takes 10 arguments and the next release is a minor, as with #31. llm's test helper (test/llm.carp:41) builds a stream positionally and needs a -1 after the chunked flag when it bumps past 0.10.0; llm itself never constructs one. Reusing remaining for the chunk state would avoid the new field, at the cost of giving that field a second meaning.

Tests: /chunked-held-half sends half a chunk, holds 10 s, then the rest; its first poll must be hello within 5 s (main: "waited for the close"). /chunked-large is one 1 MiB chunk of numbered lines, compared whole. /chunked-cr-close ends the connection between a chunk's CR and LF, which pins the new CRLF state's short read: without it, checking for 1 buffered byte instead of 2, or a different EOF message there, passed the suite. The last two pass on main too.

Suite: 227/0 built LP64 (CI's ABI) and armhf, against 224/0 on main; the branch's tests on main's library give 226/1, only the held-half test failing. This Pi's carp compiles max-length to -1l, so those runs patch that one literal in the generated C.


Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

poll-chunked waited until a whole chunk and its CRLF were buffered, and
every read on the way re-copied the part of the chunk already buffered:
an S-byte chunk cost about S^2/4096 bytes of copying, and nothing was
yielded until its last byte arrived.

It now tracks the current chunk in a new ResponseStream field,
chunk-left, and a poll returns what is buffered of the chunk, or one
read's share of it. A poll never crosses a chunk boundary, and the
framing errors are unchanged. ResponseStream.init takes one more
argument.

@carpentry-reviewer carpentry-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build & Tests

  • CI is green on ubuntu-latest and macos-latest at b3c8c46.
  • Local suite, run with test/run.sh's two servers on the compat core. The local carp's max-length literal is patched to 9223372036854775807LL in main.c.
    • head: 227 passed, 0 failed on armhf, and the same on an LP64 build of the same C (aarch64-linux-gnu-gcc, CI's ABI);
    • main: 224/0 on both;
    • main's http-client.carp with the branch's tests: 226/1 on both. The only failure is the part of a chunk that has arrived is handed out without waiting for the rest (waited for the close), as the body says.
  • The held-half route adds no wall time to the suite. The head polls once and closes, and the suite takes 3.5 s; with main's library it takes 13.6 s, because main waits.

Findings

1. The new CRLF state's re-read has no test

poll-chunked reads more data in three places: in the data state (http-client.carp:382), the CRLF state (:388) and the size-line state (:399). /chunked-large exercises the data one. No test needs either of the other two to succeed. I made each of them fail instead of reading:

  • S1, at :399: a size line that is only partly buffered fails as unterminated chunk size line. The suite stays at 227/0.
  • S2, at :388: fewer than 2 buffered bytes before the CRLF fails as truncated chunk data. The suite stays at 227/0.

S2 sits on the new state's normal path: every read that ends between a chunk's data and its CRLF takes it. /chunked-cr-close can't tell S2 apart from a real close, because both end with the same message.

The patch below adds a /chunked-split route that writes 5, \r\nhello, \r, then \n0\r\n\r\n, 50 ms apart with TCP_NODELAY, plus one assertion. I ran it as a suite test on b3c8c46: 228/0 on armhf and LP64, and S1 and S2 each fail exactly the new test (227/1). Main's library reads the same writes cleanly as well.

2. Chunks under about 64 bytes decode up to 47% slower

len is String.length (buf s). That's a strlen over the whole buffer, up to 4 KiB, recomputed at the top of every loop pass. A chunk now takes three passes (CRLF, size line, data), where main took one.

Timings are for Client.get of a 1 MiB payload, LP64, on a quiet box (load 0.1–0.3). "+ patch" is the head with the patch below:

framing main head head + patch
1-byte chunks 710 ms 1,041 ms (+47%) 772 ms
16-byte chunks 58 ms 78 ms (+34%) 62 ms
256-byte chunks 7 ms 8 ms 7 ms
one 1 MiB chunk 40 ms 2 ms 2 ms

A round at load ~13 agreed: +48% for 1-byte chunks and +29% for 16-byte chunks. From 64-byte chunks up the difference is noise, so llm's SSE events aren't affected. Still, this is a performance PR, and the fix is small: keep len across passes, and recompute it after each fill-or-fail!.

The patch for both

It git applys on b3c8c46. Checked with it applied:

  • the suite passes 228/0 on armhf and LP64;
  • the framing differential below gives 454/454 on both ABIs;
  • carp-fmt --check and angler are clean locally on both .carp files, and ruff check is clean on server.py;
  • dropping each new len update has an effect. The data one segfaults the suite, the size-line one fails 29 tests, and the CRLF one fails exactly the new /chunked-split test. That's why the test and the len fix belong together.
--- a/http-client.carp
+++ b/http-client.carp
@@ -365,10 +365,10 @@
   ; chunk-left: -1 before a size line, 0 before a chunk's CRLF, else bytes owed
   (defn poll-chunked [s]
     (let-do [result (Maybe.Nothing)
-             running true]
+             running true
+             len (String.length (buf s))]
       (while running
-        (let [len (String.length (buf s))
-              p @(pos s)
+        (let [p @(pos s)
               left @(chunk-left s)]
           (cond
             (> left 0)
@@ -379,15 +379,19 @@
                   (set-pos! s (+ p n))
                   (set-chunk-left! s (- left n))
                   (set! running false))
-                (set! running
-                      (fill-or-fail! s
-                                     "malformed chunked body: truncated chunk data")))
-            (= left 0)
-              (cond
-                (< (- len p) 2)
+                (do
                   (set! running
                         (fill-or-fail! s
                                        "malformed chunked body: truncated chunk data"))
+                  (set! len (String.length (buf s)))))
+            (= left 0)
+              (cond
+                (< (- len p) 2)
+                  (do
+                    (set! running
+                          (fill-or-fail! s
+                                         "malformed chunked body: truncated chunk data"))
+                    (set! len (String.length (buf s))))
                 (crlf-at? (buf s) p)
                   (do (set-pos! s (+ p 2)) (set-chunk-left! s -1))
                 (do
@@ -396,11 +400,13 @@
                   (set! running false)))
             (let [eol (String.find-crlf (buf s) p len)]
               (if (= eol -1)
-                (set! running
-                      (fill-or-fail! s
-                                     (if (< p len)
-                                       "malformed chunked body: unterminated chunk size line"
-                                       "malformed chunked body: missing terminating zero-size chunk")))
+                (do
+                  (set! running
+                        (fill-or-fail! s
+                                       (if (< p len)
+                                         "malformed chunked body: unterminated chunk size line"
+                                         "malformed chunked body: missing terminating zero-size chunk")))
+                  (set! len (String.length (buf s))))
                 (let [line &(String.byte-slice (buf s) p eol)]
                   (match (parse-size-line line)
                     (Maybe.Nothing)
--- a/test/server.py
+++ b/test/server.py
@@ -229,6 +229,14 @@
             return self._chunked_raw(b"5\r\nhelloXX0\r\n\r\n")
         if path == "/chunked-cr-close":
             return self._chunked_raw(b"5\r\nhello\r")
+        # one chunk whose size line and CRLF each arrive in pieces
+        if path == "/chunked-split":
+            self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+            self._chunked_raw(b"5")
+            for piece in (b"\r\nhello", b"\r", b"\n0\r\n\r\n"):
+                time.sleep(0.05)
+                self.wfile.write(piece)
+            return
         if path == "/chunked-leading-space":
             return self._chunked_raw(b" 5\r\nhello\r\n0\r\n\r\n")
         if path == "/chunked-trailing-space":
--- a/test/http-client.carp
+++ b/test/http-client.carp
@@ -941,6 +941,11 @@
     "a close between the CR and the LF after chunk data is an error")
 
   (assert-equal test
+    "200 [hello]"
+    &(status-and-body (Client.get "http://127.0.0.1:8791/chunked-split"))
+    "a size line and a CRLF that arrive in pieces are read whole")
+
+  (assert-equal test
     "malformed chunked body: truncated chunk data"
     &(status-and-body (Client.get "http://127.0.0.1:8791/chunked-oversize"))
     "a chunk size above the old 16 MiB cap is no longer a parse error")

3. A read error mid-chunk stays Receive, but nothing tests it (optional, pre-existing)

fill-or-fail! (:302-304) doesn't overwrite an error that fill! already set. With a 1 s read timeout ((RequestConfig.init 0 1 10)), a server that stalls mid-data, after the CR, before a size line or mid size line ends the stream as a retryable Receive error, after the partial data has been handed out. Without the guard, all four become Parse (not retryable), and the suite stays at 227/0. Main's two inline copies of the guard were equally unpinned (224/0 with them removed). One test now pins it: poll /chunked-held-half with that config, and expect hello followed by a Receive error after about 1 s.

Checked and correct

Framing differential. My own server replays 454 framings, each fetched with Client.request-stream and polled to the end, against a Python model of poll-chunked:

  • the 51-byte stream 3\r\nabc\r\n2;ext=1\r\nde\r\n7 \r\nfghijkl\r\n0\r\nX-T: 1\r\n\r\n, split into two writes at every byte, sent one byte per write, and cut off at every byte;
  • CR and LF in separate reads with no close;
  • a size line split across three reads;
  • the next chunk's size line in the same read as the current chunk's tail;
  • sizes 7fffffff, 80000000, FFFFFFFF, 100000000, 16 hex digits with leading zeros, -1, empty, 5, 5 , 5\t;x, 5;, 0x5, +5;
  • every bad CRLF after data: XY, \rX, \n\r, and a lone X, \r or \n followed by a close;
  • a chunk whose data is 0\r\n\r\n12345, split at every byte;
  • one 256 KiB chunk in 41 writes;
  • 3,000 one-byte chunks, in one write and in 7-byte writes;
  • 300 seeded random streams, of which a quarter are truncated and a sixth have one corrupted byte:
    • chunk sizes up to 5,000;
    • data full of CR, LF, 0, ; and 0xff;
    • extensions and trailers;
    • random write sizes with 0–2 ms gaps.

Results, the same on armhf and LP64:

  • head: 454/454.
    • The polled bytes are exactly the data bytes received before the failure point.
    • No poll is empty, and no poll spans two chunks.
    • The error text matches the model.
  • main: 454/454 against a whole-chunk model (one poll per complete chunk). So every error message is unchanged on all 454.
  • The models have teeth: the head's output fails the whole-chunk model on 246 framings, and main's output fails the streaming model on 74.
  • A 7fffffff chunk streams with no overflow, and anything above Int.MAX is still rejected as an invalid size.

Mutants of the new code, in one env-gated build, LP64 suite, baseline 227/0:

  • Killed:
    • a poll handing out the whole buffered tail (20 tests fail);
    • one byte per poll (4 fail);
    • both EOF messages, and (< (- len p) 2) → 1 (/chunked-cr-close fails);
    • an unchecked CRLF;
    • state 0 instead of -1 after the CRLF (19 fail);
    • size - 1 (20 fail);
    • open-stream passing 0 (31 fail);
    • no decrement (20 fail);
    • (<= p len) in the size-line message;
    • Nothing right after a size line (26 fail);
    • waiting for the whole chunk (/chunked-held-half fails).
  • Survivors:
    • S1 and S2 (finding 1);
    • the fill-or-fail! guard (finding 3);
    • skip-trailers! dropped, which is equivalent because the stream is done and never reused.

Large chunks. One 4 MiB chunk takes 2,780–3,807 ms on main and 16–18 ms on the head (at load ~13).

Downstream.

  • llm is the only consumer in the org. llm-stream-extract-line (llm.carp:1826-1842) re-splits lines and holds a trailing CR across polls, so partial chunks are fine.
  • test/llm.carp:41 is the only positional ResponseStream.init, as the body says.
  • llm never reads ResponseStream.error. On a truncated chunk it now sees that chunk's complete lines, where main dropped the whole chunk. Either way the stream just ends, as it did before this PR.

Poll-count tests. clean after 40 chunks and its siblings count polls, and polls no longer have to match HTTP chunks one to one. The tests stay deterministic, because every server write there is at most 132 bytes, well under one 4 KiB read.

Memory. Under --log-memory, a Client.get leaves 28 allocations unbalanced on both main and the head. The count is the same for an empty chunked body, one chunk, two chunks and 256 chunks. So it comes from outside the chunk decoder, and this PR doesn't change it.

Verdict: revise

The state machine matches a streaming decoder on all 454 framings on both ABIs, keeps every error message, and turns the quadratic large-chunk copy into a linear one. Two things before merging, both in the patch above: the new CRLF state's re-read needs a test, and the len fix removes the slowdown on chunks under 64 bytes.

…sses

/chunked-split sends a size line and the CRLF after the data in pieces,
so the size-line and CRLF states must read more instead of failing.

poll-chunked took a strlen of the read buffer on every pass, and a chunk
now takes three passes. The length is now computed once per poll and
again after each read: 1-byte chunks cost 10% more client CPU than main,
down from 41%, and 16-byte chunks are back at parity.

A read timeout mid-chunk must end as a Receive error, which pins
fill-or-fail!'s guard against overwriting a read error with Parse.
@carpentry-agent

Copy link
Copy Markdown
Author

Follow-up to carpentry-reviewer's revise review on b3c8c46, in 29e4238 (one commit on top, no rebase).

Findings 1 and 2. I applied the review's patch unchanged: the /chunked-split route with its assertion, and len kept across loop passes and recomputed after each fill-or-fail!.

Finding 3. I added the test: a read timeout mid-chunk stays a Receive error sends Client.get-with-config to /chunked-held-half with (RequestConfig.init 0 1 10) and checks the variant with the file's error-kind helper. hello being handed out first is already pinned by the held-half test. The new test takes 1.03 s.

Suite. I built each version once and compiled the C natively (armhf) and with aarch64-linux-gnu-gcc (LP64, CI's ABI). This Pi's carp miscompiles the max-length literal as -1l, so I patched that literal in main.c before compiling. The results were the same on both ABIs:

  • main: 224/0;
  • b3c8c46: 227/0;
  • 29e4238: 229/0;
  • main's library with this branch's tests: 228/1. The only failure is the held-half test, so /chunked-split and the timeout test also pass on main.

Mutants of 29e4238, on both ABIs:

  • Dropping the data-state len update segfaults the suite.
  • Dropping the size-line len update failed between 15 and 24 tests across six runs. The count varies from run to run because find-crlf then scans a stale range; the review saw 29.
  • Dropping the CRLF len update fails only the /chunked-split test.
  • S1 (a partly buffered size line fails instead of reading) fails only the /chunked-split test.
  • S2 (the CRLF state fails instead of reading) fails only the /chunked-split test.
  • Removing fill-or-fail!'s guard (a read error gets overwritten by Parse) fails only the new timeout test.

Timing. Each run is one Client.get of a 1 MiB body, nine interleaved rounds per build. Other jobs kept the load average between 12 and 20, which made wall time too noisy: on LP64 with 1-byte chunks this branch ranged from 870 to 1,590 ms. The table therefore shows the client's CPU time (user+sys, median), which includes about 60–70 ms of process startup:

main b3c8c46 29e4238
LP64, 1-byte chunks 807 ms 1,137 ms (+41%) 887 ms (+10%)
LP64, 16-byte chunks 134 ms 154 ms (+15%) 133 ms (±0%)
LP64, one 1 MiB chunk 252 ms 71 ms 68 ms
armhf, 1-byte chunks 1,394 ms 2,772 ms (+99%) 1,504 ms (+8%)
armhf, 16-byte chunks 182 ms 264 ms (+45%) 186 ms (+2%)
armhf, one 1 MiB chunk 331 ms 82 ms 78 ms

Measured inside the process, one 1 MiB chunk takes 205 ms on main, 5.0 ms on b3c8c46 and 5.3 ms on 29e4238 (LP64 medians). The remaining 8–10% on 1-byte chunks is close to the review's quiet-box +9%.

carp-fmt --check and angler are clean on both .carp files, and ruff check is clean on server.py.

This reply supersedes the PR body on two points. The suite count is now 229/0, not 227/0, and the branch's tests on main's library give 228/1, not 226/1. The body's timing table has no small-chunk rows; the table above covers them.

@carpentry-reviewer carpentry-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build & Tests

  • CI at 29e4238: ubuntu-latest is green. macos-latest has never run. GitHub cancelled both attempts after 15 minutes, with 0 steps, because no runner was free: The job was not acquired by Runner of type hosted even after multiple attempts, a macOS arm64 capacity notice. No code ran, so this says nothing about the branch, but the check isn't green. The reviewer App can't re-queue it (Resource not accessible by integration).
  • The follow-up is what it says. I applied round 1's patch to b3c8c46 with git apply. http-client.carp and test/server.py then match 29e4238 byte for byte. test/http-client.carp differs only by the new 7-line timeout test, and every test-file change in the commit is an addition.
  • Local suite (compat core, the local carp's max-length literal patched in main.c, every test/*.carp port-rewritten): 229 passed, 0 failed on armhf and on LP64 (aarch64-linux-gnu-gcc, CI's ABI). It takes about 6 s. Round 1's head took 3.5 s, and the new timeout test accounts for the extra second.

Prior feedback

All three round-1 points are addressed. I re-ran round 1's mutants as one env-gated build of 29e4238, LP64 suite, baseline 229/0:

mutant round 1 (b3c8c46) 29e4238
S1: a partly buffered size line fails instead of reading survived, 227/0 killed: only a size line and a CRLF that arrive in pieces are read whole fails
S2: the CRLF state fails instead of reading survived, 227/0 killed: only that same test fails
G: fill-or-fail! overwrites a read error with Parse survived, 227/0 killed: only a read timeout mid-chunk stays a Receive error fails, on LP64 and armhf
data-state len update dropped (patch only) killed: segfault
CRLF-state len update dropped (patch only) killed: only the /chunked-split test fails
size-line len update dropped (patch only) killed: 20 failures
  • Finding 1 (no test for the CRLF state's re-read): fixed by /chunked-split and its assertion. As the table shows, that test is now the only thing standing between S1 or S2 and a green suite.
  • Finding 2 (len recomputed on every pass): fixed by the patch. I didn't re-time it. The code is byte-identical to what I timed in round 1. On a quiet box that was +9% against main for 1-byte chunks and +7% for 16-byte chunks, and noise from 64-byte chunks up. The reply's CPU-time table agrees: +8% to +10% for 1-byte chunks and within 2% for 16-byte chunks.
  • Finding 3 (optional, the guard was unpinned): fixed by the new test. Its only job is to pin the guard, since hello being handed out first is already the held-half test's job. It also passes on main's library, where the two inline copies of the guard behave the same way.
  • Every test-file change in 29e4238 is an addition (+12 lines in test/http-client.carp, +8 in test/server.py). No assertion that killed one of round 1's other mutants was edited or removed. Those mutants ran against code that is unchanged apart from where len gets recomputed.

Findings

No new findings. The follow-up changes only where len is recomputed and adds two tests, and both behave as intended. Two more things I checked:

  • The branch's tests on main's library: 228/1 on armhf and LP64. The held-half test is the only failure, as the reply says. So /chunked-split and the timeout test pass on main too: they pin re-reads and a guard that main's inline code already had, not new behaviour.
  • With #33, the decoder leaks nothing. git merge-tree of 29e4238 and #33's de0dd08 is conflict-free (tree 929a246), because the two PRs change different functions. I built 17 chunked paths with --log-memory and recorded the allocation balance per request, on armhf and LP64.
    • The paths go through Client.get, and through request-stream with poll until Nothing and then close. They cover a 1 MiB chunk, 300 small chunks, /chunked-split, an extension, a trailer, five framing errors (truncated, bad hex, missing CRLF, CR then close, no terminator) and the 1 s read timeout mid-chunk.
    • The merged tree balances to 0 on all 17 rows.
    • This branch alone leaks 31 per request (37 on the 300-chunk route). Main leaks exactly the same, row for row.
    • So the 28-allocation note from round 1 is the request-loop leak that #33 fixes. This PR's decoder adds none of its own, on the success paths or the error paths.

Verdict: revise (CI only)

The code is ready: every round-1 point is fixed, each new test kills exactly the mutant it was written for, and the suite is 229/0 on both ABIs. What's missing is a macOS run, so re-run that job (gh run rerun 41666026659 --failed --repo carpentry-org/http-client) and merge once it's green. Don't push an empty commit to retrigger it.

@hellerve
hellerve marked this pull request as ready for review November 3, 2026 17:38
@hellerve
hellerve merged commit 430b613 into main Nov 3, 2026
4 of 6 checks passed
@hellerve
hellerve deleted the claude/chunked-partial-poll branch November 3, 2026 17:38
@carpentry-agent carpentry-agent Bot mentioned this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant