Repository navigation
Free the response and error values on every request exit - #33
Conversation
`(break)` compiles to a bare C `break;`. Carp attaches a `let`'s deleters to an inner break, but not those of `match`-bound values or of the `if` branch it leaves. The loops in `request-stream-`, `request-stream-with-jar-` and `read-headers` exited that way, so every successful request leaked the `(Pair Response String)` from `read-headers` (28 allocations for a typical head). The too-many-redirects, missing or empty Location and too-many-1xx exits leaked their response, location or message as well. The three loops now run on a `running` flag that each exit clears as the last form of its branch, so control flow is unchanged and every scope ends normally. test/memory.carp asserts an allocation balance of 0 on eight request paths; run.sh runs it with --log-memory after the suite.
|
Orchestrator check of Allocation balance per 10 requests (
Suite: 224/0 on armhf and LP64, the same as main. Every CI: ubuntu-latest is green. The macos-latest job on this run was cancelled because GitHub never assigned it a runner ("The job was not acquired by Runner of type hosted even after multiple attempts"). I re-requested the failed job. |
There was a problem hiding this comment.
Build & Tests
- CI at
de0dd08: ubuntu-latest and macos-latest are both green. The macOS job was re-queued after the no-runner cancel and has since run. Both job logs show the suite at 224/0 andtest/memory.carpat 8/0, so the memory test has now run on both OSes. - Local, armhf and LP64: every
test/*.carpwas port-rewritten, builds used private output dirs, and the max-Long literal was fixed inmain.c.- The suite passes 224/0 on head and 224/0 on main, on both ABIs.
test/memory.carppasses 8/0 on head on both ABIs. Against main's library it fails 8/8, with balances 37, 28, 37, 74, 74, 23, 37 and 37, the same as the orchestrator's comment.
- With #32:
git merge-treeofde0dd08and #32's29e4238is clean. On the merged treetest/memory.carppasses 8/0 on both ABIs, so #32's new chunk decoder adds no leak on these paths. test/run.sh: the script isset -uo pipefailwithout-e, so a red suite still runs the memory test, andexit $((suite || memory))fails the step if either run fails. CI runs it withbash test/run.sh.- http-client has no CHANGELOG, so there is nothing to update.
Findings
-
Control flow is unchanged at all 15 sites (verified).
- A token-level diff against main is exactly three
running truebindings, threewhile-do true→while running, and fifteen(break)→(set! running false). - In each loop the body is one
match(in the jar loop, aletaround onematch), and everyset!is the last form of its branch. Nothing runs after the flag is cleared that didn't run after the oldbreak, andresultgets the same value. - The generated C of
read-headers,request-stream-andrequest-stream-with-jar-has nobreak;left. Each loop compiles towhile (_n) { ... _n = running; }. - A 30-path probe prints each call's outcome (status and body length, or the
ClientError.strtext). Main and the head print the same outcome on all 30 rows.
- A token-level diff against main is exactly three
-
The leak is gone on every path I could reach (verified). The head balances to 0 on all 30 rows, on armhf and LP64.
- Coverage: the probe runs the 8 committed rows plus:
- the jar loop's error exits: no Location, with and without a Set-Cookie; empty Location; too many redirects; too many 1xx; refused; malformed status;
- a jar stream and a Set-Cookie on a 103;
- a 1 s read timeout in the middle of the head, with and without a jar;
- EOF in the middle of the head, and a 100 followed by a close;
- a header line with no colon, and a folded header;
- a redirect to a refused origin, with and without a jar;
- 101, 204, conflicting Content-Length, and a 100 followed by a 302.
- Main: 21 of the 30 rows leak, between 230 and 740 per 10 requests. The 9 rows at 0 are exits where the error value is moved into
result. - The body's "before" numbers check out where the server sends the same shape: too many 1xx is 230, no Location is 550, the jar's empty Location is 620 and 204 is 250. Rows served by
server.pyare higher (370 for a plain GET) because its responses carry more headers, as the orchestrator noted.
- Coverage: the probe runs the 8 committed rows plus:
-
Optional:
test/memory.carppins 5 of the 9 exits that leaked, and the other four can come back unnoticed.- Which exits leaked: I put
(break)back at sites in a copy of the head and ran the 30-path probe.- The six exits that hand an error or the pair straight to
resultnever leaked. Restoring all six breaks at once leaves every row at 0. Those arehttp-client.carp:586,:590,:810,:817,:1127and:1134. - The other nine did leak. The memory test covers five of them:
:599, too many 1xx;:839, empty Location;:860, too many redirects;:864, the normal exit;:1183, the jar's normal exit.
- The six exits that hand an error or the pair straight to
- Which exits aren't pinned:
request-stream-'s redirect without Location (:830), and the jar loop's no-Location (:1148), empty-Location (:1157) and too-many-redirects (:1179) exits.- A mutant that puts
(break)back at those four still passestest/memory.carp8/0. - The probe shows those paths leaking again at main's rates: 550, 550, 620 and 740 per 10 requests.
- A mutant that puts
- Suggested fix: four more rows close the gap. Three use existing routes. A redirect without Location needs one new route, because
_redirectalways sends a Location.- With the patch below, the memory test passes 12/0 on the head, on armhf and LP64.
- The four-break mutant fails exactly the four new rows (8/4).
- Main's library fails 12/12.
- It applies cleanly to
de0dd08withgit apply. Old local carp-fmt is clean on it, but CI's carp-fmt is the authority.
- Which exits leaked: I put
--- a/test/server.py
+++ b/test/server.py
@@ -162,6 +162,10 @@
code = int(qs.get("status_code", ["302"])[0])
return self._redirect(code, url)
+ # /redirect-no-location: a 302 without a Location header
+ if path == "/redirect-no-location":
+ return self._send(302)
+
# /cookies/set/<name>/<value>: set cookie then 302 to /cookies
if path.startswith("/cookies/set/"):
parts = path[len("/cookies/set/"):].split("/")
--- a/test/memory.carp
+++ b/test/memory.carp
@@ -26,6 +26,9 @@
(defn empty-location []
(ignore (Client.get "http://127.0.0.1:8791/redirect-to?url=%20")))
+(defn no-location []
+ (ignore (Client.get "http://127.0.0.1:8791/redirect-no-location")))
+
(defn too-many-interim []
(ignore (Client.get "http://127.0.0.1:8791/interim-flood/6")))
@@ -42,12 +45,37 @@
(let [jar (CookieJar.create)]
(ignore (Client.get-with-jar "http://127.0.0.1:8791/cookies/set/a/1" &jar))))
+(defn jar-no-location []
+ (let [jar (CookieJar.create)]
+ (ignore
+ (Client.get-with-jar "http://127.0.0.1:8791/redirect-no-location" &jar))))
+
+(defn jar-empty-location []
+ (let [jar (CookieJar.create)]
+ (ignore
+ (Client.get-with-jar "http://127.0.0.1:8791/redirect-to?url=%20" &jar))))
+
+(defn jar-too-many []
+ (let [jar (CookieJar.create)
+ config (RequestConfig.init 0 0 1)]
+ (ignore
+ (Client.request-with-jar-and-config "GET"
+ "http://127.0.0.1:8791/redirect/3"
+ (no-headers)
+ ""
+ &jar
+ &config))))
+
(deftest test
(assert-equal test 0l (leaked get-length) "a Content-Length response")
(assert-equal test 0l (leaked get-chunked) "a chunked response after a 103")
(assert-equal test 0l (leaked follow-redirects) "a followed redirect")
(assert-equal test 0l (leaked too-many-redirects) "too many redirects")
(assert-equal test 0l (leaked empty-location) "an empty Location")
+ (assert-equal test 0l (leaked no-location) "a redirect without Location")
(assert-equal test 0l (leaked too-many-interim) "too many 1xx responses")
(assert-equal test 0l (leaked poll-and-close) "a polled and closed stream")
- (assert-equal test 0l (leaked with-jar) "a request through a cookie jar"))
+ (assert-equal test 0l (leaked with-jar) "a request through a cookie jar")
+ (assert-equal test 0l (leaked jar-no-location) "jar: no Location")
+ (assert-equal test 0l (leaked jar-empty-location) "jar: empty Location")
+ (assert-equal test 0l (leaked jar-too-many) "jar: too many redirects"))Verdict: merge
The fix is mechanical and correct. All 15 exits keep their control flow, the head balances to 0 on every path I could reach on both ABIs, and CI is green on both OSes. Finding 3 is an optional four-row extension of the memory test.
Every request leaked.
(break)compiles to a bare Cbreak;. Carp attaches alet's deleters to an innerbreak, but not those ofmatch-bound values or of theifbranch it leaves. The loops inrequest-stream-,request-stream-with-jar-andread-headersleft through 15 such breaks:(Pair Response String)fromread-headers: the response with its header map, plus the leftover string. That is 28 allocations for a typical head.fmtmessage.The three loops now run on a
runningflag. Each exit clears it as the last form of its branch, so control flow is unchanged and every scope ends normally. The generated C has nobreak;left in these functions. The other two code breaks (parse-hex,parse-length) only have an Int, Char or Long in scope, so they stay. This is a bug fix with no API change.CI's newer compiler handles
breakthe same way:searchForInnerBreakinMemory.hsis unchanged on carp-lang master.test/memory.carpasserts an allocation balance of 0 on eight paths againstrun.sh's server.run.shruns it with--log-memoryafter the suite, and fails if either run fails. Locally it fails 8/8 on main (23 to 74 allocations per request) and passes 8/8 here, on armhf and LP64.Allocation balance per 10 requests
This is a
--log-memoryprobe: one warm-up request, then 10 calls per row, with each result dropped. armhf and LP64 gave identical numbers.request-stream,pollto the end,close: chunked / Content-Lengthget-with-jar: Set-Cookie / Set-Cookie on a redirectrequest-with-jar-and-config, too many redirects (max 2)get-with-jar: no Location / empty Location / too many 1xxget-with-jar: refused / malformed status linerequest-stream-with-jar,poll,close: Set-Cookie / chunkedNothing is left over from http, socket, tls, uri or the core on these paths.
Suite: main and this branch give identical results on armhf and LP64. The only local failures are port artifacts of running on 9791/9792 next to another session's server (2 tests in
test/cookie-jar.carphardcode 8791). carp-fmt and angler are clean.Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.