Repository navigation
Split Accept-* and Transfer-Encoding lists outside quoted-strings; resolve Cache-Control quoted-pairs - #51
Conversation
Weighted.parse-list (Accept, Accept-Encoding, Accept-Language) split on every comma, so a comma inside a quoted parameter value cut the entry short, losing its q, and turned the tail into a bogus entry. The Transfer-Encoding chunked check had the same split, so a quoted comma in a transfer-parameter could produce a `chunked` token. Cache-Control kept quoted-pairs escaped in quoted values. MediaType's and Cache-Control's identical splitting loops become one split-outside-quotes, and their two unquotes become MediaType's quoted-pair-resolving one. Weighted and the Transfer-Encoding check now use that splitter. MediaType's behaviour is unchanged.
There was a problem hiding this comment.
Build & Tests
- Head
0a95f57:carp -x test/http.carppasses 561/0 (10-02 compat core, armhf). - Master's
http.carp(fbef064) with this branch'stest/http.carp: 555/6. The 6 failures are exactly the new rows. - CI is green on ubuntu-latest and macos-latest.
What I checked
-
No public API is removed.
MediaType.split-params,MediaType.unquote,CacheControl.split-directivesandCacheControl.unquotewere allprivateandhiddenon master, and the two new top-level helpers are too. -
MediaType is unchanged, by reading.
split-outside-quotesis master'sMediaType.split-paramsloop with the separator as a parameter.unquote-stringisMediaType.unquoteverbatim. The splitter makes one pass and does onebyte-sliceper segment, so it stays linear. -
13 mutants, each run against the full 561-row suite:
mutant result killed by splitter ignores quotes 553/8 the 5 new list/TE rows plus 3 existing MediaType/Cache-Control rows splitter never skips \xpairs560/1 an escaped quote does not end a quoted parameter when splitting rangesunquote-stringresolves no pairs559/2 the new Cache-Control row and an existing MediaType row chunked?back onString.split-by560/1 chunked? ignores a chunked inside a quoted transfer-parameterWeighted.parse-listback onString.split-by557/4 the 4 new Accept-* rows CacheControl.parseback onString.split-by559/2 2 existing Cache-Control rows CacheControlwith master's old unquote560/1 CacheControl resolves a quoted-pair in a quoted directive valueMediaTypewith Cache-Control's old unquote560/1 an escaped quote inside a quoted name is unescapeda quote opens but never closes 554/7 list, Cache-Control and multipart rows chunked?splits on;560/1 chunked? true when chunked is last in a coding listsplitter skips \xpairs outside quotes too561/0, survives – unquote-stringdrops itsi + 1 < lenguard561/0, survives – unquote-string'slen > 1guard becomeslen > 0561/0, survives – The three survivors are malformed-input guards. All three were moved unchanged from master's MediaType code, where they were not pinned either, so this PR doesn't regress them. By reading,
Cache-Control: private="abc\"reaches the second guard, because the splitter keeps the escaped final quote inside the string.Cache-Control: private="reaches the third, which on that mutant would callString.byte-slice v 1 0. Rows for those two inputs would be a cheap follow-up, but I haven't built them. -
Unterminated quotes. The body names the class. The orchestrator's probe, built on master and on this head, gives the concrete cases:
field master head Accept-Language: da;x="a, en[da] [en][da]Accept-Encoding: gzip;x="a, br;q=0.5[gzip] [br q=0.5][gzip]Transfer-Encoding: gzip;p="x, chunkedchunked not chunked Transfer-Encoding: "gzip, chunkedchunked not chunked Transfer-Encoding: x"y, chunkedchunked not chunked Transfer-Encoding: gzip, chunked;p="xnot chunked not chunked All of these values are invalid: a
"can't appear in a token, and the quoted-string never closes. So neither answer is wrong by the RFC, and the head matches what MediaType and Cache-Control already did. The one consumer this reaches is framing in http-client:Response.chunked?decides how a response body is read (http-client.carp:725). A server that sends such a header is broken, though, and it controls its own framing anyway. web frames requests itself and doesn't callchunked?. So I'd leave the behaviour as it is. If Veit prefers master's split for an unbalanced quote, the splitter could treat a quote that never closes as an ordinary byte. That would change MediaType and Cache-Control on the same inputs.
Findings
None required. The fix matches the body, each changed call site is pinned by a row that fails on master, and every mutant that changes behaviour on well-formed input is killed.
Optional: the two guards above (private="abc\" and private=") and the outside-quotes pair-skip are unpinned malformed-input arms, inherited from master's MediaType code.
Verdict: merge
Quoted commas no longer split Accept-*, Transfer-Encoding or Cache-Control entries, and quoted-pairs are resolved. The 6 rows fail on master, MediaType's path is unchanged, and the only behaviour change outside the intended class is on invalid unterminated quotes, which the body discloses.
Bugs
Probed on master (fbef064) and on this branch:
Accept: text/html;level="1,2";q=0.5, application/jsontext/html@1.0, application/json@1.0text/html@0.5, application/json@1.0Accept-Encoding: gzip;x="a,b";q=0.3, brgzip@1.0, b"@1.0, br@1.0gzip@0.3, br@1.0Accept-Language: en;x="a,b";q=0.2, deen@1.0, b"@1.0, de@1.0en@0.2, de@1.0Cache-Control: private="a\"b", directiveprivatea\"ba"bTransfer-Encoding: gzip;p=",chunked,",chunked?truefalseWeighted.parse-list, which backsAccept.parse/negotiate,AcceptEncodingandAcceptLanguage, split the field on every comma, including one inside a quoted parameter value. The entry lost everything after that comma, itsqincluded, and the tail became an entry of its own.Acceptdrops that tail because2"is not a media type, butAcceptEncodingandAcceptLanguagekeep it as a bogus coding or language rangeb"at weight 1.CacheControlstripped the quotes from a quoted value but left its quoted-pairs (RFC 9110 §5.6.4) escaped.MediaTypealready resolved them.Request.chunked?/Response.chunked?split the lower-casedTransfer-Encodingvalue on every comma, so a quoted comma inside a transfer-parameter produced achunkedtoken, and that token decides how the body is framed. The rule itself is unchanged: any trimmed token equal tochunkedstill counts. Only the splitting changed.Consolidation
MediaType.split-paramsandCacheControl.split-directiveswere the same loop with different separators.CacheControl.unquotewasMediaType.unquoteminus the quoted-pair step. They are now onesplit-outside-quotes(splits on a given separator outside quoted-strings and skips\xpairs inside them) and oneunquote-string(MediaType's version). Both are top-level private+hidden next toparse-digits, which, liketransfer-encoding-chunked?, is already used from other modules this way.Weighted.parse-listand theTransfer-Encodingcheck use the splitter in place ofString.split-by. MediaType's path is the old code moved unchanged, soMediaType.parse/parse-params(which web calls) behave exactly as before; the differential below confirms it.http.carpis 32 lines shorter (+53/−85).CacheControl.parse's doc now says that quoted-pairs are resolved.One side effect on malformed input: in a weighted list or
Transfer-Encoding, an unbalanced"now runs to the end of the field, as it already did in Cache-Control and in media-type parameters. For example,gzip;x="a, bris now one entry where it was two.Left alone
ETagList.split-list: in an entity-tag,\is an ordinaryetagc, not an escape (RFC 9110 §8.8.3). The shared splitter's quoted-pair rule would read"a\", "b"as one tag where it is two.scan-quoted): it reads quoted-strings positionally, with a comma separating both auth-params and challenges. It is a scanner, not a list split.cookie-octetexcludes",\,,and;, so quoted-string rules have nothing to apply to, and a\rule could only misfire.Form: urlencoded bodies have no quoted-strings.ByteRange's list has none either.Tests
There are 6 new rows in
test/http.carp. I ran master'shttp.carpagainst this test file: 555 passed, 6 failed, and the 6 failures are exactly the new rows:a comma inside a quoted parameter does not split the rangetext/html@1.0, application/json@1.0an escaped quote does not end a quoted parameter when splitting rangestext/html@1.0, application/json@1.0a comma inside a quoted parameter does not split the codinggzip@1.0, b"@1.0, br@1.0a comma inside a quoted parameter does not split the language rangeen@1.0, b"@1.0, de@1.0chunked? ignores a chunked inside a quoted transfer-parametertrueCacheControl resolves a quoted-pair in a quoted directive valuea\"bOn this branch the suite passes, 561/0.
I broke the code one way at a time and ran the new rows against each version. Every mutant is caught:
chunked?row (5 of 6)\xquoted-pairsan escaped quote does not end a quoted parameter when splitting rangesunquote-stringdoes not resolve quoted-pairsCacheControl resolves a quoted-pair in a quoted directive valueDifferential
I built a driver against master and against this branch and ran it on 2,500 field values. 2,000 are random strings of 1 to 12 tokens drawn from
a b x t/h */* ; , = " \ q=0.5 ;q=0.5 chunked p=and a space. The other 500 are Cache-Control-shaped strings (name="…"lists whose inner text uses the same alphabet plus\"and\\). The random set alone produced no quoted Cache-Control value containing a quoted-pair. For each value the driver recordsMediaType.parse-params,CacheControl.parse,Accept.parse,AcceptEncoding.parse,AcceptLanguage.parseandchunked?with thatTransfer-Encoding.MediaType.parse-paramsCacheControl.parseAccept.parseAcceptEncoding.parseAcceptLanguage.parsechunked?No difference falls outside these classes. Some values are in a class but produce no difference, for example when a later duplicate directive overwrites the changed value, or when the quoted comma sits in a range that
Acceptdrops anyway.Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.