Skip to content

Split Accept-* and Transfer-Encoding lists outside quoted-strings; resolve Cache-Control quoted-pairs - #51

Merged
hellerve merged 1 commit into
masterfrom
claude/quoted-header-lists
Nov 13, 2026
Merged

hellerve merged 1 commit into
masterfrom
claude/quoted-header-lists

Conversation

@carpentry-agent

Copy link
Copy Markdown

Bugs

Probed on master (fbef064) and on this branch:

field value master this branch
Accept: text/html;level="1,2";q=0.5, application/json text/html@1.0, application/json@1.0 text/html@0.5, application/json@1.0
Accept-Encoding: gzip;x="a,b";q=0.3, br gzip@1.0, b"@1.0, br@1.0 gzip@0.3, br@1.0
Accept-Language: en;x="a,b";q=0.2, de en@1.0, b"@1.0, de@1.0 en@0.2, de@1.0
Cache-Control: private="a\"b", directive private a\"b a"b
Transfer-Encoding: gzip;p=",chunked,", chunked? true false
  • Weighted.parse-list, which backs Accept.parse/negotiate, AcceptEncoding and AcceptLanguage, split the field on every comma, including one inside a quoted parameter value. The entry lost everything after that comma, its q included, and the tail became an entry of its own. Accept drops that tail because 2" is not a media type, but AcceptEncoding and AcceptLanguage keep it as a bogus coding or language range b" at weight 1.
  • CacheControl stripped the quotes from a quoted value but left its quoted-pairs (RFC 9110 §5.6.4) escaped. MediaType already resolved them.
  • Request.chunked?/Response.chunked? split the lower-cased Transfer-Encoding value on every comma, so a quoted comma inside a transfer-parameter produced a chunked token, and that token decides how the body is framed. The rule itself is unchanged: any trimmed token equal to chunked still counts. Only the splitting changed.

Consolidation

MediaType.split-params and CacheControl.split-directives were the same loop with different separators. CacheControl.unquote was MediaType.unquote minus the quoted-pair step. They are now one split-outside-quotes (splits on a given separator outside quoted-strings and skips \x pairs inside them) and one unquote-string (MediaType's version). Both are top-level private+hidden next to parse-digits, which, like transfer-encoding-chunked?, is already used from other modules this way. Weighted.parse-list and the Transfer-Encoding check use the splitter in place of String.split-by. MediaType's path is the old code moved unchanged, so MediaType.parse/parse-params (which web calls) behave exactly as before; the differential below confirms it. http.carp is 32 lines shorter (+53/−85). CacheControl.parse's doc now says that quoted-pairs are resolved.

One side effect on malformed input: in a weighted list or Transfer-Encoding, an unbalanced " now runs to the end of the field, as it already did in Cache-Control and in media-type parameters. For example, gzip;x="a, br is now one entry where it was two.

Left alone

  • ETagList.split-list: in an entity-tag, \ is an ordinary etagc, not an escape (RFC 9110 §8.8.3). The shared splitter's quoted-pair rule would read "a\", "b" as one tag where it is two.
  • Auth's challenge scanner (scan-quoted): it reads quoted-strings positionally, with a comma separating both auth-params and challenges. It is a scanner, not a list split.
  • Cookie splitting: cookie-octet excludes ", \, , and ;, so quoted-string rules have nothing to apply to, and a \ rule could only misfire.
  • Form: urlencoded bodies have no quoted-strings. ByteRange's list has none either.

Tests

There are 6 new rows in test/http.carp. I ran master's http.carp against this test file: 555 passed, 6 failed, and the 6 failures are exactly the new rows:

row master's actual value
a comma inside a quoted parameter does not split the range text/html@1.0, application/json@1.0
an escaped quote does not end a quoted parameter when splitting ranges text/html@1.0, application/json@1.0
a comma inside a quoted parameter does not split the coding gzip@1.0, b"@1.0, br@1.0
a comma inside a quoted parameter does not split the language range en@1.0, b"@1.0, de@1.0
chunked? ignores a chunked inside a quoted transfer-parameter true
CacheControl resolves a quoted-pair in a quoted directive value a\"b

On this branch the suite passes, 561/0.

I broke the code one way at a time and ran the new rows against each version. Every mutant is caught:

mutant rows that fail
splitter ignores quotes the 4 list rows and the chunked? row (5 of 6)
splitter does not skip \x quoted-pairs an escaped quote does not end a quoted parameter when splitting ranges
unquote-string does not resolve quoted-pairs CacheControl resolves a quoted-pair in a quoted directive value

Differential

I built a driver against master and against this branch and ran it on 2,500 field values. 2,000 are random strings of 1 to 12 tokens drawn from a b x t/h */* ; , = " \ q=0.5 ;q=0.5 chunked p= and a space. The other 500 are Cache-Control-shaped strings (name="…" lists whose inner text uses the same alphabet plus \" and \\). The random set alone produced no quoted Cache-Control value containing a quoted-pair. For each value the driver records MediaType.parse-params, CacheControl.parse, Accept.parse, AcceptEncoding.parse, AcceptLanguage.parse and chunked? with that Transfer-Encoding.

parser values that differ of which in the intended class values in the class
MediaType.parse-params 0 none expected –
CacheControl.parse 283 283 (a quoted value with a quoted-pair) 309
Accept.parse 68 68 (a comma inside a quoted-string) 376
AcceptEncoding.parse 358 358 (same) 376
AcceptLanguage.parse 358 358 (same) 376
chunked? 3 3 (same) 376

No difference falls outside these classes. Some values are in a class but produce no difference, for example when a later duplicate directive overwrites the changed value, or when the quoted comma sits in a range that Accept drops anyway.


Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

Weighted.parse-list (Accept, Accept-Encoding, Accept-Language) split on
every comma, so a comma inside a quoted parameter value cut the entry
short, losing its q, and turned the tail into a bogus entry. The
Transfer-Encoding chunked check had the same split, so a quoted comma in a
transfer-parameter could produce a `chunked` token. Cache-Control kept
quoted-pairs escaped in quoted values.

MediaType's and Cache-Control's identical splitting loops become one
split-outside-quotes, and their two unquotes become MediaType's
quoted-pair-resolving one. Weighted and the Transfer-Encoding check now
use that splitter. MediaType's behaviour is unchanged.

@carpentry-reviewer carpentry-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build & Tests

  • Head 0a95f57: carp -x test/http.carp passes 561/0 (10-02 compat core, armhf).
  • Master's http.carp (fbef064) with this branch's test/http.carp: 555/6. The 6 failures are exactly the new rows.
  • CI is green on ubuntu-latest and macos-latest.

What I checked

  • No public API is removed. MediaType.split-params, MediaType.unquote, CacheControl.split-directives and CacheControl.unquote were all private and hidden on master, and the two new top-level helpers are too.

  • MediaType is unchanged, by reading. split-outside-quotes is master's MediaType.split-params loop with the separator as a parameter. unquote-string is MediaType.unquote verbatim. The splitter makes one pass and does one byte-slice per segment, so it stays linear.

  • 13 mutants, each run against the full 561-row suite:

    mutant result killed by
    splitter ignores quotes 553/8 the 5 new list/TE rows plus 3 existing MediaType/Cache-Control rows
    splitter never skips \x pairs 560/1 an escaped quote does not end a quoted parameter when splitting ranges
    unquote-string resolves no pairs 559/2 the new Cache-Control row and an existing MediaType row
    chunked? back on String.split-by 560/1 chunked? ignores a chunked inside a quoted transfer-parameter
    Weighted.parse-list back on String.split-by 557/4 the 4 new Accept-* rows
    CacheControl.parse back on String.split-by 559/2 2 existing Cache-Control rows
    CacheControl with master's old unquote 560/1 CacheControl resolves a quoted-pair in a quoted directive value
    MediaType with Cache-Control's old unquote 560/1 an escaped quote inside a quoted name is unescaped
    a quote opens but never closes 554/7 list, Cache-Control and multipart rows
    chunked? splits on ; 560/1 chunked? true when chunked is last in a coding list
    splitter skips \x pairs outside quotes too 561/0, survives –
    unquote-string drops its i + 1 < len guard 561/0, survives –
    unquote-string's len > 1 guard becomes len > 0 561/0, survives –

    The three survivors are malformed-input guards. All three were moved unchanged from master's MediaType code, where they were not pinned either, so this PR doesn't regress them. By reading, Cache-Control: private="abc\" reaches the second guard, because the splitter keeps the escaped final quote inside the string. Cache-Control: private=" reaches the third, which on that mutant would call String.byte-slice v 1 0. Rows for those two inputs would be a cheap follow-up, but I haven't built them.

  • Unterminated quotes. The body names the class. The orchestrator's probe, built on master and on this head, gives the concrete cases:

    field master head
    Accept-Language: da;x="a, en [da] [en] [da]
    Accept-Encoding: gzip;x="a, br;q=0.5 [gzip] [br q=0.5] [gzip]
    Transfer-Encoding: gzip;p="x, chunked chunked not chunked
    Transfer-Encoding: "gzip, chunked chunked not chunked
    Transfer-Encoding: x"y, chunked chunked not chunked
    Transfer-Encoding: gzip, chunked;p="x not chunked not chunked

    All of these values are invalid: a " can't appear in a token, and the quoted-string never closes. So neither answer is wrong by the RFC, and the head matches what MediaType and Cache-Control already did. The one consumer this reaches is framing in http-client: Response.chunked? decides how a response body is read (http-client.carp:725). A server that sends such a header is broken, though, and it controls its own framing anyway. web frames requests itself and doesn't call chunked?. So I'd leave the behaviour as it is. If Veit prefers master's split for an unbalanced quote, the splitter could treat a quote that never closes as an ordinary byte. That would change MediaType and Cache-Control on the same inputs.

Findings

None required. The fix matches the body, each changed call site is pinned by a row that fails on master, and every mutant that changes behaviour on well-formed input is killed.

Optional: the two guards above (private="abc\" and private=") and the outside-quotes pair-skip are unpinned malformed-input arms, inherited from master's MediaType code.

Verdict: merge

Quoted commas no longer split Accept-*, Transfer-Encoding or Cache-Control entries, and quoted-pairs are resolved. The 6 rows fail on master, MediaType's path is unchanged, and the only behaviour change outside the intended class is on invalid unterminated quotes, which the body discloses.

@hellerve
hellerve marked this pull request as ready for review November 13, 2026 20:21
@hellerve
hellerve merged commit 7eb4ee1 into master Nov 13, 2026
2 checks passed
@hellerve
hellerve deleted the claude/quoted-header-lists branch November 13, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant