Repository navigation
Distributor: reject oversize decompressed request bodies instead of truncating them - #7902
sarankumar1325 wants to merge 3 commits into
Conversation
The request readers are capped at maxSize+1 bytes, but a body that was cut at that cap was only rejected when the truncation also caused a read/unmarshal error. Bodies truncated at a message boundary were silently accepted with data dropped. Always reject bodies longer than maxSize, apply the same check to the OTLP JSON path, and decode gzip bodies in the buffered (httpgrpc) path instead of returning an empty body. Fixes cortexproject#7900 Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
sandy2008
left a comment
There was a problem hiding this comment.
Thanks for fixing #7900. The fix looks right to me: the deferred check in decompressRequest now rejects any read that hit the maxSize+1 cap, the new Gzip case in decompressFromBuffer decodes httpgrpc bodies with the same cap, and the OTLP JSON branch gets the same check, all returning 400 like the existing Content-Length check. Remote write and remote read only use RawSnappy, whose decoded length is checked before decoding, so they're unaffected (./pkg/util/push/ and the remote read tests pass). With http.go and otlp.go reverted to master, the bomb test, the gzip over-limit and buffered-gzip subtests of the new util test, and all four new OTLP subtests fail, as they should. Two small things inline.
| t.Run(tc.name+" over limit", func(t *testing.T) { | ||
| // Every limit below the decoded size must be reported as a size error, | ||
| // whether or not the cut lands on a message boundary. | ||
| for maxSize := len(tc.data); maxSize < len(body); maxSize++ { |
There was a problem hiding this comment.
For the noCompression case tc.data is body, so this loop starts at len(body) and runs zero times: "noCompression over limit" passes vacuously, and it still passes with http.go reverted to master. Starting every case at the compressed size keeps the gzip cases as they are and gives the uncompressed stream the same ~5k limits, message-boundary cuts included; with this change the subtest fails on master and passes with the fix.
| for maxSize := len(tc.data); maxSize < len(body); maxSize++ { | |
| for maxSize := gz.Len(); maxSize < len(body); maxSize++ { |
| * [ENHANCEMENT] Ruler: Add rule information to query parameters sent to the Query Frontend when `-ruler.frontend-address` is configured, so the Query Frontend query stats logs (`-frontend.enabled-ruler-query-stats`) can identify the evaluated rule. In this mode, the Ruler query stats log (`-ruler.query-stats-enabled`) omits the fetched statistics fields, which are always zero. #6539 | ||
| * [ENHANCEMENT] gRPC: When `-auth.enabled=true`, a gRPC request without the `X-Scope-OrgID` metadata is now rejected with the `UNAUTHENTICATED` status code instead of `UNKNOWN`. #7873 | ||
| * [ENHANCEMENT] Upgrade promql-engine and Thanos to latest. #7889 | ||
| * [BUGFIX] Distributor: Reject Remote Write and OTLP request bodies whose decompressed size exceeds the configured limit instead of silently truncating them, and decode gzip-compressed bodies received over httpgrpc instead of treating them as empty. #7902 |
There was a problem hiding this comment.
Nit: remote write (v1 and v2) and remote read always parse with util.RawSnappy, whose decoded size was already checked against the limit through snappy.DecodedLen, so they never truncated. Only the OTLP handler uses NoCompression/Gzip, so maybe scope the entry to OTLP:
| * [BUGFIX] Distributor: Reject Remote Write and OTLP request bodies whose decompressed size exceeds the configured limit instead of silently truncating them, and decode gzip-compressed bodies received over httpgrpc instead of treating them as empty. #7902 | |
| * [BUGFIX] Distributor: Reject OTLP request bodies whose decompressed size exceeds `-distributor.otlp-max-recv-msg-size` instead of silently truncating them, and decode gzip-compressed OTLP protobuf bodies received over httpgrpc instead of treating them as empty. #7902 |
…cope the CHANGELOG entry to OTLP Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
What this PR does:
decompressRequestcaps the request reader atmaxSize+1bytes, but the deferred size check only fired when reading or unmarshalling also returned an error. A body that got cut at the cap on a message boundary was accepted with the extra data silently dropped. This happens with uncompressed bodies that have noContent-Length, and with gzip bodies, where the compressed size is under the limit but the decompressed size is over it.Changes:
pkg/util/http.go: always reject a body longer thanmaxSize, even if reading it did not fail.pkg/util/http.go:decompressFromBuffer(the httpgrpc path, where the body comes in as a*bytes.Buffer) had noGzipcase and returned a nil body, so gzip requests on this path were treated as empty. It now decodes gzip with the samemaxSize+1cap.pkg/util/push/otlp.go: the OTLP JSON branch reads the body itself, so it gets the same check.With this change, oversize requests return 400
received message larger than max (...)instead of a partial write.Tests:
TestParseProtoReader_GzipDecompressionBombnow checks for the size error. Before, it only checked for some error; it passed because of an incidentalillegal tagunmarshal error.TestParseProtoReader_OversizeBodyIsRejectedcovers uncompressed and gzip streams and gzip in abytes.Buffer. It checks every limit between the encoded size and the decompressed size. Without the fix it fails: the body is accepted at message boundaries, and the buffered gzip path decodes 0 series.TestOTLPWriteHandler_OversizeBodyIsRejectedcovers protobuf and JSON, each uncompressed and gzip. All 4 subtests fail without the fix.go test -tags "netgo slicelabels" ./pkg/util/ ./pkg/util/push/,go vetandgofmtall pass.AI disclosure (per GENAI_POLICY.md): I used an AI coding assistant for most of this change. I checked the diff and the tests myself, and confirmed the tests fail before the fix and pass after it.
Which issue(s) this PR fixes:
Fixes #7900
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags