Skip to content

Distributor: reject oversize decompressed request bodies instead of truncating them - #7902

Open
sarankumar1325 wants to merge 3 commits into
cortexproject:masterfrom
sarankumar1325:fix/reject-oversize-decompressed-bodies
Open

sarankumar1325 wants to merge 3 commits into
cortexproject:masterfrom
sarankumar1325:fix/reject-oversize-decompressed-bodies

Conversation

@sarankumar1325

Copy link
Copy Markdown
Contributor

What this PR does:

decompressRequest caps the request reader at maxSize+1 bytes, but the deferred size check only fired when reading or unmarshalling also returned an error. A body that got cut at the cap on a message boundary was accepted with the extra data silently dropped. This happens with uncompressed bodies that have no Content-Length, and with gzip bodies, where the compressed size is under the limit but the decompressed size is over it.

Changes:

  • pkg/util/http.go: always reject a body longer than maxSize, even if reading it did not fail.
  • pkg/util/http.go: decompressFromBuffer (the httpgrpc path, where the body comes in as a *bytes.Buffer) had no Gzip case and returned a nil body, so gzip requests on this path were treated as empty. It now decodes gzip with the same maxSize+1 cap.
  • pkg/util/push/otlp.go: the OTLP JSON branch reads the body itself, so it gets the same check.

With this change, oversize requests return 400 received message larger than max (...) instead of a partial write.

Tests:

  • TestParseProtoReader_GzipDecompressionBomb now checks for the size error. Before, it only checked for some error; it passed because of an incidental illegal tag unmarshal error.
  • New TestParseProtoReader_OversizeBodyIsRejected covers uncompressed and gzip streams and gzip in a bytes.Buffer. It checks every limit between the encoded size and the decompressed size. Without the fix it fails: the body is accepted at message boundaries, and the buffered gzip path decodes 0 series.
  • New TestOTLPWriteHandler_OversizeBodyIsRejected covers protobuf and JSON, each uncompressed and gzip. All 4 subtests fail without the fix.
  • go test -tags "netgo slicelabels" ./pkg/util/ ./pkg/util/push/, go vet and gofmt all pass.

AI disclosure (per GENAI_POLICY.md): I used an AI coding assistant for most of this change. I checked the diff and the tests myself, and confirmed the tests fail before the fix and pass after it.

Which issue(s) this PR fixes:
Fixes #7900

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

The request readers are capped at maxSize+1 bytes, but a body that was
cut at that cap was only rejected when the truncation also caused a
read/unmarshal error. Bodies truncated at a message boundary were
silently accepted with data dropped. Always reject bodies longer than
maxSize, apply the same check to the OTLP JSON path, and decode gzip
bodies in the buffered (httpgrpc) path instead of returning an empty
body.

Fixes cortexproject#7900

Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>
@sarankumar1325
sarankumar1325 requested a review from a team as a code owner October 29, 2026 08:25
Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>

@sandy2008 sandy2008 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing #7900. The fix looks right to me: the deferred check in decompressRequest now rejects any read that hit the maxSize+1 cap, the new Gzip case in decompressFromBuffer decodes httpgrpc bodies with the same cap, and the OTLP JSON branch gets the same check, all returning 400 like the existing Content-Length check. Remote write and remote read only use RawSnappy, whose decoded length is checked before decoding, so they're unaffected (./pkg/util/push/ and the remote read tests pass). With http.go and otlp.go reverted to master, the bomb test, the gzip over-limit and buffered-gzip subtests of the new util test, and all four new OTLP subtests fail, as they should. Two small things inline.

Comment thread pkg/util/http_test.go Outdated
t.Run(tc.name+" over limit", func(t *testing.T) {
// Every limit below the decoded size must be reported as a size error,
// whether or not the cut lands on a message boundary.
for maxSize := len(tc.data); maxSize < len(body); maxSize++ {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the noCompression case tc.data is body, so this loop starts at len(body) and runs zero times: "noCompression over limit" passes vacuously, and it still passes with http.go reverted to master. Starting every case at the compressed size keeps the gzip cases as they are and gives the uncompressed stream the same ~5k limits, message-boundary cuts included; with this change the subtest fails on master and passes with the fix.

Suggested change
for maxSize := len(tc.data); maxSize < len(body); maxSize++ {
for maxSize := gz.Len(); maxSize < len(body); maxSize++ {

Comment thread CHANGELOG.md Outdated
* [ENHANCEMENT] Ruler: Add rule information to query parameters sent to the Query Frontend when `-ruler.frontend-address` is configured, so the Query Frontend query stats logs (`-frontend.enabled-ruler-query-stats`) can identify the evaluated rule. In this mode, the Ruler query stats log (`-ruler.query-stats-enabled`) omits the fetched statistics fields, which are always zero. #6539
* [ENHANCEMENT] gRPC: When `-auth.enabled=true`, a gRPC request without the `X-Scope-OrgID` metadata is now rejected with the `UNAUTHENTICATED` status code instead of `UNKNOWN`. #7873
* [ENHANCEMENT] Upgrade promql-engine and Thanos to latest. #7889
* [BUGFIX] Distributor: Reject Remote Write and OTLP request bodies whose decompressed size exceeds the configured limit instead of silently truncating them, and decode gzip-compressed bodies received over httpgrpc instead of treating them as empty. #7902

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: remote write (v1 and v2) and remote read always parse with util.RawSnappy, whose decoded size was already checked against the limit through snappy.DecodedLen, so they never truncated. Only the OTLP handler uses NoCompression/Gzip, so maybe scope the entry to OTLP:

Suggested change
* [BUGFIX] Distributor: Reject Remote Write and OTLP request bodies whose decompressed size exceeds the configured limit instead of silently truncating them, and decode gzip-compressed bodies received over httpgrpc instead of treating them as empty. #7902
* [BUGFIX] Distributor: Reject OTLP request bodies whose decompressed size exceeds `-distributor.otlp-max-recv-msg-size` instead of silently truncating them, and decode gzip-compressed OTLP protobuf bodies received over httpgrpc instead of treating them as empty. #7902

…cope the CHANGELOG entry to OTLP

Signed-off-by: sarankumar1325 <120080287+sarankumar1325@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTLP ingestion: oversize gzip/uncompressed bodies are truncated instead of rejected, and gzip bodies arriving over httpgrpc are ignored

2 participants