Skip to content

policer: bound EC parent payload size declared by peer before allocating - #4287

Open
fabwa wants to merge 5 commits into
nspcc-dev:masterfrom
fabwa:fix/policer-ec-part-allocation-bound
Open

fabwa wants to merge 5 commits into
nspcc-dev:masterfrom
fabwa:fix/policer-ec-part-allocation-bound

Conversation

@fabwa

@fabwa fabwa commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

checkECParts (pkg/services/policer/ec.go) reassembles/restores an EC object's parent header from a part received from another EC container member. The parent header's declared PayloadSize is used, completely unchecked, to size a buffer allocation:

parentHdr = *ph
partLen = (parentHdr.PayloadSize() + uint64(rule.DataPartNum) - 1) / uint64(rule.DataPartNum)
...
buf := make([]byte, int(partLen)*totalParts)

ph comes from hdr.Parent(), where hdr is a part object header received via HEAD from another container member (local or remote, depending on routing) - the PayloadSize field is just an integer in that response, never checked against anything before it drives an allocation. Another member of the same EC container (or an attacker who has placed themselves as a member of the same container as the victim) can declare an arbitrarily large payload size there to force a huge allocation on the node checking that container's EC parts, regardless of what it actually stores - up to an OOM kill or a makeslice panic above the runtime's allocation threshold. The same partLen also sizes the buffer in fetchObjectFromECRule (needLen := int(partLen) * totalParts), reached from the same parentHdr.

Fix

Add a MaxSizeSource to Policer, mirroring the existing MaxSizeSource/WithMaxSizeSource shape already used by the put service for the exact same purpose, and reject the declared parent payload size against it immediately when the parent header is first established - before computing partLen or allocating anything. A single check at that one point covers both downstream allocations, since they're both deterministic functions of the same parentHdr. Wired cmd/neofs-node/object.go to share the same cached max-object-size source the put service already uses, rather than creating a second, redundant cache instance.

Test plan

  • Added TestPolicer_checkECParts_RejectsOversizedParentPayload (pkg/services/policer/ec_internal_test.go), calling checkECParts directly against a crafted remote header whose parent declares a payload size far beyond a configured maximum, asserting the rejection happens (via its log entry) rather than falling through to the allocation.
  • Confirmed it fails (the rejection log entry is absent) against the unpatched code by temporarily removing just the new check's call site, then restored the fix.
  • Updated the existing TestPolicer_Run_EC/parts_restoration test, which exercises this same code path with a legitimate parent size, to supply a MaxSizeSource - it had none previously and would otherwise now be rejected by the new check.
  • go build ./..., go vet ./... clean; go test ./pkg/services/policer/... ./cmd/neofs-node/... passes in full.

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.82%. Comparing base (fd6e171) to head (e6a59be).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4287      +/-   ##
==========================================
+ Coverage   32.78%   32.82%   +0.04%     
==========================================
  Files         673      673              
  Lines       41707    41727      +20     
==========================================
+ Hits        13673    13698      +25     
+ Misses      28034    28029       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread pkg/services/policer/ec.go Resolved
Comment thread CHANGELOG.md Resolved
Comment thread pkg/services/policer/policer.go Resolved
Comment thread pkg/services/policer/ec.go Resolved
fabwa pushed a commit to fabwa/neofs-node that referenced this pull request Nov 9, 2026
Review learnings from nspcc-dev#4287: the test asserted concurrency relative to
storagerConcurrency, so it was satisfied by any value of the constant,
serial included. It now requires a literal minimum of overlapping Head
calls (and no more than the bound), and fails with "expected at least 2
concurrent Head calls, saw 0" when the constant is set to 1.

Shortened the comments to the contract and added the PR number to the
CHANGELOG entry.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread pkg/services/policer/ec.go Outdated
if p.maxSizeSrc == nil {
return errors.New("max object size source is not configured")
}
max := p.maxSizeSrc.MaxObjectSize()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in PUT service, it looks expected, but this is a validate-after-PUT check, what if max object size was bigger in the past?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be a historic Neo call i think. but not this PR's responsibility

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and MaxObjectSize is gone from this PR entirely - the bound is now derived from the EC part this node already holds, so no config history is involved.

}

parentHdr = *ph
partLen = (parentHdr.PayloadSize() + uint64(rule.DataPartNum) - 1) / uint64(rule.DataPartNum)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a duality then, we think our local part is the true part size and calculate maxParentSz based on it, but then recalculate partLen based on the parent just received (but we already know part size and believe in it). i would expect that they should never differ.

not critical

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They can differ, because parts are padded: maxParentSz is only an upper bound, so any parent size from localPartLen*(DataPartNum-1)+1 up to localPartLen*DataPartNum passes it. With localPartLen=10, DataPartNum=2 a declared 11 is in bounds but yields partLen=6. The existing partLen equality check rejects that, including against our own local part, so I left it as the single place that pins the exact size.

fabwa pushed a commit to fabwa/neofs-node that referenced this pull request Nov 10, 2026
Review learnings from nspcc-dev#4287: the test asserted concurrency relative to
storagerConcurrency, so it was satisfied by any value of the constant,
serial included. It now requires a literal minimum of overlapping Head
calls (and no more than the bound), and fails with "expected at least 2
concurrent Head calls, saw 0" when the constant is set to 1.

Shortened the comments to the contract and added the PR number to the
CHANGELOG entry.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fabwa
fabwa force-pushed the fix/policer-ec-part-allocation-bound branch from f872bc0 to b7f1a88 Compare November 10, 2026 16:17

@End-rey End-rey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conflicts.

Comment thread pkg/services/policer/ec.go Resolved
Comment thread pkg/services/policer/ec.go Resolved
fabwa pushed a commit to fabwa/neofs-node that referenced this pull request Nov 10, 2026
Review learnings from nspcc-dev#4287: the test asserted concurrency relative to
storagerConcurrency, so it was satisfied by any value of the constant,
serial included. It now requires a literal minimum of overlapping Head
calls (and no more than the bound), and fails with "expected at least 2
concurrent Head calls, saw 0" when the constant is set to 1.

Shortened the comments to the contract and added the PR number to the
CHANGELOG entry.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fabwa
fabwa force-pushed the fix/policer-ec-part-allocation-bound branch from 53db53d to 6b20092 Compare November 10, 2026 19:35
Fabian Wahle and others added 5 commits November 11, 2026 12:44
checkECParts (pkg/services/policer/ec.go) reassembles/restores an EC
object's parent header from a part received from another EC container
member. The parent header's declared PayloadSize is used, completely
unchecked, to size a buffer allocation:

    parentHdr = *ph
    partLen = (parentHdr.PayloadSize() + uint64(rule.DataPartNum) - 1) / uint64(rule.DataPartNum)
    ...
    buf := make([]byte, int(partLen)*totalParts)

ph comes from hdr.Parent(), where hdr is a part object header received
via HEAD from another container member (local or remote, depending on
routing) - the PayloadSize field is just an integer in that response,
never checked against anything before it drives an allocation. Another
member of the same EC container (or an attacker who has placed
themselves as a member of the same container as the victim) can
declare an arbitrarily large payload size there to force a huge
allocation on the node checking that container's EC parts, regardless
of what it actually stores - up to an OOM kill or a makeslice panic
above the runtime's allocation threshold. The same partLen also sizes
the buffer in fetchObjectFromECRule (needLen := int(partLen) *
totalParts), reached from the same parentHdr.

Fix: add a MaxSizeSource to Policer (mirroring the existing
MaxSizeSource/WithMaxSizeSource shape already used by the put service
for the exact same purpose) and reject the declared parent payload
size against it immediately when the parent header is first
established, before computing partLen or allocating anything. A
single check at that one point covers both downstream allocations,
since they're both deterministic functions of the same parentHdr.
Wired cmd/neofs-node/object.go to share the same cached max-object-size
source the put service already uses, rather than creating a second,
redundant cache instance.

Added TestPolicer_checkECParts_RejectsOversizedParentPayload
(pkg/services/policer/ec_internal_test.go), calling checkECParts
directly against a crafted remote header whose parent declares a
payload size far beyond a configured maximum, and asserting the
rejection happens (via its log entry) rather than falling through to
the allocation. Confirmed it fails (the rejection log entry is absent)
against the unpatched code by temporarily removing just the new
check's call site, then restored the fix. Also updated the existing
TestPolicer_Run_EC/parts_restoration test, which exercises this same
code path with a legitimate parent size, to supply a MaxSizeSource -
it has none previously and would otherwise now be rejected by the new
check.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Review feedback.

The regression fixture never reached the allocation it was meant to
guard: with a 2/0 rule and no part payload size set, the part/parent
size mismatch check returned first. Rebuilt it per review as a 2/1 rule
with one healthy remote part whose own payload size matches the derived
part length, and one missing part, so missingIdx is non-empty and
checkECParts reaches buf := make([]byte, int(partLen)*totalParts).
Confirmed it panics with "makeslice: len out of range" when the check
is removed.

Also guard the local parent-header fallback, which fed the same
allocation unchecked. Its size is not bounded by storage validation:
put/local.go validates an object's own PayloadSize against the maximum,
never the size its embedded parent header declares, so a locally stored
EC part can carry an arbitrary parent size. Covered by a 1/1-rule test
(the only rule reaching that fallback), likewise confirmed to panic
without the check.

Moved both tests into policer_test.go alongside the existing EC tests
and mocks, dropped the unexplained M15 reference, and shortened the
comments and the CHANGELOG entry, which now carries the PR number.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MaxObjectSize is current network config, but the policer validates objects
stored earlier: lowering it would make the policer refuse to restore parts
of legitimate older objects, and a transient config read failure would stop
it too. Parts are ceil(parentSize/DataPartNum) long and padded, so the part
this node holds bounds the parent without consulting any config. Drops the
MaxSizeSource plumbing again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
Review feedback: max shadows the predeclared identifier, which the
repository's linter rejects.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
The listing already fetches attributes per object, and payload size is
indexed for every one of them, so requesting it there replaces a HEAD
per local EC part.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fabwa pushed a commit to fabwa/neofs-node that referenced this pull request Nov 11, 2026
Review learnings from nspcc-dev#4287: the test asserted concurrency relative to
storagerConcurrency, so it was satisfied by any value of the constant,
serial included. It now requires a literal minimum of overlapping Head
calls (and no more than the bound), and fails with "expected at least 2
concurrent Head calls, saw 0" when the constant is set to 1.

Shortened the comments to the contract and added the PR number to the
CHANGELOG entry.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fabwa
fabwa force-pushed the fix/policer-ec-part-allocation-bound branch from 6b20092 to e6a59be Compare November 11, 2026 11:51
obj object.Object
objECRule int
objECPart int
headObj *object.Object

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems unnecessary now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants