Repository navigation
cmd/neofs-node: fix gRPC reload server misattribution, register Netmap properly - #4286
Merged
roman-khimov merged 4 commits intoNov 11, 2026
Merged
Conversation
fabwa
requested review from
End-rey,
carpawell,
cthulhu-rider and
roman-khimov
as code owners
November 8, 2026 18:39
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4286 +/- ##
==========================================
+ Coverage 32.78% 32.83% +0.05%
==========================================
Files 673 673
Lines 41707 41706 -1
==========================================
+ Hits 13673 13696 +23
+ Misses 28034 28010 -24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
End-rey
reviewed
Nov 9, 2026
carpawell
reviewed
Nov 10, 2026
fabwa
force-pushed
the
fix/grpc-reload-netmap-and-alignment
branch
from
November 10, 2026 16:16
5aa5f5e to
7d5776e
Compare
End-rey
approved these changes
Nov 10, 2026
fabwa
force-pushed
the
fix/grpc-reload-netmap-and-alignment
branch
from
November 10, 2026 19:35
cfa2344 to
f42faef
Compare
Member
|
Conflicts. |
…p properly H9: SIGHUP reload can permanently misattribute a running server to the wrong endpoint after a prior partial failure reloadGRPC matched currently-running servers to a freshly recomputed config snapshot by pairing c.cfgGRPC.servers[i] with oldCfg[i] - plain positional index. c.appCfg (and so oldCfg, recomputed from it on every reload) always reflects the full declared config file regardless of which servers actually started; c.cfgGRPC.servers only ever contains the ones that did. The moment any single endpoint ever fails to build - a temporarily unbindable address, a bad cert path - that slice has one fewer entry than oldCfg, and every positional pairing from that point on is off by one, silently reusing the wrong server under the wrong endpoint's label on every subsequent reload until restart. Fix: track each server together with its own endpoint/config snapshot in a single cfgGRPCServer struct (cmd/neofs-node/config.go), instead of in parallel slices indexed separately from the config. reloadGRPC now builds its "old" lookup directly from c.cfgGRPC.servers's own stored snapshots - each entry is self-describing, so there's no positional pairing left to desync, regardless of how many prior reloads partially failed. The matching/reconciliation logic itself is extracted into reconcileGRPCServers for direct testing without needing a real nCli/TLS-dependent build path. H10: NetmapService missing from gRPC servers (re)created by reload initNetmapService registered NetmapServiceServer by looping c.cfgGRPC.servers directly, instead of using cfgGRPC.registerService like every other service (Accounting, Container, Session, Object, Reputation all use it). registerService both applies the registrator to current servers and stores it for future ones; looping the current servers directly does only the former. Any server (re)created by a later reload - endpoint, conn_limit, or TLS cert change, not just a brand new endpoint - therefore serves every other registered service but not Netmap, until the whole node is restarted. Fix: register NetmapServiceServer the same way every other service does, via cfgGRPC.registerService. Added a regression test reproducing H9's exact two-reload scenario: build servers for endpoints A/B/C, fail B's rebuild on reload #1 (A/C reused, B dropped), then reload #2 with no further config changes - asserting C's original server is correctly reused (not misattributed to B) and only B is rebuilt. Signed-off-by: Fabian Wahle <fabian@nash.io>
Review learnings: comments cut to the contract and the one non-obvious point (endpoint matching, not positional), unexplained H9 reference dropped, CHANGELOG entries shortened and numbered. Signed-off-by: Fabian Wahle <fabian@nash.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback: the struct comment explained why it exists rather than what it is, and e said nothing about what it holds. entry is already used for the same type elsewhere in the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fabian Wahle <fabian@nash.io>
Review feedback. The self-signed TLS check must compare the previously declared configuration, which is what the caller passed before: the certificate's SANs are fixed at startup, so an endpoint whose rebuild failed still has one and needs no restart, yet it is absent from the running servers and looked newly added. Only the per-endpoint matching uses each running server's own snapshot. Also store the reconciled set before bailing out on an empty one. Every old server has been stopped by then, so the previous entries would be offered for reuse; reaching that path currently terminates the node, so this is not observable today. Reuse the first reload's config in the test, as its comment claims. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fabian Wahle <fabian@nash.io>
fabwa
force-pushed
the
fix/grpc-reload-netmap-and-alignment
branch
from
November 11, 2026 11:51
f42faef to
a193853
Compare
Contributor
Author
|
Rebased onto fd6e171. Only CHANGELOG conflicted; the code patch is unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related bugs in SIGHUP config reload handling for gRPC servers.
H9: reload can permanently misattribute a running server to the wrong endpoint after a prior partial failure
reloadGRPCmatched currently-running servers to a freshly recomputed config snapshot by pairingc.cfgGRPC.servers[i]witholdCfg[i]- plain positional index.c.appCfg(and sooldCfg, recomputed from it on every reload) always reflects the full declared config file regardless of which servers actually started;c.cfgGRPC.serversonly ever contains the ones that did. The moment any single endpoint ever fails to build - a temporarily unbindable address, a bad cert path - that slice has one fewer entry thanoldCfg, and every positional pairing from that point on is off by one, silently reusing the wrong server under the wrong endpoint's label on every subsequent reload until restart.H10: NetmapService missing from gRPC servers (re)created by reload
initNetmapServiceregisteredNetmapServiceServerby loopingc.cfgGRPC.serversdirectly, instead of usingcfgGRPC.registerServicelike every other service (Accounting, Container, Session, Object, Reputation all use it).registerServiceboth applies the registrator to current servers and stores it for future ones; looping the current servers directly does only the former. Any server (re)created by a later reload - endpoint, conn_limit, or TLS cert change, not just a brand new endpoint - therefore serves every other registered service but not Netmap, until the whole node is restarted.Fix
cfgGRPCServerstruct, instead of in parallel slices indexed separately from the config.reloadGRPCnow builds its "old" lookup directly fromc.cfgGRPC.servers's own stored snapshots - each entry is self-describing, so there's no positional pairing left to desync, regardless of how many prior reloads partially failed. The matching/reconciliation logic itself is extracted intoreconcileGRPCServersfor direct testing without needing a realnCli/TLS-dependent build path.NetmapServiceServerthe same way every other service does, viacfgGRPC.registerService.Test plan
go vet ./...clean;go test ./cmd/neofs-node/...passes; full repo build (go build ./...) clean.mastertip.🤖 Generated with Claude Code