Skip to content

cmd/neofs-node: fix gRPC reload server misattribution, register Netmap properly - #4286

Merged
roman-khimov merged 4 commits into
nspcc-dev:masterfrom
fabwa:fix/grpc-reload-netmap-and-alignment
Nov 11, 2026
Merged

roman-khimov merged 4 commits into
nspcc-dev:masterfrom
fabwa:fix/grpc-reload-netmap-and-alignment

Conversation

@fabwa

@fabwa fabwa commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Two related bugs in SIGHUP config reload handling for gRPC servers.

H9: reload can permanently misattribute a running server to the wrong endpoint after a prior partial failure

reloadGRPC matched currently-running servers to a freshly recomputed config snapshot by pairing c.cfgGRPC.servers[i] with oldCfg[i] - plain positional index. c.appCfg (and so oldCfg, recomputed from it on every reload) always reflects the full declared config file regardless of which servers actually started; c.cfgGRPC.servers only ever contains the ones that did. The moment any single endpoint ever fails to build - a temporarily unbindable address, a bad cert path - that slice has one fewer entry than oldCfg, and every positional pairing from that point on is off by one, silently reusing the wrong server under the wrong endpoint's label on every subsequent reload until restart.

H10: NetmapService missing from gRPC servers (re)created by reload

initNetmapService registered NetmapServiceServer by looping c.cfgGRPC.servers directly, instead of using cfgGRPC.registerService like every other service (Accounting, Container, Session, Object, Reputation all use it). registerService both applies the registrator to current servers and stores it for future ones; looping the current servers directly does only the former. Any server (re)created by a later reload - endpoint, conn_limit, or TLS cert change, not just a brand new endpoint - therefore serves every other registered service but not Netmap, until the whole node is restarted.

Fix

  • Track each server together with its own endpoint/config snapshot in a single cfgGRPCServer struct, instead of in parallel slices indexed separately from the config. reloadGRPC now builds its "old" lookup directly from c.cfgGRPC.servers's own stored snapshots - each entry is self-describing, so there's no positional pairing left to desync, regardless of how many prior reloads partially failed. The matching/reconciliation logic itself is extracted into reconcileGRPCServers for direct testing without needing a real nCli/TLS-dependent build path.
  • Register NetmapServiceServer the same way every other service does, via cfgGRPC.registerService.

Test plan

  • Added a regression test reproducing H9's exact two-reload scenario: build servers for endpoints A/B/C, fail B's rebuild on reload Add Issue template #1 (A/C reused, B dropped), then reload [#1] Add GitHub issue templates #2 with no further config changes - asserting C's original server is correctly reused (not misattributed to B) and only B is rebuilt.
  • go vet ./... clean; go test ./cmd/neofs-node/... passes; full repo build (go build ./...) clean.
  • Patch verified to apply cleanly and pass tests/build in a separate clean worktree against the current master tip.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.00000% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.83%. Comparing base (fd6e171) to head (a193853).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
cmd/neofs-node/grpc.go 26.82% 30 Missing ⚠️
cmd/neofs-node/config.go 0.00% 2 Missing ⚠️
cmd/neofs-node/netmap.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4286      +/-   ##
==========================================
+ Coverage   32.78%   32.83%   +0.05%     
==========================================
  Files         673      673              
  Lines       41707    41706       -1     
==========================================
+ Hits        13673    13696      +23     
+ Misses      28034    28010      -24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread cmd/neofs-node/grpc.go Resolved
Comment thread cmd/neofs-node/grpc.go Resolved
Comment thread cmd/neofs-node/grpc_internal_test.go Resolved
Comment thread CHANGELOG.md Resolved
Comment thread cmd/neofs-node/config.go Resolved
Comment thread cmd/neofs-node/config.go Resolved
@fabwa
fabwa force-pushed the fix/grpc-reload-netmap-and-alignment branch from 5aa5f5e to 7d5776e Compare November 10, 2026 16:16
@fabwa
fabwa force-pushed the fix/grpc-reload-netmap-and-alignment branch from cfa2344 to f42faef Compare November 10, 2026 19:35
@roman-khimov

Copy link
Copy Markdown
Member

Conflicts.

Fabian Wahle and others added 4 commits November 11, 2026 12:44
…p properly

H9: SIGHUP reload can permanently misattribute a running server to the
wrong endpoint after a prior partial failure

reloadGRPC matched currently-running servers to a freshly recomputed
config snapshot by pairing c.cfgGRPC.servers[i] with oldCfg[i] - plain
positional index. c.appCfg (and so oldCfg, recomputed from it on every
reload) always reflects the full declared config file regardless of
which servers actually started; c.cfgGRPC.servers only ever contains
the ones that did. The moment any single endpoint ever fails to build
- a temporarily unbindable address, a bad cert path - that slice has
one fewer entry than oldCfg, and every positional pairing from that
point on is off by one, silently reusing the wrong server under the
wrong endpoint's label on every subsequent reload until restart.

Fix: track each server together with its own endpoint/config snapshot
in a single cfgGRPCServer struct (cmd/neofs-node/config.go), instead of
in parallel slices indexed separately from the config. reloadGRPC now
builds its "old" lookup directly from c.cfgGRPC.servers's own stored
snapshots - each entry is self-describing, so there's no positional
pairing left to desync, regardless of how many prior reloads partially
failed. The matching/reconciliation logic itself is extracted into
reconcileGRPCServers for direct testing without needing a real
nCli/TLS-dependent build path.

H10: NetmapService missing from gRPC servers (re)created by reload

initNetmapService registered NetmapServiceServer by looping
c.cfgGRPC.servers directly, instead of using cfgGRPC.registerService
like every other service (Accounting, Container, Session, Object,
Reputation all use it). registerService both applies the registrator
to current servers and stores it for future ones; looping the current
servers directly does only the former. Any server (re)created by a
later reload - endpoint, conn_limit, or TLS cert change, not just a
brand new endpoint - therefore serves every other registered service
but not Netmap, until the whole node is restarted.

Fix: register NetmapServiceServer the same way every other service
does, via cfgGRPC.registerService.

Added a regression test reproducing H9's exact two-reload scenario:
build servers for endpoints A/B/C, fail B's rebuild on reload #1 (A/C
reused, B dropped), then reload #2 with no further config changes -
asserting C's original server is correctly reused (not misattributed
to B) and only B is rebuilt.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Review learnings: comments cut to the contract and the one non-obvious
point (endpoint matching, not positional), unexplained H9 reference
dropped, CHANGELOG entries shortened and numbered.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback: the struct comment explained why it exists rather than
what it is, and e said nothing about what it holds. entry is already used
for the same type elsewhere in the file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
Review feedback. The self-signed TLS check must compare the previously
declared configuration, which is what the caller passed before: the
certificate's SANs are fixed at startup, so an endpoint whose rebuild
failed still has one and needs no restart, yet it is absent from the
running servers and looked newly added. Only the per-endpoint matching
uses each running server's own snapshot.

Also store the reconciled set before bailing out on an empty one. Every
old server has been stopped by then, so the previous entries would be
offered for reuse; reaching that path currently terminates the node, so
this is not observable today.

Reuse the first reload's config in the test, as its comment claims.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
@fabwa
fabwa force-pushed the fix/grpc-reload-netmap-and-alignment branch from f42faef to a193853 Compare November 11, 2026 11:51
@fabwa

fabwa commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto fd6e171. Only CHANGELOG conflicted; the code patch is unchanged.

@roman-khimov
roman-khimov merged commit b9c6b05 into nspcc-dev:master Nov 11, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants