Skip to content

meta: stop the metadata chain before the service reading it - #4299

Open
fabwa wants to merge 2 commits into
nspcc-dev:masterfrom
fabwa:fix/meta-chain-shutdown-order
Open

fabwa wants to merge 2 commits into
nspcc-dev:masterfrom
fabwa:fix/meta-chain-shutdown-order

Conversation

@fabwa

@fabwa fabwa commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Replaces #4283, which tried to solve this inside Meta.Run by unsubscribing. That deadlocks, and as @carpawell put it there, the ordering "is hard to say inside this Meta service" — so this does it in cmd/neofs-node, where both lifetimes are owned.

Problem

SideChain.Stop is what actually takes the chain down (core.Close), and it is registered as a node closer. Closers run in shutdown(), after wait() has already cancelled the node context — and the metadata service, like every other worker, stops the moment that context is done. So on every shutdown the consumer tears down first and the still-running chain keeps writing into the channels it was handed, which is where send on closed channel comes from.

Nothing drains those subscriptions in that window either. If a buffer fills, the chain's dispatcher blocks, the next AddBlock holds addLock waiting for it, and core.Close — i.e. the closer itself — hangs with it, as @End-rey pointed out on #4283.

Fix

The service now runs until the closer stops it, after the chain:

c.closers = append(c.closers, func() {
    c.sidechain.Stop()
    stopMeta()
})

so the order is: chain stops delivering → service stops reading → c.wg.Wait() collects it. The service keeps draining while Stop runs, so the dispatcher never blocks and core.Close cannot wedge on addLock. Its context is deliberately not the node's; that is the one deviation from the other workers and the comment says why.

The two chain-fed channels are no longer closed. Blockchain.Close closes stopCh and waits on runToExitCh, which is the Run loop only — notificationDispatcher observes the same stopCh but nothing waits for it, and its sends sit outside its select. So there is no point at which a consumer can know the sends have stopped, whatever the ordering. m.taskQueue is still closed: its only writer is notifyReceived, called from notificationHandler, which Run does wait on.

Test plan

  • TestMeta_Run_DoesNotCloseChainFedChannels feeds both subscriptions from goroutines Run does not wait on, as the real dispatcher does, and fails with send on closed channel if either close is restored — confirmed over -count=3 -race.
  • -race green on pkg/services/meta and cmd/neofs-node, including -shuffle=1794309779113128012 (the seed from meta: unsubscribe from the chain before closing its delivery channels #4283's metrics-registry report).
  • golangci-lint with the shared config, go build ./... and go vet ./... clean.
  • The shutdown ordering itself is not unit-tested: it needs a real cfg and a real chain, and cmd/neofs-node has no harness for either.

🤖 Generated with Claude Code

fabwa pushed a commit to fabwa/neofs-node that referenced this pull request Nov 10, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.83%. Comparing base (34d007f) to head (0b95959).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
cmd/neofs-node/meta.go 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4299      +/-   ##
==========================================
+ Coverage   32.70%   32.83%   +0.12%     
==========================================
  Files         673      673              
  Lines       41698    41697       -1     
==========================================
+ Hits        13638    13691      +53     
+ Misses      28060    28006      -54     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fabian Wahle and others added 2 commits November 10, 2026 20:26
The chain keeps delivering until SideChain.Stop, which runs as a node
closer, while the service stops as soon as the node context is done -
so the consumer tore down first and the chain kept writing into its
channels. The service now runs until that closer stops it, after the
chain, which also keeps someone draining the subscriptions while the
chain shuts down.

The two chain-fed channels are no longer closed either: neo-go's Close
waits for its Run loop, not for the goroutine dispatching to
subscribers, so a consumer can never tell that those sends have ceased.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
@fabwa
fabwa force-pushed the fix/meta-chain-shutdown-order branch from fe013f1 to 0b95959 Compare November 10, 2026 19:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant