Skip to content

meta: unsubscribe from the chain before closing its delivery channels - #4283

Closed
fabwa wants to merge 4 commits into
nspcc-dev:masterfrom
fabwa:fix/meta-unsubscribe-before-close
Closed

fabwa wants to merge 4 commits into
nspcc-dev:masterfrom
fabwa:fix/meta-unsubscribe-before-close

Conversation

@fabwa

@fabwa fabwa commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Meta.Run subscribes m.bCh/m.evsCh to the chain's block header and notification delivery, waits for its own handler goroutines to stop on context cancellation, then closes both channels. The chain keeps sending to them from its own, independently-running delivery goroutine for as long as they're subscribed - stopping Run's handler goroutines does not stop that. If the chain sends while or after Run closes the channel, it panics with send on closed channel, crashing the node during what should be a graceful shutdown.

neo-go's Blockchain already provides the counterpart for exactly this: UnsubscribeFromHeadersOfAddedBlocks/UnsubscribeFromNotifications, whose own doc comments say "you can close it afterwards", and whose implementation drains any in-flight send before returning - built for precisely this shutdown sequence. SideChain (this package's MetaChain implementation) wraps the Subscribe* half of that pair but never gained the Unsubscribe* half.

Fix

Add UnsubscribeFromBlocks/UnsubscribeFromNotifications to SideChain, mirroring its own existing SubscribeForBlocks/SubscribeForNotifications naming, delegating to neo-go's methods exactly as the existing Subscribe* methods do. Add the same two methods to the MetaChain interface, and call them in Run between wg.Wait() and the close() calls.

Test plan

  • Added a regression test with a fake MetaChain whose Subscribe* methods hammer the channel from an independent goroutine exactly like the real chain does, confirming Run no longer panics under this pressure.
  • Confirmed it reliably panics with the exact same mechanism (send on closed channel, from the chain's delivery goroutine) when Run's two new Unsubscribe* calls are temporarily removed, across repeated runs - then restored the fix.
  • go vet ./... clean; go test ./pkg/services/meta/... ./pkg/services/sidechain/... ./cmd/neofs-node/... passes.
  • Patch verified to apply cleanly and pass tests in a separate clean worktree against the current master tip.

🤖 Generated with Claude Code

Comment thread pkg/services/meta/meta.go Resolved
Comment thread pkg/services/meta/meta.go Resolved
Comment thread pkg/services/meta/meta.go Resolved
Comment thread pkg/services/meta/run_internal_test.go Resolved
Comment thread pkg/services/meta/meta.go
close(m.evsCh)
// m.bCh and m.evsCh are not closed: the chain writes into them and keeps
// sending after the handlers above are gone, so closing them here races
// with those sends. m.taskQueue's only writers are the handlers.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving these channels open avoids the panic, but leaves active subscriptions without consumers. Once a buffer fills, the dispatcher blocks; a subsequent AddBlock holds addLock waiting for it, so core.Close also hangs. Delivery and consumer shutdown still need coordination.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, we have never seen this panic before in practice, but i agree that such fixes do not change the situation. Sidechain must be stopped first, then readers should be closed. it may even be true, but it is hard to say inside this Meta service

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, on both your point and @carpawell's - closing this in favour of #4299, which does the ordering in cmd/neofs-node instead: the chain is stopped first and the service keeps draining until it is down, so the dispatcher cannot block and core.Close cannot wedge on addLock.

Fabian Wahle and others added 3 commits November 10, 2026 17:11
Meta.Run subscribes m.bCh/m.evsCh to the chain's block header and
notification delivery, waits for its own handler goroutines to stop on
context cancellation, then closes both channels. The chain keeps
sending to them from its own, independently-running delivery
goroutine for as long as they're subscribed - stopping Run's handler
goroutines does not stop that. If the chain sends while or after Run
closes the channel, it panics with "send on closed channel", crashing
the node during what should be a graceful shutdown.

neo-go's Blockchain already provides the counterpart for exactly this:
UnsubscribeFromHeadersOfAddedBlocks/UnsubscribeFromNotifications, whose
own doc comments say "you can close it afterwards", and whose
implementation drains any in-flight send before returning - built for
precisely this shutdown sequence. SideChain (this package's MetaChain
implementation) wraps the Subscribe* half of that pair but never
gained the Unsubscribe* half.

Fix: add UnsubscribeFromBlocks/UnsubscribeFromNotifications to SideChain,
mirroring its own existing SubscribeForBlocks/SubscribeForNotifications
naming, delegating to neo-go's methods exactly as the existing Subscribe*
methods do. Add the same two methods to the MetaChain interface, and call
them in Run between wg.Wait() and the close() calls.

Added a regression test with a fake MetaChain whose Subscribe* methods
hammer the channel from an independent goroutine exactly like the real
chain does, confirming Run no longer panics under this pressure.
Confirmed it reliably panics with the exact same mechanism
("send on closed channel", from the chain's delivery goroutine) when
Run's two new Unsubscribe* calls are temporarily removed, across
repeated runs - then restored the fix.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Review learnings: comments cut to the contract, unexplained M43 reference
dropped, CHANGELOG entry shortened and numbered.

Signed-off-by: Fabian Wahle <fabian@nash.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unsubscribing before the close cannot work: shutdown() runs sidechain.Stop()
before waiting for workers, so the dispatcher may already be gone and the
unsubscribe loop blocks forever; and both channels share that one dispatcher,
so unsubscribing them in sequence deadlocks whenever it is blocked sending to
the second one. Meta does not own the write end of either channel and neither
handler needs a close - they return on ctx.Done - so just leave them open.
Reverts the MetaChain interface change, which also broke mockMetadataChain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
@fabwa
fabwa force-pushed the fix/meta-unsubscribe-before-close branch from 31928f4 to 225529c Compare November 10, 2026 16:16
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.77%. Comparing base (ae56c6a) to head (81a7fbf).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4283      +/-   ##
==========================================
+ Coverage   32.63%   32.77%   +0.13%     
==========================================
  Files         673      673              
  Lines       41690    41688       -2     
==========================================
+ Hits        13606    13662      +56     
+ Misses      28084    28026      -58     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The repository's linter rejects the manual Add/go/Done form.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Fabian Wahle <fabian@nash.io>
@fabwa

fabwa commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #4299, which fixes this as a shutdown-ordering change in cmd/neofs-node rather than inside Meta.Run.

@fabwa fabwa closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants