Repository navigation
Conversation
| close(m.evsCh) | ||
| // m.bCh and m.evsCh are not closed: the chain writes into them and keeps | ||
| // sending after the handlers above are gone, so closing them here races | ||
| // with those sends. m.taskQueue's only writers are the handlers. |
There was a problem hiding this comment.
Leaving these channels open avoids the panic, but leaves active subscriptions without consumers. Once a buffer fills, the dispatcher blocks; a subsequent AddBlock holds addLock waiting for it, so core.Close also hangs. Delivery and consumer shutdown still need coordination.
There was a problem hiding this comment.
TBH, we have never seen this panic before in practice, but i agree that such fixes do not change the situation. Sidechain must be stopped first, then readers should be closed. it may even be true, but it is hard to say inside this Meta service
There was a problem hiding this comment.
Agreed, on both your point and @carpawell's - closing this in favour of #4299, which does the ordering in cmd/neofs-node instead: the chain is stopped first and the service keeps draining until it is down, so the dispatcher cannot block and core.Close cannot wedge on addLock.
Meta.Run subscribes m.bCh/m.evsCh to the chain's block header and
notification delivery, waits for its own handler goroutines to stop on
context cancellation, then closes both channels. The chain keeps
sending to them from its own, independently-running delivery
goroutine for as long as they're subscribed - stopping Run's handler
goroutines does not stop that. If the chain sends while or after Run
closes the channel, it panics with "send on closed channel", crashing
the node during what should be a graceful shutdown.
neo-go's Blockchain already provides the counterpart for exactly this:
UnsubscribeFromHeadersOfAddedBlocks/UnsubscribeFromNotifications, whose
own doc comments say "you can close it afterwards", and whose
implementation drains any in-flight send before returning - built for
precisely this shutdown sequence. SideChain (this package's MetaChain
implementation) wraps the Subscribe* half of that pair but never
gained the Unsubscribe* half.
Fix: add UnsubscribeFromBlocks/UnsubscribeFromNotifications to SideChain,
mirroring its own existing SubscribeForBlocks/SubscribeForNotifications
naming, delegating to neo-go's methods exactly as the existing Subscribe*
methods do. Add the same two methods to the MetaChain interface, and call
them in Run between wg.Wait() and the close() calls.
Added a regression test with a fake MetaChain whose Subscribe* methods
hammer the channel from an independent goroutine exactly like the real
chain does, confirming Run no longer panics under this pressure.
Confirmed it reliably panics with the exact same mechanism
("send on closed channel", from the chain's delivery goroutine) when
Run's two new Unsubscribe* calls are temporarily removed, across
repeated runs - then restored the fix.
Signed-off-by: Fabian Wahle <fabian@nash.io>
Review learnings: comments cut to the contract, unexplained M43 reference dropped, CHANGELOG entry shortened and numbered. Signed-off-by: Fabian Wahle <fabian@nash.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unsubscribing before the close cannot work: shutdown() runs sidechain.Stop() before waiting for workers, so the dispatcher may already be gone and the unsubscribe loop blocks forever; and both channels share that one dispatcher, so unsubscribing them in sequence deadlocks whenever it is blocked sending to the second one. Meta does not own the write end of either channel and neither handler needs a close - they return on ctx.Done - so just leave them open. Reverts the MetaChain interface change, which also broke mockMetadataChain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fabian Wahle <fabian@nash.io>
31928f4 to
225529c
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4283 +/- ##
==========================================
+ Coverage 32.63% 32.77% +0.13%
==========================================
Files 673 673
Lines 41690 41688 -2
==========================================
+ Hits 13606 13662 +56
+ Misses 28084 28026 -58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The repository's linter rejects the manual Add/go/Done form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fabian Wahle <fabian@nash.io>
|
Superseded by #4299, which fixes this as a shutdown-ordering change in |
Summary
Meta.Runsubscribesm.bCh/m.evsChto the chain's block header and notification delivery, waits for its own handler goroutines to stop on context cancellation, then closes both channels. The chain keeps sending to them from its own, independently-running delivery goroutine for as long as they're subscribed - stoppingRun's handler goroutines does not stop that. If the chain sends while or afterRuncloses the channel, it panics withsend on closed channel, crashing the node during what should be a graceful shutdown.neo-go's
Blockchainalready provides the counterpart for exactly this:UnsubscribeFromHeadersOfAddedBlocks/UnsubscribeFromNotifications, whose own doc comments say "you can close it afterwards", and whose implementation drains any in-flight send before returning - built for precisely this shutdown sequence.SideChain(this package'sMetaChainimplementation) wraps theSubscribe*half of that pair but never gained theUnsubscribe*half.Fix
Add
UnsubscribeFromBlocks/UnsubscribeFromNotificationstoSideChain, mirroring its own existingSubscribeForBlocks/SubscribeForNotificationsnaming, delegating to neo-go's methods exactly as the existingSubscribe*methods do. Add the same two methods to theMetaChaininterface, and call them inRunbetweenwg.Wait()and theclose()calls.Test plan
MetaChainwhoseSubscribe*methods hammer the channel from an independent goroutine exactly like the real chain does, confirmingRunno longer panics under this pressure.send on closed channel, from the chain's delivery goroutine) whenRun's two newUnsubscribe*calls are temporarily removed, across repeated runs - then restored the fix.go vet ./...clean;go test ./pkg/services/meta/... ./pkg/services/sidechain/... ./cmd/neofs-node/...passes.mastertip.🤖 Generated with Claude Code