Skip to content

fix: prevent deadlock when deleting a datastore - #551

Merged
alexandernorth merged 6 commits into
mainfrom
fix-datastore-delete-deadlock
Nov 17, 2026
Merged

alexandernorth merged 6 commits into
mainfrom
fix-datastore-delete-deadlock

Conversation

@alexandernorth

Copy link
Copy Markdown
Contributor

Deleting a datastore could hang forever and wedge the whole server (DeadlineExceeded on every retry). An abandoned IntentGetAll stream left the filesystem store locked, so InstanceDelete blocked while holding the datastore map and cache locks.

  • Bump sdcio/cache to include fix: avoid cache-wide deadlock when an IntentGetAll consumer stops reading cache#173 and simplify LocalCache.InstanceIntentGetAll (also no longer drops the stream error).
  • Cancel ctx when LoadAllButRunningIntents returns early so the store is released.
  • Use the locked GetDataStore in intent/transaction RPCs and StopAll (unsynchronized map access).
  • DeleteDatastore returns cache delete errors and keeps the entry so retries can clean up; not-found counts as deleted.
  • initCache respects ctx and accepts already-exists; startup no longer adds a nil datastore on failure.
  • NETCONF driver Close is idempotent (scrapligo blocks forever on a second Close).

Note: sdcio/cache is pinned to a pseudo-version; bump to the release tag once cut.

@alexandernorth
alexandernorth requested a review from a team as a code owner November 16, 2026 13:39
@alexandernorth
alexandernorth force-pushed the fix-datastore-delete-deadlock branch from 42f9b29 to 3941fd3 Compare November 16, 2026 14:52
Signed-off-by: Alexander North <alexandernorth@users.noreply.github.com>
Signed-off-by: Alexander North <alexandernorth@users.noreply.github.com>
Signed-off-by: Alexander North <alexandernorth@users.noreply.github.com>
@alexandernorth
alexandernorth force-pushed the fix-datastore-delete-deadlock branch from 3941fd3 to a180ac3 Compare November 16, 2026 14:56
… streams

- Release DatastoreMap lock during Stop/Delete I/O using per-name pending serialization so slow deletes do not block unrelated datastores, while restoring the entry on delete failure.
- Guard Datastore.sbi with sbiMutex and close late-connecting SBI targets if Stop already cancelled the datastore.
- Guard LocalCache.InstanceIntentGetAll error delivery against unbuffered errChan callers and cancel the store stream before waiting on errChan.
- Simplify LoadAllButRunningIntents to range over intentChan and read errChan afterwards.
- Add unit tests for unbuffered/unmarshal stream cases, LoadAllButRunningIntents cancellation, and concurrent datastore delete/recreate/retry.

Signed-off-by: Markus Vahlenkamp <markus.vahlenkamp@nokia.com>
@steiler

steiler commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Pushed a small follow-up commit (3cd0611b) on top of this branch:

  • DatastoreMap lock scope: serialize add/delete per datastore name (pending) and run Stop/Delete outside md so slow deletes cannot block unrelated datastores (restoring the entry if cache deletion fails).
  • Datastore.sbi race: guard sbi with an RWMutex and close late-connecting targets if Stop already cancelled the datastore.
  • LocalCache.InstanceIntentGetAll: avoid blocking on unbuffered errChan when ctx is cancelled and cancel the underlying store stream before waiting to deliver stream/unmarshal errors.
  • LoadAllButRunningIntents: simplify the consumer loop to range intentChan + <-errChan.
  • Tests: cover unbuffered/unmarshal InstanceIntentGetAll cases, early-return cancellation in LoadAllButRunningIntents, and concurrent DeleteDataStore / recreate / retry paths.

Combine scrapligo Close timeout from main with closeOnce idempotency
from this branch.

Signed-off-by: Markus Vahlenkamp <markus.vahlenkamp@nokia.com>
Comment thread pkg/server/datastore.go Resolved
Comment thread pkg/server/server.go Resolved
@steiler

steiler commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Why DatastoreMap.pending was added

DeleteDatastore has two requirements that conflict if we only use d.md:

  1. Don't hold d.md.Lock() across Stop / Delete I/O: d.md is server-wide and taken by every RPC (GetDataStore, ListDataStore, TransactionSet, GetIntent, etc.). In sync.RWMutex, a writer holding Lock() blocks all RLock() readers across the server. sdcio/cache#173 added Cache.pending so InstanceDelete no longer holds Cache.instancesMutex during store I/O — without doing the same in DatastoreMap, a slow/stuck SBI Close or cache InstanceDelete on ds-a would still wedge RPCs to ds-b, ds-c, etc. at the DatastoreMap layer.
  2. Keep per-name lifecycle operations serialized once d.md is dropped: while Stop/Delete runs outside d.md, ds-a is already unmapped yet its teardown is still in flight — and if ds.Delete fails, putDataStore restores ds-a into the map. Without pending[name]:
    • Delete + immediate recreate (CreateDataStore("ds-a")): CreateDataStore would see ds-a as absent, create newDS, and insert it; if the in-flight Delete then fails, putDataStore("ds-a", oldDS) would overwrite newDS with the stopped oldDS (and even on success, CreateDataStore would call datastore.New(s.ctx, ...) without waiting bounded by the caller's RPC ctx).
    • Concurrent / retried DeleteDataStore("ds-a"): a retry arriving just before a failing Delete restores ds-a would see "unknown datastore" and stop retrying right before ds-a gets put back into the map.

pending[name] serializes AddDatastore / DeleteDatastore / WaitPending per datastore name while keeping d.md strictly to short in-memory map operations.

Resolve go.mod conflict: scrapligo v1.4.4 from main, cache v0.0.39 from branch.

Signed-off-by: Markus Vahlenkamp <markus.vahlenkamp@nokia.com>
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.06897% with 66 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/datastore/datastore_rpc.go 9.30% 39 Missing ⚠️
pkg/server/server.go 71.87% 14 Missing and 4 partials ⚠️
pkg/cache/local.go 89.28% 2 Missing and 1 partial ⚠️
pkg/server/datastore.go 66.66% 1 Missing and 1 partial ⚠️
pkg/server/intent.go 0.00% 2 Missing ⚠️
pkg/datastore/intent_rpc.go 66.66% 0 Missing and 1 partial ⚠️
pkg/server/transaction.go 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alexandernorth
alexandernorth merged commit 28a79cb into main Nov 17, 2026
11 checks passed
@alexandernorth
alexandernorth deleted the fix-datastore-delete-deadlock branch November 17, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants