Skip to content

Fix/persistency ingest lock#193

Merged
viktorbeck98 merged 3 commits into
developmentfrom
fix/persistency-ingest-lock
Jul 1, 2026
Merged

Fix/persistency ingest lock#193
viktorbeck98 merged 3 commits into
developmentfrom
fix/persistency-ingest-lock

Conversation

@viktorbeck98

Copy link
Copy Markdown
Collaborator

Task

#188

Description

EventPersistency.ingest_event() held no lock, so it could run concurrently with _save() / _load() on other threads:

  • save vs ingest — the background _SaveTimer thread calls save()_save() iterates ep.events_data while a concurrent ingest adds a new event_idRuntimeError: dictionary changed size during iteration. This is swallowed by save()'s try/except, so the symptom is a silently-skipped save (logged as a warning), not a crash.
  • load vs ingest_load() rebinds events_data/event_templates/events_seen while ingest reads-then-writes them → torn/lost state. Currently prevented operationally by stopping the engine before /load (the service-api 409 guard).

Fix: give EventPersistency a single threading.RLock, acquire it in ingest_event(), and have PersistencySaver share the same lock so ingest / save / load are mutually exclusive.

  • RLock (not Lock) is required: ingest fires the on_ingest callback → _check_event_count()save() on the same thread, which re-acquires the lock. A plain Lock self-deadlocks here.
  • PersistencySaver._lock is now aliased to persistency._lock, so existing save() / load() / locked() call sites are unchanged.

Out of scope (intentionally kept):

  • The /load 409 guard stays. The lock makes load crash-safe, but load still wholesale-replaces live state — requiring the engine stopped is a semantics decision, not just a crash workaround.
  • self._events_since_save += 1 remains unsynchronized (cosmetic save-cadence drift only).

Cost: one uncontended RLock acquire per event — negligible vs the existing msgpack/parquet work in add_data/dump.

How Has This Been Tested?

  • Added TestPersistencySaverConcurrency::test_ingest_blocks_while_saver_lock_held (TDD): holds the saver lock on the main thread and asserts a concurrent ingest_event blocks until release. Fails on the old code (ingest_event ran while the saver lock was held), passes with the fix.
  • The pre-existing test_events_until_save_triggers_save guards the RLock reentrancy regression — it deadlocks under a plain Lock, passes under RLock.
  • Full suites green: 101 persistency tests + 109 detector tests (all of which drive ingest_event).
  • prek passed locally (mypy, flake8, bandit, docformatter, etc.).

Checklist

  • This Pull-Request goes to the development branch.
  • I have successfully run prek locally.
  • I have added tests to cover my changes.
  • I have linked the issue-id to the task-description.
  • I have performed a self-review of my own code.

ingest_event held no lock, so the background save timer could iterate
events_data while ingest mutated it (silently-skipped saves), and /load
could replace state mid-ingest (torn state). Give EventPersistency an
RLock, acquire it in ingest_event, and have PersistencySaver share it so
ingest/save/load are mutually exclusive. RLock (not Lock) because ingest
can re-enter save() via the events_until_save callback on the same thread.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@viktorbeck98 viktorbeck98 changed the base branch from main to development June 30, 2026 13:17

@thorinaboenke thorinaboenke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can things can actually also get jumbled while data is exported?
import_state() uses saver and then does things with saver.locked() but export_state does plain persistency.save(),
should the export function also do

if saver is not None:
          with saver.locked():

?


data = data_structure.to_data(all_variables)
data_structure.add_data(data)
with self._lock:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same lock as persistency and event count timer use , so ingest_event now has to wait periodically for the state serialization (necessary) AND file writes. maybe worth moving the file write out of the critical section?

viktorbeck98 and others added 2 commits July 1, 2026 15:05
Addresses PR review: ingest_event held the shared lock across save
serialization AND file writes, and export_state ran unlocked.

- ingest_event fires on-ingest callbacks outside the lock, so a
  count-triggered save no longer holds the ingest lock during I/O.
- Split the save path into _serialize (CPU, under lock) and _write
  (I/O, no lock); PersistencySaver.save() serializes+resets the counter
  under the lock, then writes outside it.
- export_state acquires saver.locked(), symmetric with import_state.
  Counter reset moved into PersistencySaver.save(), so export no longer
  zeroes events_since_save.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code review flagged the unsynchronized read of _events_since_save as
looking like a bug. It is intentional: firing the on-ingest callback
outside the ingest lock is what keeps save()'s file I/O off that lock.
Comment names the TOCTOU (late/redundant save, never a missed save or
lost event) and warns against re-adding a lock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@viktorbeck98 viktorbeck98 merged commit b9e6720 into development Jul 1, 2026
4 checks passed
@viktorbeck98 viktorbeck98 deleted the fix/persistency-ingest-lock branch July 1, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants