Repository navigation
feat(events): persist events to disk so they survive the process (tier 3) - #538
abelonogov-ld wants to merge 237 commits into
Conversation
Tiers 1 and 2 both need the process to survive. When it does not, everything recorded since the last delivery goes with it -- including the crash report an application was recording when it died. Tier 3 is the answer to that: events are written to a log under Application Support, and a later launch delivers whatever the previous run left behind. The write is a plain append into the page cache, never an fsync, so a commit point costs tens of microseconds rather than milliseconds. track and identify are commit points: at .immediate the write happens on the calling thread, so a process that dies the instant after track returns still reports that event. Evaluations are not -- they are held as objects and encoded in runs, so the hottest path in the SDK pays a counter bump and nothing else. This is off by default. Writing events changes what a track costs and leaves a directory of the application's data on the device, and a minor version cannot decide either on an application's behalf. LDConfig.eventPersistence has three levels: disabled, which is what Tiers 1 and 2 describe and is the default; deferred, which writes off the calling thread; and immediate, which is the whole of the above. Delivery is held for the round trip rather than just its start, because a batch now stays on disk until its response is handled and a second delivery beginning inside that window would send it again under the same payload ID. Spec: Event Durability §13, §14; Page-Cache Event Persistence. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in memory, so the assertion that nothing was buffered has to ask the store.
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence Also renames handWrittenEncoder to eventJSONWriter, after its type.
…y/event-durability-tier3-persistence
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventJSONWriter.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence
…y/event-durability-tier3-persistence
…atomically; reserve capacity during encode - Without persistence, nothing but a delivery drains the held run: no threshold commit, and track, identify, flush and backgrounding no longer encode on the caller's thread. - An evaluation's summary counter and its full/debug events are recorded under pendingLock, and a commit takes both under it. - A commit reserves capacity for the run it takes, so events being encoded still count against eventCapacity. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/Mocks/DarklyServiceMock.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
… failed write left in memory A read can fail on a file that is intact, as it does in a process out of file descriptors, and every failure was treated as a batch that can never be delivered and deleted. Listing now skips such a batch until a later listing can read it, delivery retries it, and startup recovery closes the previous run's log off uncounted rather than deleting it. Only a file that is gone, or one written in a format this version does not read, is nothing to keep. closeBatch also closed only the log where a failed write had left events both there and in memory, so a flush could report success and close() discard the rest. Co-authored-by: Cursor <cursoragent@cursor.com>
…d not be kept A batch LaunchDarkly refuses in a way that may pass is kept and retried, so it is not a loss. What is lost is what the store never received: an event or a summary that could not be serialized when it was committed, or that serialized too large to store. A delivery that finds nothing to send cannot tell those apart from events that arrived, so flushAndWait reported success for them. Co-authored-by: Cursor <cursoragent@cursor.com>
Android pins the failed write at the processor level, where the bug was visible: the flush reports success and both the written and the held events arrive. It also pins that a body reads correctly once reads work again, and that a previous run's log moved aside leaves this run the name to append to. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Tier 2 replaced the delivery gate this branch was built on, so the two designs had to be reconciled rather than taken side by side. Coalescing comes from tier 2: flushes arriving before a delivery starts join it, and one that has already started is never joined, so nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice. Tier 2 empties its buffer before publishing, so a second delivery finds nothing; here a batch stays on disk until its response arrives, which is what `isDelivering` used to guard. `batchesInFlight` takes its place. A delivery claims the batches it lists and releases them when it has an answer, so a later delivery still runs and still sends what it closed for itself, and only the batches already on their way are left out of it. A caller whose own events are all that is left unsent is answered `true`, which is the contract tier 2 documents: an answer is about the batch the caller's own delivery took. The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whichever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. `flush` stays a commit point on both paths. This branch had also fixed the duplicate event reporter on its own, as `eventSyncLogger`; that now comes from the precursor branch instead, so the local copy goes. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence
Functions that must run under a lock now have plain names, keep "Requires `xLock`" in their docs, and check it with UnfairLock.assertOwned(), which traps in debug builds when the calling thread does not hold the lock. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/UnfairLock.swift
…hare of it Co-authored-by: Cursor <cursoragent@cursor.com>
…n posting it once the gate opens Co-authored-by: Cursor <cursoragent@cursor.com>
| let waiters = launchGateWaiters | ||
| launchGateWaiters = [] | ||
| waiters.forEach { DispatchQueue.global().async(execute: $0) } | ||
| } |
There was a problem hiding this comment.
Waiters leak processor after close
Medium Severity
whenLaunchGateOpens now keeps waiters until the processor is online again, and closing never drops them. The diagnostic init callback stored there holds DiagnosticReporter, which already holds this processor, so a client closed during the launch delay (or taken offline and never brought back) never releases the processor or its event store.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit bde99d3. Configure here.
There was a problem hiding this comment.
DiagnosticReporter now holds the event processor weakly
… flag and being online, and log only there Co-authored-by: Cursor <cursoragent@cursor.com>
…g offline before it starts keeps them for the next time online Co-authored-by: Cursor <cursoragent@cursor.com>
…er counterpart covers Co-authored-by: Cursor <cursoragent@cursor.com>
…empted Co-authored-by: Cursor <cursoragent@cursor.com>
…nit held at the launch gate cannot keep a closed processor alive Co-authored-by: Cursor <cursoragent@cursor.com>
…batch's turn away rather than hand it on Co-authored-by: Cursor <cursoragent@cursor.com>
…an calling deliver again for each batch on whichever thread a wait ended on A run of batches passed by or lost without a post took a stack frame each, and overflowed the stack after a few hundred. Each wait of a pass, for its turn, its retry or a post's answer, now ends back on deliveryQueue, so a pass never goes on on the URL session's shared delegate queue. Co-authored-by: Cursor <cursoragent@cursor.com>
…entBatchDeliverer The deliverer reads, drops, posts and retries the batches of each pass, and owns the sender, the backoff and the counts of refused payloads and failed reads. The processor keeps deciding when a delivery runs and what it lists, and the deliverer asks it, as its owner, whether it is online, and tells it of losses, answered posts and the wait a failure set. Co-authored-by: Cursor <cursoragent@cursor.com>
…n, keeping a refusal's wait and a Retry-After Co-authored-by: Cursor <cursoragent@cursor.com>
… row, taking its events back into memory Co-authored-by: Cursor <cursoragent@cursor.com>
…w event settings are Co-authored-by: Cursor <cursoragent@cursor.com>
… conditions Co-authored-by: Cursor <cursoragent@cursor.com>
…ith its size Co-authored-by: Cursor <cursoragent@cursor.com>
…en, rather than after them all Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
| self.releaseLaunchGateWaiters() | ||
| if self.deliverer.endOrdinaryWait() && self.launchGateOpen { | ||
| self.queueScheduledDelivery() | ||
| } |
There was a problem hiding this comment.
Reconnect delivery skipped during launch delay
Medium Severity
Going online ends an ordinary backoff wait, but the follow-up delivery is queued only when launchGateOpen is already true. endOrdinaryWait still clears nextAttempt if the gate is closed, and checkLaunchGate later only calls deliverWhatAPreviousRunLeft. After a first 503, the in-pass retry is aborted by going offline and never calls scheduleDeliveryWhenWaitEnds, so with no previous-run batches the next send waits for the report timer instead of running when the gate opens.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 94b53fa. Configure here.
…her than claiming they match Co-authored-by: Cursor <cursoragent@cursor.com>
…a FlushResult, and cap every flush's wait at 30 seconds Co-authored-by: Cursor <cursoragent@cursor.com>
…one event, for the delivery to count the loss, rather than deleting it uncounted Co-authored-by: Cursor <cursoragent@cursor.com>
…an holding every delivery behind it Co-authored-by: Cursor <cursoragent@cursor.com>
…ion's notification, leaving its run mode as it was Co-authored-by: Cursor <cursoragent@cursor.com>
…us run's events for the next time online Co-authored-by: Cursor <cursoragent@cursor.com>
…othing is written after it A client closed while a replacement in the same process was already running left its events in a log the replacement's recovery skipped as open in this process, until the next launch. The deinit after close could also write events recorded after close into that log. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 6 total unresolved issues (including 5 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 71ca05d. Configure here.
| self.encodeAndCommitPending() | ||
| self.store.close() | ||
| self.reportVanished(self.store.takeVanishedLogEvents()) | ||
| } |
There was a problem hiding this comment.
Close drops events recorded during commit
High Severity
commitAndCloseWithinBudget drains pending and encodes it, and only then calls store.close(), which is what makes admitsEvent fail. Recordings that arrive in that gap are still accepted because hasStarted is true, including an .immediate track that waits on commitLock and then returns, and those events are discarded instead of written. That is the shutdown window this persistence work is meant to cover.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 71ca05d. Configure here.
…me of no bytes A zero-length frame was staged, and then cut off every frame behind it when read: in memory the batch kept a count larger than its body, or the frames behind it were dropped uncounted. Co-authored-by: Cursor <cursoragent@cursor.com>


Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.
What changes
EventStoreis a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety.NullEventStorestands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once
pendingCommitThresholdof them accumulate, when a delivery starts, or on flush.LDConfig.eventPersistencelets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.
Reconciling with tier 2
Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.
Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what
isDeliveringguarded.batchesInFlighttakes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answeredtrue, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.
Verification
does not send a batch again when a delivery is asked for while one is in flight.DiagnosticReporterSpec.testRetriesflaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.Note
Overview
Replaces
EventReporterwith anEventProcessorpipeline centered onDirectEventProcessor, backed by an on-diskEventStore(DiskEventLog, framing, multi-process safety) so events can survive process death and be delivered on the next launch.LDConfiggains experimentaleventPersistence(.disabled/.deferred/.immediate),eventCapacityBytes,eventMaxAge, andeventDeliveryDelay, and raises defaulteventCapacityto 2000. Recording/commit semantics shift toward batched encoding and explicit commits (flush, background, close).Flush APIs now use
FlushResultand bounded waits (up to 30s):flush(timeout:completion:)can skip backoff;flushAndWaitbypasses backoff for last-chance sends.LDClient.close()coordinates with in-flight closes viaclosingClientsand callsflushAtClose()instead of inline flush-on-queue.Background delivery moves to shared
BackgroundEventDelivery(commit all environments, then deliver under one background assertion on platforms that suspend). Diagnostic init is gated behind the event processor launch delay so flag fetch gets the network first.Supporting pieces include
AnalyticsEventSender, HTTPRetry-After/ payload-refusal handling,Crc32+Advance, and test/mock updates (EventProcessorMock, hand-writtenDiagnosticCachingMock).Reviewed by Cursor Bugbot for commit 2aa561f. Bugbot is set up for automated code reviews on this repo. Configure here.