Make explicit sync a strict durability boundary

CHANGELOG.md

@@ -1,4 +1,5 @@
 ## 0.1.5 (unreleased)
+- Treat explicit `:sync`, `{:sync, metadata}`, and `sync: true` callback returns as strict durability boundaries. Built-in backends first exhaust their bounded transient retry policy; if the write still fails, the DurableServer exits with a structured `{:sync_failed, reason}` fatal-exit reason before acknowledging the callback. Automatic and periodic sync remain best effort for transient failures, while storage conflicts remain fatal.
 - Honor the caller-supplied `ensure_started_child/3` timeout while waiting for a live storage owner to finish Group registration, and preserve the caller's remaining overall deadline when sticky placement falls back to a local start instead of applying fresh fixed 5-second waits.
 - Add `:heartbeat_future_skew_tolerance_ms` supervisor option (default: `5_000`). Node heartbeats stamped further than this into the future are ignored for liveness decisions instead of being treated as always-fresh, and local heartbeat/watchdog deadlines now use monotonic time so wall-clock (NTP) adjustments no longer stretch or shrink safety windows.
 - Reject child keys in the reserved internal `__nodes/` namespace: `start_child/3`, `ensure_started_child/3`, and `rehome_child/3` now raise `ArgumentError` for keys that would collide with node heartbeat storage.

README.md

@@ -194,10 +194,21 @@ DurableServer supports these options in the `init/1` return tuple:
 
 State is synchronized to storage in these scenarios:
 
-1. **Manual sync**: Return `:sync` from any callback: `{:noreply, state, :sync}`
-2. **Automatic sync**: When `:auto_sync` is enabled, changes sync on the `:sync_every_ms` interval
+1. **Manual sync**: Return `:sync` from any callback: `{:noreply, state, :sync}`. Manual sync,
+   `{:sync, metadata}`, and the `sync: true` callback option are strict durability boundaries.
+   The backend retries classified transient failures within its bounded retry policy. If the
+   write still fails, the server exits with a structured `{:sync_failed, reason}` fatal-exit
+   reason before acknowledging the callback.
+2. **Automatic sync**: When `:auto_sync` is enabled, changes sync on callback return. Periodic
+   sync uses the configured `:sync_every_ms` interval. After the backend exhausts transient
+   retries, automatic and periodic sync log the failure and keep the dirty in-memory state
+   eligible for a later sync. Storage conflicts remain fatal.
 3. **Graceful shutdown**: State is always synced before termination
 
+Manual synchronization performs storage work inline. A `GenServer.call/3` timeout should be
+long enough to cover the configured backend's retry window. A caller timeout does not cancel
+storage work already running in the DurableServer process.
+
 ## Group
 
 `Group` provides distributed process groups, registry, lifecycle monitoring, and isolated subclusters.

lib/durable_server.ex

@@ -192,9 +192,14 @@ defmodule DurableServer do
 
   1. **Manual sync**: Return `:sync` from any callback, ie: `{:noreply, state, :sync}`
      You can also combine sync with other actions via callback options,
-     e.g. `{:noreply, state, {:continue, term}, sync: true}`.
+     e.g. `{:noreply, state, {:continue, term}, sync: true}`. Manual sync is a strict
+     durability boundary: the configured backend exhausts its bounded transient retry
+     policy before returning an error, and DurableServer terminates without acknowledging
+     the callback if the write still fails.
   2. **Automatic sync**: When `:auto_sync` is enabled all changes are immediately written when
     any callback returns, or the `:sync_every_ms` interval can be provided to periodically sync changes.
+    Automatic and periodic syncs log transient failures and keep the server alive so a later
+    sync can persist the dirty state. Storage conflicts remain fatal.
   3. **Graceful shutdown**: Automatically synced during normal termination, ie: cold deploys
   4. **Before stopping**: When returning `{:stop, reason, state}` from callbacks
 
@@ -3292,18 +3297,16 @@ defmodule DurableServer do
       {:ok, %DurableServer{} = synced_state} ->
         synced_state
 
-      {:error, :conflict} ->
-        fatal_sync_conflict!(state)
-
       {:error, reason} ->
         if is_map(metadata) do
           Logger.error("Failed to sync state with metadata: #{inspect(reason)}")
         else
           Logger.error("Failed to sync state: #{inspect(reason)}")
         end
 
-        # continue with updated state even if sync failed for transient reason (ie timeout)
-        state
+        # The backend has already exhausted its bounded retry policy. Explicit sync is a
+        # durability boundary, so never acknowledge the callback after the final write fails.
+        fatal_exit!({:sync_failed, reason})
     end
   end
 

test/durable_server/object_store_retry_test.exs

@@ -34,6 +34,27 @@ defmodule DurableServer.ObjectStoreRetryTest do
     assert Process.get(responses_key) == [:unexpected_retry]
   end
 
+  test "put stops after the configured transient retry limit" do
+    responses_key = make_ref()
+
+    Process.put(responses_key, [
+      %Req.Response{status: 503},
+      %Req.Response{status: 503},
+      :unexpected_retry
+    ])
+
+    assert {:error, %Req.Response{status: 503}} =
+             ObjectStore.put_object(
+               store(adapter(responses_key)),
+               "__nodes/test@localhost",
+               "heartbeat",
+               max_retries: 1,
+               timeout: 1_000
+             )
+
+    assert Process.get(responses_key) == [:unexpected_retry]
+  end
+
   test "finite operation deadlines cap each HTTP receive attempt" do
     parent = self()