diff --git a/pages/clustering/replication/best-practices.mdx b/pages/clustering/replication/best-practices.mdx index cd671628e..0e3c0af81 100644 --- a/pages/clustering/replication/best-practices.mdx +++ b/pages/clustering/replication/best-practices.mdx @@ -83,6 +83,13 @@ For predictable performance, all instances (MAIN and REPLICAs) should have: This ensures consistent workload distribution and prevents unexpected bottlenecks. +The MAIN instance additionally keeps two background threads per registered +REPLICA: one that encodes and ships transactions on the commit path (these run +in parallel with each other and with the MAIN instance's own WAL write, see +[parallel WAL writes and delta +shipping](/clustering/replication/how-replication-works#parallel-wal-writes-and-delta-shipping)), +and one that runs state checks and recovery for that replica. + ## Deployment requirements When running multiple instances, each on its own machine, run Memgraph as you @@ -219,4 +226,4 @@ demoted to become a REPLICA instance of the new MAIN instance. In the worst case, restarting that instance clean with new fresh storage is needed, in order for the REPLICA registration to pass successfully. - \ No newline at end of file + diff --git a/pages/clustering/replication/how-replication-works.mdx b/pages/clustering/replication/how-replication-works.mdx index 82e8acbe8..cac73e0cb 100644 --- a/pages/clustering/replication/how-replication-works.mdx +++ b/pages/clustering/replication/how-replication-works.mdx @@ -145,6 +145,14 @@ aborted on all instances. The error message identifies the specific replica and the reason, for example: `Failed to replicate to STRICT_SYNC replica 'instance_2': replica is not reachable or not in sync with the main`. +Both phases fan out across replicas in parallel: the prepare phase encodes and +ships the transaction to every STRICT_SYNC replica at once (overlapping with the +MAIN instance's WAL write, see [parallel WAL writes and delta +shipping](#parallel-wal-writes-and-delta-shipping)), and the commit decision of +the second phase is sent to all of them concurrently. The throughput cost of the +protocol therefore comes from the extra round trip rather than from the number +of replicas. + **STRICT_SYNC mode ensures consistency and partition tolerance (CP).** ### ASYNC replication mode @@ -154,11 +162,15 @@ without receiving confirmation from REPLICA instances that they have received the same transaction. This means that the **MAIN instance does not wait for the response from the REPLICA instances** in the main thread but in some other thread. -Each REPLICA instance has one permanent thread connecting it with -the MAIN instance for ASYNC replication. Using this background thread, the MAIN instance pushes -replication tasks to the REPLICA instance, creates a custom thread pool pattern, -and receives confirmations of successful replication from the REPLICATION -instance. +For each registered REPLICA, the MAIN instance keeps a permanent background +worker that carries the commit-path work for that replica: it encodes the +transaction, ships it, and receives the confirmation of successful replication. +In ASYNC mode the commit thread does not wait for that worker, so confirmations +are handled entirely off the main storage thread. A second per-replica worker +runs the periodic state checks and recovery, so a long recovery cannot delay +replication of new transactions. See [parallel WAL writes and delta +shipping](#parallel-wal-writes-and-delta-shipping) for how the commit-path +workers overlap with the MAIN instance's own WAL write. **ASYNC mode ensures system availability and partition tolerance (AP).** @@ -342,6 +354,48 @@ synchronization process. ![](/pages/clustering/replication/memgraph-replication-buffer.png) +### Parallel WAL writes and delta shipping + +When the MAIN instance commits a transaction, it has to do two things: make the +transaction durable in its own WAL file, and hand the transaction's deltas to +every registered REPLICA. Both happen on the commit path, but they do not happen +one after another. + +The commit thread walks the transaction's deltas once and records everything +that has to be written. It then schedules one task per streaming REPLICA — each +task encodes the whole transaction (metadata deltas first, then data deltas) +into that replica's RPC stream, on that replica's own background worker — and +writes its own WAL inline, at the same time. Encoding for all replicas and the +WAL write therefore overlap instead of running in sequence. This matters most on +clusters with several replicas, where commit latency no longer grows linearly +with the number of registered replicas. + +Shipping remains gated on durability. After it finishes encoding, each +per-replica task waits for the MAIN instance's WAL write to complete before it +sends the transaction end that allows the REPLICA to apply the transaction. If +the WAL write fails, that gate never opens, the task drops its stream instead of +shipping, and **no REPLICA can commit a transaction the MAIN instance did not +make durable**. + +Ordering and failure containment are preserved: + +- Each replica has a single commit-path worker, so transactions reach a given + REPLICA in the same order the MAIN instance committed them. +- WAL order still follows the MAIN instance's commit order, because the WAL + write happens while the commit holds the storage engine lock. +- A failure while encoding or shipping to one REPLICA is contained to that + REPLICA: its stream is dropped, it is marked as behind, and it is picked up by + [recovery](#instance-synchronization). The MAIN instance and the remaining + replicas still commit, and the failure is reported for SYNC and STRICT_SYNC + replicas exactly as described in [replication + modes](#replication-modes). + +The commit thread collects every per-replica task before it publishes the +transaction's commit timestamp, because encoding reads live node and edge state +that later transactions may overwrite in place. Waiting for a replica's *reply* +still follows the replica's replication mode: SYNC and STRICT_SYNC block the +commit, ASYNC does not. + ### Fixing timestamp consistency Timestamps are used to compare the state of the REPLICA instance in comparison diff --git a/pages/release-notes.mdx b/pages/release-notes.mdx index d25979c47..173360cb1 100644 --- a/pages/release-notes.mdx +++ b/pages/release-notes.mdx @@ -426,6 +426,18 @@ guide. - Indexed `STARTS WITH` filters no longer re-check the prefix after the index scan, so those queries do less work per row. Results are unchanged. [#4643](https://github.com/memgraph/memgraph/pull/4643) +- Commits on a MAIN instance with replicas are faster: the MAIN instance now + encodes a transaction for every replica in parallel with writing its own WAL, + instead of doing the WAL write and each replica in turn. Commit latency no + longer grows linearly with the number of registered replicas. Durability is + still what gates replication — a replica only receives the transaction end + after the WAL write succeeded — and per-replica ordering, WAL order, and the + reported SYNC / STRICT_SYNC failures are unchanged. Each replica also gets a + separate background worker for state checks and recovery, so a long recovery + no longer delays replication of new transactions. See [parallel WAL writes and + delta + shipping](/clustering/replication/how-replication-works#parallel-wal-writes-and-delta-shipping). + [#4645](https://github.com/memgraph/memgraph/pull/4645) - Indexed `CONTAINS`, `ENDS WITH`, and `REGEX` filters are faster when many nodes share the same property value: after one value fails the filter, the scan skips the rest of that value instead of checking every duplicate.