From ea1136cc77bcd01116acf8b1c7325e2c3f5eb146 Mon Sep 17 00:00:00 2001 From: as51340 Date: Wed, 2 Sep 2026 07:42:58 +0200 Subject: [PATCH] docs: report SYNC replication failure as a notification A transaction that cannot reach every SYNC replica is still committed on the main and on every reachable replica, so memgraph/memgraph#4601 stopped failing the query and reports the failure as a WARNING notification with the code SyncReplicationFailure instead. ReplicationException is kept only for the case where the transaction was rolled back everywhere (STRICT_SYNC two-phase commit abort). - query-metadata: add the SyncReplicationFailure code and a section on commit-time notifications, which reach the client in the summary of the last query of an implicit transaction or of the explicit COMMIT - how-replication-works: SYNC failures now succeed with a notification; name ReplicationException explicitly in the STRICT_SYNC section - help-center errors: split notification from exception by transaction outcome rather than by replica mode - setup-ha-cluster-k8s: rolling upgrades no longer require applications to catch a replication exception on SYNC writes - release-notes: v3.13.0 breaking change entry --- .../how-high-availability-works.mdx | 3 ++ .../setup-ha-cluster-k8s.mdx | 18 ++++---- .../replication/how-replication-works.mdx | 29 +++++++++---- pages/database-management/query-metadata.mdx | 33 +++++++++++++++ .../help-center/errors/high-availability.mdx | 19 +++++++-- pages/help-center/errors/replication.mdx | 42 +++++++++++++++---- pages/release-notes.mdx | 11 +++++ 7 files changed, 129 insertions(+), 26 deletions(-) diff --git a/pages/clustering/high-availability/how-high-availability-works.mdx b/pages/clustering/high-availability/how-high-availability-works.mdx index 822a5539b..8afdffca2 100644 --- a/pages/clustering/high-availability/how-high-availability-works.mdx +++ b/pages/clustering/high-availability/how-high-availability-works.mdx @@ -881,6 +881,9 @@ mode: - The MAIN continues accepting writes. - Reads remain available everywhere. + - A write that couldn't reach a SYNC replica still succeeds and is reported + with a `SyncReplicationFailure` warning + [notification](/database-management/query-metadata#notifications). Thus, only STRICT_SYNC replicas can directly impact write availability. diff --git a/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx b/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx index 0cc48433b..dc15743ae 100644 --- a/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx +++ b/pages/clustering/high-availability/setup-ha-cluster-k8s.mdx @@ -2047,13 +2047,17 @@ What they will observe depends on the query type and the replication mode: ``` - **Write queries depend on the replication mode:** - **ASYNC** — writes keep working without errors. - - **SYNC** — writes keep working, but a **ReplicationException** can happen - because a commit couldn't be replicated to the instance that is being - restarted (e.g. the old main while it is coming back up). The transaction - is still committed on the MAIN, so your application should catch the - exception and continue. You can parse the error and check for the exact - message, for example: `Failed to replicate to SYNC replica 'instance_1': - replica is not reachable or not in sync with the main`. + - **SYNC** — writes keep working. A commit that couldn't be replicated to the + instance being restarted (e.g. the old main while it is coming back up) is + still committed on the MAIN, and the failure is reported as a + `SyncReplicationFailure` warning + [notification](/database-management/query-metadata#notifications) in the + query summary rather than as an error. Your application doesn't need to + handle anything to keep working; read the notification if you want to log + which replica fell behind, for example: `Failed to replicate to SYNC replica + 'instance_1': replica is not reachable or not in sync with the main`. Before + Memgraph 3.13, the same situation raised a `ReplicationException` that + applications had to catch. - **STRICT_SYNC** — writes won't work while a data instance is being restarted. Because of the two-phase commit protocol, a transaction cannot be committed unless every STRICT_SYNC replica confirms it, so writes are diff --git a/pages/clustering/replication/how-replication-works.mdx b/pages/clustering/replication/how-replication-works.mdx index 82e8acbe8..35033b521 100644 --- a/pages/clustering/replication/how-replication-works.mdx +++ b/pages/clustering/replication/how-replication-works.mdx @@ -117,11 +117,26 @@ received or a timeout is reached. If the REPLICA fails, MAIN instance will still the data and move forward. This behaviour does not block writes on REPLICA failure, and still ensures other REPLICAs to receive new data. -When a SYNC replica fails to confirm a transaction, Memgraph raises an error -that identifies the specific replica and the reason for the failure, for example: -`Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main`. -The transaction is still committed on the MAIN instance and other alive replicas. -Failed replicas will be recovered automatically. +When a SYNC replica fails to confirm a transaction, the transaction is still +committed on the MAIN instance and on every alive replica, so the query +**succeeds**. Memgraph reports the failure as a `WARNING` +[notification](/database-management/query-metadata#notifications) with the code +`SyncReplicationFailure`, which identifies the specific replica and the reason +for the failure, for example: `Failed to replicate to SYNC replica +'instance_1': replica is not reachable or not in sync with the main`. Failed +replicas will be recovered automatically. + + + +**Breaking change in Memgraph 3.13**: a SYNC replication failure used to be +reported as a `ReplicationException`, which failed the query even though the +write had been committed. It is now reported as a warning notification instead. +Applications that caught the exception around writes should read the +notification from the query summary instead of relying on an error. A +`ReplicationException` is now raised only when the transaction was actually +rolled back everywhere, which happens in a STRICT_SYNC cluster. + + The following diagrams express the behavior of the MAIN instance in cases when SYNC REPLICA doesn't answer within the expected timeout. @@ -141,8 +156,8 @@ scenario in which a failover is the most critical operation to support. Such a m allows you a failover **without the fear of experiencing a data loss**. When a STRICT_SYNC replica fails to confirm a transaction, the transaction is -aborted on all instances. The error message identifies the specific replica and -the reason, for example: +aborted on all instances and the query fails with a `ReplicationException`. The +error message identifies the specific replica and the reason, for example: `Failed to replicate to STRICT_SYNC replica 'instance_2': replica is not reachable or not in sync with the main`. **STRICT_SYNC mode ensures consistency and partition tolerance (CP).** diff --git a/pages/database-management/query-metadata.mdx b/pages/database-management/query-metadata.mdx index 30dbaa700..40a42aef9 100644 --- a/pages/database-management/query-metadata.mdx +++ b/pages/database-management/query-metadata.mdx @@ -98,7 +98,40 @@ should be used only as messages. - `RegisterReplica` - `ReplicaPortWarning` - `SetReplica` +- `SyncReplicationFailure` - `StartStream` - `StartAllStreams` - `StopStream` - `StopAllStreams` + +### Commit-time notifications + +Most notifications are produced while the query is being executed, but a +notification can also be produced by the commit itself. The +`SyncReplicationFailure` notification is reported when a transaction committed +on the MAIN instance but couldn't be replicated to every SYNC replica. + +Because it is produced by the commit, it is delivered: + +- in the summary of the last query of an implicit (autocommit) transaction, and +- in the summary of the explicit `COMMIT` query, when you manage transactions + yourself. + +``` +{ + "severity": "WARNING", + "code": "SyncReplicationFailure", + "title": "Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main. Replica will be recovered automatically. Transaction is still committed on the main instance and other alive replicas. Check the status of the replicas using 'SHOW REPLICAS' query.", + "description": "" +} +``` + + + +A `SyncReplicationFailure` notification means the write **succeeded**. When a +transaction is actually rolled back everywhere, which happens when a +STRICT_SYNC replica can't confirm it, Memgraph raises a `ReplicationException` +instead. For details, see [replication +errors](/help-center/errors/replication#error-4). + + diff --git a/pages/help-center/errors/high-availability.mdx b/pages/help-center/errors/high-availability.mdx index 1c71064b5..948e61f86 100644 --- a/pages/help-center/errors/high-availability.mdx +++ b/pages/help-center/errors/high-availability.mdx @@ -14,26 +14,37 @@ import {CommunityLinks} from '/components/social-card/CommunityLinks' ### Troubleshooting replication failure errors [#error-1] -If you're writing to the main instance and encounter an error message like +If you're writing to the main instance and encounter a message like **"Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main"**, several issues could be causing this. -The error message identifies exactly which replicas failed, the replication mode +The message identifies exactly which replicas failed, the replication mode (SYNC or STRICT_SYNC), and the specific failure reason. For a full list of possible failure reasons, see the [replication errors reference](/help-center/errors/replication#error-4). + + +Since Memgraph 3.13, a failure that only affects SYNC replicas does **not** fail +the query. The transaction is committed on the main instance and on every alive +replica, and the failure is delivered as a `SyncReplicationFailure` warning +[notification](/database-management/query-metadata#notifications) in the query +summary. A `ReplicationException` is raised only when the transaction was rolled +back everywhere, which happens when a STRICT_SYNC replica can't confirm it. + + + Below are common causes and how to resolve them: 1. **Network isn't correctly configured between MAIN and REPLICAs** — Check if hostnames/IPs can be reached from the MAIN instance. -2. **Replica is behind MAIN** — It is possible that the replica is behind MAIN and that the recovery of the replica is in progress. Wait for a bit until the replica catches up with MAIN. If the replica is registered as a SYNC one, the transaction will still get eventually committed. +2. **Replica is behind MAIN** — It is possible that the replica is behind MAIN and that the recovery of the replica is in progress. Wait for a bit until the replica catches up with MAIN. If the replica is registered as a SYNC one, the transaction is already committed on MAIN and the replica is recovered automatically. 3. **Replica has diverged from MAIN** — If the error indicates the replica has diverged, manual recovery or a force sync may be needed. See the [force sync documentation](/clustering/high-availability/how-high-availability-works#replication-scenarios). 4. **RPC timeout** — If the error mentions an RPC timeout, the replica may be overloaded, the network latency may be too high, or the replica may have - stopped making progress altogether. The error message may suggest adjusting + stopped making progress altogether. The message may suggest adjusting the `deltas_batch_progress_size` coordinator setting; as of 3.13 [that setting no longer has any effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size), diff --git a/pages/help-center/errors/replication.mdx b/pages/help-center/errors/replication.mdx index 2db65c711..359c52270 100644 --- a/pages/help-center/errors/replication.mdx +++ b/pages/help-center/errors/replication.mdx @@ -19,13 +19,14 @@ import {CommunityLinks} from '/components/social-card/CommunityLinks' 3. [Write query forbidden on the replica!](#error-2) 4. [Query forbidden on the replica!](#error-2) 5. [Replication clause not allowed in multicommand transactions.](#error-3) -6. [Failed to replicate to SYNC/STRICT_SYNC replica 'replica_name': reason.](#error-4) +6. [Failed to replicate to STRICT_SYNC replica 'replica_name': reason.](#error-4) ## Warning 1. [Snapshots are disabled for replicas. For more details, visit: memgr.ph/replication.](#warning-1) 2. [Multi-tenant replication is currently not supported!](#warning-2) +3. [Failed to replicate to SYNC replica 'replica_name': reason.](#error-4) ## Troubleshooting replication errors [#error-1] @@ -44,16 +45,38 @@ MAIN is the only source of truth. System queries cannot be executed inside a multicommand/explicit transaction. -## Replication failure error [#error-4] +## Replication failure [#error-4] When a transaction fails to replicate to one or more SYNC or STRICT_SYNC -replicas, Memgraph returns an error that identifies exactly which replicas -failed and why. The error message follows this format: +replicas, Memgraph reports exactly which replicas failed and why. Whether that +report is a warning or an error depends on what happened to the transaction: + +| Situation | How it is reported | +|-----------|--------------------| +| The transaction is committed on MAIN, but one or more **SYNC** replicas couldn't confirm it | A `WARNING` [notification](/database-management/query-metadata#notifications) with the code `SyncReplicationFailure`. The query **succeeds**. | +| The transaction is rolled back on all instances because a **STRICT_SYNC** replica couldn't confirm it | A `ReplicationException`. The query **fails**. | + +What decides between the two is the outcome of the transaction: if the write is +committed on MAIN, it is a notification, and if it was rolled back everywhere, +it is an exception. + + + +**Breaking change in Memgraph 3.13**: a SYNC replication failure used to be +reported as a `ReplicationException` as well, even though the write had already +been committed on MAIN. It is now reported as a notification. If your +application caught the replication exception around writes, read the +`SyncReplicationFailure` notification from the query summary instead. + + + +The message text is the same in both cases and follows this format: ``` Failed to replicate to SYNC replica 'instance_1': . Replica will be recovered automatically. Transaction is still committed on the main instance and other alive replicas. +Check the status of the replicas using 'SHOW REPLICAS' query. ``` When multiple replicas fail, each failure is listed with its replica name, @@ -63,6 +86,7 @@ replication mode, and specific reason: Failed to replicate to SYNC replicas 'instance_1' and 'instance_2': . Replicas will be recovered automatically. Transaction is still committed on the main instance and other alive replicas. +Check the status of the replicas using 'SHOW REPLICAS' query. ``` The possible failure reasons are: @@ -75,20 +99,22 @@ The possible failure reasons are: | `replica has diverged from main` | The replica's data has diverged and manual recovery may be needed. | | `RPC timeout while replicating` | The MAIN timed out waiting for a response from the replica. | -The error message also indicates the transaction outcome: +The message also indicates the transaction outcome: - **"Transaction is still committed on the main instance and other alive replicas."** — This appears for SYNC replicas. The transaction succeeded on MAIN despite the - replica failure. + replica failure, and it is delivered as a `SyncReplicationFailure` warning + notification. - **"Transaction was aborted on all instances."** — This appears for STRICT_SYNC replicas. The two-phase commit protocol ensures that if any STRICT_SYNC replica - fails, the transaction is rolled back everywhere. + fails, the transaction is rolled back everywhere, and the query fails with a + `ReplicationException`. Failed replicas will be recovered automatically. Check the status of replicas using the `SHOW REPLICAS` query. -If the failure reason is an RPC timeout, the error message may include guidance +If the failure reason is an RPC timeout, the message may include guidance about adjusting the `deltas_batch_progress_size` coordinator setting. As of 3.13 [that setting no longer has any effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size) diff --git a/pages/release-notes.mdx b/pages/release-notes.mdx index d25979c47..4c7b8c0a0 100644 --- a/pages/release-notes.mdx +++ b/pages/release-notes.mdx @@ -129,6 +129,17 @@ guide. or set `--metrics-format=JSON` explicitly. The deprecated JSON endpoint remains available for several releases. [#4678](https://github.com/memgraph/memgraph/pull/4678) +- A transaction that couldn't be replicated to every SYNC replica no longer + fails the query. Because the transaction is committed on the main instance and + on every reachable replica, the failure is now reported as a `WARNING` + notification with the code `SyncReplicationFailure` carrying the same message, + delivered in the summary of the last query of an implicit transaction or of + the explicit `COMMIT`. A `ReplicationException` is now raised only when the + transaction was actually rolled back everywhere, which happens when a + STRICT_SYNC cluster aborts the two-phase commit. Applications that caught the + replication exception around writes should read the notification from the + query summary instead. + [#4601](https://github.com/memgraph/memgraph/pull/4601) {

🐞 Bug fixes

}