Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -881,6 +881,9 @@ mode:

- The MAIN continues accepting writes.
- Reads remain available everywhere.
- A write that couldn't reach a SYNC replica still succeeds and is reported
with a `SyncReplicationFailure` warning
[notification](/database-management/query-metadata#notifications).

Thus, only STRICT_SYNC replicas can directly impact write availability.

Expand Down
18 changes: 11 additions & 7 deletions pages/clustering/high-availability/setup-ha-cluster-k8s.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2047,13 +2047,17 @@ What they will observe depends on the query type and the replication mode:
```
- **Write queries depend on the replication mode:**
- **ASYNC** — writes keep working without errors.
- **SYNC** — writes keep working, but a **ReplicationException** can happen
because a commit couldn't be replicated to the instance that is being
restarted (e.g. the old main while it is coming back up). The transaction
is still committed on the MAIN, so your application should catch the
exception and continue. You can parse the error and check for the exact
message, for example: `Failed to replicate to SYNC replica 'instance_1':
replica is not reachable or not in sync with the main`.
- **SYNC** — writes keep working. A commit that couldn't be replicated to the
instance being restarted (e.g. the old main while it is coming back up) is
still committed on the MAIN, and the failure is reported as a
`SyncReplicationFailure` warning
[notification](/database-management/query-metadata#notifications) in the
query summary rather than as an error. Your application doesn't need to
handle anything to keep working; read the notification if you want to log
which replica fell behind, for example: `Failed to replicate to SYNC replica
'instance_1': replica is not reachable or not in sync with the main`. Before
Memgraph 3.13, the same situation raised a `ReplicationException` that
applications had to catch.
- **STRICT_SYNC** — writes won't work while a data instance is being
restarted. Because of the two-phase commit protocol, a transaction cannot
be committed unless every STRICT_SYNC replica confirms it, so writes are
Expand Down
29 changes: 22 additions & 7 deletions pages/clustering/replication/how-replication-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,11 +117,26 @@ received or a timeout is reached. If the REPLICA fails, MAIN instance will still
the data and move forward. This behaviour does not block writes on REPLICA failure, and still
ensures other REPLICAs to receive new data.

When a SYNC replica fails to confirm a transaction, Memgraph raises an error
that identifies the specific replica and the reason for the failure, for example:
`Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main`.
The transaction is still committed on the MAIN instance and other alive replicas.
Failed replicas will be recovered automatically.
When a SYNC replica fails to confirm a transaction, the transaction is still
committed on the MAIN instance and on every alive replica, so the query
**succeeds**. Memgraph reports the failure as a `WARNING`
[notification](/database-management/query-metadata#notifications) with the code
`SyncReplicationFailure`, which identifies the specific replica and the reason
for the failure, for example: `Failed to replicate to SYNC replica
'instance_1': replica is not reachable or not in sync with the main`. Failed
replicas will be recovered automatically.

<Callout type="warning">

**Breaking change in Memgraph 3.13**: a SYNC replication failure used to be
reported as a `ReplicationException`, which failed the query even though the
write had been committed. It is now reported as a warning notification instead.
Applications that caught the exception around writes should read the
notification from the query summary instead of relying on an error. A
`ReplicationException` is now raised only when the transaction was actually
rolled back everywhere, which happens in a STRICT_SYNC cluster.

</Callout>

The following diagrams express the behavior of the MAIN instance in cases when
SYNC REPLICA doesn't answer within the expected timeout.
Expand All @@ -141,8 +156,8 @@ scenario in which a failover is the most critical operation to support. Such a m
allows you a failover **without the fear of experiencing a data loss**.

When a STRICT_SYNC replica fails to confirm a transaction, the transaction is
aborted on all instances. The error message identifies the specific replica and
the reason, for example:
aborted on all instances and the query fails with a `ReplicationException`. The
error message identifies the specific replica and the reason, for example:
`Failed to replicate to STRICT_SYNC replica 'instance_2': replica is not reachable or not in sync with the main`.

**STRICT_SYNC mode ensures consistency and partition tolerance (CP).**
Expand Down
33 changes: 33 additions & 0 deletions pages/database-management/query-metadata.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,40 @@ should be used only as messages.
- `RegisterReplica`
- `ReplicaPortWarning`
- `SetReplica`
- `SyncReplicationFailure`
- `StartStream`
- `StartAllStreams`
- `StopStream`
- `StopAllStreams`

### Commit-time notifications

Most notifications are produced while the query is being executed, but a
notification can also be produced by the commit itself. The
`SyncReplicationFailure` notification is reported when a transaction committed
on the MAIN instance but couldn't be replicated to every SYNC replica.

Because it is produced by the commit, it is delivered:

- in the summary of the last query of an implicit (autocommit) transaction, and
- in the summary of the explicit `COMMIT` query, when you manage transactions
yourself.

```
{
"severity": "WARNING",
"code": "SyncReplicationFailure",
"title": "Failed to replicate to SYNC replica 'instance_1': replica is not reachable or not in sync with the main. Replica will be recovered automatically. Transaction is still committed on the main instance and other alive replicas. Check the status of the replicas using 'SHOW REPLICAS' query.",
"description": ""
}
```

<Callout type="info">

A `SyncReplicationFailure` notification means the write **succeeded**. When a
transaction is actually rolled back everywhere, which happens when a
STRICT_SYNC replica can't confirm it, Memgraph raises a `ReplicationException`
instead. For details, see [replication
errors](/help-center/errors/replication#error-4).

</Callout>
19 changes: 15 additions & 4 deletions pages/help-center/errors/high-availability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,37 @@ import {CommunityLinks} from '/components/social-card/CommunityLinks'

### Troubleshooting replication failure errors [#error-1]

If you're writing to the main instance and encounter an error message like
If you're writing to the main instance and encounter a message like
**"Failed to replicate to SYNC replica 'instance_1': replica is not reachable or
not in sync with the main"**, several issues could be causing this.

The error message identifies exactly which replicas failed, the replication mode
The message identifies exactly which replicas failed, the replication mode
(SYNC or STRICT_SYNC), and the specific failure reason. For a full list of
possible failure reasons, see the
[replication errors reference](/help-center/errors/replication#error-4).

<Callout type="info">

Since Memgraph 3.13, a failure that only affects SYNC replicas does **not** fail
the query. The transaction is committed on the main instance and on every alive
replica, and the failure is delivered as a `SyncReplicationFailure` warning
[notification](/database-management/query-metadata#notifications) in the query
summary. A `ReplicationException` is raised only when the transaction was rolled
back everywhere, which happens when a STRICT_SYNC replica can't confirm it.

</Callout>

Below are common causes and how to resolve them:

1. **Network isn't correctly configured between MAIN and REPLICAs** — Check if
hostnames/IPs can be reached from the MAIN instance.
2. **Replica is behind MAIN** — It is possible that the replica is behind MAIN and that the recovery of the replica is in progress. Wait for a bit until the replica catches up with MAIN. If the replica is registered as a SYNC one, the transaction will still get eventually committed.
2. **Replica is behind MAIN** — It is possible that the replica is behind MAIN and that the recovery of the replica is in progress. Wait for a bit until the replica catches up with MAIN. If the replica is registered as a SYNC one, the transaction is already committed on MAIN and the replica is recovered automatically.
3. **Replica has diverged from MAIN** — If the error indicates the replica has
diverged, manual recovery or a force sync may be needed. See the
[force sync documentation](/clustering/high-availability/how-high-availability-works#replication-scenarios).
4. **RPC timeout** — If the error mentions an RPC timeout, the replica may be
overloaded, the network latency may be too high, or the replica may have
stopped making progress altogether. The error message may suggest adjusting
stopped making progress altogether. The message may suggest adjusting
the `deltas_batch_progress_size` coordinator setting; as of 3.13 [that
setting no longer has any
effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size),
Expand Down
42 changes: 34 additions & 8 deletions pages/help-center/errors/replication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ import {CommunityLinks} from '/components/social-card/CommunityLinks'
3. [Write query forbidden on the replica!](#error-2)
4. [Query forbidden on the replica!](#error-2)
5. [Replication clause not allowed in multicommand transactions.](#error-3)
6. [Failed to replicate to SYNC/STRICT_SYNC replica 'replica_name': reason.](#error-4)
6. [Failed to replicate to STRICT_SYNC replica 'replica_name': reason.](#error-4)


## Warning

1. [Snapshots are disabled for replicas. For more details, visit: memgr.ph/replication.](#warning-1)
2. [Multi-tenant replication is currently not supported!](#warning-2)
3. [Failed to replicate to SYNC replica 'replica_name': reason.](#error-4)

## Troubleshooting replication errors [#error-1]

Expand All @@ -44,16 +45,38 @@ MAIN is the only source of truth.

System queries cannot be executed inside a multicommand/explicit transaction.

## Replication failure error [#error-4]
## Replication failure [#error-4]

When a transaction fails to replicate to one or more SYNC or STRICT_SYNC
replicas, Memgraph returns an error that identifies exactly which replicas
failed and why. The error message follows this format:
replicas, Memgraph reports exactly which replicas failed and why. Whether that
report is a warning or an error depends on what happened to the transaction:

| Situation | How it is reported |
|-----------|--------------------|
| The transaction is committed on MAIN, but one or more **SYNC** replicas couldn't confirm it | A `WARNING` [notification](/database-management/query-metadata#notifications) with the code `SyncReplicationFailure`. The query **succeeds**. |
| The transaction is rolled back on all instances because a **STRICT_SYNC** replica couldn't confirm it | A `ReplicationException`. The query **fails**. |

What decides between the two is the outcome of the transaction: if the write is
committed on MAIN, it is a notification, and if it was rolled back everywhere,
it is an exception.

<Callout type="warning">

**Breaking change in Memgraph 3.13**: a SYNC replication failure used to be
reported as a `ReplicationException` as well, even though the write had already
been committed on MAIN. It is now reported as a notification. If your
application caught the replication exception around writes, read the
`SyncReplicationFailure` notification from the query summary instead.

</Callout>

The message text is the same in both cases and follows this format:

```
Failed to replicate to SYNC replica 'instance_1': <reason>.
Replica will be recovered automatically.
Transaction is still committed on the main instance and other alive replicas.
Check the status of the replicas using 'SHOW REPLICAS' query.
```

When multiple replicas fail, each failure is listed with its replica name,
Expand All @@ -63,6 +86,7 @@ replication mode, and specific reason:
Failed to replicate to SYNC replicas 'instance_1' and 'instance_2': <reason>.
Replicas will be recovered automatically.
Transaction is still committed on the main instance and other alive replicas.
Check the status of the replicas using 'SHOW REPLICAS' query.
```

The possible failure reasons are:
Expand All @@ -75,20 +99,22 @@ The possible failure reasons are:
| `replica has diverged from main` | The replica's data has diverged and manual recovery may be needed. |
| `RPC timeout while replicating` | The MAIN timed out waiting for a response from the replica. |

The error message also indicates the transaction outcome:
The message also indicates the transaction outcome:
- **"Transaction is still committed on the main instance and other alive replicas."** —
This appears for SYNC replicas. The transaction succeeded on MAIN despite the
replica failure.
replica failure, and it is delivered as a `SyncReplicationFailure` warning
notification.
- **"Transaction was aborted on all instances."** — This appears for STRICT_SYNC
replicas. The two-phase commit protocol ensures that if any STRICT_SYNC replica
fails, the transaction is rolled back everywhere.
fails, the transaction is rolled back everywhere, and the query fails with a
`ReplicationException`.

<Callout type="info">
Failed replicas will be recovered automatically. Check the status of replicas
using the `SHOW REPLICAS` query.
</Callout>

If the failure reason is an RPC timeout, the error message may include guidance
If the failure reason is an RPC timeout, the message may include guidance
about adjusting the `deltas_batch_progress_size` coordinator setting. As of 3.13
[that setting no longer has any
effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size)
Expand Down
11 changes: 11 additions & 0 deletions pages/release-notes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,17 @@ guide.
or set `--metrics-format=JSON` explicitly. The deprecated JSON endpoint
remains available for several releases.
[#4678](https://github.com/memgraph/memgraph/pull/4678)
- A transaction that couldn't be replicated to every SYNC replica no longer
fails the query. Because the transaction is committed on the main instance and
on every reachable replica, the failure is now reported as a `WARNING`
notification with the code `SyncReplicationFailure` carrying the same message,
delivered in the summary of the last query of an implicit transaction or of
the explicit `COMMIT`. A `ReplicationException` is now raised only when the
transaction was actually rolled back everywhere, which happens when a
STRICT_SYNC cluster aborts the two-phase commit. Applications that caught the
replication exception around writes should read the notification from the
query summary instead.
[#4601](https://github.com/memgraph/memgraph/pull/4601)

{<h4 className="custom-header">🐞 Bug fixes</h4>}

Expand Down