Skip to content
79 changes: 79 additions & 0 deletions docs/server-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,85 @@ connection or delivery failures are logged. This lifecycle handling does not pro
acknowledged delivery or a distributed cache consistency protocol. Validate actual DML
and cache behavior against the deployment's shared backend and topology.

## Schema caches across HStore Servers

With HStore, each Server keeps the schema of a graph in memory and never expires it.
The schema itself is stored in PD. After a Server writes a schema change to PD, it
queues a clear event for that graph; every other Server drops what it cached when it
receives the event and reads PD on the next use.

What a deployment can rely on once every Server runs this version:

- Every committed schema change queues a notification after its write: creating, updating and
removing property keys, labels and index labels, the status changes made by index and
label jobs, and clearing or truncating a graph. A removal is published whatever
`task.sync_deletion` is set to.
- Normal change notifications on the same Server share one scheduled clear per
`(graphSpace, graph)`. The first change schedules it after 1 second without an
immediate broadcast; later changes do not replace or postpone that callback. They
can still join the same clear until its callback removes the window. A shared
scheduler or a slow PD call can therefore extend both the effective batching window
and propagation latency; retries and watch delivery can add more time. A change
after the window is removed, including during publication, schedules another clear.
Local cache updates and invalidation still happen immediately. Different Servers
batch independently, and one long-running script can span several windows.
- A schema change that fails after it changed part of what PD stores is published
too. The Server that made it drops its own copy of that element, and the caller
still gets the error.
- A Server whose PD watch was down drops all its cached schema when the watch is back,
for example after a PD restart or a PD leader change. Events published in between
are not replayed, and this is how they are made up for.
- A value read from PD before a clear is never cached after that clear.
- A clear event that could not be published does not fail the schema change, because
the change is already committed. The Server publishes it again after 1 second, then
with a delay that doubles up to 30 seconds, and at once when its PD watch is back.
Each Server also publishes once, without batching, when it opens a graph, which
covers a Server that stopped between a schema write and its event.

Each event makes the other Servers load the whole schema of that graph again on their
next read by name. Batching reduces repeated reloads during bursts, at the cost of
delaying visibility on other Servers, including for an isolated edit. Monitor read
latency and PD load during large schema changes; a successful schema change does not
guarantee that other requests stay within their configured timeouts.

What it does not provide:

- Writes during index creation. A Server learns about a new index when it handles the
event for the label that owns the index. A vertex or edge write that such a Server
prepared before that, and that reaches storage after the index build has read that
part of the label, is missing from the index. Normally this is a write in flight for
a few milliseconds without batching; the notification window also extends it.
It can be longer on a Server whose PD watch is down while the index is created.
Batching provides no index-completeness guarantee. For a complete index, stop
writes to the label and drain in-flight writes, wait for the create or rebuild
task to succeed, then verify on every write-serving Server that the index and its
association with the base label are visible before resuming writes. Do not replace
that verification with a fixed wait.
- Two Servers changing the same schema element at the same time. An update writes the
whole element, so one of the two changes can be lost. Send schema changes for one
element to the same Server, and wait for each to return before sending the next.
Pin these requests to that Server: one client behind a load balancer can still reach
different Servers before a previous change has propagated. Before moving schema
writes to another Server, verify that it reads the previously committed schema.
- A Server that stops for good between a schema write and its event, including while
the event waits for its scheduled clear. The other Servers keep what they cached
until the next schema change of that graph or their next watch restart.
- A Server that cannot reach PD keeps serving its cached schema until it reconnects.
- A PD watch that dies without an error is noticed only when another PD call from that
Server fails.
- A PD watch that ends with an error the PD client does not retry, such as
`UNAUTHENTICATED`, `PERMISSION_DENIED` or `UNIMPLEMENTED`, is not started again. The
Server logs the stopped watch at ERROR level and keeps its cached schema until it is
restarted. PD does not answer a watch with these codes itself; they come from the
gRPC layer, for example from a proxy between Server and PD.
- With the etcd meta driver the watch does not report restarts. A Server then does
not drop its caches after a watch restart, and a failed publish is sent again by the
timed retry only.

A Server on an older version publishes nothing on update, so finish upgrading every
Server before relying on this. Index entries skipped by an older version stay missing:
rebuild indexes that were created on a cluster with several Servers.

## Deleting a graph with local tasks

Stop application writes before deletion and keep them stopped during retries. The local
Expand Down
13 changes: 8 additions & 5 deletions helm/hugegraph/docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,11 +526,14 @@ the full index cache. This does not detect missing/stale label associations,
orphaned indexes or incorrect index data, and does not prove range-query results.
Run it after index creation/rebuild jobs finish, with no schema changes during
the test. Like indexed queries, the check rejects every other status, including
`CREATING`, `REBUILDING` and `INVALID`. On HStore, reading an unfinished index can
cache its temporary status on Pods other than the job runner. Later attempts
reuse that cache; the retry window does not repair it. Indexed queries on those
Pods may keep failing after the job succeeds; replace affected Server Pods only
after the jobs finish. This check does not fix cross-Server schema cache consistency.
`CREATING`, `REBUILDING` and `INVALID`. On HStore, the Pod that runs the job
publishes each status change, and the other Pods drop their cached schema and read
it again, so they see `CREATED` shortly after the job succeeds. A Pod whose PD
watch is down sees it when the watch is back. This check does not prove that, and
it does not cover the limits listed under
[Schema caches across HStore Servers](../../../docs/server-runtime.md#schema-caches-across-hstore-servers).
An index created before this behavior on a cluster with several Server Pods may
miss entries; rebuild it after the upgrade.

For a graph with existing test data, provide a read-only Gremlin script that
asserts the expected result. For example, with 40 existing `person` vertices
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,22 @@ private static void sleepBeforeRetry() {
}
}

/**
* Restart one PD node on its own data. It waits for the old process to
* exit and returns without waiting for the new one to be ready.
* Restarting the PD leader makes the leader change and every PD watch
* start again, which is what a test of that window needs to race.
*/
public void restartPDNode(int index) {
PDNodeWrapper node = this.pdNodeWrappers.get(index);
System.out.printf("[cluster-test] restarting %s%n", node.getID());
node.restart();
}

public void awaitPDNode(int index) {
waitUntilStarted(this.pdNodeWrappers.get(index));
}

public void stopCluster() {
for (ServerNodeWrapper serverNodeWrapper : serverNodeWrappers) {
serverNodeWrapper.stop();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
Expand Down Expand Up @@ -55,6 +56,8 @@ public abstract class AbstractNodeWrapper implements BaseNodeWrapper {
protected int index;
protected List<String> fileNames;
protected String startLine;
// Where in the startup log the running process began to write
private long startupLogOffset;

public AbstractNodeWrapper() {
this.clusterIndex = 1;
Expand Down Expand Up @@ -154,8 +157,10 @@ public boolean isStarted() {
return false;
}

try (Scanner sc = new Scanner(Paths.get(getLogPath()),
StandardCharsets.UTF_8.name())) {
try (InputStream log = Files.newInputStream(Paths.get(getLogPath()))) {
// Skip what an earlier run of this node wrote before a restart
log.skipNBytes(this.startupLogOffset);
Scanner sc = new Scanner(log, StandardCharsets.UTF_8.name());
while (sc.hasNextLine()) {
String line = sc.nextLine();
if (line.contains(startLine)) return true;
Expand All @@ -165,6 +170,32 @@ public boolean isStarted() {
return false;
}

/**
* Stop the process and start it again in the same directory, so the node
* comes back with its data. {@link #stop()} is for the end of a test: it
* also removes the directory.
*/
public void restart() {
if (this.instance != null && this.instance.isAlive()) {
this.instance.destroy();
try {
if (!this.instance.waitFor(20, TimeUnit.SECONDS)) {
this.instance.destroyForcibly().waitFor(10, TimeUnit.SECONDS);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new AssertionError("Interrupted while stopping " + getID(), e);
}
}
try {
Path log = Paths.get(getLogPath());
this.startupLogOffset = Files.isRegularFile(log) ? Files.size(log) : 0L;
} catch (IOException e) {
throw new AssertionError("Can't read the startup log of " + getID(), e);
}
this.start();
}

protected String moduleOptions(Path options) throws IOException {
if (!Files.isRegularFile(options) || !Files.isReadable(options)) {
throw new IOException("Missing or unreadable JVM module options: " + options);
Expand Down
Loading