Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/server-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,67 @@ connection or delivery failures are logged. This lifecycle handling does not pro
acknowledged delivery or a distributed cache consistency protocol. Validate actual DML
and cache behavior against the deployment's shared backend and topology.

## Schema caches across HStore Servers

With HStore, each Server keeps the schema of a graph in memory and never expires it.
The schema itself is stored in PD. After a Server writes a schema change to PD, it
publishes a clear event for that graph; every other Server then drops what it cached
for the graph and reads PD on the next use.

What a deployment can rely on once every Server runs this version:

- Every committed schema change is published after its write: creating, updating and
removing property keys, labels and index labels, the status changes made by index and
label jobs, and clearing or truncating a graph. A removal is published whatever
`task.sync_deletion` is set to.
- A schema change that fails after it changed part of what PD stores is published
too. The Server that made it drops its own copy of that element, and the caller
still gets the error.
- A Server whose PD watch was down drops all its cached schema when the watch is back,
for example after a PD restart or a PD leader change. Events published in between
are not replayed, and this is how they are made up for.
- A value read from PD before a clear is never cached after that clear.
- A clear event that could not be published does not fail the schema change, because
the change is already committed. The Server publishes it again after 1 second, then
with a delay that doubles up to 30 seconds, and at once when its PD watch is back.
Each Server also publishes once when it opens a graph, which covers a Server that
stopped between a schema write and its event.

Each event makes the other Servers load the whole schema of that graph again on their
next read by name. Schema changes that arrive back to back therefore slow schema reads
on the other Servers until the burst ends; no request fails because of it.

What it does not provide:

- Writes during index creation. A Server learns about a new index when it handles the
event for the label that owns the index. A vertex or edge write that such a Server
prepared before that, and that reaches storage after the index build has read that
part of the label, is missing from the index. Normally this is a write in flight for
a few milliseconds. It can be longer on a Server whose PD watch is down while the
index is created. Create indexes while writes to the label are stopped, or rebuild
the index afterwards, when a complete index is required.
- Two Servers changing the same schema element at the same time. An update writes the
whole element, so one of the two changes can be lost. Send schema changes for one
element through one client, and wait for each to return before sending the next.
- A Server that stops for good between a schema write and its event. The other Servers
keep what they cached until the next schema change of that graph or their next watch
restart.
- A Server that cannot reach PD keeps serving its cached schema until it reconnects.
- A PD watch that dies without an error is noticed only when another PD call from that
Server fails.
- A PD watch that ends with an error the PD client does not retry, such as
`UNAUTHENTICATED`, `PERMISSION_DENIED` or `UNIMPLEMENTED`, is not started again. The
Server logs the stopped watch at ERROR level and keeps its cached schema until it is
restarted. PD does not answer a watch with these codes itself; they come from the
gRPC layer, for example from a proxy between Server and PD.
- With the etcd meta driver the watch does not report restarts. A Server then does
not drop its caches after a watch restart, and a failed publish is sent again by the
timed retry only.

A Server on an older version publishes nothing on update, so finish upgrading every
Server before relying on this. Index entries skipped by an older version stay missing:
rebuild indexes that were created on a cluster with several Servers.

## Deleting a graph with local tasks

Stop application writes before deletion and keep them stopped during retries. The local
Expand Down
13 changes: 8 additions & 5 deletions helm/hugegraph/docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,11 +526,14 @@ the full index cache. This does not detect missing/stale label associations,
orphaned indexes or incorrect index data, and does not prove range-query results.
Run it after index creation/rebuild jobs finish, with no schema changes during
the test. Like indexed queries, the check rejects every other status, including
`CREATING`, `REBUILDING` and `INVALID`. On HStore, reading an unfinished index can
cache its temporary status on Pods other than the job runner. Later attempts
reuse that cache; the retry window does not repair it. Indexed queries on those
Pods may keep failing after the job succeeds; replace affected Server Pods only
after the jobs finish. This check does not fix cross-Server schema cache consistency.
`CREATING`, `REBUILDING` and `INVALID`. On HStore, the Pod that runs the job
publishes each status change, and the other Pods drop their cached schema and read
it again, so they see `CREATED` shortly after the job succeeds. A Pod whose PD
watch is down sees it when the watch is back. This check does not prove that, and
it does not cover the limits listed under
[Schema caches across HStore Servers](../../../docs/server-runtime.md#schema-caches-across-hstore-servers).
An index created before this behavior on a cluster with several Server Pods may
miss entries; rebuild it after the upgrade.

For a graph with existing test data, provide a read-only Gremlin script that
asserts the expected result. For example, with 40 existing `person` vertices
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,22 @@ private static void sleepBeforeRetry() {
}
}

/**
* Restart one PD node on its own data. It waits for the old process to
* exit and returns without waiting for the new one to be ready.
* Restarting the PD leader makes the leader change and every PD watch
* start again, which is what a test of that window needs to race.
*/
public void restartPDNode(int index) {
PDNodeWrapper node = this.pdNodeWrappers.get(index);
System.out.printf("[cluster-test] restarting %s%n", node.getID());
node.restart();
}

public void awaitPDNode(int index) {
waitUntilStarted(this.pdNodeWrappers.get(index));
}

public void stopCluster() {
for (ServerNodeWrapper serverNodeWrapper : serverNodeWrappers) {
serverNodeWrapper.stop();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
Expand Down Expand Up @@ -55,6 +56,8 @@ public abstract class AbstractNodeWrapper implements BaseNodeWrapper {
protected int index;
protected List<String> fileNames;
protected String startLine;
// Where in the startup log the running process began to write
private long startupLogOffset;

public AbstractNodeWrapper() {
this.clusterIndex = 1;
Expand Down Expand Up @@ -154,8 +157,10 @@ public boolean isStarted() {
return false;
}

try (Scanner sc = new Scanner(Paths.get(getLogPath()),
StandardCharsets.UTF_8.name())) {
try (InputStream log = Files.newInputStream(Paths.get(getLogPath()))) {
// Skip what an earlier run of this node wrote before a restart
log.skipNBytes(this.startupLogOffset);
Scanner sc = new Scanner(log, StandardCharsets.UTF_8.name());
while (sc.hasNextLine()) {
String line = sc.nextLine();
if (line.contains(startLine)) return true;
Comment on lines +163 to 166

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this. The Scanner only wraps log, and log is the resource of the enclosing try, so the file handle is closed on every path, the early return true included. What the Scanner holds besides that is a heap buffer that goes away with the local variable. Listing it as a resource too would create it before log.skipNBytes(...), which is correct only because Scanner reads lazily.

Expand All @@ -165,6 +170,32 @@ public boolean isStarted() {
return false;
}

/**
* Stop the process and start it again in the same directory, so the node
* comes back with its data. {@link #stop()} is for the end of a test: it
* also removes the directory.
*/
public void restart() {
if (this.instance != null && this.instance.isAlive()) {
this.instance.destroy();
try {
if (!this.instance.waitFor(20, TimeUnit.SECONDS)) {
this.instance.destroyForcibly().waitFor(10, TimeUnit.SECONDS);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new AssertionError("Interrupted while stopping " + getID(), e);
}
}
try {
Path log = Paths.get(getLogPath());
this.startupLogOffset = Files.isRegularFile(log) ? Files.size(log) : 0L;
} catch (IOException e) {
throw new AssertionError("Can't read the startup log of " + getID(), e);
}
this.start();
}

protected String moduleOptions(Path options) throws IOException {
if (!Files.isRegularFile(options) || !Files.isReadable(options)) {
throw new IOException("Missing or unreadable JVM module options: " + options);
Expand Down
Loading
Loading