Repository navigation
Description
Bug Type (问题类型)
logic (逻辑设计问题)
Before submit
- I have confirmed and searched that there are no similar problems in the historical issue and documents
Environment (环境信息)
- Server Version:
masterat8828d8238(1.8.0-SNAPSHOT), Server with HStore and the PD meta driver - Backend: HStore (PD + Store)
- OS: found by reading the code, not reproduced on a cluster (see "How this was checked")
- Data Size: N/A
Expected & Actual behavior (期望与实际表现)
Expected: when a PD watch behind a Server meta listener ends, the Server either starts it again or makes the loss visible to the code that depends on it, so a long-lived Server does not keep working on events it will never get.
Actual: KvClient treats nine gRPC status codes as non-retryable. A watch that ends with one of them is removed and never started again, and the only notice is an error handler that the Server leaves empty. Every listener registered through PdMetaDriver is affected.
The path on master:
KvClientlists the codes it does not retry:INVALID_ARGUMENT,NOT_FOUND,ALREADY_EXISTS,PERMISSION_DENIED,FAILED_PRECONDITION,OUT_OF_RANGE,UNIMPLEMENTED,DATA_LOSS,UNAUTHENTICATED.
hugegraph/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java
Lines 73 to 82 in 8828d82
private static final Set<Status.Code> NON_RETRYABLE_WATCH_ERRORS = Set.of(Status.Code.INVALID_ARGUMENT, Status.Code.NOT_FOUND, Status.Code.ALREADY_EXISTS, Status.Code.PERMISSION_DENIED, Status.Code.FAILED_PRECONDITION, Status.Code.OUT_OF_RANGE, Status.Code.UNIMPLEMENTED, Status.Code.DATA_LOSS, Status.Code.UNAUTHENTICATED); - For one of these,
stopWatchremoves the subscription, logsWatch for key {} stopped after a non-retryable errorat ERROR level, and calls the subscription'serrorConsumeronce.
hugegraph/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java
Lines 472 to 486 in 8828d82
private void stopWatch(WatchSubscription subscription, StreamObserver<WatchResponse> sourceObserver, Throwable throwable) { synchronized (subscription) { if (subscription.observer.get() != sourceObserver) { return; } subscription.observer.set(null); cancelStartTimeout(subscription); } subscriptions.remove(subscription); log.error("Watch for key {} stopped after a non-retryable error: {}", subscription.key, Status.fromThrowable(throwable), throwable); notifyWatchStopped(subscription, unwrapWatchError(throwable)); } PdMetaDriver.listenandlistenPrefixcall the two-argumentKvClientmethods, whoseerrorConsumeristhrowable -> { }.
hugegraph/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/meta/PdMetaDriver.java
Lines 111 to 125 in 8828d82
public <T> void listen(String key, Consumer<T> consumer) { try { this.client.listen(key, (Consumer<WatchResponse>) consumer); } catch (PDException e) { throw new HugeException("Failed to listen '%s' to pd", e, key); } } @Override public <T> void listenPrefix(String prefix, Consumer<T> consumer) { try { this.client.listenPrefix(prefix, (Consumer<WatchResponse>) consumer); } catch (PDException e) { throw new HugeException("Failed to listenPrefix '%s' to pd", e, prefix); }
hugegraph/hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java
Lines 252 to 263 in 8828d82
public void listen(String key, Consumer<T> consumer) throws PDException { listen(key, consumer, throwable -> { }, false); } public void listen(String key, Consumer<T> consumer, Consumer<Throwable> errorConsumer) throws PDException { listen(key, consumer, errorConsumer, false); } public void listenPrefix(String prefix, Consumer<T> consumer) throws PDException { listen(prefix, consumer, throwable -> { }, true); } - So nothing above the PD client learns that the watch is gone. For the schema cache this is permanent:
CachedSchemaTransactionV2registers its listener once per JVM and never resetsmetaEventListenerRegistered, so no later transaction registers again.
Lines 234 to 255 in 8828d82
private static void listenSchemaCacheClear() { synchronized (META_LISTENER_LOCK) { if (metaEventListenerRegistered.get()) { return; } try { MetaManager.instance().listenSchemaCacheClear( CachedSchemaTransactionV2::handleSchemaCacheClearEvent); // Set AFTER the underlying watch is live so a concurrent // caller that observes the flag is guaranteed an active // subscription, and a failure leaves the flag false so the // next caller retries registration. metaEventListenerRegistered.set(true); } catch (Exception e) { throw e instanceof RuntimeException ? (RuntimeException) e : new RuntimeException( "Failed to register schema cache clear listener", e); } } }
The result for the schema cache: that Server keeps the schema it has cached until it is restarted, and the one sign is a single ERROR line in its log. The same holds for the other listeners that MetaManager registers through the PD driver (graph, graph space and service add/update/remove, graph clear, the graph, vertex and edge cache clears, auth events, REST and Gremlin configuration updates).
Where the codes can come from
- PD does not answer a watch with any of them itself. Outside
KvClientand its test, the sources underhugegraph-pduse none of the nine as a gRPC status, and a refused watch on a follower isresponseObserver.onError(new PDException(-1, msg)), which the client sees asUNKNOWNand retries. The auth interceptor rejects a call by throwing a plainRuntimeException, not a gRPC status. - They can come from the gRPC layer. gRPC Java 1.55.3 (the version in the root
pom.xml) maps an HTTP answer without a gRPC status as follows: 401 toUNAUTHENTICATED, 403 toPERMISSION_DENIED, 404 toUNIMPLEMENTED. So a proxy, gateway or mesh sidecar between Server and PD that answers the watch stream with 401, 403 or 404 once ends the watch for good. A Server whose PD address points at an endpoint without the KV watch service getsUNIMPLEMENTEDas well.
So a deployment where Server talks to PD directly, such as the Helm chart, is not expected to hit this. It matters once something sits between them, or if PD starts to return real auth statuses.
How this was checked
- Code read at
8828d8238, links above. KvClientTest.testPermanentErrorStopsReconnectonmasterasserts the client behaviour for all nine codes: one watch call, no reconnect scheduled.- The search over PD sources:
git grep -nwE "(PERMISSION_DENIED|UNAUTHENTICATED|FAILED_PRECONDITION|INVALID_ARGUMENT|NOT_FOUND|ALREADY_EXISTS|OUT_OF_RANGE|UNIMPLEMENTED|DATA_LOSS)" 8828d8238 -- 'hugegraph-pd/*.java'. It matchesKvClient.java,KvClientTest.java, andNOT_FOUNDas the protobufPdpb.ErrorType.NOT_FOUND, which is a response field and not a gRPC status. - The HTTP mapping:
javap -p -conio.grpc.internal.GrpcUtil.httpStatusToGrpcCodefromgrpc-core-1.55.3.jar. - Not done: a cluster run with a proxy that injects one of these answers.
Related
- [Bug] PD KvClient watch can permanently stop after reconnect failure #3152 and fix(pd): keep KvClient watches alive after reconnect failures #3157 made the retryable errors retry until the watch is back, and introduced the split into retryable and non-retryable codes.
- [Improvement] Re-register HStore schema cache clear listener after a Meta reconnect #3036 asked for the schema cache listener to come back after a meta reconnect.
- Found while going through review comments on fix(server): keep schema caches current across HStore Servers #3477. That PR adds a callback that runs each time the watch starts and clears the schema caches there. A stopped watch never starts again, so the callback does not run for this case, and fix(server): keep schema caches current across HStore Servers #3477 lists it under its limits without changing it.
Possible directions
- Let
MetaDriver.listentake a callback for a stopped watch and pass it down toKvClientas theerrorConsumer.CachedSchemaTransactionV2would drop its caches there and allow the listener to be registered again. - Or handle it once in
PdMetaDriverfor every listener: register the watch again with a slow backoff after a stop, so each listener does not need its own recovery.
Either one needs a limit on how often a watch that keeps getting refused is registered again, since a wrong endpoint answers the same way every time.
Vertex/Edge example (问题点 / 边数据举例)
N/A
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
N/A
Activity
docs(server): list the stopped-watch limit and tighten the index test