Skip to content

[Bug] A PD watch stopped by a non-retryable gRPC status is never started again, so Server meta listeners stay deaf until restart #3478

Open

Description

@bitflicker64

Bug Type (问题类型)

logic (逻辑设计问题)

Before submit

  • I have confirmed and searched that there are no similar problems in the historical issue and documents

Environment (环境信息)

  • Server Version: master at 8828d8238 (1.8.0-SNAPSHOT), Server with HStore and the PD meta driver
  • Backend: HStore (PD + Store)
  • OS: found by reading the code, not reproduced on a cluster (see "How this was checked")
  • Data Size: N/A

Expected & Actual behavior (期望与实际表现)

Expected: when a PD watch behind a Server meta listener ends, the Server either starts it again or makes the loss visible to the code that depends on it, so a long-lived Server does not keep working on events it will never get.

Actual: KvClient treats nine gRPC status codes as non-retryable. A watch that ends with one of them is removed and never started again, and the only notice is an error handler that the Server leaves empty. Every listener registered through PdMetaDriver is affected.

The path on master:

  1. KvClient lists the codes it does not retry: INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, FAILED_PRECONDITION, OUT_OF_RANGE, UNIMPLEMENTED, DATA_LOSS, UNAUTHENTICATED.
    private static final Set<Status.Code> NON_RETRYABLE_WATCH_ERRORS =
    Set.of(Status.Code.INVALID_ARGUMENT,
    Status.Code.NOT_FOUND,
    Status.Code.ALREADY_EXISTS,
    Status.Code.PERMISSION_DENIED,
    Status.Code.FAILED_PRECONDITION,
    Status.Code.OUT_OF_RANGE,
    Status.Code.UNIMPLEMENTED,
    Status.Code.DATA_LOSS,
    Status.Code.UNAUTHENTICATED);
  2. For one of these, stopWatch removes the subscription, logs Watch for key {} stopped after a non-retryable error at ERROR level, and calls the subscription's errorConsumer once.
    private void stopWatch(WatchSubscription subscription,
    StreamObserver<WatchResponse> sourceObserver,
    Throwable throwable) {
    synchronized (subscription) {
    if (subscription.observer.get() != sourceObserver) {
    return;
    }
    subscription.observer.set(null);
    cancelStartTimeout(subscription);
    }
    subscriptions.remove(subscription);
    log.error("Watch for key {} stopped after a non-retryable error: {}",
    subscription.key, Status.fromThrowable(throwable), throwable);
    notifyWatchStopped(subscription, unwrapWatchError(throwable));
    }
  3. PdMetaDriver.listen and listenPrefix call the two-argument KvClient methods, whose errorConsumer is throwable -> { }.
    public <T> void listen(String key, Consumer<T> consumer) {
    try {
    this.client.listen(key, (Consumer<WatchResponse>) consumer);
    } catch (PDException e) {
    throw new HugeException("Failed to listen '%s' to pd", e, key);
    }
    }
    @Override
    public <T> void listenPrefix(String prefix, Consumer<T> consumer) {
    try {
    this.client.listenPrefix(prefix, (Consumer<WatchResponse>) consumer);
    } catch (PDException e) {
    throw new HugeException("Failed to listenPrefix '%s' to pd", e, prefix);
    }

    public void listen(String key, Consumer<T> consumer) throws PDException {
    listen(key, consumer, throwable -> { }, false);
    }
    public void listen(String key, Consumer<T> consumer,
    Consumer<Throwable> errorConsumer) throws PDException {
    listen(key, consumer, errorConsumer, false);
    }
    public void listenPrefix(String prefix, Consumer<T> consumer) throws PDException {
    listen(prefix, consumer, throwable -> { }, true);
    }
  4. So nothing above the PD client learns that the watch is gone. For the schema cache this is permanent: CachedSchemaTransactionV2 registers its listener once per JVM and never resets metaEventListenerRegistered, so no later transaction registers again.
    private static void listenSchemaCacheClear() {
    synchronized (META_LISTENER_LOCK) {
    if (metaEventListenerRegistered.get()) {
    return;
    }
    try {
    MetaManager.instance().listenSchemaCacheClear(
    CachedSchemaTransactionV2::handleSchemaCacheClearEvent);
    // Set AFTER the underlying watch is live so a concurrent
    // caller that observes the flag is guaranteed an active
    // subscription, and a failure leaves the flag false so the
    // next caller retries registration.
    metaEventListenerRegistered.set(true);
    } catch (Exception e) {
    throw e instanceof RuntimeException
    ? (RuntimeException) e
    : new RuntimeException(
    "Failed to register schema cache clear listener",
    e);
    }
    }
    }

The result for the schema cache: that Server keeps the schema it has cached until it is restarted, and the one sign is a single ERROR line in its log. The same holds for the other listeners that MetaManager registers through the PD driver (graph, graph space and service add/update/remove, graph clear, the graph, vertex and edge cache clears, auth events, REST and Gremlin configuration updates).

Where the codes can come from

  • PD does not answer a watch with any of them itself. Outside KvClient and its test, the sources under hugegraph-pd use none of the nine as a gRPC status, and a refused watch on a follower is responseObserver.onError(new PDException(-1, msg)), which the client sees as UNKNOWN and retries. The auth interceptor rejects a call by throwing a plain RuntimeException, not a gRPC status.
  • They can come from the gRPC layer. gRPC Java 1.55.3 (the version in the root pom.xml) maps an HTTP answer without a gRPC status as follows: 401 to UNAUTHENTICATED, 403 to PERMISSION_DENIED, 404 to UNIMPLEMENTED. So a proxy, gateway or mesh sidecar between Server and PD that answers the watch stream with 401, 403 or 404 once ends the watch for good. A Server whose PD address points at an endpoint without the KV watch service gets UNIMPLEMENTED as well.

So a deployment where Server talks to PD directly, such as the Helm chart, is not expected to hit this. It matters once something sits between them, or if PD starts to return real auth statuses.

How this was checked

  • Code read at 8828d8238, links above.
  • KvClientTest.testPermanentErrorStopsReconnect on master asserts the client behaviour for all nine codes: one watch call, no reconnect scheduled.
  • The search over PD sources: git grep -nwE "(PERMISSION_DENIED|UNAUTHENTICATED|FAILED_PRECONDITION|INVALID_ARGUMENT|NOT_FOUND|ALREADY_EXISTS|OUT_OF_RANGE|UNIMPLEMENTED|DATA_LOSS)" 8828d8238 -- 'hugegraph-pd/*.java'. It matches KvClient.java, KvClientTest.java, and NOT_FOUND as the protobuf Pdpb.ErrorType.NOT_FOUND, which is a response field and not a gRPC status.
  • The HTTP mapping: javap -p -c on io.grpc.internal.GrpcUtil.httpStatusToGrpcCode from grpc-core-1.55.3.jar.
  • Not done: a cluster run with a proxy that injects one of these answers.

Related

Possible directions

  1. Let MetaDriver.listen take a callback for a stopped watch and pass it down to KvClient as the errorConsumer. CachedSchemaTransactionV2 would drop its caches there and allow the listener to be registered again.
  2. Or handle it once in PdMetaDriver for every listener: register the watch again with a slow backoff after a stop, so each listener does not need its own recovery.

Either one needs a limit on how often a watch that keeps getting refused is registered again, since a wrong endpoint answers the same way every time.

Vertex/Edge example (问题点 / 边数据举例)

N/A

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions