Skip to content

feat(server): report per-graph readiness of every server through pd - #3139

Draft
bitflicker64 wants to merge 6 commits into
apache:masterfrom
bitflicker64:feat/graph-readiness-3137
Draft

feat(server): report per-graph readiness of every server through pd#3139
bitflicker64 wants to merge 6 commits into
apache:masterfrom
bitflicker64:feat/graph-readiness-3137

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Purpose of the PR

This is step 2 of the roadmap in #3137, and only step 2.

  • Step 1, making the creating server wait for its own GRAPH_CREATE event, is a
    separate, independent change and is not included. This PR applies to
    master on its own rather than stacking on it.
  • Step 3, moving graph creation ownership into PD so that servers become pure
    converging replicas, is not included either. Nothing here presumes that
    redesign, and the reporting added here is what step 3 would keep.

In distributed mode a graph is created on one server while the others converge on
it independently through PD. CreateGraph answers before the graph is bound
everywhere, so a client behind a load balancer can reach a server that still
answers Could not rebind [g]. The window has no upper bound and no completion
signal, which leaves a client no way to know when a new graph is usable.

This adds that signal.

Main Changes

Every server reports the status of a graph it opens. LOADING before the
backend is opened, READY once the graph is bound to the gremlin server,
FAILED when either step fails. Reporting is best effort throughout: every write
is wrapped, so it can never fail or delay a graph creation.

  • New GraphStatus enum and GraphStatusEntry, stored as JSON under
    HUGEGRAPH/{cluster}/GRAPHSPACE/{graphspace}/GRAPH_STATUS/{graph}/{server}.
  • GraphMetaManager gains update/get/remove/clear for it, plus a graph-space
    wide clear used when a graph space is dropped.
  • The key is per server, so the report of one server never overwrites another's.
    The id is server.id when configured, otherwise the host name and the rest
    server port. The rest server url alone is not usable as an identity: the
    shipped image bakes restserver.url=http://0.0.0.0:8080 into every replica.
  • Events.GRAPH_BOUND and Events.GRAPH_BIND_FAILED are published by
    ContextGremlinServer.injectGraph once the three gremlin bindings are done, or
    when they throw. READY is written from that event only, so it means this
    server can actually serve the graph rather than merely that it opened it. The
    graph create event is notified last, after the schema template is applied, so
    nothing that can still fail runs after the event that reports the graph ready.

GET graphspaces/{graphspace}/graphs/{name}/status aggregates what the
servers reported against the servers currently registered for the graph space:

{
  "graphspace": "DEFAULT",
  "graph": "hugegraph",
  "status": "LOADING",
  "ready_count": 2,
  "total_count": 3,
  "expected_count": 3,
  "servers": [
    {"server": "server-0_8080", "status": "READY", "update_time": 1754300000000},
    {"server": "server-1_8080", "status": "READY", "update_time": 1754300000001},
    {"server": "server-2_8080", "status": "LOADING", "update_time": 1754300000002}
  ]
}

It is a readiness gate, so it fails closed:

  • READY only when every registered server reported READY. The servers that
    reported are compared by identity, not counted, so the status left behind by a
    server that is gone can never stand in for a registered server that never
    reported.
  • When the registered servers can't be listed, expected_count is rendered as
    null and the status stays below READY rather than guessing.
  • The status left by a server that is neither registered nor recent is dropped,
    so a server that is gone doesn't hold a healthy graph down forever. A server
    that is merely slow keeps its status: registrations lapse, and dropping the
    status of a loading server would answer ready too early.
  • status is UNKNOWN when no server has reported yet, which is distinct from a
    graph that doesn't exist (404).
  • A graph config that can't be read is not treated as a graph that was dropped.
  • A server whose id can't be derived, because the host name can't be resolved
    and server.id isn't set, is left out of the readiness of its graphs rather
    than reporting under an id it shares with every server started the same way.
  • A server drops the status it reported before it restarted, while it is not
    registered yet: its id is stable across a restart, so a graph it had reported
    ready would otherwise be counted ready again before it had reopened it.

The failure message published in the status is the exception type only, never its
text: this is served to every member of the graph space while opening a graph is
allowed to the owner only, and backend messages carry data paths and connection
strings. The caller's read permission on the graph is checked explicitly, since
unlike the other reads of this resource this one has to answer for a graph that
is not open yet and so can't rely on opening it to verify.

Verifying these changes

  • Need tests and can be verified as follows:
    • 56 new unit tests in hugegraph-test, registered in UnitTestSuite:
      GraphStatusAggregateTest, GraphStatusEntryTest,
      GraphMetaManagerStatusTest, GraphManagerStatusTest,
      GraphStatusAPITest. They cover the aggregation rules and their
      boundaries, the metadata key layout, the reported state machine driven
      through a real event hub, and the endpoint including the 403 and 404 paths.
    • Full unit suite locally: 648 run, with only the pre-existing
      SecurityManagerTest (11) and CollectionFactoryTest (3) failures, which
      reproduce identically on a clean checkout of master under the same JDK.

Does this PR potentially affect the following parts?

  • The public API

Additive only: one new read-only endpoint. No existing response shape changes.
server.id is now read in production; its description should be updated, as it
is still documented as a legacy option.

Documentation Status

  • Doc - TODO

The endpoint carries its swagger annotations, from which the served OpenAPI
document is generated. Prose documentation belongs in a follow up on
apache/hugegraph-doc.

Known limitations

Raising these here rather than leaving them to be found:

  • Nothing reaps the status of a server that never comes back, and there is no
    TTL; entries are filtered when read, not deleted. A deployment that mints a new
    host name per reschedule accumulates one entry per graph per generation.
  • Nothing enforces that server.id is unique when it is set explicitly. Two
    servers configured with the same one are a single server to this API. The
    derived case is handled: a server that can't derive a distinct id is left out
    of readiness rather than reporting under a shared one.
  • The staleness window is a constant rather than an option.
  • The two events published by ContextGremlinServer are not covered by a test;
    both listeners on the GraphManager side are.
  • A graph of the local config directory reports nothing: its name comes from a
    file name, so reporting it would let a local file named after a graph of the
    cluster report over that graph. The status API answers for such a graph from
    the local instance instead.
  • Opening a graph this server already has is refused before any status is
    reported, so the guard that keeps a losing concurrent attempt from reporting
    failed over the attempt that succeeded is not covered by a unit test; it
    needs a real race to reach.

Happy to take direction on which of these to fold in here and which belong in
step 3, where PD owns the creation and most of them go away by construction.

In distributed mode a graph is created on one server and the others
converge on it independently, so CreateGraph answers before the graph is
bound everywhere and a client behind a load balancer can reach a server
that still answers "Could not rebind [g]". The window has no bound and
no completion signal.

Each server now reports the status of a graph to pd once it opens it:
LOADING before the backend is opened, READY once the graph is bound to
the gremlin server, FAILED when either step fails. The status is keyed
by the id the server registers with, so the report of one server never
overwrites the report of another.

GET graphspaces/{graphspace}/graphs/{name}/status aggregates what the
servers reported against the servers currently registered for the graph
space, which lets a client wait for a graph to be usable cluster wide.
The aggregate is a readiness gate so it fails closed: it answers READY
only when every registered server reported READY, and the status left
behind by a server that is gone is dropped rather than counted.

Relates to apache#3137
Notify the graph create event last, so that nothing that can still fail
runs after the event that reports the graph ready. The event is handled
without waiting for it, so a failure after it used to have the request
thread report the graph failed while the event thread reported it ready,
and which of the two answers stayed was a matter of timing.

Keep the status of a server that is missing from the registration until
it is old enough to be taken as gone. A registration is refreshed
periodically and lapses for a while when a server is merely slow, and
dropping the status of such a server left only the servers that were
ready, answering ready while it was still loading. A server that
reported also stays counted, so it can't be dropped from both sides of
the comparison at once.

Check the role of the caller on the graph. The other reads of the
resource verify it while opening the graph, which this one can't do
since it has to answer for a graph that is still loading, so a member
of the graph space could read the status of a graph it holds no
permission on.

Read a single config key rather than listing the graphs of the space to
tell whether a graph exists, and keep stack traces out of the log on
both paths: clients poll this API in a loop.

Relates to apache#3137
Reaching ready by counting let the status of a server that is gone
stand in for a registered server that never reported: a replaced server
whose status was recent enough to be kept balanced out the server that
took its place, and the graph read ready while that server couldn't
serve it. Ready now asks that every registered server reported ready.

Tell a graph config that can't be read apart from one that isn't there.
Both answered not found, so a metadata failure was served to the client
as a dropped graph, on an API clients poll and are meant to trust.

Relates to apache#3137
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. api Changes of API feature New feature pd PD module tests Add or improve test cases labels Aug 4, 2026
Drop what this server reported before it restarted, while it is not
registered yet. Its id is stable across a restart, so the status it left
behind read as the status of the server that was starting, and a graph
it had reported ready was counted ready again before it had opened it.

Don't report a graph failed when the attempt that failed is not the one
that registered it. Opening a graph another attempt of this server
registered first fails, and the graph is served all the same, so the
report used to overwrite the status of the attempt that succeeded and
stay: this server has the graph and won't open it again. An attempt
that registered the graph itself and failed afterwards still reports.

Say so when the id of this server falls back to the rest server url.
That url is the same for every replica of a container image, so the
servers share one id, report their graphs over each other and are
counted as one, and nothing said that the status was keyed by an
identity that can't tell them apart.

Relates to apache#3137
The config directory is read whether or not this server takes its
graphs from it, and the name of such a graph comes from a file name, so
a local file named after a graph of the cluster reported over the graph
of the cluster: ready for one that no other server had opened yet, or
failed for a healthy one when the local open failed. A graph of the
config directory is known to this server alone, and the status API
already answers for it from the local instance rather than from the
cluster metadata.

Drop a test that asserted nothing: opening a graph this server already
has is refused before any status is reported, so the test never reached
the case it was written for and passed with the case removed.

Relates to apache#3137
 an id that isn't unique

A graph that failed after it was registered stayed in the map of this
server: it was answered for while it reported failed, and this server
never opened it again, so the failure was final. Take it back out and
close it, so that opening it can be tried again and the report of the
attempt that succeeds replaces the one that didn't.

Leave a server whose id can't be derived out of the readiness of its
graphs. The rest server url is the same for every server started from
one image, so the servers that fall back to it are one id to the
cluster, and one of them reporting ready used to answer for all of
them. Reporting the servers as unknown instead keeps their graphs below
ready, which is what the rest of this API does when it can't tell.

Relates to apache#3137
@bitflicker64
bitflicker64 marked this pull request as draft August 4, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Changes of API feature New feature pd PD module size:XXL This PR changes 1000+ lines, ignoring generated files. tests Add or improve test cases

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

1 participant