feat(server): report per-graph readiness of every server through pd - #3139
Draft
bitflicker64 wants to merge 6 commits into
Draft
feat(server): report per-graph readiness of every server through pd#3139bitflicker64 wants to merge 6 commits into
bitflicker64 wants to merge 6 commits into
Conversation
In distributed mode a graph is created on one server and the others
converge on it independently, so CreateGraph answers before the graph is
bound everywhere and a client behind a load balancer can reach a server
that still answers "Could not rebind [g]". The window has no bound and
no completion signal.
Each server now reports the status of a graph to pd once it opens it:
LOADING before the backend is opened, READY once the graph is bound to
the gremlin server, FAILED when either step fails. The status is keyed
by the id the server registers with, so the report of one server never
overwrites the report of another.
GET graphspaces/{graphspace}/graphs/{name}/status aggregates what the
servers reported against the servers currently registered for the graph
space, which lets a client wait for a graph to be usable cluster wide.
The aggregate is a readiness gate so it fails closed: it answers READY
only when every registered server reported READY, and the status left
behind by a server that is gone is dropped rather than counted.
Relates to apache#3137
Notify the graph create event last, so that nothing that can still fail runs after the event that reports the graph ready. The event is handled without waiting for it, so a failure after it used to have the request thread report the graph failed while the event thread reported it ready, and which of the two answers stayed was a matter of timing. Keep the status of a server that is missing from the registration until it is old enough to be taken as gone. A registration is refreshed periodically and lapses for a while when a server is merely slow, and dropping the status of such a server left only the servers that were ready, answering ready while it was still loading. A server that reported also stays counted, so it can't be dropped from both sides of the comparison at once. Check the role of the caller on the graph. The other reads of the resource verify it while opening the graph, which this one can't do since it has to answer for a graph that is still loading, so a member of the graph space could read the status of a graph it holds no permission on. Read a single config key rather than listing the graphs of the space to tell whether a graph exists, and keep stack traces out of the log on both paths: clients poll this API in a loop. Relates to apache#3137
Reaching ready by counting let the status of a server that is gone stand in for a registered server that never reported: a replaced server whose status was recent enough to be kept balanced out the server that took its place, and the graph read ready while that server couldn't serve it. Ready now asks that every registered server reported ready. Tell a graph config that can't be read apart from one that isn't there. Both answered not found, so a metadata failure was served to the client as a dropped graph, on an API clients poll and are meant to trust. Relates to apache#3137
Drop what this server reported before it restarted, while it is not registered yet. Its id is stable across a restart, so the status it left behind read as the status of the server that was starting, and a graph it had reported ready was counted ready again before it had opened it. Don't report a graph failed when the attempt that failed is not the one that registered it. Opening a graph another attempt of this server registered first fails, and the graph is served all the same, so the report used to overwrite the status of the attempt that succeeded and stay: this server has the graph and won't open it again. An attempt that registered the graph itself and failed afterwards still reports. Say so when the id of this server falls back to the rest server url. That url is the same for every replica of a container image, so the servers share one id, report their graphs over each other and are counted as one, and nothing said that the status was keyed by an identity that can't tell them apart. Relates to apache#3137
The config directory is read whether or not this server takes its graphs from it, and the name of such a graph comes from a file name, so a local file named after a graph of the cluster reported over the graph of the cluster: ready for one that no other server had opened yet, or failed for a healthy one when the local open failed. A graph of the config directory is known to this server alone, and the status API already answers for it from the local instance rather than from the cluster metadata. Drop a test that asserted nothing: opening a graph this server already has is refused before any status is reported, so the test never reached the case it was written for and passed with the case removed. Relates to apache#3137
an id that isn't unique A graph that failed after it was registered stayed in the map of this server: it was answered for while it reported failed, and this server never opened it again, so the failure was final. Take it back out and close it, so that opening it can be tried again and the report of the attempt that succeeds replaces the one that didn't. Leave a server whose id can't be derived out of the readiness of its graphs. The rest server url is the same for every server started from one image, so the servers that fall back to it are one id to the cluster, and one of them reporting ready used to answer for all of them. Reporting the servers as unknown instead keeps their graphs below ready, which is what the rest of this API does when it can't tell. Relates to apache#3137
bitflicker64
marked this pull request as draft
August 4, 2026 14:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the PR
This is step 2 of the roadmap in #3137, and only step 2.
GRAPH_CREATEevent, is aseparate, independent change and is not included. This PR applies to
master on its own rather than stacking on it.
converging replicas, is not included either. Nothing here presumes that
redesign, and the reporting added here is what step 3 would keep.
In distributed mode a graph is created on one server while the others converge on
it independently through PD.
CreateGraphanswers before the graph is boundeverywhere, so a client behind a load balancer can reach a server that still
answers
Could not rebind [g]. The window has no upper bound and no completionsignal, which leaves a client no way to know when a new graph is usable.
This adds that signal.
Main Changes
Every server reports the status of a graph it opens.
LOADINGbefore thebackend is opened,
READYonce the graph is bound to the gremlin server,FAILEDwhen either step fails. Reporting is best effort throughout: every writeis wrapped, so it can never fail or delay a graph creation.
GraphStatusenum andGraphStatusEntry, stored as JSON underHUGEGRAPH/{cluster}/GRAPHSPACE/{graphspace}/GRAPH_STATUS/{graph}/{server}.GraphMetaManagergains update/get/remove/clear for it, plus a graph-spacewide clear used when a graph space is dropped.
The id is
server.idwhen configured, otherwise the host name and the restserver port. The rest server url alone is not usable as an identity: the
shipped image bakes
restserver.url=http://0.0.0.0:8080into every replica.Events.GRAPH_BOUNDandEvents.GRAPH_BIND_FAILEDare published byContextGremlinServer.injectGraphonce the three gremlin bindings are done, orwhen they throw.
READYis written from that event only, so it means thisserver can actually serve the graph rather than merely that it opened it. The
graph create event is notified last, after the schema template is applied, so
nothing that can still fail runs after the event that reports the graph ready.
GET graphspaces/{graphspace}/graphs/{name}/statusaggregates what theservers reported against the servers currently registered for the graph space:
{ "graphspace": "DEFAULT", "graph": "hugegraph", "status": "LOADING", "ready_count": 2, "total_count": 3, "expected_count": 3, "servers": [ {"server": "server-0_8080", "status": "READY", "update_time": 1754300000000}, {"server": "server-1_8080", "status": "READY", "update_time": 1754300000001}, {"server": "server-2_8080", "status": "LOADING", "update_time": 1754300000002} ] }It is a readiness gate, so it fails closed:
READYonly when every registered server reportedREADY. The servers thatreported are compared by identity, not counted, so the status left behind by a
server that is gone can never stand in for a registered server that never
reported.
expected_countis rendered asnulland the status stays belowREADYrather than guessing.so a server that is gone doesn't hold a healthy graph down forever. A server
that is merely slow keeps its status: registrations lapse, and dropping the
status of a loading server would answer ready too early.
statusisUNKNOWNwhen no server has reported yet, which is distinct from agraph that doesn't exist (404).
and
server.idisn't set, is left out of the readiness of its graphs ratherthan reporting under an id it shares with every server started the same way.
registered yet: its id is stable across a restart, so a graph it had reported
ready would otherwise be counted ready again before it had reopened it.
The failure message published in the status is the exception type only, never its
text: this is served to every member of the graph space while opening a graph is
allowed to the owner only, and backend messages carry data paths and connection
strings. The caller's read permission on the graph is checked explicitly, since
unlike the other reads of this resource this one has to answer for a graph that
is not open yet and so can't rely on opening it to verify.
Verifying these changes
hugegraph-test, registered inUnitTestSuite:GraphStatusAggregateTest,GraphStatusEntryTest,GraphMetaManagerStatusTest,GraphManagerStatusTest,GraphStatusAPITest. They cover the aggregation rules and theirboundaries, the metadata key layout, the reported state machine driven
through a real event hub, and the endpoint including the 403 and 404 paths.
SecurityManagerTest(11) andCollectionFactoryTest(3) failures, whichreproduce identically on a clean checkout of master under the same JDK.
Does this PR potentially affect the following parts?
Additive only: one new read-only endpoint. No existing response shape changes.
server.idis now read in production; its description should be updated, as itis still documented as a legacy option.
Documentation Status
Doc - TODOThe endpoint carries its swagger annotations, from which the served OpenAPI
document is generated. Prose documentation belongs in a follow up on
apache/hugegraph-doc.
Known limitations
Raising these here rather than leaving them to be found:
TTL; entries are filtered when read, not deleted. A deployment that mints a new
host name per reschedule accumulates one entry per graph per generation.
server.idis unique when it is set explicitly. Twoservers configured with the same one are a single server to this API. The
derived case is handled: a server that can't derive a distinct id is left out
of readiness rather than reporting under a shared one.
ContextGremlinServerare not covered by a test;both listeners on the
GraphManagerside are.file name, so reporting it would let a local file named after a graph of the
cluster report over that graph. The status API answers for such a graph from
the local instance instead.
reported, so the guard that keeps a losing concurrent attempt from reporting
failed over the attempt that succeeded is not covered by a unit test; it
needs a real race to reach.
Happy to take direction on which of these to fold in here and which belong in
step 3, where PD owns the creation and most of them go away by construction.