Repository navigation
fix(server): keep schema caches current across HStore Servers - #3481
Open
bitflicker64 wants to merge 7 commits into
Open
bitflicker64 wants to merge 7 commits into
bitflicker64 wants to merge 7 commits into
Conversation
With several Servers on HStore, a schema change made through one Server reached the others only for adds. Updates published no clear event, removes published one only with task.sync_deletion=false, and an event sent while a Server's PD watch was reconnecting was lost. A Server that kept an old copy then wrote it back over newer schema. Complete the existing clear event instead of adding a new mechanism: - publish after every committed add, update, remove and store clear or truncate, and once when a graph is opened - KvClient.listen() accepts a callback that runs each time a watch goes live; the Server clears its V2 schema caches when it runs - a publish that fails after the schema write no longer fails the call; the graph is kept in a pending set and published again with backoff and when the watch is live again - a generation counter and a lock on the shared cache keep a read, a listing or a write-through that started before a clear from caching its value after the clear - every event clears at once, events are not merged Not covered here: a write in flight on another Server while an index is created can still miss the index, a writer that stops for good between the schema write and its publish, two Servers changing the same element at once, serving cached schema while cut off from PD, and a watch stream that dies without an error. They are documented in docs/server-runtime.md.
A second pass over the schema cache clear path found four defects. - A schema write is two or three PD puts. One that failed after the first put left the change partly in PD, published nothing and kept the writer's own copy. It now drops that copy, publishes, and still reports the error. The publish also runs if caching the written element fails. - The generation was advanced before the caches were changed. A read running at that moment could note the new generation, take the old value from the id cache and copy it into the array cache afterwards, where it stayed until the next clear. The generation is now advanced as the last step of every change. - A watch start during a running retry scheduled a second retry chain, which doubled the publishes and the backoff. Only the newest retry reschedules now. - After a refused write-through the "whole list is cached" flag of that schema type was never set again, so each listing read PD until the next clear. Tests: one per fix, plus tests for the publish order, the callback passed by PdMetaDriver, the guard across transactions of one graph and local invalidation events. The listing-lock test now holds a real listing, and the concurrency test checks every read while it runs. A V2 graph opened without PD no longer leaves a retry behind that fires in a later test.
Follow-up to review comments on this change. - MultiClusterSchemaCacheTest: the readers that run while an index is created were submitted and never checked, so a reader that stopped on an error left the test passing with fewer reads. Their futures are read once the index is created. - docs/server-runtime.md: a PD watch that ends with a gRPC status the PD client does not retry is not started again, so that Server keeps its cached schema until it is restarted. This is how master behaves for every meta listener and is tracked in apache#3478. - A TODO in the test's waitFor notes that the shared RestClient sets no timeouts.
- Distinguish watch readiness from transport heartbeats. - Bound cluster HTTP calls and assert successful reader responses. - Drain in-flight retries before replacing the test publisher. - Clarify schema writer routing and propagation limits.
Batch normal clear events per graph in a fixed one-second window. Reuse the existing scheduler and retry path, keeping graph-open recovery immediate. Cover batching, concurrent flushes and test cleanup races; document delayed visibility.
Serialize manual window flushes with the existing scheduler so assertions wait for an already-running publication. Add a regression that blocks a detached window and verifies the helper waits for its publication. Exercise concurrent retry failure through the immediate graph-open publish path.
This was referenced Nov 15, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3481 +/- ##
============================================
+ Coverage 58.52% 58.60% +0.08%
- Complexity 2717 3130 +413
============================================
Files 779 780 +1
Lines 69439 69634 +195
Branches 9577 9604 +27
============================================
+ Hits 40637 40812 +175
- Misses 25108 25131 +23
+ Partials 3694 3691 -3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A Server that stops for good while a clear event waits in the 1-second window never sends it. Before the window this gap lasted milliseconds. Measured on a 3-Server cluster: after one schema change and a kill of the writing Server within the window, the other Servers kept the old schema.
imbajin
force-pushed
the
fix/schema-cache-events-3235
branch
4 times, most recently
from
November 19, 2026 06:45
f6966ea to
6dc4691
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the PR
With several Servers on HStore, a schema change made through one Server does not reliably reach the others, and a Server that keeps an old copy can write it back over newer schema. #3235 has the failures measured on master
8828d8238.This PR completes the existing schema-cache-clear event. It does not add a new sync mechanism. There is no change to PD raft, the PD store or
kv.proto; the only PD code touched is the client library (KvClient).It is a new change written from master. It is not built on #3237 (polling, closed) and contains nothing from #3376, which stays separate follow-up work.
Scope at a glance
The right column is not solved here and this PR does not claim it. Each item is explained under "What this PR does not implement".
Next
Main Changes
What this PR fixes
All changes are in
CachedSchemaTransactionV2unless noted.updateSchema()published nothing, so a property append,user_data, the index list of a base label and every status change stayed on the writer.removeSchema()and store clear or truncate published only withtask.sync_deletion=false. Now every committed add, update, remove and store clear or truncate publishes a clear event after its write. A Server merges the events of one graph in a 1-second window, added by the window follow-up below; the writer still updates its own cache at once.A schema write is two or three PD puts. If it fails after the first one, PD holds part of the change. That case publishes too, and the writer drops its own copy of the element; the caller still gets the error.
KvClientsubscribes again one second after a disconnect without telling the listener.KvClient.listen()has a new overload with a callback that runs each time a watch goes live (the firstStartingorStartedframe confirming registration on a new stream, including after a leader change or a stream error; anAliveheartbeat alone does not confirm registration). The Server clears all its V2 schema caches when it runs, so nothing missed while the watch was down survives.Plumbing for the callback:
MetaDriver.listen(key, consumer, onStarted)as a default method that falls back to the plain listen, overridden inPdMetaDriver, and passed throughMetaManagerandGraphMetaManager. The etcd meta driver uses the fallback and gets no watch-start clear.Changed behavior of schema calls
Once the schema write has committed, a failed event publication no longer fails the schema call. The call returns success and the publication is retried in the background (item 3).
Before this PR the add path threw after the commit when the publish failed, so the caller saw an error for a change that was already stored. Inside an index rebuild job the same throw would fail the job after its status write had committed. A publish failure is logged at WARN with the graph name.
What a caller can now observe: the call succeeds while other Servers still hold the old copy until the retry goes through or their watch restarts.
How this relates to the review of #3237
That review asked for a notification design in place of polling and listed points to settle first. This is where each one stands here. Nothing from #3237 itself is revived.
Cost of an event
One event makes every other Server drop the schema cache of that graph. Its next read by name then loads the whole schema again (four prefix scans in PD).
Measured on 1 PD + 1 Store + 3 Servers on one host (12 CPUs), fresh cluster with 300 property keys, 2026-11-13: 200 updates in a row through S0 while S1 and S2 each serve schema reads by name on two threads. Two runs per build.
8828d8238So for as long as schema changes arrive back to back, schema reads on the other Servers were about 11 times slower in this setup, and they recovered when the burst ended. The cost grows with the schema: the same burst on a graph with about 950 property keys gave 180 to 190 reads per second. Ten index creations in a row cost about the same on both builds (140 to 149 ms per creation and 2,450 to 2,580 reads per second on the other Servers). I did not measure the effect on graph reads and writes, only on schema reads. Lowering this cost means sending which element changed, or merging events. Merging is done by the window follow-up below, which has its own measurements; sending which element changed stays follow-up.
What this PR does not implement
These are known limits. They are listed in
docs/server-runtime.mdas limits, and none of them is implied to be solved by this PR. Follow-up issues so far: the existing #3376, and #3478 for a watch that stops for good.GraphTransaction.prepareCommit()takes the index list from the cached label, andIndexLabelRebuildJobscans the label once and starts as soon as the task is scheduled.UNAUTHENTICATED,PERMISSION_DENIED,UNIMPLEMENTEDand six more) is removed and not started again. The watch-start clear then never runs, and that Server keeps its cached schema until it is restarted; its log has one ERROR line for the stopped watch. PD does not answer a watch with these codes itself. They can come from a proxy between Server and PD: gRPC maps HTTP 401, 403 and 404 to three of them.Two more points:
Found while reviewing this change and left out because they are independent of it:
IntObjectMap.set()grows its array by one doubling only, so the first write of a schema id above twice the current size throws. It is on master and is hit by the array cache of both schema transactions. It can no longer skip a publish, because the publish runs in afinally.SchemaDriver) without a watch-start callback. I did not examine what their cache is used for.Review scope
This PR is meant to stay at its current size and shape. For a review finding I will do the following:
The second commit is the result of such a pass over the first one. It fixes four things, each with a test: no event after a schema write that failed partway, a retry that could schedule itself twice, a generation that advanced before the caches had changed (a concurrent read could copy the old value into the array cache), and a listing flag that never recovered after a refused write-through.
The third commit follows the review comments on the mirror, hugegraph#374. It changes no production code. The index test now fails when one of its background readers stops on an error (their results were never read, so the test could pass with fewer reads than it claims), and the stopped-watch limit above is added to
docs/server-runtime.mdwith #3478 as its follow-up. The remaining comments were checked against the code and not taken: theResponsein the reader loop is released byreadEntity, theScannerreads a stream that itstrycloses, a stopped watch is already logged at ERROR by the PD client, a nullonStartedis already rejected by name in the PD client, and the shared test helper was retained. HTTP timeouts were initially deferred; the maintainer follow-up now sets finite timeouts on this test class's targets and removes the TODO.A second round of comments on the third commit initially led to no code change. Two points were new: isolation from an already-running retry, and the retry thread remaining after the pending set is empty. The maintainer follow-up fixes the isolation issue by invalidating the retry epoch and draining the executor outside the monitor before replacing the test publisher, with a deterministic regression test. The executor remains one idle daemon thread, created only after a first failed publish, consistent with other Server-core schedulers.
Size before maintainer follow-up
16 files, +2,692 / -121 (
git diff --shortstat 8828d8238 7df011acb).CachedSchemaTransactionV2,KvClient, meta plumbing)Verifying these changes
CachedSchemaTransactionV2Test(new, 42 tests with the window follow-up): publish on every write path and after the write, failed and partial writes, failed publish and retry, watch start, and the generation guard on every cache insertion and write-through pathKvClientTest(10 new tests): the watch-start callback on first start, leader change, stream error and a replaced watch, including anAliveheartbeat arriving before registration confirmationMultiClusterSchemaCacheTest(new, 5 tests): 3 PD, 3 Store and 3 ServersReproductions before and after
Same scripts for both columns, through the REST API, 1 PD + 1 Store + 3 Servers (S0, S1, S2) as local processes on one host (12 CPUs, Java 17), 2026-11-13. "Before" is master
8828d8238, "after" is the pre-maintainer PR revision used for these 2026-11-13 measurements.user_datathrough S0p1, then S1 appendsp2to the same label[a, p2]; a vertex write that setsp1returns 400 on S2[a, p1, p2]; the vertex is read back withp1through each ServerCREATINGin 30 of 30task.sync_deletion=trueIn the "after" run each Server logged the watch-start clear four times: once at start and once per PD restart. No Server logged a failed publish.
All of this ran on one host with sub-millisecond latency. It shows that the cases happen and stop happening, not how often they occur in production.
Author test suites before maintainer follow-up
Java 17, 2026-11-13. Format and multi cluster ran on the tree of the head commit. The other suites ran on the second commit; the third changes one cluster test and one doc file and adds no file. Server unit and HStore core ran before one doc paragraph and one comment in the cluster-test harness were added to the second commit.
mvn editorconfig:check -pl hugegraph-pd/hg-pd-test,hugegraph-server/hugegraph-test,hugegraph-cluster-test/hugegraph-clustertest-test -am -ntpmvn apache-rat:check -ntpmvn test -pl hugegraph-server/hugegraph-test -am -P unit-test -ntpmvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hstore -ntpmvn test -pl hugegraph-cluster-test/hugegraph-clustertest-test -am -P multi-cluster-test -ntpMultiClusterSchemaCacheTest); the runtime log check passes. Run again on the third commit with the same resultmvn test -pl hugegraph-pd/hg-pd-test -am -P pd-client-test -ntpKvClientTest: 48), run on the first commit; historical evidence, superseded for the modified PD client by the follow-up validation belowThe two cache test classes were also run 12 times in a row in fresh JVMs: 12 of 12 passed.
One existing test changed:
CachedSchemaTransactionTest.testV2WarmRequestsRetainSchemaWithoutMetadataReadsAndReceiveInvalidationsexpected one PD put and now expects one at graph open and one more for the store clear.Do the tests detect the bugs
MultiClusterSchemaCacheTeston unpatched master8828d8238: 4 of 5 fail (update, append, index creation, PD restart).CachedSchemaTransactionV2Testwas run against it:testEveryPublishComesAfterItsStoreWrite,testAddPublishestestWriteThatFailsHalfwayPublishesAndDropsTheElement,testRemoveThatFailsHalfwayPublishesAndDropsTheElement,testFailedWriteKeepsItsEventWhenThePublishFailsTootestWatchStartDuringARetryLeavesOneRetrytestReadDuringALocalWriteDoesNotPutTheOldValueBack(3 of 3 runs)testRefusedWriteThroughEvictsWhatWasCachedMeanwhile,testReadsNeverGoBackBehindAWriteUnderConcurrentClearstestAClearWaitsForAListingReadFromTheCache,testListingIsNeverPartialWhileClearsRun,testReadsNeverGoBackBehindAWriteUnderConcurrentClearstestRefusedWriteThroughEvictsWhatWasCachedMeanwhileKnown gaps in the evidence
testRemoveReachesEveryServerWithSyncDeletionpasses on unpatched master, because a new graph publishes events while its system schema is set up. The unit test and the manual check in the table above cover that gate.testChangeDuringPDRestartReachesEveryServerdoes not prove that a watch was down when the change committed. It failed on unpatched master in the one control run. The evidence for the watch-start clear is the unit tests plus the Server log lines of the manual run.testReadsNeverGoBackBehindAWriteUnderConcurrentClearsandtestListingIsNeverPartialWhileClearsRundepend on timing. The first one failed in about half of the runs before the generation was advanced last, and in none of 12 after.finallyafter a failure inside the cache update has no test.Maintainer follow-up and validation:
d2d54feReviewed from three independent angles (cache correctness, watch/retry state transitions, and existing comments/test quality), followed by adversarial cross-checks. The follow-up changes 5 files, +156 / -15 relative to
7df011acb; only +15 / -7 are in production code. It adds no PD-server/proto change, polling, ACK/barrier, outbox, or extra retry abstraction.Starting/Startedof the current attempt.Validation on the exact tree pushed as
d2d54fe, Java 17, 2026-11-15 (Asia/Singapore):CachedSchemaTransactionV2Test+CachedSchemaTransactionTestKvClientTestagainst an actual PD serviceMultiClusterSuiteTest, actual 3 PD + 3 Store + 3 Server processesThe fixed code was restored and its tests passed after both mutation checks. Cache tests used Mockito's supported explicit startup Java agent because self-attachment is unavailable in this environment. Multi-node child JVMs used a 512 MiB heap cap, 2 active processors, and 128 MiB direct-memory/RocksDB limits; the test configuration uses Store replica count 1. These are functional checks, not a performance benchmark or evidence for three-replica behavior. The author's earlier throughput measurements were not rerun.
The successful ASF workflow run 43432139802 applies to
7df011acb, the head of #3477. The CI of this PR is the first ASF run ond2d54fe.The permanent stopped-watch recovery gap remains tracked by #3478. Strong concurrent DDL protection, an index-rebuild barrier, atomic schema/event publication, and offline-serving policy remain separate design work.
Maintainer follow-up: 1-second clear-event window (
4f6941e,d37e1af)Up to
d2d54fea Server sent one clear event per schema change, so a burst of changes made the other Servers load the whole schema again once per change (see "Cost of an event"). Three commits change that:fix(server): coalesce schema cache clears, by the maintainer: +45 / -10 inCachedSchemaTransactionV2, plus its tests and docs.fix(test): await running schema clear windows, by the maintainer: tests only.docs/server-runtime.mdon a Server that stops for good while its event waits in the window.How the window works:
The first change waits too, by a decision of the maintainer: a schema change does not have to reach the other Servers within a second, and sending the first change of each window at once costs one more event per window. I built that variant and measured it next to the two builds below. It is not in this PR.
Measured on the repo's
docker/docker-compose-3pd-3store-3server.yml(3 PD, 3 Store, 3 Server) on one host with 12 CPUs and 15 GB, auth off, 2026-11-17. Both builds are from this branch and differ only inhugegraph-core. The setup is not the one under "Cost of an event", so the two tables do not compare with each other.d2d54fe, one event per changed37e1af, windowWhat the window costs:
docs/server-runtime.md.The cluster scripts used for this PR were run again on
d37e1af: 30 index creations in a row gave 0 notCREATED, 0 missing in the base label and 0 unindexed vertices, and a removed property key was gone on every Server 1 second later.Unit tests on
d37e1af, Java 17, 2026-11-18:CachedSchemaTransactionV2Test42 andCachedSchemaTransactionTest21: 63 passed, 0 failed, 0 skipped.6dc4691changes docs only. Not run again for this follow-up: the full unit suite,KvClientTestandMultiClusterSuiteTest. The CI run of this PR on6dc4691is the first full run with the window.Does this PR potentially affect the following parts?
KvClient.listen()overload with a started callback and a default methodMetaDriver.listen(key, consumer, onStarted). Each Server publishes one clear event per graph when it opens the graph.Documentation Status
Doc - TODO: required documentation is pending; complete it before merging.Doc - Done: documentation is included here or linked below.Doc - No Need: no user-visible documentation is affected.Documentation files in this PR or paired hugegraph-doc PR:
docs/server-runtime.md: new section "Schema caches across HStore Servers" with the guarantees and the limitshelm/hugegraph/docs/operations.md: the paragraph on index status across Pods