Skip to content

fix(server): keep schema caches current across HStore Servers - #3481

Open
bitflicker64 wants to merge 7 commits into
apache:masterfrom
hugegraph:fix/schema-cache-events-3235
Open

bitflicker64 wants to merge 7 commits into
apache:masterfrom
hugegraph:fix/schema-cache-events-3235

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

Replaces #3477, which was opened from a personal fork and stopped at 7df011acb. This PR is opened from the org branch hugegraph/hugegraph:fix/schema-cache-events-3235, the branch of hugegraph#374, so both PRs have one head. It has the three commits of #3477 and the maintainer follow-up d2d54fe. The review threads of #3477 are answered there.

image

With several Servers on HStore, a schema change made through one Server does not reliably reach the others, and a Server that keeps an old copy can write it back over newer schema. #3235 has the failures measured on master 8828d8238.

This PR completes the existing schema-cache-clear event. It does not add a new sync mechanism. There is no change to PD raft, the PD store or kv.proto; the only PD code touched is the client library (KvClient).

It is a new change written from master. It is not built on #3237 (polling, closed) and contains nothing from #3376, which stays separate follow-up work.

Scope at a glance

In this PR Not in this PR, separate follow-up
Publish a clear event after every schema add, update, remove and store clear or truncate, also when the write fails partway ACK or barrier between Servers before an index rebuild starts
Clear the V2 schema caches when the PD watch starts or restarts Schema write and event as one atomic PD commit
Retry a publish that failed Protection for two Servers changing the same element at once (#3376)
Generation guard so an old read cannot refill a cleared cache Refusing requests while cut off from PD, and a keepalive on the watch

The right column is not solved here and this PR does not claim it. Each item is explained under "What this PR does not implement".

schema-cache1

Next

schema-cache2

Main Changes

What this PR fixes

All changes are in CachedSchemaTransactionV2 unless noted.

  1. Schema update and removal propagation. updateSchema() published nothing, so a property append, user_data, the index list of a base label and every status change stayed on the writer. removeSchema() and store clear or truncate published only with task.sync_deletion=false. Now every committed add, update, remove and store clear or truncate publishes a clear event after its write. A Server merges the events of one graph in a 1-second window, added by the window follow-up below; the writer still updates its own cache at once.
    A schema write is two or three PD puts. If it fails after the first one, PD holds part of the change. That case publishes too, and the writer drops its own copy of the element; the caller still gets the error.
  2. Cache invalidation on PD watch start and recovery. Events are delivered at most once, and KvClient subscribes again one second after a disconnect without telling the listener. KvClient.listen() has a new overload with a callback that runs each time a watch goes live (the first Starting or Started frame confirming registration on a new stream, including after a leader change or a stream error; an Alive heartbeat alone does not confirm registration). The Server clears all its V2 schema caches when it runs, so nothing missed while the watch was down survives.
  3. Failed-publication retry. A graph whose publish failed goes into an in-memory pending set and is published again after 1 s, doubling up to 30 s, and at once when the watch is live again. Each Server also publishes once when it opens a graph, which repairs a publish lost to a restart of the writer.
  4. Generation guard. A generation counter and a read-write lock on the shared cache. Every clear, removal and local write advances the generation as the last step of its change. A read by id, a read by name, a full listing or the writer's own write-through that started before a clear cannot cache its value after the clear. A name miss now triggers one reload per graph at a time instead of one per waiting thread.

Plumbing for the callback: MetaDriver.listen(key, consumer, onStarted) as a default method that falls back to the plain listen, overridden in PdMetaDriver, and passed through MetaManager and GraphMetaManager. The etcd meta driver uses the fallback and gets no watch-start clear.

Changed behavior of schema calls

Once the schema write has committed, a failed event publication no longer fails the schema call. The call returns success and the publication is retried in the background (item 3).

Before this PR the add path threw after the commit when the publish failed, so the caller saw an error for a change that was already stored. Inside an index rebuild job the same throw would fail the job after its status write had committed. A publish failure is logged at WARN with the graph name.

What a caller can now observe: the call succeeds while other Servers still hold the old copy until the retry goes through or their watch restarts.

How this relates to the review of #3237

That review asked for a notification design in place of polling and listed points to settle first. This is where each one stands here. Nothing from #3237 itself is revived.

Point from that review In this PR
Notify, do not poll Done. Nothing runs while the schema is idle. A retry timer exists only while a publish is pending.
Restore update and status notifications Done, item 1. This reverses the choice in #3011 to keep status changes local. The cost is measured under "Cost of an event".
Retry without blocking DDL Done, item 3.
A reliable link between a schema commit and its notification Partly. A failed publish is retried, a restarted writer publishes again, and a write that fails partway publishes too. The write and the event are still separate PD puts: see the limits.
A Server that was away reloads before it serves Partly. It drops all its cached schema when its watch is live again. It is not kept from serving while it is cut off: see the limits.
Batch redundant invalidations within one operation Done by time, not by operation: the window follow-up below merges the events of one graph on one Server in a 1-second window. This delays the clear of the base label, which widens the index window described in the limits.

Cost of an event

One event makes every other Server drop the schema cache of that graph. Its next read by name then loads the whole schema again (four prefix scans in PD).

Measured on 1 PD + 1 Store + 3 Servers on one host (12 CPUs), fresh cluster with 300 property keys, 2026-11-13: 200 updates in a row through S0 while S1 and S2 each serve schema reads by name on two threads. Two runs per build.

master 8828d8238 this PR
One update on S0, mean 7.8 and 8.0 ms 11.8 and 11.8 ms
Reads per second on S1 and S2 while idle 3,900 and 4,140 3,850 and 4,310
Reads per second during the burst 3,610 and 3,760 320 and 340
Median read during the burst 1.0 and 1.1 ms 10.6 and 10.8 ms
Reads per second right after the burst 4,780 and 4,890 4,950 and 4,970
Failed reads 0 0
S1 and S2 show the last update 3 s after the burst no yes

So for as long as schema changes arrive back to back, schema reads on the other Servers were about 11 times slower in this setup, and they recovered when the burst ended. The cost grows with the schema: the same burst on a graph with about 950 property keys gave 180 to 190 reads per second. Ten index creations in a row cost about the same on both builds (140 to 149 ms per creation and 2,450 to 2,580 reads per second on the other Servers). I did not measure the effect on graph reads and writes, only on schema reads. Lowering this cost means sending which element changed, or merging events. Merging is done by the window follow-up below, which has its own measurements; sending which element changed stays follow-up.

What this PR does not implement

These are known limits. They are listed in docs/server-runtime.md as limits, and none of them is implied to be solved by this PR. Follow-up issues so far: the existing #3376, and #3478 for a watch that stops for good.

Not implemented What can still go wrong Why it is excluded
ACK or barrier before an index rebuild A write on another Server that is prepared from the old base label and committed after the rebuild scan has passed it is missing from the new index. GraphTransaction.prepareCommit() takes the index list from the cached label, and IndexLabelRebuildJob scans the label once and starts as soon as the task is scheduled. The rebuild would have to wait until every Server has confirmed the new base label. That needs Server membership, confirmations and a rule for a Server that never answers. It is a protocol of its own and belongs in a separate design.
Atomic schema commit plus event publication A writer that stops for good between its schema write and its publish, which now includes up to 1 second in the window, leaves the other Servers on the old copy until the next event for that graph or their next watch restart. If the writer restarts, its publish on graph open repairs this. The write and the event are two PD puts. Making them one needs a PD-side change (one raft entry for both, or an outbox), which this PR avoids on purpose.
Concurrent same-element schema changes An update still writes the whole element from the writer's copy. Two Servers changing the same element at the same moment can lose one change. It needs a conditional write in PD. That is #3376, which is +3,711 lines on its own.
Offline-serving gate A Server that cannot reach PD keeps serving its cached schema until it reconnects. Refusing requests in that state trades availability for freshness. That is a product decision, not part of a cache fix.
Restart of a watch that stopped for good A PD watch that ends with a gRPC status the PD client does not retry (UNAUTHENTICATED, PERMISSION_DENIED, UNIMPLEMENTED and six more) is removed and not started again. The watch-start clear then never runs, and that Server keeps its cached schema until it is restarted; its log has one ERROR line for the stopped watch. PD does not answer a watch with these codes itself. They can come from a proxy between Server and PD: gRPC maps HTTP 401, 403 and 404 to three of them. This is how the PD client on master behaves for every Server meta listener, not only this one. Starting the watch again needs a stop callback through the meta driver and a limit on how often a refused watch is retried. Tracked in #3478.
Watch keepalive A watch stream that dies without an error is noticed only when another PD call from that Server fails. It needs keepalive settings on the PD client channels, which affects every user of the PD client and needs its own tests.

Two more points:

  • A reload right after a PD leader change also needs fix(pd): wait for a ReadIndex before serving get and scanPrefix #3383, so the read cannot be served by a leader that has not applied its log yet.
  • The guarantees start when the last Server runs this version. A Server on the old version still publishes nothing on update. Index entries skipped before the upgrade stay missing until that index is rebuilt.

Found while reviewing this change and left out because they are independent of it:

  • IntObjectMap.set() grows its array by one doubling only, so the first write of a schema id above twice the current size throws. It is on master and is hit by the array cache of both schema transactions. It can no longer skip a publish, because the publish runs in a finally.
  • Store nodes listen on the same clear key (SchemaDriver) without a watch-start callback. I did not examine what their cache is used for.

Review scope

This PR is meant to stay at its current size and shape. For a review finding I will do the following:

  • A bug in this patch, or something that breaks one of the four items under "What this PR fixes": fixed here, with the smallest change and a regression test.
  • Anything that needs one of the excluded items above (ACK or barrier, atomic commit, conditional write, offline gate, keepalive), a PD server change, or a refactor of code this PR does not touch: answered with the reason and proposed as a follow-up issue, not added here.

The second commit is the result of such a pass over the first one. It fixes four things, each with a test: no event after a schema write that failed partway, a retry that could schedule itself twice, a generation that advanced before the caches had changed (a concurrent read could copy the old value into the array cache), and a listing flag that never recovered after a refused write-through.

The third commit follows the review comments on the mirror, hugegraph#374. It changes no production code. The index test now fails when one of its background readers stops on an error (their results were never read, so the test could pass with fewer reads than it claims), and the stopped-watch limit above is added to docs/server-runtime.md with #3478 as its follow-up. The remaining comments were checked against the code and not taken: the Response in the reader loop is released by readEntity, the Scanner reads a stream that its try closes, a stopped watch is already logged at ERROR by the PD client, a null onStarted is already rejected by name in the PD client, and the shared test helper was retained. HTTP timeouts were initially deferred; the maintainer follow-up now sets finite timeouts on this test class's targets and removes the TODO.

A second round of comments on the third commit initially led to no code change. Two points were new: isolation from an already-running retry, and the retry thread remaining after the pending set is empty. The maintainer follow-up fixes the isolation issue by invalidating the retry epoch and draining the executor outside the monitor before replacing the test publisher, with a deterministic regression test. The executor remains one idle daemon thread, created only after a first failed publish, consistent with other Server-core schedulers.

Size before maintainer follow-up

16 files, +2,692 / -121 (git diff --shortstat 8828d8238 7df011acb).

Part Added Removed
Production code (CachedSchemaTransactionV2, KvClient, meta plumbing) 628 111
Tests 1,946 3
Cluster-test harness (restart one PD node in place) 49 2
Docs 69 5

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • CachedSchemaTransactionV2Test (new, 42 tests with the window follow-up): publish on every write path and after the write, failed and partial writes, failed publish and retry, watch start, and the generation guard on every cache insertion and write-through path
    • KvClientTest (10 new tests): the watch-start callback on first start, leader change, stream error and a replaced watch, including an Alive heartbeat arriving before registration confirmation
    • MultiClusterSchemaCacheTest (new, 5 tests): 3 PD, 3 Store and 3 Servers

Reproductions before and after

Same scripts for both columns, through the REST API, 1 PD + 1 Store + 3 Servers (S0, S1, S2) as local processes on one host (12 CPUs, Java 17), 2026-11-13. "Before" is master 8828d8238, "after" is the pre-maintainer PR revision used for these 2026-11-13 measurements.

Case Before After
Update user_data through S0 Still missing on S1 and S2 after 30 s Visible on S1 and S2 at the first check, after 1 s
S0 appends p1, then S1 appends p2 to the same label PD ends with [a, p2]; a vertex write that sets p1 returns 400 on S2 All three Servers show [a, p1, p2]; the vertex is read back with p1 through each Server
Index status, 30 index creations through S0 S1 and S2 keep CREATING in 30 of 30 0 of 30
Base label, same 30 runs Index missing from the label on S1 in 14 of 30 and on S2 in 15 of 30; 29 of 29 vertices written through such a Server are not indexed 0 of 30; 0 vertices not indexed
Add through S0 right after a PD restart Missing on S1 and S2 after 60 s, 3 of 3 runs Visible within 2 s, 3 of 3 runs
Remove with task.sync_deletion=true Still listed on S1 and S2 after 30 s Gone on all Servers within 1 s

In the "after" run each Server logged the watch-start clear four times: once at start and once per PD restart. No Server logged a failed publish.

All of this ran on one host with sub-millisecond latency. It shows that the cases happen and stop happening, not how often they occur in production.

Author test suites before maintainer follow-up

Java 17, 2026-11-13. Format and multi cluster ran on the tree of the head commit. The other suites ran on the second commit; the third changes one cluster test and one doc file and adds no file. Server unit and HStore core ran before one doc paragraph and one comment in the cluster-test harness were added to the second commit.

Suite Command Result
Format mvn editorconfig:check -pl hugegraph-pd/hg-pd-test,hugegraph-server/hugegraph-test,hugegraph-cluster-test/hugegraph-clustertest-test -am -ntp BUILD SUCCESS
Licence headers mvn apache-rat:check -ntp BUILD SUCCESS
Server unit mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test -ntp 1,477 run, 0 failures, 8 skipped
HStore core (PD and Store running) mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hstore -ntp 1,033 run, 0 failures, 77 skipped
Multi cluster mvn test -pl hugegraph-cluster-test/hugegraph-clustertest-test -am -P multi-cluster-test -ntp 11 run, 0 failures (5 are MultiClusterSchemaCacheTest); the runtime log check passes. Run again on the third commit with the same result
PD client (PD running) mvn test -pl hugegraph-pd/hg-pd-test -am -P pd-client-test -ntp 122 run, 0 failures (KvClientTest: 48), run on the first commit; historical evidence, superseded for the modified PD client by the follow-up validation below

The two cache test classes were also run 12 times in a row in fresh JVMs: 12 of 12 passed.

One existing test changed: CachedSchemaTransactionTest.testV2WarmRequestsRetainSchemaWithoutMetadataReadsAndReceiveInvalidations expected one PD put and now expects one at graph open and one more for the store clear.

Do the tests detect the bugs

  • MultiClusterSchemaCacheTest on unpatched master 8828d8238: 4 of 5 fail (update, append, index creation, PD restart).
  • Each fix was taken out in turn and CachedSchemaTransactionV2Test was run against it:
Change taken out Tests that fail
Publish after the write (publish moved before it) testEveryPublishComesAfterItsStoreWrite, testAddPublishes
Handling of a write that failed partway testWriteThatFailsHalfwayPublishesAndDropsTheElement, testRemoveThatFailsHalfwayPublishesAndDropsTheElement, testFailedWriteKeepsItsEventWhenThePublishFailsToo
Only the newest retry reschedules testWatchStartDuringARetryLeavesOneRetry
Generation advanced last in a local write testReadDuringALocalWriteDoesNotPutTheOldValueBack (3 of 3 runs)
Eviction after a refused write-through testRefusedWriteThroughEvictsWhatWasCachedMeanwhile, testReadsNeverGoBackBehindAWriteUnderConcurrentClears
Read lock around a cached listing testAClearWaitsForAListingReadFromTheCache, testListingIsNeverPartialWhileClearsRun, testReadsNeverGoBackBehindAWriteUnderConcurrentClears
Whole-list flag set after a full load testRefusedWriteThroughEvictsWhatWasCachedMeanwhile

Known gaps in the evidence

  • testRemoveReachesEveryServerWithSyncDeletion passes on unpatched master, because a new graph publishes events while its system schema is set up. The unit test and the manual check in the table above cover that gate.
  • testChangeDuringPDRestartReachesEveryServer does not prove that a watch was down when the change committed. It failed on unpatched master in the one control run. The evidence for the watch-start clear is the unit tests plus the Server log lines of the manual run.
  • testReadsNeverGoBackBehindAWriteUnderConcurrentClears and testListingIsNeverPartialWhileClearsRun depend on timing. The first one failed in about half of the runs before the generation was advanced last, and in none of 12 after.
  • The generation order is tested with a deterministic test for a local write. For a clear it is covered only by the timing test above.
  • The publish in finally after a failure inside the cache update has no test.
  • The in-flight write window during index creation and the writer that never returns are derived from the code and were not reproduced.
  • Not tested: a real network partition, a silent stream death, Servers on mixed versions.
  • Not rerun locally for the maintainer follow-up: api-test, the TinkerPop suites, the PD core, common and REST suites, the Store suites and the simple-cluster suite. Historical CI and current-head validation are recorded separately below.

Maintainer follow-up and validation: d2d54fe

Reviewed from three independent angles (cache correctness, watch/retry state transitions, and existing comments/test quality), followed by adversarial cross-checks. The follow-up changes 5 files, +156 / -15 relative to 7df011acb; only +15 / -7 are in production code. It adds no PD-server/proto change, polling, ACK/barrier, outbox, or extra retry abstraction.

  • Separate receipt of a transport frame from confirmed watch registration; run the started callback only on the first Starting / Started of the current attempt.
  • Assert HTTP 2xx in background readers, and set this cluster test's targets to a 5 s connect timeout and 15 s read timeout.
  • Invalidate the retry epoch and wait for in-flight test publication to finish before replacing the test publisher.
  • Document that DDL for the same schema element must be serialized through one pinned Server; verify visibility before changing writers. Remove the unconditional no-request-failure claim.

Validation on the exact tree pushed as d2d54fe, Java 17, 2026-11-15 (Asia/Singapore):

Validation Result
Full 38-module clean compile, packaging, editorconfig format/check, and diff whitespace check Passed
CachedSchemaTransactionV2Test + CachedSchemaTransactionTest 36 + 21 = 57 passed
Full KvClientTest against an actual PD service 51 passed
MultiClusterSuiteTest, actual 3 PD + 3 Store + 3 Server processes 11 passed, including all 5 schema-cache tests; runtime log check passed
Total distinct positive tests in this follow-up 119 passed; 0 failures, 0 errors, 0 skipped
Restore old watch-start logic while retaining new regressions All 3 new tests fail at their intended assertions
Restore old reset helper while retaining the in-flight regression The 1 new test fails at its intended assertion

The fixed code was restored and its tests passed after both mutation checks. Cache tests used Mockito's supported explicit startup Java agent because self-attachment is unavailable in this environment. Multi-node child JVMs used a 512 MiB heap cap, 2 active processors, and 128 MiB direct-memory/RocksDB limits; the test configuration uses Store replica count 1. These are functional checks, not a performance benchmark or evidence for three-replica behavior. The author's earlier throughput measurements were not rerun.

The successful ASF workflow run 43432139802 applies to 7df011acb, the head of #3477. The CI of this PR is the first ASF run on d2d54fe.

The permanent stopped-watch recovery gap remains tracked by #3478. Strong concurrent DDL protection, an index-rebuild barrier, atomic schema/event publication, and offline-serving policy remain separate design work.

Maintainer follow-up: 1-second clear-event window (4f6941e, d37e1af)

Up to d2d54fe a Server sent one clear event per schema change, so a burst of changes made the other Servers load the whole schema again once per change (see "Cost of an event"). Three commits change that:

  • 4f6941e fix(server): coalesce schema cache clears, by the maintainer: +45 / -10 in CachedSchemaTransactionV2, plus its tests and docs.
  • d37e1af fix(test): await running schema clear windows, by the maintainer: tests only.
  • 6dc4691, mine: one sentence in docs/server-runtime.md on a Server that stops for good while its event waits in the window.

How the window works:

  • The changes of one graph on one Server share a fixed 1-second window. The first change starts the window and sends nothing yet. Later changes do not move its end. One event is sent when it ends.
  • A change that arrives while that event is being sent starts the next window, so it gets an event of its own.
  • The writer updates its own cache at once, as before. Only the event to the other Servers waits.
  • The publish on graph open and the retry on watch recovery skip the window. An event that cannot be sent at the end of a window goes into the existing retry.
  • There is no new setting, no PD change and no new thread: the window runs on the scheduler of the retry.

The first change waits too, by a decision of the maintainer: a schema change does not have to reach the other Servers within a second, and sending the first change of each window at once costs one more event per window. I built that variant and measured it next to the two builds below. It is not in this PR.

Measured on the repo's docker/docker-compose-3pd-3store-3server.yml (3 PD, 3 Store, 3 Server) on one host with 12 CPUs and 15 GB, auth off, 2026-11-17. Both builds are from this branch and differ only in hugegraph-core. The setup is not the one under "Cost of an event", so the two tables do not compare with each other.

d2d54fe, one event per change d37e1af, window
Clear events one other Server received for 200 updates in a row, 3 runs 200, 200, 200 3, 2, 3
Schema reads per second on the other two Servers during those updates 708, 932, 671 2,798, 3,244, 2,842
One update on the writing Server, mean 17.7, 13.7, 20.3 ms 11.8, 9.7, 13.8 ms
One change on its own, time until the other Servers show it, median of 10 rounds 9, 13, 8 ms 1,021, 1,020, 1,018 ms
600 updates with the PD leader killed 1 s into them, 2 runs: keys wrong on a Server 30 s and 60 s later 0 0
Writing Server killed right after one change on its own, 3 trials: the other Servers during the 20 s it is down show the change, 3 of 3 keep the old value, 3 of 3
The same trials after the writing Server is started again not needed show the change 2.8 s after the start, 3 of 3

What the window costs:

  • Every schema change reaches the other Servers about 1 second later, a change on its own included. It can take longer: the scheduler is shared, and a slow PD call or a retry adds time.
  • A Server that is killed inside the window never sends that event. The other Servers get the change when it opens the graph again (last two rows), or with the next event for that graph. Before the window this gap was the time between the write and its publish. It is in the limits table above and in docs/server-runtime.md.
  • The in-flight write window during index creation, described in the limits, is longer by up to the window.

The cluster scripts used for this PR were run again on d37e1af: 30 index creations in a row gave 0 not CREATED, 0 missing in the base label and 0 unindexed vertices, and a removed property key was gone on every Server 1 second later.

Unit tests on d37e1af, Java 17, 2026-11-18:

mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test \
    -Dtest=CachedSchemaTransactionV2Test,CachedSchemaTransactionTest

CachedSchemaTransactionV2Test 42 and CachedSchemaTransactionTest 21: 63 passed, 0 failed, 0 skipped. 6dc4691 changes docs only. Not run again for this follow-up: the full unit suite, KvClientTest and MultiClusterSuiteTest. The CI run of this PR on 6dc4691 is the first full run with the window.

Does this PR potentially affect the following parts?

  • Dependencies (add/update license info & regenerate_known_dependencies.sh)
  • Modify configurations
  • The public API
  • Other affects: a schema call no longer fails when only the event publication failed after the commit (see "Changed behavior of schema calls"). Clear events of one graph are merged in a 1-second window on each Server, so a schema change reaches the other Servers about 1 second later than without it (see the window follow-up). New Java methods: a KvClient.listen() overload with a started callback and a default method MetaDriver.listen(key, consumer, onStarted). Each Server publishes one clear event per graph when it opens the graph.
  • Nope

Documentation Status

  • Doc - TODO: required documentation is pending; complete it before merging.
  • Doc - Done: documentation is included here or linked below.
  • Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

  • docs/server-runtime.md: new section "Schema caches across HStore Servers" with the guarantees and the limits
  • helm/hugegraph/docs/operations.md: the paragraph on index status across Pods

bitflicker64 and others added 6 commits November 13, 2026 11:26
With several Servers on HStore, a schema change made through one
Server reached the others only for adds. Updates published no clear
event, removes published one only with task.sync_deletion=false, and
an event sent while a Server's PD watch was reconnecting was lost.
A Server that kept an old copy then wrote it back over newer schema.

Complete the existing clear event instead of adding a new mechanism:

- publish after every committed add, update, remove and store clear
  or truncate, and once when a graph is opened
- KvClient.listen() accepts a callback that runs each time a watch
  goes live; the Server clears its V2 schema caches when it runs
- a publish that fails after the schema write no longer fails the
  call; the graph is kept in a pending set and published again with
  backoff and when the watch is live again
- a generation counter and a lock on the shared cache keep a read,
  a listing or a write-through that started before a clear from
  caching its value after the clear
- every event clears at once, events are not merged

Not covered here: a write in flight on another Server while an index
is created can still miss the index, a writer that stops for good
between the schema write and its publish, two Servers changing the
same element at once, serving cached schema while cut off from PD,
and a watch stream that dies without an error. They are documented
in docs/server-runtime.md.
A second pass over the schema cache clear path found four defects.

- A schema write is two or three PD puts. One that failed after the
  first put left the change partly in PD, published nothing and kept
  the writer's own copy. It now drops that copy, publishes, and still
  reports the error. The publish also runs if caching the written
  element fails.
- The generation was advanced before the caches were changed. A read
  running at that moment could note the new generation, take the old
  value from the id cache and copy it into the array cache afterwards,
  where it stayed until the next clear. The generation is now advanced
  as the last step of every change.
- A watch start during a running retry scheduled a second retry chain,
  which doubled the publishes and the backoff. Only the newest retry
  reschedules now.
- After a refused write-through the "whole list is cached" flag of
  that schema type was never set again, so each listing read PD until
  the next clear.

Tests: one per fix, plus tests for the publish order, the callback
passed by PdMetaDriver, the guard across transactions of one graph and
local invalidation events. The listing-lock test now holds a real
listing, and the concurrency test checks every read while it runs. A
V2 graph opened without PD no longer leaves a retry behind that fires
in a later test.
Follow-up to review comments on this change.

- MultiClusterSchemaCacheTest: the readers that run while an index is
  created were submitted and never checked, so a reader that stopped on
  an error left the test passing with fewer reads. Their futures are
  read once the index is created.
- docs/server-runtime.md: a PD watch that ends with a gRPC status the
  PD client does not retry is not started again, so that Server keeps
  its cached schema until it is restarted. This is how master behaves
  for every meta listener and is tracked in apache#3478.
- A TODO in the test's waitFor notes that the shared RestClient sets no
  timeouts.
- Distinguish watch readiness from transport heartbeats.
- Bound cluster HTTP calls and assert successful reader responses.
- Drain in-flight retries before replacing the test publisher.
- Clarify schema writer routing and propagation limits.
Batch normal clear events per graph in a fixed one-second window.
Reuse the existing scheduler and retry path, keeping graph-open recovery immediate.
Cover batching, concurrent flushes and test cleanup races; document delayed visibility.
Serialize manual window flushes with the existing scheduler so assertions wait for an already-running publication.

Add a regression that blocks a detached window and verifies the helper waits for its publication.

Exercise concurrent retry failure through the immediate graph-open publish path.
@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.76471% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.60%. Comparing base (8828d82) to head (24d1151).

Files with missing lines Patch % Lines
...graph/backend/cache/CachedSchemaTransactionV2.java 92.62% 7 Missing and 9 partials ⚠️
...n/java/org/apache/hugegraph/meta/PdMetaDriver.java 50.00% 3 Missing ⚠️
...ain/java/org/apache/hugegraph/meta/MetaDriver.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3481      +/-   ##
============================================
+ Coverage     58.52%   58.60%   +0.08%     
- Complexity     2717     3130     +413     
============================================
  Files           779      780       +1     
  Lines         69439    69634     +195     
  Branches       9577     9604      +27     
============================================
+ Hits          40637    40812     +175     
- Misses        25108    25131      +23     
+ Partials       3694     3691       -3     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø)
hstore 36.14% <62.71%> (-0.08%) ⬇️
pd 57.00% <100.00%> (+0.16%) ⬆️
server_hbase 42.60% <0.00%> (+0.11%) ⬆️ Carriedforward from 6dc4691
server_memory 57.06% <90.78%> (+0.40%) ⬆️
server_rocksdb 42.84% <3.07%> (-0.17%) ⬇️
store 36.14% <0.00%> (-0.07%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A Server that stops for good while a clear event waits in the 1-second
window never sends it. Before the window this gap lasted milliseconds.
Measured on a 3-Server cluster: after one schema change and a kill of
the writing Server within the window, the other Servers kept the old
schema.
@imbajin
imbajin force-pushed the fix/schema-cache-events-3235 branch 4 times, most recently from f6966ea to 6dc4691 Compare November 19, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Stale schema caches on HStore Servers: updates stay invisible, appends get overwritten, new indexes miss data

2 participants