[test] ReplicaTest is flaky due to stale gauges in shared test metric groups

[Shawn-Hx]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

Problem

ReplicaTest may read a stale physicalStorage/localLogSize gauge left by another test running in the same reused Maven Surefire JVM.

The following assertions can fail:

ReplicaTest.testBucketPhysicalStorageLocalLogSizeIncludesFollower
expected: 450L
 but was: 10830L

ReplicaTest.testPhysicalStorageLocalLogSizeIsScopedPerBucket
expected: 450L
 but was: 10830L

Running ReplicaTest alone, for example directly from IntelliJ IDEA, normally succeeds. The failure only occurs when another test using the same table and bucket registers the metric first in the same reused Surefire fork.

The default test configuration uses two Surefire forks with fork reuse enabled. Therefore, whether the affected tests fail depends on test-class assignment and execution order, making the CI failure flaky.

Expected behavior

Each test should use metrics associated with the LogTablet created by that test. The local log size gauge should return 450L.

Actual behavior

The test may reuse a bucket metric group created by an earlier test and read its stale gauge. In the observed failure, the gauge still references a previous LogTablet whose size is 10830L.

Root cause

ReplicaTestBase uses the static singleton instances:

TestingMetricGroups.TABLET_SERVER_METRICS
TestingMetricGroups.USER_METRICS

These objects are shared by all tests executed in the same JVM.

TabletServerMetricGroup and TableMetricGroup cache metric groups using computeIfAbsent. When another replica for the same table and bucket registers the same gauge, the existing metric is retained. The gauge therefore continues referencing the LogTablet created by an earlier test.

ReplicaManager.shutdown() does not remove all table and bucket metric groups, so this state survives until another test runs in the reused fork.

How to reproduce

Use JDK 11 and force the polluting test and the affected tests to run sequentially in one reused fork:

JAVA_HOME=/path/to/jdk-11 \
./mvnw -pl fluss-server \
  -DskipITs \
  -Dtest='RemoteLogFetcherTest#testFetchOverlappingSegmentsFromReplicasWithDifferentBoundaries,ReplicaTest#testBucketPhysicalStorageLocalLogSizeIncludesFollower+testPhysicalStorageLocalLogSizeIsScopedPerBucket' \
  -Dfluss.forkCount=1 \
  -Dfluss.reuseForks=true \
  -Dsurefire.runOrder=alphabetical \
  test

The first test succeeds, while both ReplicaTest methods fail with:

expected: 450L
 but was: 10830L

Running only the two ReplicaTest methods succeeds, confirming that the failure is caused by state leaked from the preceding test.

Solution

Create a fresh TabletServerMetricGroup and its associated UserMetrics for every ReplicaTestBase test instead of using the static instances from TestingMetricGroups.

Are you willing to submit a PR?

• I'm willing to submit a PR!