[tests] Fix cluster tests on main #3267

Activity

1. chernser

   added this to the 0.12.0 milestone on Oct 29, 2026
2. chernser

   added

   test

   on Oct 29, 2026
3. chernser

   added the

   Task

   issue type on Oct 29, 2026
4. chernser

   assigned

   polyglotAI-bot

   on Oct 29, 2026
5. chernser commented on Oct 29, 2026

   

   chernser

   on Oct 29, 2026

   ContributorAuthor

   More actions

   @polyglotAI-bot please fix the issue. Read description carefully and create a short plan.

   👀React with 👀1Reacted by polyglotAI-bot
6. polyglotAI-bot commented on Oct 29, 2026

   

   polyglotAI-bot

   on Oct 29, 2026

   Collaborator

   More actions

   Plan for this issue. I checked the cluster failures on main and reproduced the main one locally with the repo's own cluster configs (3 replicas + Keeper + nginx, CH 26.10.3.7).

   What fails on main (cluster legs)

   1. Users and roles: HttpTransportTests.testPasswordAuthentication, ClientTests.testOperationCustomRoles / testClientCustomRoles (also Flaky test: client-v2 cluster legs — testOperationCustomRoles / testClientCustomRoles / testPasswordAuthentication use a new user or role before it reaches every replica #3237).
   2. Quorum insert right after CREATE TABLE hangs for 1800 s: RowBinaryFormatWriterTest.* (Flaky test: client-v2 cluster legs — insert right after CREATE TABLE hangs 1800 s with UNKNOWN_STATUS_OF_INSERT (quorum never reached) #3246).
   3. jdbc-v2 + CH cluster latest: testAllTableEnginesFromSystemTableEnginesAreMapped (new Elasticsearch engine in 26.10). This is not cluster-specific. It also fails on the single-server leg and is already [jdbc-v2] Add new engine to the list #3265 / [jdbc-v2] Add RETURNING, SHARED keywords and Elasticsearch engine for ClickHouse HEAD #3274. Not in this plan.

   Cause of 1 (verified)

   Each server keeps the users and roles from <replicated> in an in-memory cache. Login, role lookup and DROP ... IF EXISTS read only this cache (ZooKeeperReplicator::findEntity). A Keeper watch thread updates the cache asynchronously. nginx sends each request to any server, so a request just after CREATE USER can go to a server that does not have the user yet.

   The error in the description is the same race. DROP USER IF EXISTS some_user went to a server whose cache did not have the user from the previous run, so it did nothing. Then CREATE USER found the user in Keeper and failed with ACCESS_ENTITY_ALREADY_EXISTS.

   No server setting makes this synchronous. More CPU makes the window smaller, but does not close it.

   Local results (create a user, then 3 logins through nginx):

   CPU per replica	No sync	With SYSTEM RELOAD USERS ON CLUSTER
   no limit	0/100 failed	0/30 failed
   0.3	2/150 failed	0/100 failed
   0.15	23/120 failed (22 auth failures, 1 ACCESS_ENTITY_ALREADY_EXISTS)	0/120 failed

   CREATE USER ... ON CLUSTER does not help. With replicated storage it fails with ACCESS_ENTITY_ALREADY_EXISTS on the other servers (10/10).

   Plan

   PR 1: a sync point after user/role changes (test code only)

   • Add a helper to BaseIntegrationTest that runs SYSTEM RELOAD USERS ON CLUSTER 'test_cluster' on the cluster and does nothing on a single server.
   • This is not a sleep. On each server, SYSTEM RELOAD USERS syncs with Keeper and reads all users and roles again before it returns (ZooKeeperReplicator::reload). The ON CLUSTER query returns only after all 3 servers did this. It takes about 0.2 s.
   • Call it at the end of the setup in the tests that create, drop, alter or grant users and roles (client-v2 and jdbc-v2). The requests after it still go to any server, so the tests still check the replicated user directory on all 3 servers. No replica pinning and no skipped tests.

   PR 2: cluster resources (containers/clickhouse-cluster and the container starter)

   • Turn off the system logs that the tests do not read (client-v2 and jdbc-v2 tests read only system.query_log): metric_log, asynchronous_metric_log, trace_log, processors_profile_log, background_schedule_pool_log, and similar. Each of the 3 servers flushes and merges these tables all the time.
   • Set a memory limit on each container, so that each server sizes its memory and caches for its share of the runner, not for all 16 GB. I will measure the peak memory in a full local run first and choose the limit from that.
   • Compare CPU time and failure counts in a full local cluster run before and after, with a 4-CPU limit like the GitHub runner.

   Quorum hang (#3246): I found no setting that fixes it. It is a server race: a replica that is still cloning counts as active for the quorum, but the clone fetches the part without the quorum mark, so the quorum never completes. PR 2 makes the window smaller. A real fix must be in the server, so a human should report it to ClickHouse/ClickHouse.

   Questions

   1. Is the SYSTEM RELOAD USERS ON CLUSTER sync point acceptable for PR 1?
   2. Cloud: the helper does nothing there (we probably cannot run SYSTEM RELOAD USERS on Cloud). Is it OK to keep Cloud as it is for now?
   3. Is it OK to handle the quorum hang under Flaky test: client-v2 cluster legs — insert right after CREATE TABLE hangs 1800 s with UNKNOWN_STATUS_OF_INSERT (quorum never reached) #3246, separately from this issue?

   Neither PR changes .github/workflows or production code.
7. chernser commented on Oct 29, 2026

   

   chernser

   on Oct 29, 2026

   ContributorAuthor

   More actions

   @polyglotAI-bot
   PR 1 should use check instead of delay to verify user is synced across all replicas (do limited number of attempts). Here is SQL example

   SELECT
       hostName() AS host,
       countIf(name = {user:String}) AS matching_users
   FROM clusterAllReplicas('test_cluster', system.users)
   GROUP BY host

   Also this PR should use unique user names if it doesn't matter for tests. In this case we make sure tests are verifying exact user not the one created by another tests.

   All administrative commands should be avoided in tests because:

   • tests should be close to ordinary use. In this case we doing too many operations while in reality there is a small delay between them
   • it is not allowed to run any such impacting command on the Cloud - so test will start failing again.

   P2: completely what I think

   Questions/Answers:

   1. already answered above
   2. already answered above
   3. this waits so yes.

   Please proceed

   👀React with 👀1Reacted by polyglotAI-bot
8. odp-ops

   added a commit that references this issue on Nov 1, 2026

   tests: turn off unused system logs and limit replica memory in the cl…

   ...

   2e11d9c
9. chernser commented on Nov 2, 2026

   

   chernser

   on Nov 2, 2026

   ContributorAuthor

   More actions

   Closed by #3285
10. chernser

    closed this as completedon Nov 2, 2026