Repository navigation
[tests] Turn off unused system logs and limit replica memory in the cluster setup - #3278
Merged
Merged
Conversation
…uster setup The cluster tests read only system.query_log. Each of the 3 replicas still wrote and merged about 22 other system logs. Each replica also sized its memory limit and caches for the whole runner (15 GiB), not for its share. - clickhouse-cluster/config.d/custom_config.xml: remove all default system logs except query_log (a remove of a section that a version does not have is a no-op, so the list works for 25.8 to latest). - ClusterClickHouseContainerStarter: 2 GiB memory limit on each replica container, so each server sets max_server_memory_usage to 1.8 GiB and lowers its caches to 1 GiB. Keeper and nginx stay unlimited. Part of: #3267 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Client V2 CoverageCoverage Report
Class Coverage
|
JDBC V2 CoverageCoverage Report
Class Coverage
|
JDBC V1 CoverageCoverage Report
Class Coverage
|
Client V1 CoverageCoverage Report
Class Coverage
|
chernser
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #3267 (PR 2 of the plan in #3267 (comment), approved). PR 1 is #3277.
The cluster legs run 3 replicas, Keeper and nginx on one GitHub runner (4 vCPU, 16 GB). Two things made each replica use more of the runner than the tests need:
asynchronous_metric_log,metric_log,text_log,trace_log, …) and merges their parts. The tests read onlysystem.query_log.max_server_memory_usageto 13.5 GiB and allows its caches to grow to 5 GiB, so 3 replicas size themselves for about 3× the runner.Changes
containers/clickhouse-cluster/config.d/custom_config.xml:remove="1"for every default system log exceptquery_log. Aremovefor a section that a server version does not have is a no-op, so one list works for all versions in the matrix. Single-server mode is not changed.ClusterClickHouseContainerStarter: 2 GiB memory limit on each replica container. Each server then logsAvailable RAM: 2.00 GiB, setsmax_server_memory_usageto 1.80 GiB and lowers its caches to 1 GiB (checked on 25.8 and 26.10). The measured peak is about 0.43 GB per replica (see below), so the limit is about 5× the peak. The total for 3 replicas is 6 GiB of the 16 GB runner.Not changed: no workflow files, no production code, no test code. Quorum insert hang stays with #3246. No CHANGELOG entry: test setup only. No section of
docs/changes_checklist.mdapplies.Test
Local cluster made by the repo's own
ClusterClickHouseContainerStarter(TEST_CLICKHOUSE_CONTAINER=cluster, CH 26.10). The host has 4 vCPU and 15 GiB, the same shape asubuntu-latest, with no extra CPU limit. Full integration-test runs (mvn -pl <module> -DskipUTs=true verify). CPU time and memory come from the cgroup of each container.mainmainnumbers(1000000000)queries inStatementTest). The saving from the logs is within the run-to-run variation there.SYSTEM FLUSH LOGS, onlysystem.query_logexists on a replica (25.8 and 26.10).ClientTests+InsertTests(they readsystem.query_log): 66/66 pass, 2 GiB limit applied.maineither on this host. This PR does not fix one specific test. It lowers the load and memory of each replica on a busy runner. The users/roles race is fixed by [tests] Wait until users, roles and grants reach all cluster replicas #3277.Pre-PR validation gate
AGENTS.md(test setup only, no workflow changes, Java 8 compatible)🤖 Generated with Claude Code
Note
Low Risk
Test-only container/config tweaks; no production or workflow changes, with measured integration runs still passing.
Overview
Reduces load from the 3-replica cluster integration setup on shared CI runners by targeting two causes of oversized replica behavior.
In
custom_config.xml(cluster containers only), itremoves every default system log exceptquery_log, so replicas stop continuously flushing/merging logs the tests never read. Unknown log sections on older matrix versions are no-ops.In
ClusterClickHouseContainerStarter, each replica gets a 2 GiB Docker memory limit, so ClickHouse derivesmax_server_memory_usageand cache sizes from the container instead of the full host (~15 GiB × 3). Keeper and nginx stay unlimited.Reviewed by Cursor Bugbot for commit 2e11d9c. Bugbot is set up for automated code reviews on this repo. Configure here.