25.8 Stable backport of #91574: Add cleanup thread to MergeTree to avoid cleanup starvation

[ilejn]

This is just a try to backport this PR.

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add a dedicated cleanup thread for MergeTree to prevent cleanup delays under heavy merge load. This resolves ClickHouse#86181.

The MergeTree cleanup logic was tightly coupled with the merge scheduler, which caused cleanup tasks to be delayed or completely blocked when the table was under heavy merge load. ReplicatedMergeTree already had a separate cleanup thread, so this change aligns MergeTree with that design. @azat You might be interested in this one.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [4580acc]

[ilejn]

Review notes

commit b0e1bc6
look_behind_lines in test_merge_tree_load_parts

In upstream the equivalent change is done by ea67e82 from ClickHouse#87719 , but it is not directly applicable to 25.8

commit 9f01fd8
Hardening tests

Fixes fluky check. Actually not related to the backport: if test files are altered artificially, e.g. via adding fake comments, we have same issue.

commit 79107eb
Author: Mikhail Artemenko mikhail.artemenko@clickhouse.com
fix integration tests

Some good test fixes

commit 792180a
Call clearCaches

The trickiest change. Fixes failures in e.g. 03174_exact_rows_before_aggregation

commit 02fecfe
Author: Azat Khuzhin a3at.mail@gmail.com
tests: make test test_merge_tree_load_parts retriable

Some required test fixes

commit 2643af8
Author: Azat Khuzhin a3at.mail@gmail.com
Merge pull request ClickHouse#91574 from amosbird/merge-tree-clean-up-thread

Actual backport with some changes.