fix: don't drop a non-empty Atomic default database during Replicated conversion#255
Merged
GrigoryPervakov merged 2 commits intoJun 30, 2026
Merged
Conversation
…cated conversion
ensureReplicaDefaultDatabaseEngine converts the `default` database from
Atomic to Replicated when enableDatabaseSync is on. When `default` already
existed as a non-Replicated engine *and contained tables*, the count > 0
branch logged "data loss is possible" but then fell through and ran
`DROP DATABASE default SYNC` anyway, destroying those tables before
recreating `default` as Replicated.
This races any client that seeds `default` while it is still Atomic on a
fresh cluster (e.g. an OTel collector that auto-creates tables on first
export): if the seed lands before the operator's first reconcileDatabaseSync
pass, the operator drops the seeded schema and it never comes back.
Make the behavior match the documented intent ("never drops tables or
recreates the default database as Replicated if any tables have been
created"): when `default` is non-empty and not Replicated, leave it as-is
and skip the conversion. The engine is still switched automatically on an
empty `default`, so fresh clusters keep getting the Replicated engine.
Skip (return nil) rather than error so an otherwise-healthy cluster running
on an Atomic `default` is not held with SchemaInSync=False forever.
Add an integration test asserting a seeded, non-empty Atomic `default`
survives EnsureDefaultDatabaseEngine with its engine and rows intact.
… default Per review: rather than silently skipping (return nil), return an error when `default` is non-empty and not Replicated so the operator sets the SchemaInSync condition to false and surfaces the unconvertible database instead of hiding it. The database is still left intact (never dropped). Update the integration test to assert EnsureDefaultDatabaseEngine reports failure while the seeded table and Atomic engine are preserved.
GrigoryPervakov
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ensureReplicaDefaultDatabaseEngineconverts thedefaultdatabase from Atomic to Replicated whenenableDatabaseSyncis enabled. Whendefaultalready existed with a non-Replicated engine and contained tables, thecount > 0branch logged"data loss is possible"but then fell through and ranDROP DATABASE default SYNCanyway, destroying those tables before recreatingdefaultas Replicated.This makes the code match its documented intent — "It never drops tables or recreates the default database as Replicated if any tables have been created" — by refusing to convert when
defaultis non-empty and not Replicated, and returning an error so the operator reports the problem.The race this fixes
On a fresh cluster,
defaultstarts as Atomic and empty. The operator only converts it to Replicated on a reconcile pass after the replica isReady(reconcileDatabaseSyncoperates onreadyReplicas). Any client that connects and seedsdefaultin that window — e.g. an OTel collector that auto-creates tables on first export — creates tables in the still-Atomicdefault. If that seed lands before the operator's first conversion pass, the operator finds a non-empty Atomicdefault, warns, and drops it. The seeded schema never comes back; downstream consumers loop onTable default.<x> does not exist.This was hit in the ClickStack helm-charts on the
clickhouse-operator-helmv0.0.6 bump (single-replica deployment): allotel_*tables were dropped on first boot. Ref: ClickHouse/ClickStack-helm-charts#240.Behavior change
defaultstateReplicatedSchemaInSync=false)Fresh/empty clusters still get the Replicated engine automatically; only a populated
defaultis now preserved.Why return an error (not skip)
When
defaultis non-empty and not Replicated, the operator cannot perform the requested conversion without losing data. Returning an error propagates throughEnsureDefaultDatabaseEngineso the operator setsClickHouseConditionTypeSchemaInSync = false, surfacing the unconvertible database rather than silently leaving the cluster in a state that does not match the requested topology. The database itself is never dropped. (Per review feedback.)Test
Added an integration spec (
internal/controller/clickhouse/commands_test.go): seed a table with rows into the Atomicdefault, runEnsureDefaultDatabaseEngine, and assert it reports failure (false) while the engine is stillAtomicand the rows survived. The existing Atomic→Replicated conversion spec (emptydefault) continues to pass.Full
commandersuite passes locally (testcontainers).Notes
enableDatabaseSyncsemantics are unchanged.