Skip to content

fix: don't drop a non-empty Atomic default database during Replicated conversion#255

Merged
GrigoryPervakov merged 2 commits into
mainfrom
warren/fix-default-db-drop-on-nonempty-atomic
Jun 30, 2026
Merged

fix: don't drop a non-empty Atomic default database during Replicated conversion#255
GrigoryPervakov merged 2 commits into
mainfrom
warren/fix-default-db-drop-on-nonempty-atomic

Conversation

@wrn14897

@wrn14897 wrn14897 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

ensureReplicaDefaultDatabaseEngine converts the default database from Atomic to Replicated when enableDatabaseSync is enabled. When default already existed with a non-Replicated engine and contained tables, the count > 0 branch logged "data loss is possible" but then fell through and ran DROP DATABASE default SYNC anyway, destroying those tables before recreating default as Replicated.

This makes the code match its documented intent — "It never drops tables or recreates the default database as Replicated if any tables have been created" — by refusing to convert when default is non-empty and not Replicated, and returning an error so the operator reports the problem.

The race this fixes

On a fresh cluster, default starts as Atomic and empty. The operator only converts it to Replicated on a reconcile pass after the replica is Ready (reconcileDatabaseSync operates on readyReplicas). Any client that connects and seeds default in that window — e.g. an OTel collector that auto-creates tables on first export — creates tables in the still-Atomic default. If that seed lands before the operator's first conversion pass, the operator finds a non-empty Atomic default, warns, and drops it. The seeded schema never comes back; downstream consumers loop on Table default.<x> does not exist.

This was hit in the ClickStack helm-charts on the clickhouse-operator-helm v0.0.6 bump (single-replica deployment): all otel_* tables were dropped on first boot. Ref: ClickHouse/ClickStack-helm-charts#240.

Behavior change

default state Before After
does not exist create Replicated create Replicated (unchanged)
already Replicated no-op no-op (unchanged)
Atomic, empty drop + recreate Replicated drop + recreate Replicated (unchanged)
Atomic, non-empty warn, then DROP + recreate Replicated (data loss) leave intact, return error (operator sets SchemaInSync=false)

Fresh/empty clusters still get the Replicated engine automatically; only a populated default is now preserved.

Why return an error (not skip)

When default is non-empty and not Replicated, the operator cannot perform the requested conversion without losing data. Returning an error propagates through EnsureDefaultDatabaseEngine so the operator sets ClickHouseConditionTypeSchemaInSync = false, surfacing the unconvertible database rather than silently leaving the cluster in a state that does not match the requested topology. The database itself is never dropped. (Per review feedback.)

Test

Added an integration spec (internal/controller/clickhouse/commands_test.go): seed a table with rows into the Atomic default, run EnsureDefaultDatabaseEngine, and assert it reports failure (false) while the engine is still Atomic and the rows survived. The existing Atomic→Replicated conversion spec (empty default) continues to pass.

commander preserves a non-empty Atomic default instead of dropping it [integration]  status="passed"
commander converts default database from Atomic to Replicated [integration]          status="passed"

Full commander suite passes locally (testcontainers).

Notes

  • No CRD/API changes; enableDatabaseSync semantics are unchanged.
  • This is a data-loss bugfix.

…cated conversion

ensureReplicaDefaultDatabaseEngine converts the `default` database from
Atomic to Replicated when enableDatabaseSync is on. When `default` already
existed as a non-Replicated engine *and contained tables*, the count > 0
branch logged "data loss is possible" but then fell through and ran
`DROP DATABASE default SYNC` anyway, destroying those tables before
recreating `default` as Replicated.

This races any client that seeds `default` while it is still Atomic on a
fresh cluster (e.g. an OTel collector that auto-creates tables on first
export): if the seed lands before the operator's first reconcileDatabaseSync
pass, the operator drops the seeded schema and it never comes back.

Make the behavior match the documented intent ("never drops tables or
recreates the default database as Replicated if any tables have been
created"): when `default` is non-empty and not Replicated, leave it as-is
and skip the conversion. The engine is still switched automatically on an
empty `default`, so fresh clusters keep getting the Replicated engine.

Skip (return nil) rather than error so an otherwise-healthy cluster running
on an Atomic `default` is not held with SchemaInSync=False forever.

Add an integration test asserting a seeded, non-empty Atomic `default`
survives EnsureDefaultDatabaseEngine with its engine and rows intact.
@wrn14897 wrn14897 requested a review from GrigoryPervakov June 30, 2026 17:34
@GrigoryPervakov GrigoryPervakov changed the title Don't drop a non-empty Atomic default database during Replicated conversion fix: don't drop a non-empty Atomic default database during Replicated conversion Jun 30, 2026
Comment thread internal/controller/clickhouse/commands.go Outdated
@wrn14897 wrn14897 marked this pull request as ready for review June 30, 2026 17:54
… default

Per review: rather than silently skipping (return nil), return an error when
`default` is non-empty and not Replicated so the operator sets the
SchemaInSync condition to false and surfaces the unconvertible database
instead of hiding it. The database is still left intact (never dropped).

Update the integration test to assert EnsureDefaultDatabaseEngine reports
failure while the seeded table and Atomic engine are preserved.
@GrigoryPervakov GrigoryPervakov enabled auto-merge (squash) June 30, 2026 18:28
@GrigoryPervakov GrigoryPervakov merged commit d9ad252 into main Jun 30, 2026
57 of 61 checks passed
@GrigoryPervakov GrigoryPervakov deleted the warren/fix-default-db-drop-on-nonempty-atomic branch June 30, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants