diff --git a/README.md b/README.md index 0d9a6c9..35bdcc6 100644 --- a/README.md +++ b/README.md @@ -563,7 +563,7 @@ from the network at startup instead of bundling it). | `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. | | `COTEL_ART_REPRO_SPANS` | `20000` | Read by the ART-corruption reproducer test, not by the binary: how many synthetic spans it builds before the migration under test. The default keeps the test inside the ordinary `go test ./...` budget; raise it when hunting the defect. See [the reproducer](docs/operations/duckdb-recovery.md#the-synthetic-reproducer). | | `COTEL_ART_REPRO_CYCLES` | `2` | Read by the ART-corruption reproducer test, not by the binary: how many replace-transaction-then-restart cycles it runs. The damage accumulates across cycles, so this is the other knob to raise. | -| `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. | +| `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume held a database the binary could no longer open, kept as evidence until the owner had it deleted on 2026-11-15. A fresh install that has neither volume gets the default created empty, which is correct. | ## Maintenance commands diff --git a/docker-compose.yml b/docker-compose.yml index f86be41..915d0b0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -49,7 +49,8 @@ volumes: # database the binary cannot open otherwise means overwriting the only copy of # it in place — the evidence and the fallback both. The default is a restored # copy rather than the name compose derives on its own (`cotel_cotel-data`), - # because that one holds a database the binary can no longer open. Compose + # because that one held a database the binary could no longer open (deleted + # 2026-11-15, see docs/operations/duckdb-recovery.md). Compose # warns that it did not create a hand-made volume; that warning is expected. cotel-data: name: ${COTEL_DATA_VOLUME:-cotel-data-repaired-20261004} diff --git a/docs/operations/duckdb-recovery.md b/docs/operations/duckdb-recovery.md index 6ba7209..a8081e1 100644 --- a/docs/operations/duckdb-recovery.md +++ b/docs/operations/duckdb-recovery.md @@ -249,7 +249,7 @@ it at the same moment. |---|---|---|---| | `cotel-data-repaired-20261004` | live production, mounted at `/data`, the compose default | 146 MB | never, it is prod | | `cotel-snapshots` | the timed Parquet snapshots written by the running service, the actual backup | bounded by `COTEL_SNAPSHOT_KEEP` | never; it is the only restore point here | -| `cotel_cotel-data` | the damaged original, untouched: the only artifact that reproduces the `abort()` | 147 MB | a landed upstream fix *and* a DuckDB in cotel proven not to hit it, see below | +| `cotel_cotel-data` | the damaged original, untouched: the only artifact that reproduced the `abort()` | 147 MB | deleted 2026-11-15 by the owner's decision, before either release condition below held; see below | | `cotel-data-backup-20261004` | byte-identical copy of the damaged original, from step 2 | 147 MB | immediately, see below. For the 2026-10-04 recovery: deleted 2026-10-13, hash-verified against the original first | Delete by name, never with `docker volume prune`. Prune takes every volume no @@ -321,8 +321,18 @@ same two hashes were confirmed a second time, so the 2026-10-04 recovery now lea `cotel-data-repaired-20261004`, `cotel_cotel-data` and `cotel-snapshots`. A `…-backup-…` volume appearing after that date is not this one: classify it. +On 2026-11-15 `cotel_cotel-data` was deleted as well (next section), so the host +now holds only the live volume and `cotel-snapshots`. + ### The damaged original has two separate lifetimes +> **Deleted 2026-11-15.** The owner decided the damaged original was no longer +> worth keeping and had `cotel_cotel-data` removed, with neither release condition +> below met. The rest of this section is the reasoning that held it until then, +> kept so that what the deletion gave up is on record: the only file that aborted +> DuckDB 1.1.3 on open is gone, and the [synthetic reproducer](#the-synthetic-reproducer) +> is now the only way left to chase this defect. + As a **data fallback** it expires on its own. Its newest span was 2026-09-27 and raw spans are purged at `COTEL_RETENTION_RAW_DAYS` (default 30), so from roughly 2026-10-27 re-repairing it recovers nothing retention would have kept anyway. @@ -338,8 +348,8 @@ indexed table, then a large replace transaction. `internal/storage/schema.sql` d exactly that — line 53 adds `spans.user_id VARCHAR`, and line 74 puts an ART index on that same added column. -Therefore: **hold `cotel_cotel-data` indefinitely**, mounted read-only only. Both -release conditions have to hold, and each one has a trap: +Therefore the policy was: **hold `cotel_cotel-data` indefinitely**, mounted +read-only only, until both release conditions held. Each one has a trap: - **A landed fix upstream, not a closed issue.** duckdb#25360 is not ours — it was filed by an unaffiliated reporter, against a ~19 GB database, and we have never