Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,7 @@ from the network at startup instead of bundling it).
| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. |
| `COTEL_ART_REPRO_SPANS` | `20000` | Read by the ART-corruption reproducer test, not by the binary: how many synthetic spans it builds before the migration under test. The default keeps the test inside the ordinary `go test ./...` budget; raise it when hunting the defect. See [the reproducer](docs/operations/duckdb-recovery.md#the-synthetic-reproducer). |
| `COTEL_ART_REPRO_CYCLES` | `2` | Read by the ART-corruption reproducer test, not by the binary: how many replace-transaction-then-restart cycles it runs. The damage accumulates across cycles, so this is the other knob to raise. |
| `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. |
| `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume held a database the binary could no longer open, kept as evidence until the owner had it deleted on 2026-11-15. A fresh install that has neither volume gets the default created empty, which is correct. |

## Maintenance commands

Expand Down
3 changes: 2 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ volumes:
# database the binary cannot open otherwise means overwriting the only copy of
# it in place — the evidence and the fallback both. The default is a restored
# copy rather than the name compose derives on its own (`cotel_cotel-data`),
# because that one holds a database the binary can no longer open. Compose
# because that one held a database the binary could no longer open (deleted
# 2026-11-15, see docs/operations/duckdb-recovery.md). Compose
# warns that it did not create a hand-made volume; that warning is expected.
cotel-data:
name: ${COTEL_DATA_VOLUME:-cotel-data-repaired-20261004}
Expand Down
16 changes: 13 additions & 3 deletions docs/operations/duckdb-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ it at the same moment.
|---|---|---|---|
| `cotel-data-repaired-20261004` | live production, mounted at `/data`, the compose default | 146 MB | never, it is prod |
| `cotel-snapshots` | the timed Parquet snapshots written by the running service, the actual backup | bounded by `COTEL_SNAPSHOT_KEEP` | never; it is the only restore point here |
| `cotel_cotel-data` | the damaged original, untouched: the only artifact that reproduces the `abort()` | 147 MB | a landed upstream fix *and* a DuckDB in cotel proven not to hit it, see below |
| `cotel_cotel-data` | the damaged original, untouched: the only artifact that reproduced the `abort()` | 147 MB | deleted 2026-11-15 by the owner's decision, before either release condition below held; see below |
| `cotel-data-backup-20261004` | byte-identical copy of the damaged original, from step 2 | 147 MB | immediately, see below. For the 2026-10-04 recovery: deleted 2026-10-13, hash-verified against the original first |

Delete by name, never with `docker volume prune`. Prune takes every volume no
Expand Down Expand Up @@ -321,8 +321,18 @@ same two hashes were confirmed a second time, so the 2026-10-04 recovery now lea
`cotel-data-repaired-20261004`, `cotel_cotel-data` and `cotel-snapshots`. A
`…-backup-…` volume appearing after that date is not this one: classify it.

On 2026-11-15 `cotel_cotel-data` was deleted as well (next section), so the host
now holds only the live volume and `cotel-snapshots`.

### The damaged original has two separate lifetimes

> **Deleted 2026-11-15.** The owner decided the damaged original was no longer
> worth keeping and had `cotel_cotel-data` removed, with neither release condition
> below met. The rest of this section is the reasoning that held it until then,
> kept so that what the deletion gave up is on record: the only file that aborted
> DuckDB 1.1.3 on open is gone, and the [synthetic reproducer](#the-synthetic-reproducer)
> is now the only way left to chase this defect.

As a **data fallback** it expires on its own. Its newest span was 2026-09-27 and raw
spans are purged at `COTEL_RETENTION_RAW_DAYS` (default 30), so from roughly
2026-10-27 re-repairing it recovers nothing retention would have kept anyway.
Expand All @@ -338,8 +348,8 @@ indexed table, then a large replace transaction. `internal/storage/schema.sql` d
exactly that — line 53 adds `spans.user_id VARCHAR`, and line 74 puts an ART index
on that same added column.

Therefore: **hold `cotel_cotel-data` indefinitely**, mounted read-only only. Both
release conditions have to hold, and each one has a trap:
Therefore the policy was: **hold `cotel_cotel-data` indefinitely**, mounted
read-only only, until both release conditions held. Each one has a trap:

- **A landed fix upstream, not a closed issue.** duckdb#25360 is not ours — it was
filed by an unaffiliated reporter, against a ~19 GB database, and we have never
Expand Down
Loading