Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2 — without `user_id`, and *with* the mid-table `duration_ms` generated column, so that the one ALTER production ran on a long-lived filled table (v9 → v10, six weeks before the abort) runs here too instead of passing over a column the fixture never had, and so the indexes are built over the logical/physical column skew they were built over in production — with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts that `duration_ms` was there before the migration and is gone after it (a `DROP COLUMN IF EXISTS` that no-ops must not be able to come back unnoticed), that the reopen succeeds, that both checkpoints fold — a present-but-empty `.wal` counts as folded, so an engine that truncates rather than unlinks is not reported as this defect — that all four indexes survive, that a bare `user_id = <constant>` folded into the scan agrees with the same predicate wrapped in an expression, which is not, and that re-inserting a stored `span_id` is still rejected while an unused one is accepted. The last two are the silent form of the damage: a wrong answer rather than an error. They are phrased that way because no `SELECT` over `spans` is planned as an ART index scan on this engine — measured, including on the primary key and on a textbook unique match with the thresholds raised — so an “indexed probe agrees with an unindexed probe” assertion would have been two sequential scans agreeing; the pushdown pair is what [ADR-0013](docs/decisions/0013-spans-has-no-derived-columns.md)'s wrong answers actually came out of, and unique-constraint enforcement is the one path that provably traverses an ART index. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.1 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))

### Changed
- Database snapshots are taken once a day and fourteen are kept (`COTEL_SNAPSHOT_INTERVAL=24h`, `COTEL_SNAPSHOT_KEEP=14`), down from every six hours with 56 kept. Reach stays fourteen days; a restore now loses up to a day of telemetry instead of six hours, for about a quarter of the disk. The probe's `SNAPSHOT_STALE_AFTER_SECONDS` default moves from 12 h to 48 h with it, so a healthy daily worker does not read as stale ([ADR-0023 amendment](docs/decisions/0023-production-database-snapshots.md))
- **Users are now actors, down to the schema** ([ADR-0027](docs/decisions/0027-users-become-actors.md)). What a token belongs to was never only a person: production holds one person, one host and fifteen Paperclip agents. Schema v11 renames the `users` table to `actors` and `spans.user_id` / `daily_usage.user_id` to `actor_id`; the API moves to `/api/v1/actors`, `?actor_id=` and `actors_count`; the dashboard's Users page becomes Actors at `/actors`; export archives become `format_version` 2. The rename runs in Go before `schema.sql`, because a rename cannot be idempotent in SQL and `CREATE TABLE IF NOT EXISTS actors` would otherwise create an empty twin beside the populated table; it checks the catalog per step, drops and lets `schema.sql` recreate the `spans` indexes (DuckDB will not ALTER an indexed table), commits as one transaction, and refuses to start if both tables exist. Requests using the old names keep working - `/api/v1/users`, `?user_id=`, `mode=user_only|user_and_history`, the `/users` dashboard URLs, and v1 export archives - but **response bodies use the new field names only**, and **an older binary cannot open a migrated file without losing its tokens**: rolling back across v11 means restoring a pre-v11 snapshot
- `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -539,15 +539,15 @@ from the network at startup instead of bundling it).
| `COTEL_RETENTION_AGGREGATE_DAYS` | `90` | Daily aggregate retention in days |
| `COTEL_RETENTION_INTERVAL` | `6h` | Retention worker tick interval (Go duration) |
| `COTEL_SNAPSHOT_DIR` | _(unset - snapshots off)_ | Directory the snapshot worker exports the whole database into, one dated subdirectory per snapshot. Empty disables snapshots; `docker-compose.yml` sets `/snapshots`, backed by its own volume. See [Database Snapshots and Restore](docs/operations/duckdb-snapshots.md). |
| `COTEL_SNAPSHOT_INTERVAL` | `6h` | How often a snapshot is taken (Go duration). A snapshot is only taken when the newest complete one is older than this, so a restart cannot churn through the retained window. |
| `COTEL_SNAPSHOT_KEEP` | `56` | How many complete snapshots to keep; older ones and incomplete ones are pruned after each successful export. At the default interval, 56 is 14 days of reach for about 235 MB. The last snapshot standing is never pruned. |
| `COTEL_SNAPSHOT_INTERVAL` | `24h` | How often a snapshot is taken (Go duration). A snapshot is only taken when the newest complete one is older than this, so a restart cannot churn through the retained window. |
| `COTEL_SNAPSHOT_KEEP` | `14` | How many complete snapshots to keep; older ones and incomplete ones are pruned after each successful export. At the default interval, 14 is 14 days of reach for about 60 MB. The last snapshot standing is never pruned. |
| `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/snapshots`. Note that `docker volume prune` on a stopped deploy deletes it - the volume counts as in use only while the container exists. |
| `COTEL_WAL_AUTOCHECKPOINT` | `4MB` | DuckDB `checkpoint_threshold`: the write-ahead log is folded into the main file once it grows past this size. Lower values bound how much WAL an ungraceful kill leaves to replay on the next open; higher values checkpoint less often during ingest. DuckDB's own default is `16MB`. |
| `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel |
| `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) |
| `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. |
| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `172800` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `24h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. |
| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. |
| `COTEL_ART_REPRO_SPANS` | `20000` | Read by the ART-corruption reproducer test, not by the binary: how many synthetic spans it builds before the migration under test. The default keeps the test inside the ordinary `go test ./...` budget; raise it when hunting the defect. See [the reproducer](docs/operations/duckdb-recovery.md#the-synthetic-reproducer). |
| `COTEL_ART_REPRO_CYCLES` | `2` | Read by the ART-corruption reproducer test, not by the binary: how many replace-transaction-then-restart cycles it runs. The damage accumulates across cycles, so this is the other knob to raise. |
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ services:
# the default outside this compose file; here they are always on, because
# a deploy without a recovery point is how production ended up with none.
COTEL_SNAPSHOT_DIR: ${COTEL_SNAPSHOT_DIR:-/snapshots}
COTEL_SNAPSHOT_INTERVAL: ${COTEL_SNAPSHOT_INTERVAL:-6h}
COTEL_SNAPSHOT_KEEP: ${COTEL_SNAPSHOT_KEEP:-56}
COTEL_SNAPSHOT_INTERVAL: ${COTEL_SNAPSHOT_INTERVAL:-24h}
COTEL_SNAPSHOT_KEEP: ${COTEL_SNAPSHOT_KEEP:-14}

volumes:
# Named explicitly so the deploy can be pointed at a different volume without
Expand Down
21 changes: 21 additions & 0 deletions docs/decisions/0023-production-database-snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,3 +210,24 @@ first of which is fatal on its own:
Parquet, so a snapshot outlives the engine version that wrote it. That is the
property the native-file copy does not have, and the reason this ADR exists rather
than a cron line in a README.

---

## Amendment, 2026-11-15: one snapshot a day, fourteen kept

**Decider:** Flop (board)

The cadence moves from `6h` × 56 to `24h` × 14 (`COTEL_SNAPSHOT_INTERVAL=24h`,
`COTEL_SNAPSHOT_KEEP=14`). The reach this ADR argued for is unchanged - fourteen
days, past the six an unnoticed failure has already lasted. What changes is the
recovery point: a restore now loses up to a day of telemetry instead of six
hours, which the owner accepts for this service. The probe's
`SNAPSHOT_STALE_AFTER_SECONDS` moves with it, from 12 h to 48 h (two intervals,
for the reason given in [the snapshots runbook](../operations/duckdb-snapshots.md#the-threshold)).

The prune rule keeps the N *newest* snapshots, so on the first daily export after
this change a volume full of 6-hourly snapshots would shrink to the 14 newest -
about three days of reach - and take two weeks to grow back. To keep the reach
through the switch, thin an existing 6-hourly backlog to the newest snapshot of
each UTC day before the first daily export, so the fourteen kept are fourteen
days.
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,11 @@ Restart Claude Code. Telemetry starts flowing immediately.
| `COTEL_RETENTION_AGGREGATE_DAYS` | `90` | Daily aggregate retention in days |
| `COTEL_RETENTION_INTERVAL` | `6h` | Retention worker tick interval |
| `COTEL_SNAPSHOT_DIR` | _(unset - snapshots off)_ | Where the snapshot worker exports the whole database (compose sets `/snapshots`); empty disables snapshots. See [Database Snapshots and Restore](./operations/duckdb-snapshots) |
| `COTEL_SNAPSHOT_INTERVAL` | `6h` | How often a snapshot is taken (Go duration) |
| `COTEL_SNAPSHOT_KEEP` | `56` | How many complete snapshots to keep - 14 days at the default interval |
| `COTEL_SNAPSHOT_INTERVAL` | `24h` | How often a snapshot is taken (Go duration) |
| `COTEL_SNAPSHOT_KEEP` | `14` | How many complete snapshots to keep - 14 days at the default interval |
| `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not the binary: the Docker volume mounted at `/snapshots` |
| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not the binary: `auto` pages only on an affirmatively broken snapshot, `require` also when the instance claims none, `off` never asks. See [Production health probes](./operations/health-probe#the-snapshot-claim-asked-of-a-second-endpoint) |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not the binary: how old the newest snapshot may be - two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h` |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `172800` | Read by `scripts/probe-healthz.sh`, not the binary: how old the newest snapshot may be - two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `24h` |
| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not the binary: overrides the derived `/api/v1/health` address |

## Data & retention
Expand Down
6 changes: 3 additions & 3 deletions docs/operations/duckdb-snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ an endpoint that has stopped answering all land in those two rows, and under

### The threshold

`SNAPSHOT_STALE_AFTER_SECONDS` defaults to **43200 s (12 h)** — two
`COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`. Two and not one because
`SNAPSHOT_STALE_AFTER_SECONDS` defaults to **172800 s (48 h)** — two
`COTEL_SNAPSHOT_INTERVAL` periods at the shipped `24h`. Two and not one because
a cycle is scheduled *from* the last one, so a single interval leaves no room
for the export itself and a tick landing just before the next run would be red
every time. If you change `COTEL_SNAPSHOT_INTERVAL`, change this with it: the
Expand Down Expand Up @@ -159,7 +159,7 @@ from it, so there is no second address to keep in step on the host. A green line
names the snapshot it found:

```
probe-healthz: OK — HTTP 200 ingest age 2s (threshold 21600s) url=http://robmini.local:8080/healthz | snapshot last run 2026-10-06T17:39:54Z (age 909s, threshold 43200s) dir=/snapshots/2026-10-06T17-39-54Z
probe-healthz: OK — HTTP 200 ingest age 2s (threshold 21600s) url=http://robmini.local:8080/healthz | snapshot last run 2026-10-06T17:39:54Z (age 909s, threshold 172800s) dir=/snapshots/2026-10-06T17-39-54Z
```

## Restoring from a snapshot
Expand Down
4 changes: 2 additions & 2 deletions docs/operations/health-probe.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ host. `API_HEALTH_URL` overrides the derivation.
| `snapshot` object | Verdict |
|---|---|
| `status: "error"` | red, exit 6, with `last_error` quoted |
| `status: "ok"` and `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 12 h) | red, exit 6 |
| `status: "ok"` and `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 48 h) | red, exit 6 |
| `status: "ok"` and `last_run_at` absent or unparseable | red, exit 6 |
| `status: "unknown"` | not asserted; said on the green line |
| no `snapshot` field, or `/api/v1/health` not readable | not asserted; said on the green line |
Expand Down Expand Up @@ -238,7 +238,7 @@ a probe-configuration problem), and the ingest probe, which never reads
| Variable | Default | Role |
|---|---|---|
| `SNAPSHOT_CHECK` | `auto` | `auto` pages on affirmative failure only; `require` also pages when the instance makes no snapshot claim; `off` never asks. `probe-loopback` sets **`require`** — it only ever asks the deploy host, where snapshots are expected, so "no claim" is a failure there and not a local default |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | How old the newest snapshot may be. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; move it with the interval |
| `SNAPSHOT_STALE_AFTER_SECONDS` | `172800` | How old the newest snapshot may be. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `24h`; move it with the interval |
| `API_HEALTH_URL` | derived from the `/healthz` URL | Override when the snapshot report is not at the same entry point |

What counts as red, why the threshold is two intervals, and how to check it by
Expand Down
11 changes: 6 additions & 5 deletions internal/storage/snapshot.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ import (
"time"
)

// Snapshot defaults. 6h × 56 kept snapshots is 14 days of reach: the September
// Snapshot defaults. 24h × 14 kept snapshots is 14 days of reach: the September
// 2026 corruption went unnoticed for six days, so depth past the moment someone
// notices is worth more than a tighter recovery point. At the measured 4.2 MiB
// per snapshot that whole window costs ~235 MB.
// notices is what matters. One a day was chosen by the owner on 2026-11-15 over
// the original 6h × 56: a restore loses at most a day of telemetry, which is
// acceptable here, for a quarter of the snapshots (~60 MB at 4.2 MiB each).
const (
DefaultSnapshotKeep = 56
DefaultSnapshotInterval = 6 * time.Hour
DefaultSnapshotKeep = 14
DefaultSnapshotInterval = 24 * time.Hour
)

// SnapshotManifestName is written last in a snapshot directory, and nothing
Expand Down
Loading
Loading