Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- `cotel --backfill-cost` (dry run) and `--backfill-cost-apply` recompute `cost_usd` for every span of a priced model from its stored token counts at today's rates, revived from the closed PR #67 as reviewed there (FLO-551/552). Spans with an unknown or empty model are counted and left alone; `daily_usage` is not touched, because a day rolled up after the backfill derives its cost from the corrected spans. The apply needs the database to itself, so it runs while the server is stopped; the dry run restores the newest snapshot into a throwaway file instead of copying a live one. README → Maintenance commands has both procedures for the current compose deploy, with `--entrypoint` set so a one-off does not start a second Cloudflare tunnel connector
- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2 — without `user_id`, and *with* the mid-table `duration_ms` generated column, so that the one ALTER production ran on a long-lived filled table (v9 → v10, six weeks before the abort) runs here too instead of passing over a column the fixture never had, and so the indexes are built over the logical/physical column skew they were built over in production — with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts that `duration_ms` was there before the migration and is gone after it (a `DROP COLUMN IF EXISTS` that no-ops must not be able to come back unnoticed), that the reopen succeeds, that both checkpoints fold — a present-but-empty `.wal` counts as folded, so an engine that truncates rather than unlinks is not reported as this defect — that all four indexes survive, that a bare `user_id = <constant>` folded into the scan agrees with the same predicate wrapped in an expression, which is not, and that re-inserting a stored `span_id` is still rejected while an unused one is accepted. The last two are the silent form of the damage: a wrong answer rather than an error. They are phrased that way because no `SELECT` over `spans` is planned as an ART index scan on this engine — measured, including on the primary key and on a textbook unique match with the thresholds raised — so an “indexed probe agrees with an unindexed probe” assertion would have been two sequential scans agreeing; the pushdown pair is what [ADR-0013](docs/decisions/0013-spans-has-no-derived-columns.md)'s wrong answers actually came out of, and unique-constraint enforcement is the one path that provably traverses an ART index. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.1 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))

### Changed
- **Users are now actors, down to the schema** ([ADR-0027](docs/decisions/0027-users-become-actors.md)). What a token belongs to was never only a person: production holds one person, one host and fifteen Paperclip agents. Schema v11 renames the `users` table to `actors` and `spans.user_id` / `daily_usage.user_id` to `actor_id`; the API moves to `/api/v1/actors`, `?actor_id=` and `actors_count`; the dashboard's Users page becomes Actors at `/actors`; export archives become `format_version` 2. The rename runs in Go before `schema.sql`, because a rename cannot be idempotent in SQL and `CREATE TABLE IF NOT EXISTS actors` would otherwise create an empty twin beside the populated table; it checks the catalog per step, drops and lets `schema.sql` recreate the `spans` indexes (DuckDB will not ALTER an indexed table), commits as one transaction, and refuses to start if both tables exist. Requests using the old names keep working - `/api/v1/users`, `?user_id=`, `mode=user_only|user_and_history`, the `/users` dashboard URLs, and v1 export archives - but **response bodies use the new field names only**, and **an older binary cannot open a migrated file without losing its tokens**: rolling back across v11 means restoring a pre-v11 snapshot
- `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))

### Fixed
- Spans from the Claude 5.5 / 5.2 generation are priced. From 2026-11-07 every span from `claude-opus-5-5`, `claude-fable-5-2` and `claude-haiku-5-5` was stored with `cost_usd` unset because the price table did not know them, so the dashboard showed $0 for a week of real traffic. Adds Opus 5.5 ($4/$20), Sonnet 5.5 ($2/$10), Haiku 5.5 ($0.10/$0.50; its $0.50/$2.50 rate for prompts over 100K tokens is not modelled, since a span does not say which tier billed it), Fable and Mythos 5.2 ($8/$40) and 5.1 ($10/$50), with the published cache-read rates where they depart from 0.1 x input. Sonnet 5 moves from $3/$15 to the $2/$10 Anthropic still publishes. `pricing.Known` tells an unpriced model from a priced span that costs zero. Already-stored spans keep their unset cost until `--backfill-cost-apply` runs
- The Deploy workflow stops asking the production host's GUI keychain for credentials it does not hold, and stops being able to hold the deploy queue indefinitely when something does hang. `docker compose build --pull` makes buildkit resolve the Dockerfile frontend and the base images, which makes the docker CLI consult the credential store named in the host's `~/.docker/config.json`. On the production runner that store is `desktop`, whose helper is tied to the logged-in GUI session: spawned from the runner's launchd session it blocks in `open(2)` and never returns, buildkit re-spawns it on every retry, and a single hung call also stalls image resolution for any other build on that host (measured: a hand build from an interactive shell, normally 2.6 s, sat 4 min behind one). Nothing in the run log said so, because the hang lands before the first layer and the step's last line is the frontend resolve. The same helper call from an interactive shell on the same host answers in milliseconds, so both the hosted-runner smoke build and a hand build on the deploy host passed throughout. The job now builds against a copy of the config carrying the host's contexts and CLI plugins but no credential store at all, which is faithful to what the host actually has: its `auths` map is empty, and registry auth, if it is ever needed there, belongs in an explicit `docker login` with a repository secret rather than in one person's keychain. The deploy job also gains `timeout-minutes: 20`. The first wedge ran for nine hours and, because `deploy-main` has `cancel-in-progress: false`, every push behind it sat `pending` with no job at all, which is indistinguishable from an idle queue: production kept serving a build five days old, the next runtime fix would silently not have shipped, and no run ever went red to say so. A hung build now fails loudly and releases the group instead.

## [0.4.0] - 2026-10-13
Expand Down
Loading
Loading