Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/health-probe.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ on:
- "scripts/probe-edge-ingest_test.sh"
- "scripts/page-cotel-health.sh"
- "scripts/page-cotel-health_test.sh"
- "scripts/otel-headers-helper.py"
- "scripts/otel-headers-helper_test.sh"
- ".github/workflows/health-probe.yml"

permissions:
Expand All @@ -75,6 +77,8 @@ jobs:
run: bash scripts/probe-edge-ingest_test.sh
- name: Run pager tests
run: bash scripts/page-cotel-health_test.sh
- name: Run OTEL headers helper tests
run: bash scripts/otel-headers-helper_test.sh

probe-ingest:
name: Probe the public OTLP ingest
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- **Ingest keys: one shared key authorizes, each sender's actor token identifies** ([ADR-0028](docs/decisions/0028-ingest-keys-and-actor-tokens.md)). Paperclip now strips every `OTEL_*` and `CLAUDE_CODE_*TELEMETRY*` variable from an agent's env and applies one company-wide set of OTLP headers, so a per-agent `Authorization: Bearer <actor token>` can no longer be configured, and with that setting empty it turns telemetry off, which is why agent telemetry stopped on 2026-11-07. An ingest key (`cotel_ik_<64 hex>`, created, copied and revoked under Setup → Settings → Ingest keys, stored in plaintext like actor tokens) authorizes OTLP ingest and nothing else, never export/import or the dashboard; the existing actor token, unchanged and not reissued, rides next to it in `X-Cotel-Actor-Token`. A valid key with a missing or unknown actor token is accepted as Anonymous even when allow-anonymous is off, because the request was authorized; `X-Cotel-Actor-Token` without a bearer never identifies anyone, and a bearer actor token works exactly as before. Schema v12 adds `ingest_keys`; the dashboard API gains `GET`/`POST /api/v1/ingest-keys` and `DELETE /api/v1/ingest-keys/{id}`. `POST /v1/logs` accepts and discards OTLP logs (`200 {}`), so Paperclip's log exporter stops getting 404. `scripts/otel-headers-helper.py` is a Claude Code `otelHeadersHelper` that prints the complete header set — `OTEL_EXPORTER_OTLP_HEADERS` overlaid with `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, plus `X-Cotel-Actor-Token` from `COTEL_ACTOR_TOKEN` — and always exits 0 with one JSON object, since a failing helper stops Claude Code's exports. Docs → [Actors and Authentication](docs/operations/actors-and-auth.md#ingest-keys-one-key-for-many-senders)
- `cotel --backfill-cost` (dry run) and `--backfill-cost-apply` recompute `cost_usd` for every span of a priced model from its stored token counts at today's rates, revived from the closed PR #67 as reviewed there (FLO-551/552). Spans with an unknown or empty model are counted and left alone; `daily_usage` is not touched, because a day rolled up after the backfill derives its cost from the corrected spans. The apply needs the database to itself, so it runs while the server is stopped; the dry run restores the newest snapshot into a throwaway file instead of copying a live one. README → Maintenance commands has both procedures for the current compose deploy, with `--entrypoint` set so a one-off does not start a second Cloudflare tunnel connector
- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2 — without `user_id`, and *with* the mid-table `duration_ms` generated column, so that the one ALTER production ran on a long-lived filled table (v9 → v10, six weeks before the abort) runs here too instead of passing over a column the fixture never had, and so the indexes are built over the logical/physical column skew they were built over in production — with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts that `duration_ms` was there before the migration and is gone after it (a `DROP COLUMN IF EXISTS` that no-ops must not be able to come back unnoticed), that the reopen succeeds, that both checkpoints fold — a present-but-empty `.wal` counts as folded, so an engine that truncates rather than unlinks is not reported as this defect — that all four indexes survive, that a bare `user_id = <constant>` folded into the scan agrees with the same predicate wrapped in an expression, which is not, and that re-inserting a stored `span_id` is still rejected while an unused one is accepted. The last two are the silent form of the damage: a wrong answer rather than an error. They are phrased that way because no `SELECT` over `spans` is planned as an ART index scan on this engine — measured, including on the primary key and on a textbook unique match with the thresholds raised — so an “indexed probe agrees with an unindexed probe” assertion would have been two sequential scans agreeing; the pushdown pair is what [ADR-0013](docs/decisions/0013-spans-has-no-derived-columns.md)'s wrong answers actually came out of, and unique-constraint enforcement is the one path that provably traverses an ART index. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.1 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))

Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ Open the dashboard → **Actors** (people icon in the sidebar) → **Add actor**

By default, cotel accepts spans without an `Authorization` header (allow-anonymous mode). To enforce strict auth: open **Setup** → **Settings** tab → disable **Allow anonymous OTLP**. Requests without a valid token then receive `401 Unauthorized`.

For a fleet that shares one set of OTLP headers, such as agents run by Paperclip, create an **ingest key** under **Setup** → **Settings** → **Ingest keys** and send it as the bearer instead. The key only authorizes ingest; each sender still names its actor by sending its own actor token in the `X-Cotel-Actor-Token` header, which `scripts/otel-headers-helper.py` adds from `COTEL_ACTOR_TOKEN` as a Claude Code `otelHeadersHelper`. See [Ingest keys](docs/operations/actors-and-auth.md#ingest-keys-one-key-for-many-senders).

See **[docs/operations/actors-and-auth.md](docs/operations/actors-and-auth.md)** for the full guide: one token per sender, token rotation, and security considerations.

## Publishing with Cloudflare
Expand Down
5 changes: 3 additions & 2 deletions cmd/cotel/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,11 @@ func main() {
go db.RunSnapshotWorker(snapshotCfg, envDuration("COTEL_SNAPSHOT_INTERVAL", storage.DefaultSnapshotInterval))

ingestMux := http.NewServeMux()
ingestMux.Handle("/v1/traces", auth.Middleware(db, ingest.New(db)))
ingestMux.Handle("/v1/traces", auth.IngestMiddleware(db, ingest.New(db)))
ingestMux.Handle("/v1/logs", auth.IngestMiddleware(db, ingest.NewLogsSink()))

ro := db.ReadOnly()
apiHandler := api.New(ro).SetPublicIngestURL(parsePublicIngestURL(os.Getenv("COTEL_PUBLIC_INGEST_URL"))).SetActorStore(db)
apiHandler := api.New(ro).SetPublicIngestURL(parsePublicIngestURL(os.Getenv("COTEL_PUBLIC_INGEST_URL"))).SetActorStore(db).SetIngestKeyStore(db)
dashMux := http.NewServeMux()
dashMux.Handle("/api/v1/export", auth.Middleware(db, export.NewHandler(db)))
dashMux.Handle("/api/v1/import", auth.Middleware(db, importpkg.NewHandler(db)))
Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ export default defineConfig({
{ text: 'ADR-0025 - Best-Effort Ingest Watcher in Actions', link: '/decisions/0025-best-effort-ingest-watcher-in-actions' },
{ text: 'ADR-0026 - The Deploy Host Asks /healthz on a Schedule', link: '/decisions/0026-scheduled-loopback-probe-with-a-pager' },
{ text: 'ADR-0027 - Users Become Actors', link: '/decisions/0027-users-become-actors' },
{ text: 'ADR-0028 - Ingest Keys Authorize, Actor Tokens Identify', link: '/decisions/0028-ingest-keys-and-actor-tokens' },
],
},
],
Expand Down
126 changes: 126 additions & 0 deletions docs/decisions/0028-ingest-keys-and-actor-tokens.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# ADR 0028 — Ingest keys authorize, actor tokens identify

**Date:** 2026-11-15
**Status:** Accepted
**Deciders:** Flop (board)

---

## Context

Since ADR-0008 every sender proves two things with one header:
`Authorization: Bearer <actor token>` both lets the request in and names the
actor its spans belong to. Each Paperclip agent carried its own token in its own
adapter env, set by `scripts/agent-set-cotel-token.sh`.

Paperclip now manages Claude Code's telemetry env itself. Before every agent run
it removes every `OTEL_*` and `CLAUDE_CODE_*TELEMETRY*` variable from the
agent's env and applies **one** company-wide OTLP destination with **one** set
of headers (its "Agent observability" setting): http/protobuf, traces to
`{base}/v1/traces`, logs to `{base}/v1/logs`, the headers in
`OTEL_EXPORTER_OTLP_TRACES_HEADERS` and `OTEL_EXPORTER_OTLP_LOGS_HEADERS` with
percent-encoded values, and the resource attributes `paperclip.agent.id` and
`paperclip.company.id`. With that setting empty it sets
`CLAUDE_CODE_ENABLE_TELEMETRY=0`.

That is why agent telemetry stopped on 2026-11-07: the per-agent headers were
stripped and telemetry was switched off. Configuring the setting brings
telemetry back, but only with headers that are identical for every agent. A
per-agent `Authorization` header cannot be expressed through Paperclip, so with
one header doing both jobs the choice is between no agent telemetry and every
agent merged into one actor.

## Decision

Split authorization from identification.

- An **ingest key** is a new shared credential, `cotel_ik_<64 hex>`. It is
created and revoked in the dashboard (**Setup → Settings → Ingest keys**) and
travels as `Authorization: Bearer <ingest key>`. Many senders can share one.
- The **actor token** is the existing per-actor `cotel_<64 hex>` token,
unchanged and not reissued. Alongside an ingest key it travels in
`X-Cotel-Actor-Token`. Agents keep it in the env var `COTEL_ACTOR_TOKEN`,
which Paperclip's stripping does not touch.

### How ingest decides

| Request | Result |
|---|---|
| `Authorization: Bearer <ingest key>` + `X-Cotel-Actor-Token: <valid actor token>` | Accepted, attributed to that actor |
| Valid ingest key, actor token missing, empty or unknown | Accepted as **Anonymous**, even with allow-anonymous off |
| Invalid `cotel_ik_` key | `401` |
| `Authorization: Bearer <actor token>` (legacy) | Accepted, attributed to that actor; `X-Cotel-Actor-Token` is ignored |
| Any other unknown `cotel_` bearer | `401` |
| No bearer (a bearer not starting with `cotel_` counts as none) | Allow-anonymous decides, as before; `X-Cotel-Actor-Token` alone never identifies |

A request with a valid ingest key and no usable actor token is stored as
Anonymous even when allow-anonymous is off. The toggle exists to keep out
requests nobody authorized; this one was authorized, and refusing it would drop
real telemetry over a missing label. The actor token alone is never enough:
it is an identity, not a credential, so `X-Cotel-Actor-Token` without a bearer
falls through to allow-anonymous like any request without one.

### Storage and scope

- Schema v12 adds `ingest_keys (id, name, token UNIQUE, created_at)`. The key
is stored in **plaintext** and can be copied from the dashboard at any time,
exactly like an actor token. A first draft stored only a hash and showed the
key once; the owner rejected that on review: a key that cannot be copied again
into the next sender's configuration is a chore, and the database file already
holds every actor token in clear, so hashing one more secret buys nothing.
Revoking a key deletes its row.
- An ingest key authorizes **OTLP ingest only** (`/v1/traces`, `/v1/logs`) and
is refused on export and import. It is not a dashboard credential: the rest of
the dashboard API has no authentication of its own in cotel and stays behind
Cloudflare Access (or whatever restricts the dashboard), as before.
- Management is in the dashboard API: `GET`, `POST /api/v1/ingest-keys` and
`DELETE /api/v1/ingest-keys/{id}`.
- `POST /v1/logs` exists only so Paperclip's log exporter does not get `404`. It
accepts the body and discards it (`200 {}`). cotel stores spans, not logs.

### Getting the actor header out of Claude Code

Claude Code does not turn arbitrary env vars into headers. It does run an
`otelHeadersHelper` from `settings.json`: a command that prints a JSON object of
HTTP headers, run at startup and every 29 minutes
(`CLAUDE_CODE_OTEL_HEADERS_HELPER_DEBOUNCE_MS`), for http/protobuf and
http/json only. If the helper fails or prints invalid JSON, exports fail until it
works again.

`scripts/otel-headers-helper.py` is that helper. It rebuilds the headers from
`OTEL_EXPORTER_OTLP_HEADERS` overlaid with `OTEL_EXPORTER_OTLP_TRACES_HEADERS`,
adds `X-Cotel-Actor-Token` from `COTEL_ACTOR_TOKEN`, and always prints one JSON
object and exits 0. It emits the **complete** header set, not just the actor
header, because how Claude Code merges the helper's output with the static
`OTEL_EXPORTER_OTLP_*HEADERS` vars is not documented: if the helper's output
replaced them, a helper that printed only `X-Cotel-Actor-Token` would silently
drop the ingest key and every export would get `401`.

## Alternatives considered

- **Patch our Paperclip fork to stop stripping the env.** Rejected: it fights
upstream behaviour on purpose, and the patch would have to survive the weekly
rebase indefinitely.
- **Map the `paperclip.agent.id` resource attribute to actors.** Viable, but
Paperclip-only and a second identity scheme to keep in sync with the actors
table. It may be added later as a fallback for requests without an actor
token.
- **One shared actor for all agents.** Works today with no code change, and
loses per-agent attribution, which is what ADR-0008 exists for.

## Consequences

- **The ingest key is readable by agent code.** It sits in the agent's env, so
anything the agent runs can read it. That is why its scope is ingest-only: the
worst a leaked key does is write spans.
- **Revoking a key cuts every sender using it** at once. Rotating a shared key
means creating the new one, updating the shared configuration, then revoking
the old one.
- **Unknown actor tokens land in Anonymous silently.** A typo in
`COTEL_ACTOR_TOKEN`, a rotated token or a deleted actor gives no error to the
sender; its spans appear under Anonymous. Check the Actors page after wiring a
new sender.
- **No last-used timestamp.** Ingest does not write to the database per request
to record key use, so there is no way to see which keys are still in use other
than revoking one and watching what stops.
- Senders using a bearer actor token directly keep working unchanged.
1 change: 1 addition & 0 deletions docs/decisions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially.
| [ADR-0025](./0025-best-effort-ingest-watcher-in-actions) | cotel is watched by a best-effort ingest probe in GitHub Actions | Accepted |
| [ADR-0026](./0026-scheduled-loopback-probe-with-a-pager) | The deploy host asks `/healthz` on a schedule, and it pages | Accepted |
| [ADR-0027](./0027-users-become-actors) | Users become actors, down to the schema | Accepted |
| [ADR-0028](./0028-ingest-keys-and-actor-tokens) | Ingest keys authorize, actor tokens identify | Accepted |
Loading
Loading