Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- **Paperclip agents are attributed by `paperclip.agent.id`** ([ADR-0029](docs/decisions/0029-paperclip-agents-by-agent-id.md)). Each actor gets an optional Paperclip agent id (schema v13; the actor page, or `PATCH /api/v1/actors/{id}` with `{"paperclip_agent_id"}`), and a span authorized by an ingest key with no actor token is attributed to the actor whose id matches the `paperclip.agent.id` Paperclip puts on every agent span - from the resource attributes, else the span's. Paperclip's company observability then needs only the ingest key, and agents on remote machines are covered too. The ADR-0028 helper route could not reach agents: Paperclip runs them through the Claude Agent SDK with `--setting-sources=project,local`, so they never read the user `settings.json` the helper is registered in, and putting the helper in managed settings makes Claude Code drop the endpoint Paperclip sets. The attribute is honoured only behind an ingest key, so an unauthenticated sender cannot claim an agent; an actor token still wins; lookups are memoized per request
- **Ingest keys: one shared key authorizes, each sender's actor token identifies** ([ADR-0028](docs/decisions/0028-ingest-keys-and-actor-tokens.md)). Paperclip now strips every `OTEL_*` and `CLAUDE_CODE_*TELEMETRY*` variable from an agent's env and applies one company-wide set of OTLP headers, so a per-agent `Authorization: Bearer <actor token>` can no longer be configured, and with that setting empty it turns telemetry off, which is why agent telemetry stopped on 2026-11-07. An ingest key (`cotel_ik_<64 hex>`, created, copied and revoked under Setup → Settings → Ingest keys, stored in plaintext like actor tokens) authorizes OTLP ingest and nothing else, never export/import or the dashboard; the existing actor token, unchanged and not reissued, rides next to it in `X-Cotel-Actor-Token`. A valid key with a missing or unknown actor token is accepted as Anonymous even when allow-anonymous is off, because the request was authorized; `X-Cotel-Actor-Token` without a bearer never identifies anyone, and a bearer actor token works exactly as before. Schema v12 adds `ingest_keys`; the dashboard API gains `GET`/`POST /api/v1/ingest-keys` and `DELETE /api/v1/ingest-keys/{id}`. `POST /v1/logs` accepts and discards OTLP logs (`200 {}`), so Paperclip's log exporter stops getting 404. `scripts/otel-headers-helper.py` is a Claude Code `otelHeadersHelper` that prints the complete header set — `OTEL_EXPORTER_OTLP_HEADERS` overlaid with `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, plus `X-Cotel-Actor-Token` from `COTEL_ACTOR_TOKEN` — and always exits 0 with one JSON object, since a failing helper stops Claude Code's exports. Docs → [Actors and Authentication](docs/operations/actors-and-auth.md#ingest-keys-one-key-for-many-senders)
- `cotel --backfill-cost` (dry run) and `--backfill-cost-apply` recompute `cost_usd` for every span of a priced model from its stored token counts at today's rates, revived from the closed PR #67 as reviewed there (FLO-551/552). Spans with an unknown or empty model are counted and left alone; `daily_usage` is not touched, because a day rolled up after the backfill derives its cost from the corrected spans. The apply needs the database to itself, so it runs while the server is stopped; the dry run restores the newest snapshot into a throwaway file instead of copying a live one. README → Maintenance commands has both procedures for the current compose deploy, with `--entrypoint` set so a one-off does not start a second Cloudflare tunnel connector
- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2 — without `user_id`, and *with* the mid-table `duration_ms` generated column, so that the one ALTER production ran on a long-lived filled table (v9 → v10, six weeks before the abort) runs here too instead of passing over a column the fixture never had, and so the indexes are built over the logical/physical column skew they were built over in production — with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts that `duration_ms` was there before the migration and is gone after it (a `DROP COLUMN IF EXISTS` that no-ops must not be able to come back unnoticed), that the reopen succeeds, that both checkpoints fold — a present-but-empty `.wal` counts as folded, so an engine that truncates rather than unlinks is not reported as this defect — that all four indexes survive, that a bare `user_id = <constant>` folded into the scan agrees with the same predicate wrapped in an expression, which is not, and that re-inserting a stored `span_id` is still rejected while an unused one is accepted. The last two are the silent form of the damage: a wrong answer rather than an error. They are phrased that way because no `SELECT` over `spans` is planned as an ART index scan on this engine — measured, including on the primary key and on a textbook unique match with the thresholds raised — so an “indexed probe agrees with an unindexed probe” assertion would have been two sequential scans agreeing; the pushdown pair is what [ADR-0013](docs/decisions/0013-spans-has-no-derived-columns.md)'s wrong answers actually came out of, and unique-constraint enforcement is the one path that provably traverses an ART index. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.1 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))
Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ export default defineConfig({
{ text: 'ADR-0026 - The Deploy Host Asks /healthz on a Schedule', link: '/decisions/0026-scheduled-loopback-probe-with-a-pager' },
{ text: 'ADR-0027 - Users Become Actors', link: '/decisions/0027-users-become-actors' },
{ text: 'ADR-0028 - Ingest Keys Authorize, Actor Tokens Identify', link: '/decisions/0028-ingest-keys-and-actor-tokens' },
{ text: 'ADR-0029 - Paperclip Agents by paperclip.agent.id', link: '/decisions/0029-paperclip-agents-by-agent-id' },
],
},
],
Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/0028-ingest-keys-and-actor-tokens.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# ADR 0028 — Ingest keys authorize, actor tokens identify

**Date:** 2026-11-15
**Status:** Accepted
**Status:** Accepted; amended by [ADR-0029](./0029-paperclip-agents-by-agent-id.md) for Paperclip agents
**Deciders:** Flop (board)

---
Expand Down
75 changes: 75 additions & 0 deletions docs/decisions/0029-paperclip-agents-by-agent-id.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# ADR 0029 — Paperclip agents are identified by `paperclip.agent.id`

**Date:** 2026-11-15
**Status:** Accepted
**Deciders:** Flop (board)
**Amends:** [ADR-0028](./0028-ingest-keys-and-actor-tokens.md) for Paperclip agents

---

## Context

ADR-0028 split authorization from identity: a shared ingest key authorizes, and
each sender's actor token, sent as `X-Cotel-Actor-Token`, identifies it. For
Paperclip agents the token was meant to travel as the env var
`COTEL_ACTOR_TOKEN`, turned into a header by Claude Code's `otelHeadersHelper`.
On production that did not work, for reasons on Paperclip's and Claude Code's
side that cotel cannot change:

- **Paperclip runs agents through the Claude Agent SDK with
`--setting-sources=project,local`**, so `~/.claude/settings.json` - where the
helper was registered - is never read by an agent. The agent's process did
carry the right `COTEL_ACTOR_TOKEN`; nothing sent it.
- **Registering the helper in managed settings breaks export.** When
`otelHeadersHelper` comes from managed settings, Claude Code drops
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` from every lower-trust source ("lower-trust
scopes cannot redirect the traces signal"), which is where Paperclip puts the
endpoint. Traces from the whole host stopped until the file was removed.
- **Paperclip header values are literal.** The company destination's headers are
one secret for every agent; `$COTEL_ACTOR_TOKEN` is sent as that literal text.
- **Agents cannot set their own env.** Since Paperclip #15283 an agent key gets
`403` on any env change to a `*_local` agent, its own included; only the board
can.
- The feature covers Claude Code and Codex only; per-host Claude Code settings
would never have reached Codex or the other harnesses anyway.

What Paperclip does provide is identity: it sets `OTEL_RESOURCE_ATTRIBUTES` for
every agent run, so each span carries `paperclip.agent.id` (the agent's UUID).

## Decision

**cotel maps Paperclip agent ids to actors.** Each actor has an optional
`paperclip_agent_id` (schema v13, set on the actor page or with
`PATCH /api/v1/actors/{id}`). On ingest:

1. An actor token - `Authorization: Bearer <actor token>`, or
`X-Cotel-Actor-Token` next to an ingest key - names the actor, as before.
2. Otherwise, **if and only if an ingest key authorized the request**, the span
is attributed to the actor whose `paperclip_agent_id` equals the span's
`paperclip.agent.id` - read from the resource attributes, falling back to the
span's own attributes.
3. Otherwise the span is Anonymous.

The attribute is honoured only behind an ingest key: an unauthenticated sender
(with `allow_anonymous` on) could otherwise claim any agent's identity by
setting an attribute. An agent maps to at most one actor; assigning an id that
another actor holds moves it. There is no `UNIQUE` constraint on the column
(DuckDB has raised false duplicate-key errors on updates of such rows); storage
keeps one actor per id in a transaction. Lookups are memoized per request, so a
batch from one agent costs one query.

ADR-0028 is unchanged for everything else: ingest keys, actor tokens, the header
and the helper still serve people and machines whose Claude Code reads user
settings (a developer's laptop, a host's interactive shell).

## Consequences

- Paperclip's company observability needs only the ingest key in
`Authorization`. No helper, managed settings or per-agent env is needed on the
Paperclip host, and remote agents (another machine) are covered the same way.
- Each agent's id is entered once per actor. A new Paperclip agent stays
Anonymous until its actor is mapped - visible on the Actors page.
- The `COTEL_ACTOR_TOKEN` already written into the agents' Paperclip env is
harmless and unused.
- Spans ingested before an agent was mapped stay Anonymous; the mapping is not
applied retroactively.
1 change: 1 addition & 0 deletions docs/decisions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially.
| [ADR-0026](./0026-scheduled-loopback-probe-with-a-pager) | The deploy host asks `/healthz` on a schedule, and it pages | Accepted |
| [ADR-0027](./0027-users-become-actors) | Users become actors, down to the schema | Accepted |
| [ADR-0028](./0028-ingest-keys-and-actor-tokens) | Ingest keys authorize, actor tokens identify | Accepted |
| [ADR-0029](./0029-paperclip-agents-by-agent-id) | Paperclip agents are identified by `paperclip.agent.id` | Accepted |
16 changes: 14 additions & 2 deletions docs/operations/actors-and-auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ There is no last-used time: ingest does not write to the database to record whic
| Request | Result |
|---|---|
| `Authorization: Bearer <ingest key>` + `X-Cotel-Actor-Token: <valid actor token>` | Accepted, attributed to that actor |
| Valid ingest key, actor token missing, empty or unknown | Accepted as **Anonymous** — even with allow-anonymous off, because the key authorized the request |
| Valid ingest key, no valid actor token, span carries `paperclip.agent.id` mapped to an actor | Accepted, attributed to that actor ([Paperclip agents](#paperclip-agents)) |
| Valid ingest key, no valid actor token and no mapped `paperclip.agent.id` | Accepted as **Anonymous** — even with allow-anonymous off, because the key authorized the request |
| Unknown `cotel_ik_` key | `401` |
| `Authorization: Bearer <actor token>` | Accepted, attributed to that actor; `X-Cotel-Actor-Token` is ignored |
| Any other unknown `cotel_` bearer | `401` |
Expand All @@ -126,9 +127,20 @@ A wrong actor token is not an error. A typo, a rotated token or a deleted actor

`POST /v1/logs` accepts the same credentials and discards the body (`200 {}`). It exists so that a sender exporting logs as well as traces, as Paperclip does, does not get `404`; cotel stores spans only.

### Paperclip agents

Paperclip sends every agent's telemetry with one company-wide header set, so its agents cannot each send an actor token. Instead, Paperclip tags every span with `paperclip.agent.id`, and cotel maps that id to an actor ([ADR-0029](../decisions/0029-paperclip-agents-by-agent-id.md)):

1. In Paperclip, **Company Settings → General → Agent observability**: destination **Custom OTLP**, base URL of your ingest endpoint (e.g. `https://otlp.example.com`), one header `Authorization` = `Bearer cotel_ik_<key>`, **Enable export** on. Leave the content-capture switches off unless you want prompts and tool output stored in span attributes; **Additional tool details** is useful (it gives the Tools page command detail).
2. In cotel, open each agent's actor page and fill in **Paperclip agent** with the agent's UUID (from its Paperclip page URL), or `PATCH /api/v1/actors/{id}` with `{"paperclip_agent_id": "<uuid>"}`.

That is all: no helper, managed settings or per-agent env is needed on the Paperclip host, and agents on remote machines are covered the same way. The id is honoured only when an ingest key authorized the request, so an unauthenticated sender cannot claim an agent by setting the attribute. An actor token still wins when one is sent. A new agent shows up as Anonymous until its actor is mapped.

The helper approach below does **not** work for Paperclip agents: Paperclip starts them through the Claude Agent SDK with `--setting-sources=project,local`, so they never read the user `settings.json` the helper is registered in, and registering it in managed settings makes Claude Code drop the endpoint Paperclip sets. ADR-0029 has the details.

### Configuring a sender

Each sender needs three things:
For a person's or a host's own Claude Code, which reads the user `settings.json`. Each sender needs three things:

1. **The ingest key** in its OTLP headers: `Authorization=Bearer cotel_ik_<key>` in `OTEL_EXPORTER_OTLP_HEADERS` or `OTEL_EXPORTER_OTLP_TRACES_HEADERS`. With Paperclip this is the company-wide header set; Paperclip puts it in the per-signal header vars.
2. **Its own actor token** in the env var `COTEL_ACTOR_TOKEN`, in that sender's own process env (for a Paperclip agent, its adapter env). Do not put it in the `env` block of a shared `settings.json`: that block applies to every launch and overrides the inherited env, so every sender would claim the same actor.
Expand Down
14 changes: 13 additions & 1 deletion docs/operations/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,19 @@ delete are still accepted as aliases; response bodies use the new names only
(`actors`, `actor_id`, `actors_count`).

Its rows are returned under `actors` rather than `items`. `GET /actors/{id}`
returns one actor in the same shape and accepts `range`. See
returns one actor in the same shape and accepts `range`. Each actor carries
`paperclip_agent_id` (a UUID or `null`): the Paperclip agent whose spans are
attributed to it ([ADR-0029](../decisions/0029-paperclip-agents-by-agent-id.md)).

### `PATCH /actors/{id}`

Sets or clears the actor's Paperclip agent mapping, the only writable field.
Body: `{"paperclip_agent_id": "<uuid>"}`, or `""` to clear. The value is trimmed
and lower-cased; anything but a UUID or `""` is `400`. Assigning an id another
actor holds moves it to this one. Answers `200` with the actor, as `GET` would;
`404` for an unknown or deleted actor and for `__anonymous__`.

See
[Actors and Authentication](./actors-and-auth.md) for the management endpoints
(create, rotate, delete).

Expand Down
15 changes: 15 additions & 0 deletions frontend/src/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,9 @@ export interface Actor {
cost: number
sessions: number
last_seen: string | null
// Paperclip agent UUID mapped to this actor: spans sent with an ingest key and
// no actor token are attributed here when their paperclip.agent.id matches.
paperclip_agent_id: string | null
}

export interface ActorsListResponse {
Expand Down Expand Up @@ -303,6 +306,18 @@ export async function rotateActorToken(id: string): Promise<Actor> {
return res.json()
}

// setActorPaperclipAgentId maps a Paperclip agent UUID to the actor; an empty
// string clears it. Setting an id another actor has moves it to this actor.
export async function setActorPaperclipAgentId(id: string, paperclipAgentId: string): Promise<Actor> {
const res = await fetch(`/api/v1/actors/${encodeURIComponent(id)}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ paperclip_agent_id: paperclipAgentId }),
})
if (!res.ok) throw await responseError(res)
return res.json()
}

export type DeleteMode = 'actor_only' | 'actor_and_history'

export async function deleteActor(id: string, mode?: DeleteMode): Promise<void> {
Expand Down
Loading
Loading