Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))

### Fixed
- A `paperclip.agent.id` sent in upper or mixed case now finds its actor. The dashboard stores the id lowercased and the ingest lookup compared it as sent, so such spans stayed on Anonymous.
- **Codex and Grok CLI agents get sessions, models, tokens and cost.** Paperclip's Codex (`codex-app-server`) and Grok (`grok-cli`) agents were exporting spans that ingest stored with no session, so they never reached the Sessions page, and with no tokens or model, so they cost $0. Neither harness uses Claude Code's attribute names, and both emit several spans per event. `internal/ingest/producers.go` adds an adapter per harness, chosen by `service.name` and applied only to those spans. Claude Code spans are untouched. For Codex, the session comes from `conversation.id`, or from `thread.id` when it is a UUID (on other spans it is the OS thread number). Usage comes from the turn span's `codex.turn.token_usage.*`, with cached input split out. The per-request `gen_ai.usage.*` copies are not read, since they sum to the turn. A tool call is a span with both `tool_name` and `call_id`. For Grok, the session comes from `session_id` and the model from `model_id`. Tokens are kept only on `session.process_conversation_turn`, with `cache_read_tokens` taken out of its inclusive `input_tokens`, because `sampling.stream_decode` repeats the output count. Only `tool.execution` counts as a tool call, not `tool.decision`. Prices are added for `gpt-5.6-sol` ($4 input / $0.40 cached / $20 output) and `grok-4.7` / `grok-4.6` ($2 / $0.50 / $6), at base rates, taken from third-party rate cards that cite the providers. Grok still lands on Anonymous: it does not carry `paperclip.agent.id` (ADR-0029). Cursor exports no telemetry at all. Already-stored spans are not rewritten, because `session_id` is an indexed column (see the ART incident).
- Actor and overview session counts no longer count a span stored without a session id as a session. An actor whose only spans were sessionless showed one session on Actors and none on Sessions.
- Spans from the Claude 5.5 / 5.2 generation are priced. From 2026-11-07 every span from `claude-opus-5-5`, `claude-fable-5-2` and `claude-haiku-5-5` was stored with `cost_usd` unset because the price table did not know them, so the dashboard showed $0 for a week of real traffic. Adds Opus 5.5 ($4/$20), Sonnet 5.5 ($2/$10), Haiku 5.5 ($0.10/$0.50; its $0.50/$2.50 rate for prompts over 100K tokens is not modelled, since a span does not say which tier billed it), Fable and Mythos 5.2 ($8/$40) and 5.1 ($10/$50), with the published cache-read rates where they depart from 0.1 x input. Sonnet 5 moves from $3/$15 to the $2/$10 Anthropic still publishes. `pricing.Known` tells an unpriced model from a priced span that costs zero. Already-stored spans keep their unset cost until `--backfill-cost-apply` runs
- The Deploy workflow stops asking the production host's GUI keychain for credentials it does not hold, and stops being able to hold the deploy queue indefinitely when something does hang. `docker compose build --pull` makes buildkit resolve the Dockerfile frontend and the base images, which makes the docker CLI consult the credential store named in the host's `~/.docker/config.json`. On the production runner that store is `desktop`, whose helper is tied to the logged-in GUI session: spawned from the runner's launchd session it blocks in `open(2)` and never returns, buildkit re-spawns it on every retry, and a single hung call also stalls image resolution for any other build on that host (measured: a hand build from an interactive shell, normally 2.6 s, sat 4 min behind one). Nothing in the run log said so, because the hang lands before the first layer and the step's last line is the frontend resolve. The same helper call from an interactive shell on the same host answers in milliseconds, so both the hosted-runner smoke build and a hand build on the deploy host passed throughout. The job now builds against a copy of the config carrying the host's contexts and CLI plugins but no credential store at all, which is faithful to what the host actually has: its `auths` map is empty, and registry auth, if it is ever needed there, belongs in an explicit `docker login` with a repository secret rather than in one person's keychain. The deploy job also gains `timeout-minutes: 20`. The first wedge ran for nine hours and, because `deploy-main` has `cancel-in-progress: false`, every push behind it sat `pending` with no job at all, which is indistinguishable from an idle queue: production kept serving a build five days old, the next runtime fix would silently not have shipped, and no run ever went red to say so. A hung build now fails loudly and releases the group instead.

Expand Down
6 changes: 3 additions & 3 deletions internal/api/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -551,7 +551,7 @@ func (h *Handler) handleOverview(w http.ResponseWriter, r *http.Request) {
// it; it is one principal on the actors list and counts as one here too.
if err := h.db.QueryRow(cte+`
SELECT
COUNT(DISTINCT session_id),
COUNT(DISTINCT NULLIF(session_id, '')),
COUNT(DISTINCT actor_id) + CASE WHEN COUNT(*) FILTER (WHERE actor_id IS NULL) > 0 THEN 1 ELSE 0 END,
COALESCE(SUM(cost), 0),
COALESCE(SUM(input_tokens), 0),
Expand Down Expand Up @@ -1685,7 +1685,7 @@ func (h *Handler) historyRawSeries(resp *historyResponse, r *http.Request, gran
brows, err := h.db.Query(fmt.Sprintf(`
SELECT
%s AS bucket,
COUNT(DISTINCT session_id) AS sessions,
COUNT(DISTINCT NULLIF(session_id, '')) AS sessions,
COUNT(*) AS spans,
COALESCE(SUM(cost_usd), 0) AS cost_usd,
COALESCE(SUM(input_tokens), 0) AS input_tokens,
Expand Down Expand Up @@ -1744,7 +1744,7 @@ func (h *Handler) historyUnionSeries(resp *historyResponse, r *http.Request, gra
brows, err := h.db.Query(cte+fmt.Sprintf(`
SELECT
%s AS bucket,
COUNT(DISTINCT session_id) AS sessions,
COUNT(DISTINCT NULLIF(session_id, '')) AS sessions,
CAST(SUM(spans) AS BIGINT) AS spans,
COALESCE(SUM(cost), 0) AS cost_usd,
COALESCE(SUM(input_tokens), 0) AS input_tokens,
Expand Down
5 changes: 5 additions & 0 deletions internal/ingest/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"io"
"log"
"net/http"
"strings"
"time"

"google.golang.org/protobuf/encoding/protojson"
Expand Down Expand Up @@ -136,6 +137,8 @@ func (h *Handler) agentResolver(enabled bool) func(agentID string) string {
}
seen := map[string]string{}
return func(agentID string) string {
// The dashboard stores the id lowercased; match a sender's casing to it.
agentID = strings.ToLower(strings.TrimSpace(agentID))
if agentID == "" {
return ""
}
Expand Down Expand Up @@ -225,6 +228,8 @@ func decodeSpan(sp *tracepb.Span, svcName string, resAttrs map[string]any, resJS
s.CostUSD = &f
}

normalizeProducer(svcName, sp.Name, attrs, &s)

// Derive cost from token counts when the producer did not supply cost_usd.
// Claude Code beta telemetry omits the field; we compute it ingest-time so
// the stored row is always queryable without a per-query price JOIN.
Expand Down
2 changes: 2 additions & 0 deletions internal/ingest/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"

"github.com/Flopsstuff/cotel/internal/ingest"
Expand Down Expand Up @@ -385,6 +386,7 @@ func TestPaperclipAgentAttribution(t *testing.T) {
{"ingest key, span attr only", true, "", "", daedalus, "Daedalus", 1},
{"resource attr beats span attr", true, "", wayland, daedalus, "Wayland", 1},
{"unknown agent stays anonymous", true, "", "11111111-1111-1111-1111-111111111111", "", "", 1},
{"agent id matched case-insensitively", true, "", strings.ToUpper(wayland), "", "Wayland", 1},
{"actor token wins", true, "flop", wayland, "", "flop", 0},
{"no ingest key: attribute ignored", false, "", wayland, "", "", 0},
}
Expand Down
137 changes: 137 additions & 0 deletions internal/ingest/producers.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
package ingest

import (
"fmt"
"strings"

"github.com/Flopsstuff/cotel/internal/storage"
)

// Producer adapters map the span attributes of harnesses other than Claude Code
// onto the columns decodeSpan fills from Claude Code's names. Each one was
// written against what the harness actually exports through Paperclip, not
// against a spec: neither harness follows a published convention for sessions
// or usage, and both emit several spans per logical event, so the rules below
// also choose which span carries a count, to keep sums from double counting.
//
// They run after the generic extraction and before cost is derived, and only
// for spans whose service.name names the harness. Claude Code spans are never
// touched.

// normalizeProducer applies the adapter for svcName, if there is one.
func normalizeProducer(svcName, spanName string, attrs map[string]any, s *storage.Span) {
svc := strings.ToLower(svcName)
switch {
case strings.HasPrefix(svc, "codex"):
normalizeCodex(attrs, s)
case strings.HasPrefix(svc, "grok"):
normalizeGrok(spanName, attrs, s)
}
}

// Codex (service.name "codex-app-server", Rust tracing over OTLP).
//
// - Session: conversation.id on exec spans; on the turn span and the tool
// handler spans the same id travels as thread.id. thread.id is also the
// tracing layer's OS thread number (an integer) on every other span, so it
// counts only when it is a UUID string.
// - Usage: the turn span (session_task.turn) carries the turn's totals under
// codex.turn.token_usage.*, next to model. The per-request handle_responses
// spans repeat the same tokens as gen_ai.usage.* without a model; they sum
// to the turn total, so they are not read. input_tokens includes the cached
// part; non_cached_input_tokens is what bills at the input rate.
// - Tools: a tool invocation is the span with both tool_name and call_id
// (exec, exec_command). code_mode.broker.invoke_tool names the same nested
// call again without a call_id and is not counted.
func normalizeCodex(attrs map[string]any, s *storage.Span) {
if s.SessionID == "" {
if v := str(attrs["conversation.id"]); v != "" {
s.SessionID = v
} else if v, ok := attrs["thread.id"].(string); ok && isUUID(v) {
s.SessionID = v
}
}

const p = "codex.turn.token_usage."
if _, ok := attrs[p+"input_tokens"]; ok {
total := toInt64(attrs[p+"input_tokens"])
cached := toInt64(attrs[p+"cached_input_tokens"])
in := total - cached
if v, ok := attrs[p+"non_cached_input_tokens"]; ok {
in = toInt64(v)
}
s.InputTokens = ptr(max(in, 0))
s.CacheReadTokens = ptr(cached)
s.CacheWriteTokens = ptr(toInt64(attrs[p+"cache_write_input_tokens"]))
s.OutputTokens = ptr(toInt64(attrs[p+"output_tokens"]))
}

if s.ToolName != "" && str(attrs["call_id"]) == "" {
s.ToolName = ""
}
}

// Grok CLI (service.name "grok-cli").
//
// - Session: session_id; model: model_id.
// - Usage: session.process_conversation_turn carries the turn's totals.
// sampling.stream_decode repeats output_tokens per sample and would double
// the turn's output, so token counts are kept on the turn span only.
// input_tokens includes cache_read_tokens (OpenAI-style accounting), so the
// cached part is taken out of the input column.
// - Tools: tool.execution is the call; tool.decision is the permission check
// before it and names the same tool.
func normalizeGrok(spanName string, attrs map[string]any, s *storage.Span) {
if s.SessionID == "" {
s.SessionID = str(attrs["session_id"])
}
if s.Model == "" {
s.Model = str(attrs["model_id"])
}

if spanName == "session.process_conversation_turn" {
if s.InputTokens != nil && s.CacheReadTokens != nil {
s.InputTokens = ptr(max(*s.InputTokens-*s.CacheReadTokens, 0))
}
} else {
s.InputTokens, s.OutputTokens = nil, nil
s.CacheReadTokens, s.CacheWriteTokens = nil, nil
}

if spanName != "tool.execution" {
s.ToolName = ""
}
}

func str(v any) string {
if v == nil {
return ""
}
if s, ok := v.(string); ok {
return s
}
return fmt.Sprintf("%v", v)
}

func ptr(n int64) *int64 { return &n }

// isUUID reports whether s has the 8-4-4-4-12 hex shape of a UUID.
func isUUID(s string) bool {
if len(s) != 36 {
return false
}
for i := 0; i < len(s); i++ {
c := s[i]
switch i {
case 8, 13, 18, 23:
if c != '-' {
return false
}
default:
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f' || c >= 'A' && c <= 'F') {
return false
}
}
}
return true
}
Loading
Loading