Skip to content

feat: logs and logs --follow with WebSocket reconnect and gap-fill - #13

Open
RogerCecilpot wants to merge 1 commit into
feat/244-config-contextsfrom
feat/254-logs
Open

RogerCecilpot wants to merge 1 commit into
feat/244-config-contextsfrom
feat/254-logs

Conversation

@RogerCecilpot

Copy link
Copy Markdown

Closes devlyft/devlyft#254.

Stacked on #11 (#250). It will be restacked onto #244 (config and contexts) once that PR exists.

What

  • devlyft logs <deploy-id> prints recent lines from GET /v1/deploys/{id}/logs, oldest first. --follow / -f then streams new lines until Ctrl-C (exit 130).
    • Flags: --since (a Go duration; server default 1h), --pod, --filter, --match (repeatable), --limit (at most 5000), --timestamps.
    • Output: raw lines by default; TSV is timestamp\tpod\tline; --format json prints NDJSON, one LogEntry per line; YAML prints one document per line; --jq applies to each line; --quiet is a usage error.
    • Under JSON or --jq, reconnect notices go to the debug log, so stderr only ever carries the error object.
  • internal/api/logstream is the hand-written live-tail client:
    • Dial uses github.com/coder/websocket. The token goes in the bearer subprotocol, never the URL; the handshake is debug-logged through redact. Rejected handshakes become clierr errors: 401 and 403 exit 3, 404 exits 4, and none are retried.
    • Follow is the reconnect loop, independent of the WebSocket library:
      • A 4001 close reconnects at once with a fresh token.
      • Drops and failed dials back off exponentially with full jitter (0.5s to 30s) and give up after 8 in a row.
      • After every connect it fills the gap from REST history and de-duplicates by timestamp, pod and line, since neither source has IDs.
      • Memory for de-duplication is capped at about 10k remembered lines.
  • Facade: api.Client.FollowLogs uses the same *http.Client, token source, user agent and logger as REST calls, so #244's proxy and CA bundle settings will apply to the WebSocket with no extra wiring. The gomock mock is regenerated.
  • Output: output.Printer.PrintRecord prints streamed records: NDJSON, YAML documents, --jq per record, TSV, or text.
  • Docs: docs/logs.md documents the flags, the client contract and the gap-fill algorithm; the README has a short Logs section.

WebSocket contract (from logs-api's stream handler)

  • GET /v1/deploys/{deployId}/logs/stream, with query parameters pod, filter and match (repeatable), the same filters as REST.
  • The token is sent as Sec-WebSocket-Protocol: bearer, <token>, and the server selects bearer. coder/websocket sends bearer,<token>, which the server accepts because it splits on commas and trims.
  • Before upgrading, the server can answer with a plain-text 400, 401, 403 or 404.
  • Each text frame is one entry, {"timestamp","line","pod"?}: the same shape as the REST LogEntry.
  • The stream has no resume or since: the tail starts at the moment of connection.
  • The server closes with 4001 reauthenticate after at most 30 minutes. It also sends 4001 whenever its tail ends, including on failures.

Decisions

  • since on REST is a duration, not a timestamp: the spec says it's a Go duration such as 1h. The gap-fill asks for since = now − newest line shown + 5s of slack, rounded up to whole seconds, with limit=5000. If the page comes back full, a notice says some lines may be missing.
  • Fast 4001s are backed off: a 4001 within 10 seconds of connecting counts as a failure, because the server sends 4001 when its tail fails too. Otherwise a broken tail would make the client reconnect in a tight loop.
  • The gap-fill runs after connecting, not before: nothing logged between the two is lost, and the overlap is de-duplicated.
  • IDs only: the deploy argument must be a UUID, and a name is a usage error. Name resolution comes with #244.
  • Library choice: github.com/coder/websocket rather than gorilla. It's context-aware, actively maintained, and takes an *http.Client for the handshake, which is what makes the proxy and CA settings apply.

Tests

  • logstream.Follow, against fakes:
    • reauth with gap-fill and no duplicates
    • early 4001 backed off
    • backoff growth and cap
    • permanent 401/403/404 not retried; 503 retried
    • seen entries skipped, and the same line from different pods kept
    • gap-fill errors and truncated pages warn
    • emit errors stop the loop
    • cancellation, including during backoff
    • pruning keeps memory bounded
  • logstream.Dial, against a real WebSocket test server: the subprotocol token, the token never in the URL, path, query and user agent, debug logs without the token, 4001 mapped to ErrReauth, other closes, binary frames skipped, malformed frames, every rejected-handshake status and its exit code, network and token-source errors, and URL scheme mapping.
  • Facade end to end: a real WebSocket plus the contract-checking REST fake (apitest). The first stream sends a line and closes 4001 immediately. The client reconnects with a different token, and both gap-fills are validated against the vendored spec (since in seconds, limit=5000). Every line is emitted exactly once, in order.
  • Command goldens: every output format, --timestamps, filters passed through, follow (text and JSON), interrupted (exit 130), stream 403 (exit 3), 404 (exit 4), and usage errors (exit 2).
  • Verified: make lint reports 0 issues, and make test and make vuln pass. make cover-check reports 87.6% total (logstream 96.7%, the logs command 96.2%). The Windows and macOS cross-builds and actionlint are clean.

Not done here

  • Name resolution for the deploy argument comes with #244.
  • Platform spec gap: the WebSocket isn't in the OpenAPI spec yet (devlyft/devlyft#234). docs/logs.md holds the client-side contract until it is.
  • Proxy spike: the behaviour through corporate proxies and TLS interception is the #262 spike, which this unblocks.

🤖 Generated with Claude Code

`devlyft logs <deploy-id>` prints recent lines from the REST endpoint;
--follow then streams new lines from logs-api's live-tail WebSocket until
interrupted. Flags: --since, --pod, --filter, --match, --limit,
--timestamps. --format json prints NDJSON and --jq applies per line.

internal/api/logstream holds the hand-written client:
- Dial authenticates with the token in the bearer subprotocol (never the
  URL), maps rejected handshakes to clierr (401/403 exit 3, 404 exit 4)
  and logs the handshake through redaction.
- Follow reconnects after the server's 4001 re-auth close with a fresh
  token, backs off drops and failed dials with full jitter, and after
  every connect fills the gap from REST `since` (a duration on this API),
  de-duplicating by timestamp, pod and line. A 4001 that arrives at once
  is backed off, since the server also sends it when its tail fails.

The facade gains FollowLogs, built on the same http.Client, token source
and logger as REST calls so proxy and CA settings apply to the stream.
output.Printer gains PrintRecord for streamed records.

Refs devlyft/devlyft#254

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RogerCecilpot
RogerCecilpot changed the base branch from feat/250-test-harness to feat/244-config-contexts October 25, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant