Skip to content

fix(cli): stop telemetry hangs and errors on blocked networks#5880

Open
pamelachia wants to merge 9 commits into
developfrom
pamela/cli-telemetry-network-resilience
Open

fix(cli): stop telemetry hangs and errors on blocked networks#5880
pamelachia wants to merge 9 commits into
developfrom
pamela/cli-telemetry-network-resilience

Conversation

@pamelachia

@pamelachia pamelachia commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

On networks that block or blackhole eu.posthog.com (corporate proxies, CI and Docker egress policies, air-gapped machines), CLI telemetry currently breaks the CLI itself. Reproduced on stable v2.109.1 with the PostHog host pointed at an unreachable address: every command pays a 5s trailing hang and then exits 1 with {"code":"UnknownError","message":"Timeout while shutting down PostHog. Some events may not have been sent."}, even though the command's real work succeeded. Long-running commands additionally print raw SDK stack traces to stderr, and the Go binary's shutdown wait is unbounded. The only workaround is supabase telemetry disable, which permanently removes those users from CLI analytics. This PR makes telemetry strictly fire-and-forget: a failed delivery never changes a command's output, exit code, or runtime beyond a ~2s cap.

Changes

  • Go binary: set a 2s ShutdownTimeout on the PostHog client (the SDK default waits indefinitely for the final flush, including retries) and route SDK logs to the --debug logger instead of unconditional stderr.
  • TS shells: new shared scopedPosthogClient factory in shared/telemetry/posthog-client.ts, used by both analytics.layer.ts and legacy-analytics.layer.ts. It wraps fetch to report delivery failures as delivered: posthog-node has no logger hook and prints failures through hardcoded console.error calls, so making the transport infallible from the SDK's perspective is the one supported way to get no retries, no stderr output, and no shutdown stall.
  • One shared 2s budget (EXIT_DELAY_CAP_MS) bounds both each request (requestTimeout) and the entire shutdown: _shutdown receives the same value, which the SDK races against the whole drain (the in-flight request plus the queued flush loop). The previous 5s shutdown deadline left room for two sequential 2s request timeouts, so a command with one event in flight and a second event queued exited in ~4s instead of ~2s. Covered by a two-event blackhole regression test that holds the first request in flight and asserts total shutdown stays under the cap (~4.0s before, ~2.0s after).
  • Root cause of the exit-1 failure: the shutdown finalizer was Effect.promise(() => client._shutdown(5_000)).pipe(Effect.ignore), but Effect.promise turns a promise rejection into a defect and Effect.ignore only swallows typed failures, so a shutdown timeout failed the whole command. The rejection is now caught on the promise itself, so telemetry shutdown can never fail a command.

Context for review

  • flushAt: 1, flushInterval: 0 is kept deliberately. For a short-lived CLI process, eager flush overlaps the send with command execution; batching would move the network wait to process exit where the user feels it.
  • The exit bounds are caps, not floors: on healthy networks both shells return as soon as the flush completes. Fully detaching the send instead would race process exit and silently drop events on healthy networks, which is the common case.
  • The shutdown-deadline rejection is silent: its logger call in @posthog/core is debug-gated (only active after an explicit .debug() call, which the CLI never makes), and the ungated console.error path (logFlushError) is unreachable because the wrapped fetch never throws.
  • Verified against real unreachable endpoints. Refused connection: stable v2.109.1 runs telemetry status in 5s with exit 1 and the UnknownError payload; this branch runs it instantly with exit 0 and empty stderr. Blackholed (non-routable address): Go Close() returns in 2.0s (previously unbounded), TS capture plus shutdown completes in 2.0s with zero stderr bytes.
  • Accepted tradeoff: on working networks slower than 2s round-trip to PostHog, events drop (including a queued exit event when the drain exceeds the 2s deadline). Telemetry is best-effort.
  • Delivery failures remain observable under --debug on the Go side. On the TS side they are dropped silently.

Linear

  • fixes GROWTH-1001

@pamelachia
pamelachia requested a review from a team as a code owner July 15, 2026 06:03
@pamelachia pamelachia self-assigned this Jul 15, 2026
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@878576ba751d060a0c566956cc8df3e363c31d4a

Preview package for commit 878576b.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a96c6f5826

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e38cb95741

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli-go/internal/telemetry/client.go
@avallete

Copy link
Copy Markdown
Member

@codex review

@pamelachia
pamelachia marked this pull request as draft July 15, 2026 11:21
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 77ce8398ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pamelachia
pamelachia marked this pull request as ready for review July 17, 2026 06:04
Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants