Skip to content

feat(cli): Emit cli_run once per invocation from the runner - #32

Merged
thecodedrift merged 51 commits into
mainfrom
feat/telemetry-cli-run
Jun 14, 2026
Merged

feat(cli): Emit cli_run once per invocation from the runner#32
thecodedrift merged 51 commits into
mainfrom
feat/telemetry-cli-run

Conversation

@thecodedrift

@thecodedrift thecodedrift commented Jun 12, 2026

Copy link
Copy Markdown
Member

Phase 1 of the telemetry rework (stacked on the proposal docs).

Adds the cli_run denominator: the index.ts runner now emits exactly one cli_run per invocation — on both success and failure — with { command, cli_version, success, durationMs, anonymous, loggedIn }. On failure it first emits cli_error { command, code } (CliError gains an optional CliErrorCode, falling back to INTERNAL_ERROR). The telemetry client exposes identity so the runner can stamp anonymous/loggedIn.

The runner logic is extracted to telemetry-run.ts so it's unit-testable without executing the CLI entry point — test/cli-run.test.ts covers it.

Purely additive: legacy per-command events still emit (removed in phases 2–4), so the suite stays green.

Part of the restructure-cli-telemetry OpenSpec change.


Stack generated by Git Town

thecodedrift and others added 10 commits June 12, 2026 13:22
Phase 1 of the telemetry rework: add the cli_run denominator centrally in
the index.ts runner so every invocation emits exactly one cli_run with
{ command, cli_version, success, durationMs, anonymous, loggedIn }, on both
success and failure. On failure the runner first emits cli_error
{ command, code } — CliError gains an optional CliErrorCode, falling back
to INTERNAL_ERROR. The telemetry client now exposes identity so the runner
can stamp anonymous/loggedIn.

The runner logic (command-name resolution, run-event emission) is extracted
to telemetry-run.ts so it is unit-testable without executing the CLI entry
point. Legacy per-command events still emit in this phase (removed in
phases 2–4); cli_run is purely additive here, so the suite stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 of the telemetry rework. Replace the per-rule-command start and
_completed events with concrete state-transition events fired where the
state actually changes:

- rule create  → cli_rule_created  { ruleCount }  (only when rules are written)
- rule improve → cli_rule_improved { ruleCount }
- rule delete  → cli_rule_deleted                 (only when files removed)
- rule meta / rule verify → no bespoke event; covered by cli_run

The create/improve/delete commands keep their try/finally and emit the
concrete event from the finally guarded by a state flag; meta/verify drop
their command-level telemetry entirely. Only a sample event name in
telemetry.test.ts needed updating (to cli_rule_created).

Stacked on the cli_run phase; cli_run still emits per invocation, so the
suite stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 of the telemetry rework. Replace the auth/init/update/onboard/check
start and _completed pairs with concrete state-transition events:

- auth login  → cli_authenticated  (fresh login only; already-logged-in is not)
- auth logout → cli_logged_out      (only when a saved token was removed)
- auth status → no bespoke event; covered by cli_run
- init/update → cli_installed       (interactive wizard, non-interactive, update)
- onboard --mark-complete → cli_onboarded (already-done / recipe → cli_run only)
- check       → cli_check_completed { errorCount, warningCount, findings }
                (only when a scan runs; counts only, never matched code)

Tests: wizard-integration now asserts cli_installed on completion and no
install event on cancel; telemetry.test sample event names updated to cli_run.
Full suite green (256). cli_run still emits per invocation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4 of the telemetry rework. Replace help_index / help_<topic> /
help_unknown with a single cli_help carrying a topic property (the served
topic, "(index)" for a no-arg invocation, or the attempted topic when
unknown) — help intent is now one event filtered by topic. Remove info's
bespoke cli_info / cli_info_completed events (covered by cli_run) and its
now-unused getTelemetry import.

detect.ts (cli_detect) is not on this branch's lineage — it lives in the
unmerged local-rule-routing stack and is reconciled when both land.

Adds test/help-telemetry.test.ts asserting cli_help { topic } across the
served / index / unknown cases and that no legacy help_* event fires.
Full suite green (259).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All five phases of the telemetry rework are complete, so finalize on the
tip of the stack: apply the analytics delta into the main spec (add the
cli_run denominator requirement; rewrite the cli_ taxonomy, the
wrong-topic funnel, and the standard-properties scenarios) and move the
change to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/.

Legacy event sweep is clean (the only _completed is the intentional
cli_check_completed concrete event); validate/typecheck/lint/suite green;
commands smoke-tested end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-run

* feat/rethink-posthog-telemetry:
  ci(openspec): Skip the archive check on non-tip stacked PRs
* feat/telemetry-cli-run:
  ci(openspec): Skip the archive check on non-tip stacked PRs
…fecycle

* feat/telemetry-rule-events:
  ci(openspec): Skip the archive check on non-tip stacked PRs
…help

* feat/telemetry-auth-lifecycle:
  ci(openspec): Skip the archive check on non-tip stacked PRs
* feat/telemetry-cli-help:
  ci(openspec): Skip the archive check on non-tip stacked PRs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a per-invocation telemetry “denominator” (cli_run) emitted exactly once from the CLI runner, plus a failure-side cli_error event, as Phase 1 of the CLI telemetry taxonomy restructure (stacked PRs).

Changes:

  • Extract runner-side telemetry helpers (resolveCommandName, resolveCwd, emitRunEvents) into a new testable module and call them from src/index.ts in a finally block.
  • Extend telemetry + error plumbing to support cli_error { command, code } via an optional CliError.code and a TelemetryClient.identity surface for anonymous/logged-in stamping.
  • Add unit tests for the new runner telemetry behavior and document the phased rollout plan in OpenSpec change tasks.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
packages/cli/src/index.ts Emits cli_run (and cli_error on failure) once per invocation from the top-level runner.
packages/cli/src/telemetry-run.ts New extracted runner telemetry helpers (command/cwd resolution + event emission).
packages/cli/src/telemetry.ts Exposes TelemetryClient.identity and wires it into the initialized/no-op clients.
packages/cli/src/util/cli-error.ts Adds optional CliErrorCode to CliError for stable failure attribution.
packages/cli/test/cli-run.test.ts Adds unit tests covering resolveCommandName and runner event emission.
openspec/changes/restructure-cli-telemetry/tasks.md Adds the phased implementation task plan for the telemetry restructure change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/cli/src/telemetry-run.ts
Comment thread packages/cli/test/cli-run.test.ts Outdated
Comment thread openspec/changes/restructure-cli-telemetry/tasks.md Outdated
Comment thread openspec/changes/restructure-cli-telemetry/tasks.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Comment thread packages/cli/test/cli-run.test.ts Outdated
Comment thread packages/cli/src/index.ts Outdated

@thecodedrift thecodedrift left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 comments, some more significant than others. I'm not sold on the CLI vs Cli but I believe we need to be consistent

The identity stuff IS a concern though, we want to ensure we actually have this captured if possible so we can trace errors back to a service user when required

Comment thread packages/cli/src/util/cli-error.ts Outdated
Comment thread packages/cli/src/util/cli-error.ts Outdated
Comment thread packages/cli/src/telemetry-run.ts
Comment thread packages/cli/src/telemetry.ts Outdated
thecodedrift and others added 7 commits June 13, 2026 12:16
…i_version

PR #32 review:
- HIGH: cli_run's anonymous/loggedIn came from the telemetry client's
  identity snapshot taken at getTelemetry() init, which is stale for
  commands that change auth mid-run (auth login/logout). Add
  resolveRunIdentity(cwd) that reads the token fresh, and have the runner
  resolve it at emission time so cli_run reports post-invocation auth state.
  Removed the now-unused cached `identity` from the telemetry client.
- Drop cli_version from cli_run — the version rides on the standard
  cliVersion property; no second field.
- cli_error now fires only for a thrown error; failures signalled purely via
  process.exitCode are captured by cli_run success:false (no misleading
  INTERNAL_ERROR). Added a test for that case.
- Test asserts cli_error-then-cli_run order and exactly two calls
  (toHaveBeenNthCalledWith), and that cli_run carries no cli_version.
- tasks.md: fix "commitable"→"committable" and the detect.ts reference
  (not in this lineage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #34 review: guard the createdRuleCount/improvedRuleCount assignment on
rules.length > 0, so an empty/missing status.rules no longer emits
cli_rule_created / cli_rule_improved with ruleCount: 0. The "generated"
state with zero rules now leaves the count undefined, so the concrete
event does not fire (covered by cli_run).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #35 review:
- check.ts: compute errorCount/warningCount in one loop over results and
  derive hasErrors from errorCount, instead of one `some` + two `filter`
  passes over potentially large scan output.
- wizard-integration test: assert on the event name across all capture
  calls rather than not.toHaveBeenCalledWith("cli_installed"), so a call
  with extra properties can't produce a false negative.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #36 review: the served-topic and no-arg help-telemetry tests asserted
cli_help was emitted but not that the implementation avoids dual-emitting
a legacy help_* event. Both now map captured calls to their event names
and assert none start with help_ (and specifically not help_index).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review feedback: instead of repeating a "no help_* event" assertion
inside every behavioral help-telemetry test (over-testing), keep those
tests purely behavioral (cli_help { topic }) and add a single source-scan
test that asserts no help_* event-name literal remains anywhere under
src/. That states the contract once, confidently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per review: cli_run's anonymous/loggedIn should describe who *initiated*
the run, not the post-command auth state. Resolving identity in the
finally meant `auth login` reported loggedIn:true; reviewer confirmed the
intended semantic is "the login was performed as a logged-out user".
Move resolveRunIdentity to invocation start (before runCommand) and stamp
cli_run with that, so login reports loggedIn:false and logout reports
loggedIn:true — reliably, independent of when a command first calls
getTelemetry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-run

* feat/rethink-posthog-telemetry:
  docs(openspec): Address review on the telemetry contract
thecodedrift and others added 18 commits June 13, 2026 17:17
…-run

* feat/rethink-posthog-telemetry:
  docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract
* feat/telemetry-cli-run:
  docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract
…fecycle

* feat/telemetry-rule-events:
  docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract
…help

* feat/telemetry-auth-lifecycle:
  docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract
* feat/telemetry-cli-help:
  docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract
The acronym rename landed on the runner branch, but auth.ts is owned by
this phase and its edits sat on the same lines, so the merge kept the old
casing and left the type unresolved. Rename auth.ts here so the type
checks across the stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…help

* feat/telemetry-auth-lifecycle:
  fix(cli): Carry the CLIErrorCode rename into auth.ts
* feat/telemetry-cli-help:
  fix(cli): Carry the CLIErrorCode rename into auth.ts
Same module as CLIError/CLIErrorCode — keep the error envelope type's
acronym uppercase so the whole error surface is consistent. Self-contained
to types/errors.ts (no external importers of the type name).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat/telemetry-cli-run:
  refactor(cli): Uppercase the acronym in CLIErrorEnvelope too
…fecycle

* feat/telemetry-rule-events:
  refactor(cli): Uppercase the acronym in CLIErrorEnvelope too
…help

* feat/telemetry-auth-lifecycle:
  refactor(cli): Uppercase the acronym in CLIErrorEnvelope too
* feat/telemetry-cli-help:
  refactor(cli): Uppercase the acronym in CLIErrorEnvelope too
Finish the CLI acronym sweep across the package — the api config type is
the last Cli-cased identifier. Self-contained to api/config.ts (the type
is not exported).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat/telemetry-cli-run:
  refactor(cli): Uppercase the acronym in CLIConfig
…fecycle

* feat/telemetry-rule-events:
  refactor(cli): Uppercase the acronym in CLIConfig
…help

* feat/telemetry-auth-lifecycle:
  refactor(cli): Uppercase the acronym in CLIConfig
* feat/telemetry-cli-help:
  refactor(cli): Uppercase the acronym in CLIConfig
@thecodedrift
thecodedrift deleted the branch main June 14, 2026 04:42
@thecodedrift thecodedrift reopened this Jun 14, 2026
@thecodedrift
thecodedrift changed the base branch from feat/rethink-posthog-telemetry to main June 14, 2026 04:45
chore(openspec): Finalize + archive restructure-cli-telemetry
feat(cli): Collapse help_* into cli_help { topic }; drop cli_info
feat(cli): Concrete auth + lifecycle events; drop start/_completed pairs
feat(cli): Emit concrete rule events; drop rule start/_completed pairs
@thecodedrift
thecodedrift merged commit f9d8ae3 into main Jun 14, 2026
3 checks passed
@thecodedrift
thecodedrift deleted the feat/telemetry-cli-run branch June 14, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants