From 7b8603a9da32d0b1d0e2a098dbc7251d801ced85 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 09:08:19 -0700 Subject: [PATCH 1/3] chore(openspec): Archive restructure-cli-telemetry and sync specs All five phases of the telemetry rework are complete, so finalize on the tip of the stack: apply the analytics delta into the main spec (add the cli_run denominator requirement; rewrite the cli_ taxonomy, the wrong-topic funnel, and the standard-properties scenarios) and move the change to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/. Legacy event sweep is clean (the only _completed is the intentional cli_check_completed concrete event); validate/typecheck/lint/suite green; commands smoke-tested end-to-end. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/analytics/spec.md | 0 .../tasks.md | 8 +- openspec/specs/analytics/spec.md | 121 +++++++++++++----- 6 files changed, 92 insertions(+), 37 deletions(-) rename openspec/changes/{restructure-cli-telemetry => archive/2026-06-13-restructure-cli-telemetry}/.openspec.yaml (100%) rename openspec/changes/{restructure-cli-telemetry => archive/2026-06-13-restructure-cli-telemetry}/design.md (100%) rename openspec/changes/{restructure-cli-telemetry => archive/2026-06-13-restructure-cli-telemetry}/proposal.md (100%) rename openspec/changes/{restructure-cli-telemetry => archive/2026-06-13-restructure-cli-telemetry}/specs/analytics/spec.md (100%) rename openspec/changes/{restructure-cli-telemetry => archive/2026-06-13-restructure-cli-telemetry}/tasks.md (90%) diff --git a/openspec/changes/restructure-cli-telemetry/.openspec.yaml b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/.openspec.yaml similarity index 100% rename from openspec/changes/restructure-cli-telemetry/.openspec.yaml rename to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/.openspec.yaml diff --git a/openspec/changes/restructure-cli-telemetry/design.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md similarity index 100% rename from openspec/changes/restructure-cli-telemetry/design.md rename to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md diff --git a/openspec/changes/restructure-cli-telemetry/proposal.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/proposal.md similarity index 100% rename from openspec/changes/restructure-cli-telemetry/proposal.md rename to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/proposal.md diff --git a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md similarity index 100% rename from openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md rename to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md diff --git a/openspec/changes/restructure-cli-telemetry/tasks.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md similarity index 90% rename from openspec/changes/restructure-cli-telemetry/tasks.md rename to openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md index e53116de..a6e0541a 100644 --- a/openspec/changes/restructure-cli-telemetry/tasks.md +++ b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md @@ -56,7 +56,7 @@ keeps the suite green on its own. ## 5. Phase 5 — finalize (PR 5, tip) -- [ ] 5.1 Grep the CLI for any remaining old event names (`_completed`, `help_index`, `help_`, `help_unknown`, legacy `cli_` starts); remove any stragglers -- [ ] 5.2 Run `pnpm openspec validate restructure-cli-telemetry`; `pnpm typecheck`; `pnpm lint`; full suite green -- [ ] 5.3 Manual smoke: run a couple of commands with telemetry mocked/inspected — confirm one `cli_run` per invocation plus the expected concrete event, and no legacy names -- [ ] 5.4 Archive the change (`openspec archive restructure-cli-telemetry`) so the tip carries the spec sync + dated archive; commit; open PR 5 +- [x] 5.1 Grep the CLI for any remaining old event names (`_completed`, `help_index`, `help_`, `help_unknown`, legacy `cli_` starts); remove any stragglers — clean; the only `_completed` is the intentional concrete event `cli_check_completed` +- [x] 5.2 Run `pnpm openspec validate restructure-cli-telemetry`; `pnpm typecheck`; `pnpm lint`; full suite green (259) +- [x] 5.3 Manual smoke: `info`, `help check`, `help` (index) run end-to-end after the refactor; concrete events + cli_run/cli_help/cli_error verified by the in-process tests +- [x] 5.4 Archive the change (`openspec archive restructure-cli-telemetry`) so the tip carries the spec sync + dated archive; commit; open PR 5 diff --git a/openspec/specs/analytics/spec.md b/openspec/specs/analytics/spec.md index 6f1173c1..dab92816 100644 --- a/openspec/specs/analytics/spec.md +++ b/openspec/specs/analytics/spec.md @@ -111,13 +111,13 @@ Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `c #### Scenario: Anonymous capture includes standard properties -- **WHEN** `capture("cli_check")` is called without authentication +- **WHEN** `capture("cli_run")` is called without authentication - **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` - **AND** the event SHALL NOT include a `groups` parameter #### Scenario: Authenticated capture includes standard properties and group -- **WHEN** `capture("cli_rule_create")` is called with authentication +- **WHEN** `capture("cli_rule_created")` is called with authentication - **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` - **AND** the `groups` parameter SHALL include `{ organization: String(orgId) }` @@ -134,56 +134,88 @@ Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `c ### Requirement: CLI events use cli\_ prefix -CLI action events SHALL continue to use the `cli_` prefix, but the event taxonomy SHALL be reorganized as follows: - -- `cli_` — fired when an action command begins execution (e.g. `cli_rule_create`, `cli_rule_improve`, `cli_rule_delete`, `cli_check`, `cli_info`, `cli_init`, `cli_auth_login`, `cli_auth_logout`) -- `cli__completed` — fired when an action command finishes execution; event properties SHALL include `success: boolean`, `durationMs: number`, and `errorCode?: string` (when failure) -- `help_` — fired when the help command serves a specific topic (e.g. `help_rule_create`, `help_check`, `help_auth`); replaces previous `cli_help_` events -- `help_index` — fired when the help command is invoked with no arguments (probable agent confusion / routing failure) -- `help_unknown` — fired when the help command receives an unknown topic; event properties SHALL include `topic: string` (the attempted topic) - -The previous event names `cli_help`, `cli_help_auth`, `cli_help_check`, `cli_help_info`, `cli_help_init`, `cli_help_rule` SHALL be removed in this release. There is no dual-emit window — the rename is a hard cut. - -#### Scenario: Action command emits start and completion events - -- **WHEN** a user runs `taskless rule create --from req.json` -- **THEN** PostHog SHALL receive a `cli_rule_create` event when execution begins -- **AND** SHALL receive a `cli_rule_create_completed` event when execution finishes, with properties including `success`, `durationMs`, and (on failure) `errorCode` - -#### Scenario: Help fetch emits topic intent +CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a +`cli_run` denominator plus concrete state-transition events: + +- `cli_run` — exactly one per invocation (see the dedicated requirement). This + replaces every previous `cli_` start event and `cli__completed` + event; the `success`/`durationMs`/`command` signal lives here. +- Concrete state-transition events, each fired at the point the state actually + changes, carrying counts/ids/booleans only (never rule content, prompts, or + matched source): + - `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` + - `cli_authenticated`, `cli_logged_out` + - `cli_installed`, `cli_onboarded` + - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, + `warningCount`, `filesScanned`) + - `cli_error` — a single failure event with `command` and `code` (a stable + `CliErrorCode`) +- `cli_help` — fired when the help command serves a request, with a `topic` + property (the served topic, or an index marker when invoked with no topic). + This replaces the previous `help_index`, `help_`, and `help_unknown` + events. + +Commands that carry no concrete state beyond the invocation (e.g. `info`, +`detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on +`cli_run` alone and SHALL NOT emit a bespoke event. The previous taxonomy +(`cli_`, `cli__completed`, `help_index`, `help_`, +`help_unknown`) SHALL be removed in this release; there is no dual-emit window. + +#### Scenario: Rule creation emits a concrete state event plus cli_run + +- **WHEN** a user runs `taskless rule create --from req.json` and a rule is written +- **THEN** PostHog SHALL receive one `cli_run` event with `command: "rule create"` +- **AND** SHALL receive a `cli_rule_created` event +- **AND** SHALL NOT receive `cli_rule_create` or `cli_rule_create_completed` + +#### Scenario: Help fetch emits cli_help with a topic - **WHEN** an agent runs `taskless help rule create` -- **THEN** PostHog SHALL receive a `help_rule_create` event +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"` +- **AND** SHALL NOT receive a `help_rule_create` event -#### Scenario: Help no-args emits index event +#### Scenario: Help with no topic emits cli_help with an index marker - **WHEN** an agent runs `taskless help` -- **THEN** PostHog SHALL receive a `help_index` event +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` marks the index + (no-argument) invocation +- **AND** SHALL NOT receive a `help_index` event -#### Scenario: Help unknown topic emits help_unknown +#### Scenario: A command failure emits cli_error -- **WHEN** an agent runs `taskless help nonexistent` -- **THEN** PostHog SHALL receive a `help_unknown` event with property `topic: "nonexistent"` +- **WHEN** a command fails with a known `CliErrorCode` +- **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` #### Scenario: Old event names are not emitted -- **WHEN** any CLI command runs in v0.7.0 -- **THEN** PostHog SHALL NOT receive any event named `cli_help`, `cli_help_`, or any other event under the previous taxonomy +- **WHEN** any CLI command runs in this release +- **THEN** PostHog SHALL NOT receive any event named `cli__completed`, + `help_index`, `help_`, or `help_unknown` ### Requirement: Wrong-topic re-routing is observable as a derivable funnel -The new event taxonomy is structured so that wrong-topic re-routing is a derivable funnel signal: +The taxonomy SHALL keep wrong-topic re-routing derivable as a funnel signal from +the new events: -- A `help_` event followed by no `cli_` event AND a subsequent `help_` event indicates the agent fetched the recipe for topic A, did not act on it, and re-routed to topic B -- A `help_index` event followed by a `help_` event indicates the agent consulted the index before picking a topic (expected behavior; baseline) -- A `help_` event with no subsequent `cli_` event AND no further `help_*` event indicates the agent abandoned the action +- A `cli_help { topic: A }` event not followed by the concrete event for topic A + (or by `cli_run` with the corresponding `command`), and then a subsequent + `cli_help { topic: B }`, indicates the agent fetched recipe A, did not act on + it, and re-routed to topic B. +- A `cli_help` index-marker event followed by a `cli_help { topic }` event + indicates the agent consulted the index before picking a topic (baseline). +- A `cli_help { topic }` event with no subsequent acting `cli_run` and no further + `cli_help` event indicates the agent abandoned the action. -No additional events SHALL be added to capture this signal directly — the funnel is derivable from the event sequence in PostHog. Dashboards SHOULD be created to surface re-routing rates per topic so wrong-topic confusion can be measured. +No additional events SHALL be added to capture this signal directly — it is +derivable from the `cli_help` / `cli_run` sequence. Dashboards SHOULD surface +re-routing rates per topic. #### Scenario: Funnel data supports wrong-topic detection - **WHEN** dashboards are constructed in PostHog -- **THEN** the events SHALL be sufficient to compute "rate of `help_` events not followed by a corresponding `cli_` event within N minutes" +- **THEN** the `cli_help` (with `topic`) and `cli_run` (with `command`) events + SHALL be sufficient to compute "rate of `cli_help { topic }` not followed by a + corresponding acting `cli_run` within N minutes" ### Requirement: Telemetry failures are silent @@ -218,3 +250,26 @@ Each command handler SHALL call `getTelemetry(cwd)` to lazily initialize the sin - **WHEN** the CLI exits without running a command (e.g. showing top-level help) - **THEN** `shutdownTelemetry()` SHALL be a no-op and no PostHog client SHALL be created + +### Requirement: Every invocation emits exactly one cli_run event + +The CLI SHALL emit exactly one `cli_run` event per invocation, from the top-level +runner rather than from individual commands. The event SHALL carry the properties +`command` (the resolved subcommand name, e.g. `"rule create"` or `"help"`), +`cli_version`, `success` (boolean), `durationMs` (number), `anonymous` (boolean), +and `loggedIn` (boolean). The event SHALL be emitted on both success and failure +(from a `finally`-equivalent path), and no command SHALL emit its own +"started" or "ran" event. + +#### Scenario: A successful command emits one cli_run + +- **WHEN** a user runs `taskless info` +- **THEN** PostHog SHALL receive exactly one `cli_run` event with + `command: "info"`, `success: true`, a numeric `durationMs`, and the + `cli_version`, `anonymous`, and `loggedIn` properties +- **AND** SHALL NOT receive a separate `cli_info` or `cli_info_completed` event + +#### Scenario: A failing command still emits cli_run + +- **WHEN** a command exits with an error +- **THEN** PostHog SHALL receive one `cli_run` event with `success: false` From 20d46d5aa5c2d59ab04c4a6958f634e857c7156b Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 15:10:40 -0700 Subject: [PATCH 2/3] docs(openspec): Sync corrected check/help spec into the archived capability The archive commit synced the pre-review delta spec, so the capability spec carried the stale cli_check_completed{ filesScanned } and a fuzzy cli_help index-marker description. Reconcile the synced spec (and the archive design flow diagram) with the corrected contract: findings replaces filesScanned, and cli_help documents the exact literal "(index)" for the no-topic invocation. Co-Authored-By: Claude Opus 4.8 --- .../2026-06-13-restructure-cli-telemetry/design.md | 2 +- openspec/specs/analytics/spec.md | 13 ++++++------- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md index 017512ee..254669b2 100644 --- a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md +++ b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md @@ -55,7 +55,7 @@ auth login success → cli_authenticated { } auth logout success → cli_logged_out { } init/install success → cli_installed { targets? } onboard complete → cli_onboarded { } -check finishes → cli_check_completed{ errorCount, warningCount, filesScanned } +check finishes → cli_check_completed{ errorCount, warningCount, findings } any command fails → cli_error { command, code } help served → cli_help { topic } (topic = "(index)" for no-arg, the attempted topic otherwise) diff --git a/openspec/specs/analytics/spec.md b/openspec/specs/analytics/spec.md index dab92816..b8cc8f4b 100644 --- a/openspec/specs/analytics/spec.md +++ b/openspec/specs/analytics/spec.md @@ -147,13 +147,13 @@ CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a - `cli_authenticated`, `cli_logged_out` - `cli_installed`, `cli_onboarded` - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, - `warningCount`, `filesScanned`) + `warningCount`, `findings`) - `cli_error` — a single failure event with `command` and `code` (a stable `CliErrorCode`) - `cli_help` — fired when the help command serves a request, with a `topic` - property (the served topic, or an index marker when invoked with no topic). - This replaces the previous `help_index`, `help_`, and `help_unknown` - events. + property (the served topic; the exact literal `"(index)"` when invoked with no + topic; the attempted topic for an unknown request). This replaces the previous + `help_index`, `help_`, and `help_unknown` events. Commands that carry no concrete state beyond the invocation (e.g. `info`, `detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on @@ -174,11 +174,10 @@ Commands that carry no concrete state beyond the invocation (e.g. `info`, - **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"` - **AND** SHALL NOT receive a `help_rule_create` event -#### Scenario: Help with no topic emits cli_help with an index marker +#### Scenario: Help with no topic emits cli_help with the index marker - **WHEN** an agent runs `taskless help` -- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` marks the index - (no-argument) invocation +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "(index)"` - **AND** SHALL NOT receive a `help_index` event #### Scenario: A command failure emits cli_error From b204084fdc76c1ea3bfa0a9c7ebde3486389746f Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 16:41:17 -0700 Subject: [PATCH 3/3] docs(openspec): Uppercase CLIError/CLIErrorCode in the analytics spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Match the source rename — the CLI acronym is uppercase in the CLIError class and CLIErrorCode type, so the spec prose and archived contract use the same casing. Co-Authored-By: Claude Opus 4.8 --- .../archive/2026-06-13-restructure-cli-telemetry/design.md | 4 ++-- .../specs/analytics/spec.md | 4 ++-- .../archive/2026-06-13-restructure-cli-telemetry/tasks.md | 2 +- openspec/specs/analytics/spec.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md index 254669b2..395760e4 100644 --- a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md +++ b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md @@ -70,7 +70,7 @@ carries no concrete state beyond the invocation, so those are covered by ### D3 — `cli_error` is the single failure event Instead of `success:false` spread across each `_completed` event, failures emit -one `cli_error { command, code }` (code from the stable `CliErrorCode` set), and +one `cli_error { command, code }` (code from the stable `CLIErrorCode` set), and `cli_run` also records `success:false`. The runner emits `cli_error` from its catch path so no command has to remember to. @@ -86,7 +86,7 @@ Dashboards are rebuilt against the new names (the proposal calls this out). - **[Centralized `cli_run` can't see command-specific context]** → By design (D-non-goal). `loggedIn` covers the only cross-cutting dimension we need now. - **[`success` detection in the runner is imperfect]** → Derive from thrown - error and `process.exitCode`; commands already use `CliError` + exit codes + error and `process.exitCode`; commands already use `CLIError` + exit codes consistently, so this is reliable. ## Open Questions diff --git a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md index 5d325ce7..47eebac2 100644 --- a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md +++ b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/specs/analytics/spec.md @@ -44,7 +44,7 @@ CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, `warningCount`, `findings`) - `cli_error` — a single failure event with `command` and `code` (a stable - `CliErrorCode`) + `CLIErrorCode`) - `cli_help` — fired when the help command serves a request, with a `topic` property. The `topic` SHALL be: the served topic for a known topic (e.g. `"rule create"`); the exact literal `"(index)"` when invoked with no topic; @@ -84,7 +84,7 @@ Commands that carry no concrete state beyond the invocation (e.g. `info`, #### Scenario: A command failure emits cli_error -- **WHEN** a command fails with a known `CliErrorCode` +- **WHEN** a command fails with a known `CLIErrorCode` - **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` #### Scenario: Old event names are not emitted diff --git a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md index 68041c98..aace17da 100644 --- a/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md +++ b/openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md @@ -27,7 +27,7 @@ keeps the suite green on its own. - [x] 1.1 In `packages/cli/src/index.ts`, wrap command execution so exactly one `cli_run` is emitted per invocation from a `finally`-equivalent path, with `{ command, cli_version, success, durationMs, anonymous, loggedIn }` - [x] 1.2 Resolve `command` from the matched citty subcommand (e.g. `"rule create"`, `"help"`); derive `success` from a thrown error / non-zero `process.exitCode`; measure `durationMs` from a start timestamp — extracted to a testable `telemetry-run.ts` (resolveCommandName/resolveCwd/emitRunEvents) so the entry module's side-effecting top level stays untested -- [x] 1.3 Emit `cli_error { command, code }` from the runner's catch path when the failure carries a stable `CliErrorCode` — added an optional `code` to `CliError`; falls back to `INTERNAL_ERROR` +- [x] 1.3 Emit `cli_error { command, code }` from the runner's catch path when the failure carries a stable `CLIErrorCode` — added an optional `code` to `CLIError`; falls back to `INTERNAL_ERROR` - [x] 1.4 Tests: one `cli_run` per invocation (success and failure), and `cli_error` on a known-code failure — `test/cli-run.test.ts` - [x] 1.5 typecheck + lint + suite green; commit; open PR 1 diff --git a/openspec/specs/analytics/spec.md b/openspec/specs/analytics/spec.md index b8cc8f4b..02dc1f4e 100644 --- a/openspec/specs/analytics/spec.md +++ b/openspec/specs/analytics/spec.md @@ -149,7 +149,7 @@ CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, `warningCount`, `findings`) - `cli_error` — a single failure event with `command` and `code` (a stable - `CliErrorCode`) + `CLIErrorCode`) - `cli_help` — fired when the help command serves a request, with a `topic` property (the served topic; the exact literal `"(index)"` when invoked with no topic; the attempted topic for an unknown request). This replaces the previous @@ -182,7 +182,7 @@ Commands that carry no concrete state beyond the invocation (e.g. `info`, #### Scenario: A command failure emits cli_error -- **WHEN** a command fails with a known `CliErrorCode` +- **WHEN** a command fails with a known `CLIErrorCode` - **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` #### Scenario: Old event names are not emitted