feat(cli): Emit concrete rule events; drop rule start/_completed pairs

openspec/changes/restructure-cli-telemetry/design.md → openspec/changes/archive/2026-06-13-restructure-cli-telemetry/design.md

@@ -55,7 +55,7 @@ auth login success   → cli_authenticated  { }
 auth logout success  → cli_logged_out     { }
 init/install success → cli_installed      { targets? }
 onboard complete     → cli_onboarded      { }
-check finishes       → cli_check_completed{ errorCount, warningCount, filesScanned }
+check finishes       → cli_check_completed{ errorCount, warningCount, findings }
 any command fails    → cli_error          { command, code }
 help served          → cli_help           { topic }   (topic = "(index)" for no-arg,
                                                         the attempted topic otherwise)

openspec/changes/restructure-cli-telemetry/tasks.md → openspec/changes/archive/2026-06-13-restructure-cli-telemetry/tasks.md

@@ -27,36 +27,36 @@ keeps the suite green on its own.
 
 - [x] 1.1 In `packages/cli/src/index.ts`, wrap command execution so exactly one `cli_run` is emitted per invocation from a `finally`-equivalent path, with `{ command, cli_version, success, durationMs, anonymous, loggedIn }`
 - [x] 1.2 Resolve `command` from the matched citty subcommand (e.g. `"rule create"`, `"help"`); derive `success` from a thrown error / non-zero `process.exitCode`; measure `durationMs` from a start timestamp — extracted to a testable `telemetry-run.ts` (resolveCommandName/resolveCwd/emitRunEvents) so the entry module's side-effecting top level stays untested
-- [x] 1.3 Emit `cli_error { command, code }` from the runner's catch path when the failure carries a stable `CliErrorCode` — added an optional `code` to `CliError`; falls back to `INTERNAL_ERROR`
+- [x] 1.3 Emit `cli_error { command, code }` from the runner's catch path when the failure carries a stable `CLIErrorCode` — added an optional `code` to `CLIError`; falls back to `INTERNAL_ERROR`
 - [x] 1.4 Tests: one `cli_run` per invocation (success and failure), and `cli_error` on a known-code failure — `test/cli-run.test.ts`
 - [x] 1.5 typecheck + lint + suite green; commit; open PR 1
 
 ## 2. Phase 2 — rule concrete-state events (PR 2, on PR 1)
 
-- [ ] 2.1 `commands/rules.ts`: remove `cli_rule_create(_completed)`, `cli_rule_improve(_completed)`, `cli_rule_delete(_completed)`, `cli_rule_meta(_completed)`, `cli_rule_verify(_completed)`
-- [ ] 2.2 Emit `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` at the point each state changes (counts/ids/booleans only); `verify`/`meta` are covered by `cli_run` alone
-- [ ] 2.3 Update rule command tests to the new events; assert no `cli_rule_*_completed`
-- [ ] 2.4 typecheck + lint + suite green; commit; open PR 2
+- [x] 2.1 `commands/rules.ts`: remove `cli_rule_create(_completed)`, `cli_rule_improve(_completed)`, `cli_rule_delete(_completed)`, `cli_rule_meta(_completed)`, `cli_rule_verify(_completed)`
+- [x] 2.2 Emit `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` at the point each state changes (counts/ids/booleans only); `verify`/`meta` are covered by `cli_run` alone (their command-level telemetry was removed entirely)
+- [x] 2.3 Update rule command tests to the new events; assert no `cli_rule_*_completed` — only `telemetry.test.ts` referenced an old rule name (a sample), updated to `cli_rule_created`; rule-from/verify tests assert behavior, not events
+- [x] 2.4 typecheck + lint + suite green; commit; open PR 2
 
 ## 3. Phase 3 — auth + lifecycle events (PR 3, on PR 2)
 
-- [ ] 3.1 `commands/auth.ts`: remove `cli_auth_login(_completed)`, `cli_auth_logout(_completed)`, `cli_auth_status(_completed)`; emit `cli_authenticated` and `cli_logged_out` on success (status → `cli_run` only)
-- [ ] 3.2 `commands/init.ts` + `wizard/index.ts`: remove `cli_init(_completed)`, `cli_init_cancelled`, `cli_update(_completed)`; emit `cli_installed` on a successful install
-- [ ] 3.3 `commands/onboard.ts`: remove `cli_onboard_recipe` / `cli_onboard_already_done`; emit `cli_onboarded` when onboarding is marked complete
-- [ ] 3.4 `commands/check.ts`: remove `cli_check(_completed)`; emit `cli_check_completed { errorCount, warningCount, filesScanned }` (counts only — no matched code)
-- [ ] 3.5 Update auth/init/onboard/check tests to the new events
-- [ ] 3.6 typecheck + lint + suite green; commit; open PR 3
+- [x] 3.1 `commands/auth.ts`: remove `cli_auth_login(_completed)`, `cli_auth_logout(_completed)`, `cli_auth_status(_completed)`; emit `cli_authenticated` (fresh login only) and `cli_logged_out` (token actually removed); status → `cli_run` only
+- [x] 3.2 `commands/init.ts` + `wizard/index.ts`: remove `cli_init(_completed)`, `cli_init_cancelled`, `cli_update(_completed)`; emit `cli_installed` on a successful install (interactive + non-interactive + update)
+- [x] 3.3 `commands/onboard.ts`: remove `cli_onboard_recipe` / `cli_onboard_already_done`; emit `cli_onboarded` when onboarding is marked complete
+- [x] 3.4 `commands/check.ts`: remove `cli_check(_completed)`; emit `cli_check_completed { errorCount, warningCount, findings }` only when a scan actually runs (counts only — no matched code; `findings` replaces the unavailable `filesScanned`)
+- [x] 3.5 Update auth/init/onboard/check tests to the new events — wizard-integration assertions updated to `cli_installed` / no-event-on-cancel; telemetry.test sample names → `cli_run`
+- [x] 3.6 typecheck + lint + suite green; commit; open PR 3
 
 ## 4. Phase 4 — cli_help { topic } + drop bespoke info/detect events (PR 4, on PR 3)
 
-- [ ] 4.1 `commands/help.ts`: replace `help_index`, `help_<topic>`, `help_unknown` with one `cli_help { topic }` (served topic, an index marker for no-arg, the attempted topic for unknown)
-- [ ] 4.2 `commands/info.ts`: remove its bespoke `cli_info(_completed)` events — covered by `cli_run`. (`commands/detect.ts` / `cli_detect` is NOT in this branch's lineage — it lives in the unmerged local-rule-routing stack — so there is nothing to change here; reconcile when both stacks land.)
-- [ ] 4.3 Add `test/help-telemetry.test.ts` and update info tests; assert `cli_help` carries `topic` (served / `"(index)"` / attempted) and no `help_*` event is emitted
-- [ ] 4.4 typecheck + lint + suite green; commit; open PR 4
+- [x] 4.1 `commands/help.ts`: replace `help_index`, `help_<topic>`, `help_unknown` with one `cli_help { topic }` (served topic, `"(index)"` marker for no-arg, the attempted topic for unknown)
+- [x] 4.2 `commands/info.ts`: remove bespoke `cli_info(_completed)` (covered by `cli_run`); also drop its now-unused `getTelemetry` import. NOTE: `detect.ts`/`cli_detect` is NOT on this branch's lineage (it lives in the unmerged local-rule-routing stack) — no change needed here; it will be reconciled when that stack and this one both land
+- [x] 4.3 Assert `cli_help` carries `topic` and no `help_*` event — added `test/help-telemetry.test.ts` (served topic, index marker, unknown topic, and no legacy `help_*`)
+- [x] 4.4 typecheck + lint + suite green; commit; open PR 4
 
 ## 5. Phase 5 — finalize (PR 5, tip)
 
-- [ ] 5.1 Grep the CLI for any remaining old event names (`_completed`, `help_index`, `help_<topic>`, `help_unknown`, legacy `cli_<action>` starts); remove any stragglers
-- [ ] 5.2 Run `pnpm openspec validate restructure-cli-telemetry`; `pnpm typecheck`; `pnpm lint`; full suite green
-- [ ] 5.3 Manual smoke: run a couple of commands with telemetry mocked/inspected — confirm one `cli_run` per invocation plus the expected concrete event, and no legacy names
-- [ ] 5.4 Archive the change (`openspec archive restructure-cli-telemetry`) so the tip carries the spec sync + dated archive; commit; open PR 5
+- [x] 5.1 Grep the CLI for any remaining old event names (`_completed`, `help_index`, `help_<topic>`, `help_unknown`, legacy `cli_<action>` starts); remove any stragglers — clean; the only `_completed` is the intentional concrete event `cli_check_completed`
+- [x] 5.2 Run `pnpm openspec validate restructure-cli-telemetry`; `pnpm typecheck`; `pnpm lint`; full suite green (259)
+- [x] 5.3 Manual smoke: `info`, `help check`, `help` (index) run end-to-end after the refactor; concrete events + cli_run/cli_help/cli_error verified by the in-process tests
+- [x] 5.4 Archive the change (`openspec archive restructure-cli-telemetry`) so the tip carries the spec sync + dated archive; commit; open PR 5

openspec/specs/analytics/spec.md

@@ -111,13 +111,13 @@ Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `c
 
 #### Scenario: Anonymous capture includes standard properties
 
-- **WHEN** `capture("cli_check")` is called without authentication
+- **WHEN** `capture("cli_run")` is called without authentication
 - **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: <string>, scaffoldVersion: <number> }`
 - **AND** the event SHALL NOT include a `groups` parameter
 
 #### Scenario: Authenticated capture includes standard properties and group
 
-- **WHEN** `capture("cli_rule_create")` is called with authentication
+- **WHEN** `capture("cli_rule_created")` is called with authentication
 - **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: <string>, scaffoldVersion: <number> }`
 - **AND** the `groups` parameter SHALL include `{ organization: String(orgId) }`
 
@@ -134,56 +134,87 @@ Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `c
 
 ### Requirement: CLI events use cli\_ prefix
 
-CLI action events SHALL continue to use the `cli_` prefix, but the event taxonomy SHALL be reorganized as follows:
-
-- `cli_<action>` — fired when an action command begins execution (e.g. `cli_rule_create`, `cli_rule_improve`, `cli_rule_delete`, `cli_check`, `cli_info`, `cli_init`, `cli_auth_login`, `cli_auth_logout`)
-- `cli_<action>_completed` — fired when an action command finishes execution; event properties SHALL include `success: boolean`, `durationMs: number`, and `errorCode?: string` (when failure)
-- `help_<topic>` — fired when the help command serves a specific topic (e.g. `help_rule_create`, `help_check`, `help_auth`); replaces previous `cli_help_<topic>` events
-- `help_index` — fired when the help command is invoked with no arguments (probable agent confusion / routing failure)
-- `help_unknown` — fired when the help command receives an unknown topic; event properties SHALL include `topic: string` (the attempted topic)
-
-The previous event names `cli_help`, `cli_help_auth`, `cli_help_check`, `cli_help_info`, `cli_help_init`, `cli_help_rule` SHALL be removed in this release. There is no dual-emit window — the rename is a hard cut.
-
-#### Scenario: Action command emits start and completion events
-
-- **WHEN** a user runs `taskless rule create --from req.json`
-- **THEN** PostHog SHALL receive a `cli_rule_create` event when execution begins
-- **AND** SHALL receive a `cli_rule_create_completed` event when execution finishes, with properties including `success`, `durationMs`, and (on failure) `errorCode`
-
-#### Scenario: Help fetch emits topic intent
+CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a
+`cli_run` denominator plus concrete state-transition events:
+
+- `cli_run` — exactly one per invocation (see the dedicated requirement). This
+  replaces every previous `cli_<action>` start event and `cli_<action>_completed`
+  event; the `success`/`durationMs`/`command` signal lives here.
+- Concrete state-transition events, each fired at the point the state actually
+  changes, carrying counts/ids/booleans only (never rule content, prompts, or
+  matched source):
+  - `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted`
+  - `cli_authenticated`, `cli_logged_out`
+  - `cli_installed`, `cli_onboarded`
+  - `cli_check_completed` — error/warning counts only (e.g. `errorCount`,
+    `warningCount`, `findings`)
+  - `cli_error` — a single failure event with `command` and `code` (a stable
+    `CLIErrorCode`)
+- `cli_help` — fired when the help command serves a request, with a `topic`
+  property (the served topic; the exact literal `"(index)"` when invoked with no
+  topic; the attempted topic for an unknown request). This replaces the previous
+  `help_index`, `help_<topic>`, and `help_unknown` events.
+
+Commands that carry no concrete state beyond the invocation (e.g. `info`,
+`detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on
+`cli_run` alone and SHALL NOT emit a bespoke event. The previous taxonomy
+(`cli_<action>`, `cli_<action>_completed`, `help_index`, `help_<topic>`,
+`help_unknown`) SHALL be removed in this release; there is no dual-emit window.
+
+#### Scenario: Rule creation emits a concrete state event plus cli_run
+
+- **WHEN** a user runs `taskless rule create --from req.json` and a rule is written
+- **THEN** PostHog SHALL receive one `cli_run` event with `command: "rule create"`
+- **AND** SHALL receive a `cli_rule_created` event
+- **AND** SHALL NOT receive `cli_rule_create` or `cli_rule_create_completed`
+
+#### Scenario: Help fetch emits cli_help with a topic
 
 - **WHEN** an agent runs `taskless help rule create`
-- **THEN** PostHog SHALL receive a `help_rule_create` event
+- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"`
+- **AND** SHALL NOT receive a `help_rule_create` event
 
-#### Scenario: Help no-args emits index event
+#### Scenario: Help with no topic emits cli_help with the index marker
 
 - **WHEN** an agent runs `taskless help`
-- **THEN** PostHog SHALL receive a `help_index` event
+- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "(index)"`
+- **AND** SHALL NOT receive a `help_index` event
 
-#### Scenario: Help unknown topic emits help_unknown
+#### Scenario: A command failure emits cli_error
 
-- **WHEN** an agent runs `taskless help nonexistent`
-- **THEN** PostHog SHALL receive a `help_unknown` event with property `topic: "nonexistent"`
+- **WHEN** a command fails with a known `CLIErrorCode`
+- **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code`
 
 #### Scenario: Old event names are not emitted
 
-- **WHEN** any CLI command runs in v0.7.0
-- **THEN** PostHog SHALL NOT receive any event named `cli_help`, `cli_help_<topic>`, or any other event under the previous taxonomy
+- **WHEN** any CLI command runs in this release
+- **THEN** PostHog SHALL NOT receive any event named `cli_<action>_completed`,
+  `help_index`, `help_<topic>`, or `help_unknown`
 
 ### Requirement: Wrong-topic re-routing is observable as a derivable funnel
 
-The new event taxonomy is structured so that wrong-topic re-routing is a derivable funnel signal:
+The taxonomy SHALL keep wrong-topic re-routing derivable as a funnel signal from
+the new events:
 
-- A `help_<topic_a>` event followed by no `cli_<action_a>` event AND a subsequent `help_<topic_b>` event indicates the agent fetched the recipe for topic A, did not act on it, and re-routed to topic B
-- A `help_index` event followed by a `help_<topic>` event indicates the agent consulted the index before picking a topic (expected behavior; baseline)
-- A `help_<topic>` event with no subsequent `cli_<action>` event AND no further `help_*` event indicates the agent abandoned the action
+- A `cli_help { topic: A }` event not followed by the concrete event for topic A
+  (or by `cli_run` with the corresponding `command`), and then a subsequent
+  `cli_help { topic: B }`, indicates the agent fetched recipe A, did not act on
+  it, and re-routed to topic B.
+- A `cli_help` index-marker event followed by a `cli_help { topic }` event
+  indicates the agent consulted the index before picking a topic (baseline).
+- A `cli_help { topic }` event with no subsequent acting `cli_run` and no further
+  `cli_help` event indicates the agent abandoned the action.
 
-No additional events SHALL be added to capture this signal directly — the funnel is derivable from the event sequence in PostHog. Dashboards SHOULD be created to surface re-routing rates per topic so wrong-topic confusion can be measured.
+No additional events SHALL be added to capture this signal directly — it is
+derivable from the `cli_help` / `cli_run` sequence. Dashboards SHOULD surface
+re-routing rates per topic.
 
 #### Scenario: Funnel data supports wrong-topic detection
 
 - **WHEN** dashboards are constructed in PostHog
-- **THEN** the events SHALL be sufficient to compute "rate of `help_<topic>` events not followed by a corresponding `cli_<action>` event within N minutes"
+- **THEN** the `cli_help` (with `topic`) and `cli_run` (with `command`) events
+  SHALL be sufficient to compute "rate of `cli_help { topic }` not followed by a
+  corresponding acting `cli_run` within N minutes"
 
 ### Requirement: Telemetry failures are silent
 
@@ -218,3 +249,26 @@ Each command handler SHALL call `getTelemetry(cwd)` to lazily initialize the sin
 
 - **WHEN** the CLI exits without running a command (e.g. showing top-level help)
 - **THEN** `shutdownTelemetry()` SHALL be a no-op and no PostHog client SHALL be created
+
+### Requirement: Every invocation emits exactly one cli_run event
+
+The CLI SHALL emit exactly one `cli_run` event per invocation, from the top-level
+runner rather than from individual commands. The event SHALL carry the properties
+`command` (the resolved subcommand name, e.g. `"rule create"` or `"help"`),
+`cli_version`, `success` (boolean), `durationMs` (number), `anonymous` (boolean),
+and `loggedIn` (boolean). The event SHALL be emitted on both success and failure
+(from a `finally`-equivalent path), and no command SHALL emit its own
+"started" or "ran" event.
+
+#### Scenario: A successful command emits one cli_run
+
+- **WHEN** a user runs `taskless info`
+- **THEN** PostHog SHALL receive exactly one `cli_run` event with
+  `command: "info"`, `success: true`, a numeric `durationMs`, and the
+  `cli_version`, `anonymous`, and `loggedIn` properties
+- **AND** SHALL NOT receive a separate `cli_info` or `cli_info_completed` event
+
+#### Scenario: A failing command still emits cli_run
+
+- **WHEN** a command exits with an error
+- **THEN** PostHog SHALL receive one `cli_run` event with `success: false`