From 05ee55fc12a035508f06bae979c980943e81b894 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 12 Jun 2026 12:40:01 -0700 Subject: [PATCH 1/4] docs(openspec): Propose restructure-cli-telemetry change Establish the contract for reworking the CLI's PostHog taxonomy: a single cli_run denominator emitted once per invocation, cli_* reserved for concrete state transitions (cli_rule_created, cli_error, cli_check_completed, ...), and cli_help { topic } replacing the help_* trio. Modifies the analytics capability; a hard cut with no dual-emit window. Base of a stack; implementation lands in phased PRs on top. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../restructure-cli-telemetry/.openspec.yaml | 2 + .../restructure-cli-telemetry/design.md | 94 ++++++++++++ .../restructure-cli-telemetry/proposal.md | 65 +++++++++ .../specs/analytics/spec.md | 138 ++++++++++++++++++ 4 files changed, 299 insertions(+) create mode 100644 openspec/changes/restructure-cli-telemetry/.openspec.yaml create mode 100644 openspec/changes/restructure-cli-telemetry/design.md create mode 100644 openspec/changes/restructure-cli-telemetry/proposal.md create mode 100644 openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md diff --git a/openspec/changes/restructure-cli-telemetry/.openspec.yaml b/openspec/changes/restructure-cli-telemetry/.openspec.yaml new file mode 100644 index 00000000..8fe20555 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-12 diff --git a/openspec/changes/restructure-cli-telemetry/design.md b/openspec/changes/restructure-cli-telemetry/design.md new file mode 100644 index 00000000..b8d93a26 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/design.md @@ -0,0 +1,94 @@ +## Context + +This design is intentionally lean — the taxonomy was decided before writing the +proposal. It records the model and the one non-trivial implementation decision +(centralizing `cli_run`); there is no open architecture to resolve. + +Today each command emits `cli_` at start and `cli__completed` at +end. The telemetry wrapper already attaches standard properties (`cli`, +`cliVersion`, `scaffoldVersion`, org group) and fails silently. Those guarantees +are unchanged; only the event taxonomy changes. + +## Goals / Non-Goals + +**Goals:** + +- One `cli_run` per invocation as the denominator, emitted centrally. +- Named `cli_*` events only for concrete state transitions. +- `cli_help { topic }` replacing the `help_*` trio. +- A hard cut — no dual-emit window. + +**Non-Goals:** + +- Changing identity, opt-out, standard properties, or silent-failure behavior. +- Per-command run-context enrichment (auth state on `cli_run` already implies + anonymous vs. authenticated). +- Capturing rule content, prompts, or matched source in any event. + +## Decisions + +### D1 — `cli_run` is emitted once, centrally, in the runner + +The CLI's top-level runner (`index.ts`, around `runCommand`) wraps execution: it +resolves the matched subcommand name, runs the command, and emits a single +`cli_run` with `{ command, cli_version, success, durationMs, anonymous, +loggedIn }` — on both success and failure (in a `finally`). + +_Why:_ Centralizing makes the denominator impossible to forget and removes ~20 +per-command start/`_completed` captures. The command name comes from the +resolved citty command; `success` is derived from whether the command threw (or +set a non-zero `process.exitCode`); `durationMs` from a start timestamp. + +_Alternative considered:_ keep per-command emission. Rejected — it's what we have +now, and it's the source of the scatter. + +### D2 — Concrete-state events fire at the state change, not command boundaries + +Each meaningful outcome emits its own event where the state actually changes: + +``` +rule create success → cli_rule_created { mode: "remote"|"static", ruleCount } +rule improve success → cli_rule_improved { ruleCount } +rule delete success → cli_rule_deleted { } +auth login success → cli_authenticated { } +auth logout success → cli_logged_out { } +init/install success → cli_installed { targets? } +onboard complete → cli_onboarded { } +check finishes → cli_check_completed{ errorCount, warningCount, filesScanned } +any command fails → cli_error { command, code } +help served → cli_help { topic } (topic = "(index)" for no-arg, + the attempted topic otherwise) +``` + +Properties are counts/ids/booleans only. + +_Why:_ These are the funnel-worthy moments. "Ran info/status/detect/verify/meta" +carries no concrete state beyond the invocation, so those are covered by +`cli_run` alone with no bespoke event. + +### D3 — `cli_error` is the single failure event + +Instead of `success:false` spread across each `_completed` event, failures emit +one `cli_error { command, code }` (code from the stable `CliErrorCode` set), and +`cli_run` also records `success:false`. The runner emits `cli_error` from its +catch path so no command has to remember to. + +### D4 — Hard cut, no dual-emit + +Old event names are removed in the same release; nothing emits both taxonomies. +Dashboards are rebuilt against the new names (the proposal calls this out). + +## Risks / Trade-offs + +- **[Dashboards/funnels on old names break]** → Accepted and documented; this is + a deliberate hard cut. The new taxonomy is simpler to rebuild against. +- **[Centralized `cli_run` can't see command-specific context]** → By design + (D-non-goal). `loggedIn` covers the only cross-cutting dimension we need now. +- **[`success` detection in the runner is imperfect]** → Derive from thrown + error and `process.exitCode`; commands already use `CliError` + exit codes + consistently, so this is reliable. + +## Open Questions + +- None. (`cli_check_completed` property names — `errorCount`/`warningCount` — are + a naming detail finalized during implementation against the check result shape.) diff --git a/openspec/changes/restructure-cli-telemetry/proposal.md b/openspec/changes/restructure-cli-telemetry/proposal.md new file mode 100644 index 00000000..48b3b890 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/proposal.md @@ -0,0 +1,65 @@ +## Why + +The CLI's PostHog taxonomy emits a `cli_` start event plus a +`cli__completed` event for nearly every command — roughly two events per +invocation, with the "ran" signal scattered across ~20 distinct event names. +There is no single denominator for "the CLI was invoked," so basic questions +("how many runs of each command", "overall success rate") require stitching many +event names together. Following PostHog's own guidance, we want one top-level +`cli_run` event per invocation as the denominator, and to reserve named `cli_*` +events for concrete state transitions worth analyzing as funnels. + +## What Changes + +- **NEW `cli_run`** — emitted exactly once per invocation, centrally in the CLI + runner, with properties `command`, `cli_version`, `success`, `durationMs`, + `anonymous`, and `loggedIn`. This is the universal denominator; no command + emits its own "started/ran" event anymore. +- **`cli_*` reserved for concrete state transitions** — replace the per-command + start/`_completed` pairs with events that represent real outcomes: + `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted`, + `cli_authenticated`, `cli_logged_out`, `cli_installed`, `cli_onboarded`, + `cli_check_completed` (error counts only — no rule content or matched code), + and a unified `cli_error` (`command`, `code`). +- **`cli_help { topic }`** — replace `help_index` / `help_` / `help_unknown` + with a single `cli_help` event carrying a `topic` field (parallel to + `cli_run`'s `command`), so help intent is one event filtered by topic. +- **BREAKING (analytics only)** — the previous taxonomy is a hard cut, no + dual-emit window. Dashboards/funnels built on the old names must be rebuilt. +- **No per-command enrichment context** — auth state is already on `cli_run` + (`loggedIn`), which implies `--anonymous` behavior; commands do not thread + extra run-context properties in this change. + +## Capabilities + +### New Capabilities + +(none) + +### Modified Capabilities + +- `analytics`: Rewrite the CLI event taxonomy — introduce `cli_run` as the + once-per-invocation denominator, replace start/`_completed` pairs with + concrete-state `cli_*` events, collapse the `help_*` events into + `cli_help { topic }`, and update the wrong-topic re-routing funnel to derive + from the new events. Standard-property guarantees are unchanged. + +## Impact + +- **Centralized emission**: `packages/cli/src/index.ts` runner gains the + `cli_run` emission (resolved command name + success + duration), so the event + cannot be forgotten per command. +- **Per-command call sites** (`commands/{rules,auth,init,check,info,onboard}.ts`, + `wizard/index.ts`): remove start/`_completed` captures; emit concrete-state + events at the point the state actually changes. +- **Help** (`commands/help.ts`): `help_index`/`help_`/`help_unknown` → + `cli_help { topic }`. +- **Telemetry module** (`telemetry.ts`): standard properties unchanged; the + wrapper continues to attach `cli`, `cliVersion`, `scaffoldVersion`, and org + group to every event. +- **Tests**: `test/telemetry.test.ts` and command tests that assert event names + update to the new taxonomy. +- **Privacy**: concrete events carry counts/ids/booleans only — never rule + content, prompts, or matched source. +- **No new dependencies**; no change to opt-out, identity, or silent-failure + behavior. diff --git a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md new file mode 100644 index 00000000..fc5099e5 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md @@ -0,0 +1,138 @@ +## ADDED Requirements + +### Requirement: Every invocation emits exactly one cli_run event + +The CLI SHALL emit exactly one `cli_run` event per invocation, from the top-level +runner rather than from individual commands. The event SHALL carry the properties +`command` (the resolved subcommand name, e.g. `"rule create"` or `"help"`), +`cli_version`, `success` (boolean), `durationMs` (number), `anonymous` (boolean), +and `loggedIn` (boolean). The event SHALL be emitted on both success and failure +(from a `finally`-equivalent path), and no command SHALL emit its own +"started" or "ran" event. + +#### Scenario: A successful command emits one cli_run + +- **WHEN** a user runs `taskless info` +- **THEN** PostHog SHALL receive exactly one `cli_run` event with + `command: "info"`, `success: true`, a numeric `durationMs`, and the + `cli_version`, `anonymous`, and `loggedIn` properties +- **AND** SHALL NOT receive a separate `cli_info` or `cli_info_completed` event + +#### Scenario: A failing command still emits cli_run + +- **WHEN** a command exits with an error +- **THEN** PostHog SHALL receive one `cli_run` event with `success: false` + +## MODIFIED Requirements + +### Requirement: CLI events use cli\_ prefix + +CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a +`cli_run` denominator plus concrete state-transition events: + +- `cli_run` — exactly one per invocation (see the dedicated requirement). This + replaces every previous `cli_` start event and `cli__completed` + event; the `success`/`durationMs`/`command` signal lives here. +- Concrete state-transition events, each fired at the point the state actually + changes, carrying counts/ids/booleans only (never rule content, prompts, or + matched source): + - `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` + - `cli_authenticated`, `cli_logged_out` + - `cli_installed`, `cli_onboarded` + - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, + `warningCount`, `filesScanned`) + - `cli_error` — a single failure event with `command` and `code` (a stable + `CliErrorCode`) +- `cli_help` — fired when the help command serves a request, with a `topic` + property (the served topic, or an index marker when invoked with no topic). + This replaces the previous `help_index`, `help_`, and `help_unknown` + events. + +Commands that carry no concrete state beyond the invocation (e.g. `info`, +`detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on +`cli_run` alone and SHALL NOT emit a bespoke event. The previous taxonomy +(`cli_`, `cli__completed`, `help_index`, `help_`, +`help_unknown`) SHALL be removed in this release; there is no dual-emit window. + +#### Scenario: Rule creation emits a concrete state event plus cli_run + +- **WHEN** a user runs `taskless rule create --from req.json` and a rule is written +- **THEN** PostHog SHALL receive one `cli_run` event with `command: "rule create"` +- **AND** SHALL receive a `cli_rule_created` event +- **AND** SHALL NOT receive `cli_rule_create` or `cli_rule_create_completed` + +#### Scenario: Help fetch emits cli_help with a topic + +- **WHEN** an agent runs `taskless help rule create` +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"` +- **AND** SHALL NOT receive a `help_rule_create` event + +#### Scenario: Help with no topic emits cli_help with an index marker + +- **WHEN** an agent runs `taskless help` +- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` marks the index + (no-argument) invocation +- **AND** SHALL NOT receive a `help_index` event + +#### Scenario: A command failure emits cli_error + +- **WHEN** a command fails with a known `CliErrorCode` +- **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` + +#### Scenario: Old event names are not emitted + +- **WHEN** any CLI command runs in this release +- **THEN** PostHog SHALL NOT receive any event named `cli__completed`, + `help_index`, `help_`, or `help_unknown` + +### Requirement: Wrong-topic re-routing is observable as a derivable funnel + +The taxonomy SHALL keep wrong-topic re-routing derivable as a funnel signal from +the new events: + +- A `cli_help { topic: A }` event not followed by the concrete event for topic A + (or by `cli_run` with the corresponding `command`), and then a subsequent + `cli_help { topic: B }`, indicates the agent fetched recipe A, did not act on + it, and re-routed to topic B. +- A `cli_help` index-marker event followed by a `cli_help { topic }` event + indicates the agent consulted the index before picking a topic (baseline). +- A `cli_help { topic }` event with no subsequent acting `cli_run` and no further + `cli_help` event indicates the agent abandoned the action. + +No additional events SHALL be added to capture this signal directly — it is +derivable from the `cli_help` / `cli_run` sequence. Dashboards SHOULD surface +re-routing rates per topic. + +#### Scenario: Funnel data supports wrong-topic detection + +- **WHEN** dashboards are constructed in PostHog +- **THEN** the `cli_help` (with `topic`) and `cli_run` (with `command`) events + SHALL be sufficient to compute "rate of `cli_help { topic }` not followed by a + corresponding acting `cli_run` within N minutes" + +### Requirement: All capture calls include standard properties + +Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `cliVersion` property (the `@taskless/cli` package version read from `package.json`), and the `scaffoldVersion` property (the `version` field from `.taskless/taskless.json`, or `0` if the manifest is absent or unreadable). When authenticated, the `groups` parameter SHALL include `{ organization: String(orgId) }`. The `cliVersion` and `scaffoldVersion` values SHALL be resolved once at telemetry initialization and attached to every subsequent `capture()` call without re-reading the source files. + +#### Scenario: Anonymous capture includes standard properties + +- **WHEN** `capture("cli_run")` is called without authentication +- **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` +- **AND** the event SHALL NOT include a `groups` parameter + +#### Scenario: Authenticated capture includes standard properties and group + +- **WHEN** `capture("cli_rule_created")` is called with authentication +- **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` +- **AND** the `groups` parameter SHALL include `{ organization: String(orgId) }` + +#### Scenario: Scaffold version falls back to 0 when manifest missing + +- **WHEN** `getTelemetry(cwd)` is initialized in a directory with no `.taskless/taskless.json` +- **THEN** every `capture()` call from the returned client SHALL include `scaffoldVersion: 0` + +#### Scenario: CLI version is resolved from package.json + +- **WHEN** `getTelemetry()` is initialized +- **THEN** `cliVersion` SHALL be read from `packages/cli/package.json` (bundled at build time or read at runtime) +- **AND** SHALL be attached to every event emitted through the returned client From 27c24cd060a8a4d771e2c706151407c4e54e359e Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 09:51:04 -0700 Subject: [PATCH 2/4] ci(openspec): Skip the archive check on non-tip stacked PRs Bring the stack-aware OpenSpec archive check (originally #31, landed in the local-rule-routing stack) into the telemetry stack so PRs below the tip show the check as skipped and only the tip runs it. Identical content to the local-rule-routing copy, so the two stacks won't conflict when both reach main. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/pr-check-openspec.yml | 44 +++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pr-check-openspec.yml b/.github/workflows/pr-check-openspec.yml index 40395f37..e1cf7ebe 100644 --- a/.github/workflows/pr-check-openspec.yml +++ b/.github/workflows/pr-check-openspec.yml @@ -1,14 +1,52 @@ name: PR OpenSpec Archive Check +# A change is archived exactly once, when ALL of its work has landed. In a +# stack of PRs, only the tip carries the archived change; the PRs below it still +# carry the in-flight change directory by design. So this check runs only on the +# tip of a stack (or a standalone PR) and is skipped on PRs that still have work +# stacked on top of them. Tip = no other OPEN PR targets this PR's head branch +# as its base. on: pull_request: - branches: [main] permissions: contents: read + pull-requests: read jobs: + stack-position: + name: Detect stack position + runs-on: ubuntu-latest + outputs: + is_tip: ${{ steps.detect.outputs.is_tip }} + steps: + - name: Determine whether this PR is the tip of its stack + id: detect + env: + GH_TOKEN: ${{ github.token }} + HEAD_REF: ${{ github.head_ref }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + + # Count OPEN PRs that target this PR's head branch as their base. + # Any such PR means work is still stacked on top → not the tip. + children=$(gh pr list --repo "$REPO" --state open --base "$HEAD_REF" \ + --json number --jq 'length' 2>/dev/null || echo 0) + children=${children:-0} + + if [ "$children" -gt 0 ]; then + echo "is_tip=false" >> "$GITHUB_OUTPUT" + echo "This PR has $children open PR(s) stacked on top — changes still in flight." + echo "The OpenSpec archive check is skipped until this PR is the tip of the stack." + else + echo "is_tip=true" >> "$GITHUB_OUTPUT" + echo "No PRs are stacked on top — this PR is the tip (or standalone); the archive check will run." + fi + check-openspec-archived: + needs: stack-position + if: needs.stack-position.outputs.is_tip == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -28,8 +66,8 @@ jobs: echo "::error::Unarchived OpenSpec change directories found under openspec/changes/:" echo "$UNARCHIVED" | sed 's|^| - |' echo "" - echo "Every change must be moved under openspec/changes/archive/ before merge." - echo "Run the openspec archive workflow (e.g. /openspec-archive-change ) and commit the result." + echo "This PR is the tip of its stack (all work landed), so the change must be archived." + echo "Move it under openspec/changes/archive/ (e.g. /openspec-archive-change ) and commit." exit 1 fi From cbe6c0a278d0db9c034020c95537715e05d5a56d Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 10:58:32 -0700 Subject: [PATCH 3/4] docs(openspec): Address review on the telemetry contract PR #33 review: - Drop cli_version from cli_run's property list across proposal/design/spec; the CLI version rides on the standard cliVersion property (no second field). - Define cli_help's topic precisely: served topic, the literal "(index)" for no-arg, and the attempted topic for unknown; add an unknown-topic scenario. - cliVersion is embedded at build time (no runtime read), matching the CLI spec. - Clarify loggedIn vs anonymous as two distinct run-context dimensions, neither of which is the independent --anonymous flag. - Align cli_check_completed counts on findings (matches the implementation). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../restructure-cli-telemetry/design.md | 5 +-- .../restructure-cli-telemetry/proposal.md | 15 ++++---- .../specs/analytics/spec.md | 36 +++++++++++-------- 3 files changed, 34 insertions(+), 22 deletions(-) diff --git a/openspec/changes/restructure-cli-telemetry/design.md b/openspec/changes/restructure-cli-telemetry/design.md index b8d93a26..017512ee 100644 --- a/openspec/changes/restructure-cli-telemetry/design.md +++ b/openspec/changes/restructure-cli-telemetry/design.md @@ -31,8 +31,9 @@ are unchanged; only the event taxonomy changes. The CLI's top-level runner (`index.ts`, around `runCommand`) wraps execution: it resolves the matched subcommand name, runs the command, and emits a single -`cli_run` with `{ command, cli_version, success, durationMs, anonymous, -loggedIn }` — on both success and failure (in a `finally`). +`cli_run` with `{ command, success, durationMs, anonymous, loggedIn }` — on both +success and failure (in a `finally`). The CLI version is not added here; it rides +on the standard `cliVersion` property already attached to every event. _Why:_ Centralizing makes the denominator impossible to forget and removes ~20 per-command start/`_completed` captures. The command name comes from the diff --git a/openspec/changes/restructure-cli-telemetry/proposal.md b/openspec/changes/restructure-cli-telemetry/proposal.md index 48b3b890..cb61c283 100644 --- a/openspec/changes/restructure-cli-telemetry/proposal.md +++ b/openspec/changes/restructure-cli-telemetry/proposal.md @@ -12,9 +12,10 @@ events for concrete state transitions worth analyzing as funnels. ## What Changes - **NEW `cli_run`** — emitted exactly once per invocation, centrally in the CLI - runner, with properties `command`, `cli_version`, `success`, `durationMs`, - `anonymous`, and `loggedIn`. This is the universal denominator; no command - emits its own "started/ran" event anymore. + runner, with properties `command`, `success`, `durationMs`, `anonymous`, and + `loggedIn` (the CLI version rides on the standard `cliVersion` property already + attached to every event — `cli_run` adds no second version field). This is the + universal denominator; no command emits its own "started/ran" event anymore. - **`cli_*` reserved for concrete state transitions** — replace the per-command start/`_completed` pairs with events that represent real outcomes: `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted`, @@ -26,9 +27,11 @@ events for concrete state transitions worth analyzing as funnels. `cli_run`'s `command`), so help intent is one event filtered by topic. - **BREAKING (analytics only)** — the previous taxonomy is a hard cut, no dual-emit window. Dashboards/funnels built on the old names must be rebuilt. -- **No per-command enrichment context** — auth state is already on `cli_run` - (`loggedIn`), which implies `--anonymous` behavior; commands do not thread - extra run-context properties in this change. +- **No per-command enrichment context** — `cli_run` already carries the two + cross-cutting dimensions we need: `loggedIn` (a valid token is present) and + `anonymous` (no authenticated identity resolved). These are distinct from the + `--anonymous` flag, which is an independent per-command invocation choice; + commands do not thread extra run-context properties in this change. ## Capabilities diff --git a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md index fc5099e5..5d325ce7 100644 --- a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md +++ b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md @@ -5,17 +5,19 @@ The CLI SHALL emit exactly one `cli_run` event per invocation, from the top-level runner rather than from individual commands. The event SHALL carry the properties `command` (the resolved subcommand name, e.g. `"rule create"` or `"help"`), -`cli_version`, `success` (boolean), `durationMs` (number), `anonymous` (boolean), -and `loggedIn` (boolean). The event SHALL be emitted on both success and failure -(from a `finally`-equivalent path), and no command SHALL emit its own -"started" or "ran" event. +`success` (boolean), `durationMs` (number), `anonymous` (boolean), and `loggedIn` +(boolean). The CLI version is provided by the standard `cliVersion` property +attached to every event (see the standard-properties requirement); `cli_run` +SHALL NOT introduce a second version field. The event SHALL be emitted on both +success and failure (from a `finally`-equivalent path), and no command SHALL emit +its own "started" or "ran" event. #### Scenario: A successful command emits one cli_run - **WHEN** a user runs `taskless info` - **THEN** PostHog SHALL receive exactly one `cli_run` event with `command: "info"`, `success: true`, a numeric `durationMs`, and the - `cli_version`, `anonymous`, and `loggedIn` properties + `anonymous` and `loggedIn` properties (plus the standard `cliVersion`) - **AND** SHALL NOT receive a separate `cli_info` or `cli_info_completed` event #### Scenario: A failing command still emits cli_run @@ -40,13 +42,14 @@ CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a - `cli_authenticated`, `cli_logged_out` - `cli_installed`, `cli_onboarded` - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, - `warningCount`, `filesScanned`) + `warningCount`, `findings`) - `cli_error` — a single failure event with `command` and `code` (a stable `CliErrorCode`) - `cli_help` — fired when the help command serves a request, with a `topic` - property (the served topic, or an index marker when invoked with no topic). - This replaces the previous `help_index`, `help_`, and `help_unknown` - events. + property. The `topic` SHALL be: the served topic for a known topic (e.g. + `"rule create"`); the exact literal `"(index)"` when invoked with no topic; + and the attempted topic string when the topic is unknown. This single event + replaces the previous `help_index`, `help_`, and `help_unknown` events. Commands that carry no concrete state beyond the invocation (e.g. `info`, `detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on @@ -67,13 +70,18 @@ Commands that carry no concrete state beyond the invocation (e.g. `info`, - **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"` - **AND** SHALL NOT receive a `help_rule_create` event -#### Scenario: Help with no topic emits cli_help with an index marker +#### Scenario: Help with no topic emits cli_help with the index marker - **WHEN** an agent runs `taskless help` -- **THEN** PostHog SHALL receive a `cli_help` event whose `topic` marks the index - (no-argument) invocation +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "(index)"` - **AND** SHALL NOT receive a `help_index` event +#### Scenario: Help with an unknown topic emits cli_help with the attempted topic + +- **WHEN** an agent runs `taskless help nope` +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "nope"` +- **AND** SHALL NOT receive a `help_unknown` event + #### Scenario: A command failure emits cli_error - **WHEN** a command fails with a known `CliErrorCode` @@ -131,8 +139,8 @@ Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `c - **WHEN** `getTelemetry(cwd)` is initialized in a directory with no `.taskless/taskless.json` - **THEN** every `capture()` call from the returned client SHALL include `scaffoldVersion: 0` -#### Scenario: CLI version is resolved from package.json +#### Scenario: CLI version is embedded at build time - **WHEN** `getTelemetry()` is initialized -- **THEN** `cliVersion` SHALL be read from `packages/cli/package.json` (bundled at build time or read at runtime) +- **THEN** `cliVersion` SHALL be the `@taskless/cli` version embedded at build time (no runtime file read), consistent with the CLI spec's build-time version requirement - **AND** SHALL be attached to every event emitted through the returned client From c35347df50efc02e63184c3bcf0b3b87e1e84c3d Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 13 Jun 2026 17:17:16 -0700 Subject: [PATCH 4/4] docs(openspec): Uppercase CLIError/CLIErrorCode in the change contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keep the proposed analytics spec/design prose in sync with the source rename — the CLI acronym is uppercase in CLIError/CLIErrorCode. Co-Authored-By: Claude Opus 4.8 --- openspec/changes/restructure-cli-telemetry/design.md | 4 ++-- .../changes/restructure-cli-telemetry/specs/analytics/spec.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openspec/changes/restructure-cli-telemetry/design.md b/openspec/changes/restructure-cli-telemetry/design.md index 017512ee..2c616578 100644 --- a/openspec/changes/restructure-cli-telemetry/design.md +++ b/openspec/changes/restructure-cli-telemetry/design.md @@ -70,7 +70,7 @@ carries no concrete state beyond the invocation, so those are covered by ### D3 — `cli_error` is the single failure event Instead of `success:false` spread across each `_completed` event, failures emit -one `cli_error { command, code }` (code from the stable `CliErrorCode` set), and +one `cli_error { command, code }` (code from the stable `CLIErrorCode` set), and `cli_run` also records `success:false`. The runner emits `cli_error` from its catch path so no command has to remember to. @@ -86,7 +86,7 @@ Dashboards are rebuilt against the new names (the proposal calls this out). - **[Centralized `cli_run` can't see command-specific context]** → By design (D-non-goal). `loggedIn` covers the only cross-cutting dimension we need now. - **[`success` detection in the runner is imperfect]** → Derive from thrown - error and `process.exitCode`; commands already use `CliError` + exit codes + error and `process.exitCode`; commands already use `CLIError` + exit codes consistently, so this is reliable. ## Open Questions diff --git a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md index 5d325ce7..47eebac2 100644 --- a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md +++ b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md @@ -44,7 +44,7 @@ CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, `warningCount`, `findings`) - `cli_error` — a single failure event with `command` and `code` (a stable - `CliErrorCode`) + `CLIErrorCode`) - `cli_help` — fired when the help command serves a request, with a `topic` property. The `topic` SHALL be: the served topic for a known topic (e.g. `"rule create"`); the exact literal `"(index)"` when invoked with no topic; @@ -84,7 +84,7 @@ Commands that carry no concrete state beyond the invocation (e.g. `info`, #### Scenario: A command failure emits cli_error -- **WHEN** a command fails with a known `CliErrorCode` +- **WHEN** a command fails with a known `CLIErrorCode` - **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` #### Scenario: Old event names are not emitted