diff --git a/.github/workflows/pr-check-openspec.yml b/.github/workflows/pr-check-openspec.yml index 40395f37..e1cf7ebe 100644 --- a/.github/workflows/pr-check-openspec.yml +++ b/.github/workflows/pr-check-openspec.yml @@ -1,14 +1,52 @@ name: PR OpenSpec Archive Check +# A change is archived exactly once, when ALL of its work has landed. In a +# stack of PRs, only the tip carries the archived change; the PRs below it still +# carry the in-flight change directory by design. So this check runs only on the +# tip of a stack (or a standalone PR) and is skipped on PRs that still have work +# stacked on top of them. Tip = no other OPEN PR targets this PR's head branch +# as its base. on: pull_request: - branches: [main] permissions: contents: read + pull-requests: read jobs: + stack-position: + name: Detect stack position + runs-on: ubuntu-latest + outputs: + is_tip: ${{ steps.detect.outputs.is_tip }} + steps: + - name: Determine whether this PR is the tip of its stack + id: detect + env: + GH_TOKEN: ${{ github.token }} + HEAD_REF: ${{ github.head_ref }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + + # Count OPEN PRs that target this PR's head branch as their base. + # Any such PR means work is still stacked on top → not the tip. + children=$(gh pr list --repo "$REPO" --state open --base "$HEAD_REF" \ + --json number --jq 'length' 2>/dev/null || echo 0) + children=${children:-0} + + if [ "$children" -gt 0 ]; then + echo "is_tip=false" >> "$GITHUB_OUTPUT" + echo "This PR has $children open PR(s) stacked on top — changes still in flight." + echo "The OpenSpec archive check is skipped until this PR is the tip of the stack." + else + echo "is_tip=true" >> "$GITHUB_OUTPUT" + echo "No PRs are stacked on top — this PR is the tip (or standalone); the archive check will run." + fi + check-openspec-archived: + needs: stack-position + if: needs.stack-position.outputs.is_tip == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -28,8 +66,8 @@ jobs: echo "::error::Unarchived OpenSpec change directories found under openspec/changes/:" echo "$UNARCHIVED" | sed 's|^| - |' echo "" - echo "Every change must be moved under openspec/changes/archive/ before merge." - echo "Run the openspec archive workflow (e.g. /openspec-archive-change ) and commit the result." + echo "This PR is the tip of its stack (all work landed), so the change must be archived." + echo "Move it under openspec/changes/archive/ (e.g. /openspec-archive-change ) and commit." exit 1 fi diff --git a/openspec/changes/restructure-cli-telemetry/.openspec.yaml b/openspec/changes/restructure-cli-telemetry/.openspec.yaml new file mode 100644 index 00000000..8fe20555 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-12 diff --git a/openspec/changes/restructure-cli-telemetry/design.md b/openspec/changes/restructure-cli-telemetry/design.md new file mode 100644 index 00000000..2c616578 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/design.md @@ -0,0 +1,95 @@ +## Context + +This design is intentionally lean — the taxonomy was decided before writing the +proposal. It records the model and the one non-trivial implementation decision +(centralizing `cli_run`); there is no open architecture to resolve. + +Today each command emits `cli_` at start and `cli__completed` at +end. The telemetry wrapper already attaches standard properties (`cli`, +`cliVersion`, `scaffoldVersion`, org group) and fails silently. Those guarantees +are unchanged; only the event taxonomy changes. + +## Goals / Non-Goals + +**Goals:** + +- One `cli_run` per invocation as the denominator, emitted centrally. +- Named `cli_*` events only for concrete state transitions. +- `cli_help { topic }` replacing the `help_*` trio. +- A hard cut — no dual-emit window. + +**Non-Goals:** + +- Changing identity, opt-out, standard properties, or silent-failure behavior. +- Per-command run-context enrichment (auth state on `cli_run` already implies + anonymous vs. authenticated). +- Capturing rule content, prompts, or matched source in any event. + +## Decisions + +### D1 — `cli_run` is emitted once, centrally, in the runner + +The CLI's top-level runner (`index.ts`, around `runCommand`) wraps execution: it +resolves the matched subcommand name, runs the command, and emits a single +`cli_run` with `{ command, success, durationMs, anonymous, loggedIn }` — on both +success and failure (in a `finally`). The CLI version is not added here; it rides +on the standard `cliVersion` property already attached to every event. + +_Why:_ Centralizing makes the denominator impossible to forget and removes ~20 +per-command start/`_completed` captures. The command name comes from the +resolved citty command; `success` is derived from whether the command threw (or +set a non-zero `process.exitCode`); `durationMs` from a start timestamp. + +_Alternative considered:_ keep per-command emission. Rejected — it's what we have +now, and it's the source of the scatter. + +### D2 — Concrete-state events fire at the state change, not command boundaries + +Each meaningful outcome emits its own event where the state actually changes: + +``` +rule create success → cli_rule_created { mode: "remote"|"static", ruleCount } +rule improve success → cli_rule_improved { ruleCount } +rule delete success → cli_rule_deleted { } +auth login success → cli_authenticated { } +auth logout success → cli_logged_out { } +init/install success → cli_installed { targets? } +onboard complete → cli_onboarded { } +check finishes → cli_check_completed{ errorCount, warningCount, filesScanned } +any command fails → cli_error { command, code } +help served → cli_help { topic } (topic = "(index)" for no-arg, + the attempted topic otherwise) +``` + +Properties are counts/ids/booleans only. + +_Why:_ These are the funnel-worthy moments. "Ran info/status/detect/verify/meta" +carries no concrete state beyond the invocation, so those are covered by +`cli_run` alone with no bespoke event. + +### D3 — `cli_error` is the single failure event + +Instead of `success:false` spread across each `_completed` event, failures emit +one `cli_error { command, code }` (code from the stable `CLIErrorCode` set), and +`cli_run` also records `success:false`. The runner emits `cli_error` from its +catch path so no command has to remember to. + +### D4 — Hard cut, no dual-emit + +Old event names are removed in the same release; nothing emits both taxonomies. +Dashboards are rebuilt against the new names (the proposal calls this out). + +## Risks / Trade-offs + +- **[Dashboards/funnels on old names break]** → Accepted and documented; this is + a deliberate hard cut. The new taxonomy is simpler to rebuild against. +- **[Centralized `cli_run` can't see command-specific context]** → By design + (D-non-goal). `loggedIn` covers the only cross-cutting dimension we need now. +- **[`success` detection in the runner is imperfect]** → Derive from thrown + error and `process.exitCode`; commands already use `CLIError` + exit codes + consistently, so this is reliable. + +## Open Questions + +- None. (`cli_check_completed` property names — `errorCount`/`warningCount` — are + a naming detail finalized during implementation against the check result shape.) diff --git a/openspec/changes/restructure-cli-telemetry/proposal.md b/openspec/changes/restructure-cli-telemetry/proposal.md new file mode 100644 index 00000000..cb61c283 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/proposal.md @@ -0,0 +1,68 @@ +## Why + +The CLI's PostHog taxonomy emits a `cli_` start event plus a +`cli__completed` event for nearly every command — roughly two events per +invocation, with the "ran" signal scattered across ~20 distinct event names. +There is no single denominator for "the CLI was invoked," so basic questions +("how many runs of each command", "overall success rate") require stitching many +event names together. Following PostHog's own guidance, we want one top-level +`cli_run` event per invocation as the denominator, and to reserve named `cli_*` +events for concrete state transitions worth analyzing as funnels. + +## What Changes + +- **NEW `cli_run`** — emitted exactly once per invocation, centrally in the CLI + runner, with properties `command`, `success`, `durationMs`, `anonymous`, and + `loggedIn` (the CLI version rides on the standard `cliVersion` property already + attached to every event — `cli_run` adds no second version field). This is the + universal denominator; no command emits its own "started/ran" event anymore. +- **`cli_*` reserved for concrete state transitions** — replace the per-command + start/`_completed` pairs with events that represent real outcomes: + `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted`, + `cli_authenticated`, `cli_logged_out`, `cli_installed`, `cli_onboarded`, + `cli_check_completed` (error counts only — no rule content or matched code), + and a unified `cli_error` (`command`, `code`). +- **`cli_help { topic }`** — replace `help_index` / `help_` / `help_unknown` + with a single `cli_help` event carrying a `topic` field (parallel to + `cli_run`'s `command`), so help intent is one event filtered by topic. +- **BREAKING (analytics only)** — the previous taxonomy is a hard cut, no + dual-emit window. Dashboards/funnels built on the old names must be rebuilt. +- **No per-command enrichment context** — `cli_run` already carries the two + cross-cutting dimensions we need: `loggedIn` (a valid token is present) and + `anonymous` (no authenticated identity resolved). These are distinct from the + `--anonymous` flag, which is an independent per-command invocation choice; + commands do not thread extra run-context properties in this change. + +## Capabilities + +### New Capabilities + +(none) + +### Modified Capabilities + +- `analytics`: Rewrite the CLI event taxonomy — introduce `cli_run` as the + once-per-invocation denominator, replace start/`_completed` pairs with + concrete-state `cli_*` events, collapse the `help_*` events into + `cli_help { topic }`, and update the wrong-topic re-routing funnel to derive + from the new events. Standard-property guarantees are unchanged. + +## Impact + +- **Centralized emission**: `packages/cli/src/index.ts` runner gains the + `cli_run` emission (resolved command name + success + duration), so the event + cannot be forgotten per command. +- **Per-command call sites** (`commands/{rules,auth,init,check,info,onboard}.ts`, + `wizard/index.ts`): remove start/`_completed` captures; emit concrete-state + events at the point the state actually changes. +- **Help** (`commands/help.ts`): `help_index`/`help_`/`help_unknown` → + `cli_help { topic }`. +- **Telemetry module** (`telemetry.ts`): standard properties unchanged; the + wrapper continues to attach `cli`, `cliVersion`, `scaffoldVersion`, and org + group to every event. +- **Tests**: `test/telemetry.test.ts` and command tests that assert event names + update to the new taxonomy. +- **Privacy**: concrete events carry counts/ids/booleans only — never rule + content, prompts, or matched source. +- **No new dependencies**; no change to opt-out, identity, or silent-failure + behavior. diff --git a/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md new file mode 100644 index 00000000..47eebac2 --- /dev/null +++ b/openspec/changes/restructure-cli-telemetry/specs/analytics/spec.md @@ -0,0 +1,146 @@ +## ADDED Requirements + +### Requirement: Every invocation emits exactly one cli_run event + +The CLI SHALL emit exactly one `cli_run` event per invocation, from the top-level +runner rather than from individual commands. The event SHALL carry the properties +`command` (the resolved subcommand name, e.g. `"rule create"` or `"help"`), +`success` (boolean), `durationMs` (number), `anonymous` (boolean), and `loggedIn` +(boolean). The CLI version is provided by the standard `cliVersion` property +attached to every event (see the standard-properties requirement); `cli_run` +SHALL NOT introduce a second version field. The event SHALL be emitted on both +success and failure (from a `finally`-equivalent path), and no command SHALL emit +its own "started" or "ran" event. + +#### Scenario: A successful command emits one cli_run + +- **WHEN** a user runs `taskless info` +- **THEN** PostHog SHALL receive exactly one `cli_run` event with + `command: "info"`, `success: true`, a numeric `durationMs`, and the + `anonymous` and `loggedIn` properties (plus the standard `cliVersion`) +- **AND** SHALL NOT receive a separate `cli_info` or `cli_info_completed` event + +#### Scenario: A failing command still emits cli_run + +- **WHEN** a command exits with an error +- **THEN** PostHog SHALL receive one `cli_run` event with `success: false` + +## MODIFIED Requirements + +### Requirement: CLI events use cli\_ prefix + +CLI events SHALL use the `cli_` prefix, with the taxonomy organized as a +`cli_run` denominator plus concrete state-transition events: + +- `cli_run` — exactly one per invocation (see the dedicated requirement). This + replaces every previous `cli_` start event and `cli__completed` + event; the `success`/`durationMs`/`command` signal lives here. +- Concrete state-transition events, each fired at the point the state actually + changes, carrying counts/ids/booleans only (never rule content, prompts, or + matched source): + - `cli_rule_created`, `cli_rule_improved`, `cli_rule_deleted` + - `cli_authenticated`, `cli_logged_out` + - `cli_installed`, `cli_onboarded` + - `cli_check_completed` — error/warning counts only (e.g. `errorCount`, + `warningCount`, `findings`) + - `cli_error` — a single failure event with `command` and `code` (a stable + `CLIErrorCode`) +- `cli_help` — fired when the help command serves a request, with a `topic` + property. The `topic` SHALL be: the served topic for a known topic (e.g. + `"rule create"`); the exact literal `"(index)"` when invoked with no topic; + and the attempted topic string when the topic is unknown. This single event + replaces the previous `help_index`, `help_`, and `help_unknown` events. + +Commands that carry no concrete state beyond the invocation (e.g. `info`, +`detect`, `update`, `auth status`, `rule verify`, `rule meta`) SHALL rely on +`cli_run` alone and SHALL NOT emit a bespoke event. The previous taxonomy +(`cli_`, `cli__completed`, `help_index`, `help_`, +`help_unknown`) SHALL be removed in this release; there is no dual-emit window. + +#### Scenario: Rule creation emits a concrete state event plus cli_run + +- **WHEN** a user runs `taskless rule create --from req.json` and a rule is written +- **THEN** PostHog SHALL receive one `cli_run` event with `command: "rule create"` +- **AND** SHALL receive a `cli_rule_created` event +- **AND** SHALL NOT receive `cli_rule_create` or `cli_rule_create_completed` + +#### Scenario: Help fetch emits cli_help with a topic + +- **WHEN** an agent runs `taskless help rule create` +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "rule create"` +- **AND** SHALL NOT receive a `help_rule_create` event + +#### Scenario: Help with no topic emits cli_help with the index marker + +- **WHEN** an agent runs `taskless help` +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "(index)"` +- **AND** SHALL NOT receive a `help_index` event + +#### Scenario: Help with an unknown topic emits cli_help with the attempted topic + +- **WHEN** an agent runs `taskless help nope` +- **THEN** PostHog SHALL receive a `cli_help` event with `topic: "nope"` +- **AND** SHALL NOT receive a `help_unknown` event + +#### Scenario: A command failure emits cli_error + +- **WHEN** a command fails with a known `CLIErrorCode` +- **THEN** PostHog SHALL receive a `cli_error` event with `command` and `code` + +#### Scenario: Old event names are not emitted + +- **WHEN** any CLI command runs in this release +- **THEN** PostHog SHALL NOT receive any event named `cli__completed`, + `help_index`, `help_`, or `help_unknown` + +### Requirement: Wrong-topic re-routing is observable as a derivable funnel + +The taxonomy SHALL keep wrong-topic re-routing derivable as a funnel signal from +the new events: + +- A `cli_help { topic: A }` event not followed by the concrete event for topic A + (or by `cli_run` with the corresponding `command`), and then a subsequent + `cli_help { topic: B }`, indicates the agent fetched recipe A, did not act on + it, and re-routed to topic B. +- A `cli_help` index-marker event followed by a `cli_help { topic }` event + indicates the agent consulted the index before picking a topic (baseline). +- A `cli_help { topic }` event with no subsequent acting `cli_run` and no further + `cli_help` event indicates the agent abandoned the action. + +No additional events SHALL be added to capture this signal directly — it is +derivable from the `cli_help` / `cli_run` sequence. Dashboards SHOULD surface +re-routing rates per topic. + +#### Scenario: Funnel data supports wrong-topic detection + +- **WHEN** dashboards are constructed in PostHog +- **THEN** the `cli_help` (with `topic`) and `cli_run` (with `command`) events + SHALL be sufficient to compute "rate of `cli_help { topic }` not followed by a + corresponding acting `cli_run` within N minutes" + +### Requirement: All capture calls include standard properties + +Every `capture()` call SHALL include the `cli` property (anonymous UUID), the `cliVersion` property (the `@taskless/cli` package version read from `package.json`), and the `scaffoldVersion` property (the `version` field from `.taskless/taskless.json`, or `0` if the manifest is absent or unreadable). When authenticated, the `groups` parameter SHALL include `{ organization: String(orgId) }`. The `cliVersion` and `scaffoldVersion` values SHALL be resolved once at telemetry initialization and attached to every subsequent `capture()` call without re-reading the source files. + +#### Scenario: Anonymous capture includes standard properties + +- **WHEN** `capture("cli_run")` is called without authentication +- **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` +- **AND** the event SHALL NOT include a `groups` parameter + +#### Scenario: Authenticated capture includes standard properties and group + +- **WHEN** `capture("cli_rule_created")` is called with authentication +- **THEN** the event SHALL include `{ cli: anonymousUuid, cliVersion: , scaffoldVersion: }` +- **AND** the `groups` parameter SHALL include `{ organization: String(orgId) }` + +#### Scenario: Scaffold version falls back to 0 when manifest missing + +- **WHEN** `getTelemetry(cwd)` is initialized in a directory with no `.taskless/taskless.json` +- **THEN** every `capture()` call from the returned client SHALL include `scaffoldVersion: 0` + +#### Scenario: CLI version is embedded at build time + +- **WHEN** `getTelemetry()` is initialized +- **THEN** `cliVersion` SHALL be the `@taskless/cli` version embedded at build time (no runtime file read), consistent with the CLI spec's build-time version requirement +- **AND** SHALL be attached to every event emitted through the returned client