Build a privacy-first product telemetry MVP for launch monitoring

[boycez]

Goal

Build a privacy-first telemetry MVP before launch so the team can monitor rollout health, understand adoption, prioritize product improvements, and correlate user feedback with product behavior.

Essential telemetry

• Install funnel: installer started, prerequisite checks, step duration, success/failure, failure category, upgrade vs. clean install, and uninstall.
• Launch and health: app launch, time to interactive, gateway startup duration, startup failure category, crash/unhandled error, and app version.
• Activation funnel: first successful setup, provider configured, first chat started, first successful response, and time to first value.
• Engagement and retention: privacy-safe active day/session counts, sessions per user, returning users, and coarse session duration buckets.
• Feature adoption: agent selection, skill usage, attachments, channels, scheduled tasks, sandbox enablement, and key Settings changes using enumerated event values.
• Reliability: request success/failure, latency buckets, aborts, gateway reconnects/restarts, tool failures, update checks, and update outcomes.
• Feedback linkage: optional feedback ID and app/session correlation ID so interviews, surveys, and support reports can be associated with aggregate diagnostics.

MVP pipeline

• Event contract: versioned event names, typed properties, required dimensions, ownership, retention, and deprecation rules.
• Client instrumentation: one shared telemetry API for installer, Electron main process, renderer, gateway lifecycle, and sandbox boundaries.
• Privacy controls: explicit disclosure/consent as required, stable pseudonymous installation ID, data minimization, and no prompts, responses, file paths, file contents, credentials, API keys, or direct PII.
• Local delivery: bounded on-disk queue, batching, retry with backoff, offline support, event-size limits, and failure isolation so telemetry never blocks product flows.
• Ingestion and storage: authenticated endpoint, schema validation, environment separation, deduplication, retention policy, and access controls.
• Dashboards: launch funnel, activation, DAU/WAU, retention cohorts, feature adoption, reliability, latency, and version/rollout comparisons.
• Alerts: install success regression, crash/startup failure spikes, gateway health regression, request failure spikes, and telemetry pipeline health.
• Release validation: distinguish version, channel, locale, OS version, architecture, install type, and experiment/flight without collecting identifying device details.

Acceptance criteria

• A reviewed telemetry specification defines the MVP events, properties, privacy classification, owner, and retention for each event.
• Installer, app startup, activation, engagement, feature adoption, reliability, and feedback-linkage events are implemented through one shared API.
• No prompt/response content, paths, filenames, credentials, keys, or direct PII are emitted.
• Events queue safely offline and upload in bounded batches without affecting app startup or interaction latency.
• Ingestion validates schemas and rejects malformed or disallowed properties.
• Dashboards show install funnel, activation funnel, active usage, retention, feature adoption, reliability, and version comparisons.
• Alerts exist for critical install, startup, crash, gateway, and request regressions.
• Telemetry can be disabled according to the approved privacy/consent design.
• Automated tests cover schema validation, redaction/allowlisting, queue bounds, retries, deduplication, and disabled mode.
• A pre-production environment proves end-to-end event delivery before launch.

Initial success metrics

• Install success rate and top failure categories
• Median/p95 install and startup duration
• Setup-to-first-successful-response conversion
• Day-1 and Day-7 retention
• Successful response and tool execution rates
• Crash-free and startup-successful sessions
• Adoption of agents, skills, channels, attachments, scheduling, and sandbox controls

Notes

Use allowlisted structured properties rather than arbitrary dictionaries. Telemetry must be operationally independent from support logs: support logs may contain detailed local diagnostics, while product telemetry should remain aggregate, bounded, and privacy-safe by design.