Skip to content

feat(cli): DO NOT MERGE emit telemetry for the validate action - #1864

Draft
iankhou wants to merge 1 commit into
mainfrom
telemetry-validate-event
Draft

feat(cli): DO NOT MERGE emit telemetry for the validate action#1864
iankhou wants to merge 1 commit into
mainfrom
telemetry-validate-event

Conversation

@iankhou

@iankhou iankhou commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

DO NOT MERGE until we enable the VALIDATE event on the Telemetry backend.

The cdk validate action (unstable) runs offline and online validation but currently emits no telemetry. This adds a VALIDATE telemetry event so we can measure, at minimum, that the validation engine ran (start time, duration, success/failure), and whether offline validation caught errors that would have failed a deployment.

Not included

cdk validate --watch does not emit VALIDATE events.

We must enable our Telemetry backend to accept VALIDATE events.

Checklist

  • Unit tests added/updated (count-validation-results.test.ts, cli-io-host.test.ts, validate.test.ts + io-snapshots)
  • Integration tests: n/a — no new resource types or cross-service interactions
  • No manual edits to generated files

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Adds a VALIDATE telemetry event covering the validation phase of cdk validate (offline report collection and online CloudFormation validation, excluding synthesis), with counters for offline violations per severity, offlineWouldFailDeploy (offline validation found a report that would have failed cdk deploy), and onlineViolations.
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@aws-cdk-automation
aws-cdk-automation requested a review from a team August 20, 2026 14:27
@iankhou
iankhou deployed to no-approval August 20, 2026 14:27 — with GitHub Actions Active
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.02326% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.01%. Comparing base (88f6ad1) to head (05059e6).
⚠️ Report is 19 commits behind head on main.

Files with missing lines Patch % Lines
packages/aws-cdk/lib/cli/io-host/cli-io-host.ts 33.33% 2 Missing ⚠️
packages/aws-cdk/lib/cli/cdk-toolkit.ts 95.83% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1864      +/-   ##
==========================================
+ Coverage   90.31%   91.01%   +0.69%     
==========================================
  Files          80       80              
  Lines       12159    12297     +138     
  Branches     1725     1768      +43     
==========================================
+ Hits        10982    11192     +210     
+ Misses       1143     1068      -75     
- Partials       34       37       +3     
Flag Coverage Δ
suite.unit 91.01% <93.02%> (+0.69%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@iankhou iankhou changed the title feat(cli): emit telemetry for the validate action feat(cli): DO NOT MERGE emit telemetry for the validate action Aug 20, 2026
@iankhou
iankhou requested a balanced review from Copilot August 20, 2026 20:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds telemetry for non-watch cdk validate runs, including duration, errors, and validation-result counters.

Changes:

  • Introduces the VALIDATE telemetry event and span.
  • Counts offline and online validation outcomes.
  • Adds unit tests and updated I/O snapshots.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/aws-cdk/lib/cli/cdk-toolkit.ts Wraps validation in a telemetry span.
packages/aws-cdk/lib/cli/io-host/cli-io-host.ts Converts validation messages into telemetry events.
packages/aws-cdk/lib/cli/telemetry/messages.ts Defines validation span messages.
packages/aws-cdk/lib/cli/telemetry/schema.ts Adds the VALIDATE event type.
packages/aws-cdk/lib/api-private.ts Exposes validation counter utilities internally.
packages/@aws-cdk/toolkit-lib/lib/toolkit/toolkit.ts Uses the shared online-validation plugin name.
packages/@aws-cdk/toolkit-lib/lib/toolkit/private/validation-report.ts Defines the online-validation plugin constant.
packages/@aws-cdk/toolkit-lib/lib/toolkit/private/count-validation-results.ts Calculates validation telemetry counters.
packages/@aws-cdk/toolkit-lib/test/toolkit/count-validation-results.test.ts Tests validation counter calculation.
packages/aws-cdk/test/cli/io-host/cli-io-host.test.ts Tests VALIDATE event emission.
packages/aws-cdk/test/commands/validate.test.ts Tests validation span outcomes.
packages/aws-cdk/test/commands/__io_snapshots__/validate/with_violations_reports_multiple_violations_from_multiple_plugins.ndjson Updates validation output snapshot.
packages/aws-cdk/test/commands/__io_snapshots__/validate/with_violations_reports_failure_when_validation_report_has_violations.ndjson Updates failure output snapshot.
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_ends_the_VALIDATE_span_with_the_error_name_when_the_engine_crashes.ndjson Captures crash telemetry output.
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_emits_a_VALIDATE_span_end_message_with_offline_violation_counters.ndjson Captures violation-counter output.
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_emits_a_VALIDATE_span_end_message_even_when_no_violations_are_found.ndjson Captures successful validation telemetry.
packages/aws-cdk/test/commands/__io_snapshots__/validate/stack_selection_validates_a_single_selected_stack.ndjson Updates stack-selection output.
packages/aws-cdk/test/commands/__io_snapshots__/validate/no_violations_reports_success_when_no_validation_report_exists.ndjson Updates no-report output.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +15 to +16
const offline = result.pluginReports.filter((r) => r.pluginName !== ONLINE_VALIDATION_PLUGIN_NAME);
const online = result.pluginReports.filter((r) => r.pluginName === ONLINE_VALIDATION_PLUGIN_NAME);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants