feat(cli): DO NOT MERGE emit telemetry for the validate action - #1864
Draft
iankhou wants to merge 1 commit into
Draft
feat(cli): DO NOT MERGE emit telemetry for the validate action#1864iankhou wants to merge 1 commit into
iankhou wants to merge 1 commit into
Conversation
Adds a VALIDATE telemetry event covering the validation phase of cdk validate (offline report collection and online CloudFormation validation, excluding synthesis), with counters for offline violations per severity, offlineWouldFailDeploy (offline validation found a report that would have failed cdk deploy), and onlineViolations.
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1864 +/- ##
==========================================
+ Coverage 90.31% 91.01% +0.69%
==========================================
Files 80 80
Lines 12159 12297 +138
Branches 1725 1768 +43
==========================================
+ Hits 10982 11192 +210
+ Misses 1143 1068 -75
- Partials 34 37 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds telemetry for non-watch cdk validate runs, including duration, errors, and validation-result counters.
Changes:
- Introduces the
VALIDATEtelemetry event and span. - Counts offline and online validation outcomes.
- Adds unit tests and updated I/O snapshots.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
packages/aws-cdk/lib/cli/cdk-toolkit.ts |
Wraps validation in a telemetry span. |
packages/aws-cdk/lib/cli/io-host/cli-io-host.ts |
Converts validation messages into telemetry events. |
packages/aws-cdk/lib/cli/telemetry/messages.ts |
Defines validation span messages. |
packages/aws-cdk/lib/cli/telemetry/schema.ts |
Adds the VALIDATE event type. |
packages/aws-cdk/lib/api-private.ts |
Exposes validation counter utilities internally. |
packages/@aws-cdk/toolkit-lib/lib/toolkit/toolkit.ts |
Uses the shared online-validation plugin name. |
packages/@aws-cdk/toolkit-lib/lib/toolkit/private/validation-report.ts |
Defines the online-validation plugin constant. |
packages/@aws-cdk/toolkit-lib/lib/toolkit/private/count-validation-results.ts |
Calculates validation telemetry counters. |
packages/@aws-cdk/toolkit-lib/test/toolkit/count-validation-results.test.ts |
Tests validation counter calculation. |
packages/aws-cdk/test/cli/io-host/cli-io-host.test.ts |
Tests VALIDATE event emission. |
packages/aws-cdk/test/commands/validate.test.ts |
Tests validation span outcomes. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/with_violations_reports_multiple_violations_from_multiple_plugins.ndjson |
Updates validation output snapshot. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/with_violations_reports_failure_when_validation_report_has_violations.ndjson |
Updates failure output snapshot. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_ends_the_VALIDATE_span_with_the_error_name_when_the_engine_crashes.ndjson |
Captures crash telemetry output. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_emits_a_VALIDATE_span_end_message_with_offline_violation_counters.ndjson |
Captures violation-counter output. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/telemetry_emits_a_VALIDATE_span_end_message_even_when_no_violations_are_found.ndjson |
Captures successful validation telemetry. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/stack_selection_validates_a_single_selected_stack.ndjson |
Updates stack-selection output. |
packages/aws-cdk/test/commands/__io_snapshots__/validate/no_violations_reports_success_when_no_validation_report_exists.ndjson |
Updates no-report output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+15
to
+16
| const offline = result.pluginReports.filter((r) => r.pluginName !== ONLINE_VALIDATION_PLUGIN_NAME); | ||
| const online = result.pluginReports.filter((r) => r.pluginName === ONLINE_VALIDATION_PLUGIN_NAME); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE until we enable the VALIDATE event on the Telemetry backend.
The
cdk validateaction (unstable) runs offline and online validation but currently emits no telemetry. This adds aVALIDATEtelemetry event so we can measure, at minimum, that the validation engine ran (start time, duration, success/failure), and whether offline validation caught errors that would have failed a deployment.Not included
cdk validate --watchdoes not emit VALIDATE events.We must enable our Telemetry backend to accept VALIDATE events.
Checklist
count-validation-results.test.ts,cli-io-host.test.ts,validate.test.ts+ io-snapshots)By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license