Skip to content

ci: unique identifier per integration-suite attempt; configurable stack prefix#1287

Merged
jchrostek-dd merged 1 commit into
mainfrom
john/unique-job-id
Jun 24, 2026
Merged

ci: unique identifier per integration-suite attempt; configurable stack prefix#1287
jchrostek-dd merged 1 commit into
mainfrom
john/unique-job-id

Conversation

@jchrostek-dd

@jchrostek-dd jchrostek-dd commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Overview

Integration-suite deploys intermittently fail at cdk deploy (before any test runs) because of resource conflict. On some retries, the previous run resources were not cleaned up correctly.

Fix

  • Include CI_JOB_ID in the identifier so every attempt gets unique names and there won't be resource conflict.
  • Also shorted 'integ' to 'it' so we are less likely to hit the 64 character limit on cdk resources names.

Note - The integration tests are failing due to 429 error, too many requests. Will address this in follow up PR.

@jchrostek-dd jchrostek-dd changed the title ci: unique identifier per integration-suite attempt; shorten stack prefix ci: unique identifier per integration-suite attempt; configurable stack prefix Jun 24, 2026
@jchrostek-dd jchrostek-dd force-pushed the john/unique-job-id branch 2 times, most recently from d6ca278 to b5e32de Compare June 24, 2026 12:12
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jun 24, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 7 Pipeline jobs failed

DataDog/datadog-lambda-extension | integration-suite: [lmi]   View in Datadog   GitLab

DataDog/datadog-lambda-extension | integration-suite: [on-demand]   View in Datadog   GitLab

DataDog/datadog-lambda-extension | integration-suite: [oom]   View in Datadog   GitLab

View all 7 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: af5ebda | Docs | Datadog PR Page | Give us feedback!

Integration-suite deploys intermittently failed at 'cdk deploy' (before any
test runs) because the test stack was named only after the commit SHA and the
job retries. When a retry's teardown deleted a function's log group while
another attempt was still invoking that function, the Lambda service recreated
the group as an unmanaged, never-expire group. It survived 'cdk destroy' and
blocked the next attempt, which reused the same name and failed to auto-import
it (the construct is RemovalPolicy.DESTROY, not Retain).

Include CI_JOB_ID in the identifier so every attempt (including retries) gets
unique stack, function, and log group names that never recur, so a leftover
group can no longer collide with a later deploy. CI_JOB_ID is unique per retry
and available in both script and after_script, so teardown still targets the
right stack. The identifier also carries a short 'it-' prefix (replacing the
longer 'integ-') to keep generated Lambda names under the 64-char limit; the
shared integ-auth-delegated-role and capacity provider are unchanged.
@jchrostek-dd jchrostek-dd marked this pull request as ready for review June 24, 2026 13:16
@jchrostek-dd jchrostek-dd requested a review from a team as a code owner June 24, 2026 13:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces CloudFormation/CDK resource conflicts in the integration test pipeline by ensuring each integration-suite retry attempt uses a unique stack identifier (via CI_JOB_ID) and by shortening the default stack-name prefix from integ to it to reduce the chance of hitting CDK/AWS name-length limits.

Changes:

  • Make CI stack identifiers unique per job attempt: it-${CI_COMMIT_SHORT_SHA}-${CI_JOB_ID}.
  • Replace per-file getIdentifier() usage with a shared exported IDENTIFIER constant.
  • Remove the integ- prefix from per-suite stack IDs/names and rely on the it--prefixed identifier.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated no comments.

Show a summary per file
File Description
integration-tests/config.ts Prefixes local fallback identifiers with it- and exports a shared IDENTIFIER constant derived from getIdentifier().
integration-tests/bin/app.ts Updates CDK stack IDs to use ${IDENTIFIER}-<suite> naming (dropping the integ- prefix).
integration-tests/tests/snapstart.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/payload-size.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/otlp.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/oom.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/on-demand.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/lmi.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/lmi-oom.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/custom-metrics.test.ts Uses shared IDENTIFIER to build the suite stack name.
integration-tests/tests/auth.test.ts Uses shared IDENTIFIER to build the suite stack name.
.gitlab/templates/pipeline.yaml.tpl Sets unique IDENTIFIER per job attempt and updates deploy/destroy stack name patterns to match the new scheme.

@jchrostek-dd jchrostek-dd merged commit fbd190d into main Jun 24, 2026
55 of 61 checks passed
@jchrostek-dd jchrostek-dd deleted the john/unique-job-id branch June 24, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants