Skip to content

Repair DataOps OIDC SES configuration-set tag-on-create permission #129

Description

@alexeygrigorev

Extend the DataOps OIDC deploy role for reviewed sponsor communications

Status: in progress — PM accepted exact source; awaiting commit/integration and HUMAN apply
Tags: bug, infra, testing, P1
Depends on: None — implementation may run alongside #114; acceptance must recompare the final agent-approved #114 SAM resource graph
Blocks: #128 default-off deployment/HUMAN rollout; #134 On-Call closure after its application checks passed
Current evidence: run 30591398299 passed the complete app check job, then CloudFormation was denied ses:TagResource on the exact dataops-sponsor-reviewed-send-disabled configuration-set ARN and entered rollback.

Scope

Repair only the newly proven CloudFormation tag-on-create gap in the existing SponsorCommunicationsServiceDeployPolicy in DataTalksClub/aws-infra. The denied resource is exactly:

arn:${AWS::Partition}:ses:${FullDocsRegion}:${AWS::AccountId}:configuration-set/${SponsorSesConfigurationSetName}

The observed deployed name is dataops-sponsor-reviewed-send-disabled. Add no SES sending or runtime capability and no wildcard SES resource. Preserve the already effective Gate A/Gate B policies, OIDC trust, attachments, and unrelated dirty ../aws-infra checkout. Implementation must use a fresh isolated worktree from current aws-infra/origin/main (currently ddbaf2893bf2985d32e5a33310fe57e2f4ac9452; fetch and recheck before work).

Acceptance Criteria

  • A fresh isolated aws-infra worktree starts from current origin/main; the dirty sibling checkout is untouched, and the candidate changes only sandbox/dataops/template.github-actions.yaml plus its focused policy test unless a narrowly justified public-safe fixture is required.
  • The existing SponsorCommunicationsServiceDeployPolicy adds only ses:TagResource for the exact parameterized configuration-set ARN above. No Sponsor permission moves to the inline policy, no new managed policy/attachment is created, and Gate A/Gate B trust and documents remain otherwise unchanged.
  • The implementation proves SES/CloudFormation tag-condition semantics. If aws:TagKeys safely applies, require a present key set limited by ForAllValues:StringEquals to exactly aws:cloudformation:logical-id, aws:cloudformation:stack-id, and aws:cloudformation:stack-name. If that condition would reject legitimate CloudFormation system tagging or is unsupported, record authoritative evidence and use only the exact-action/exact-ARN fallback—never a broader resource/action.
  • The deploy role gains no ses:UntagResource, ses:*, wildcard SES ARN, send/bulk/raw-send, identity, template, contact-list, suppression-list, account, or runtime permission. Existing SES lifecycle actions remain exact.
  • Parsed structural and negative-mutation tests prove the chosen tag statement/action/resource/condition, reject every prohibited broadening, preserve OIDC trust and all prior Gate A/Gate B capabilities, and keep policy-size plus attachment headroom. SAM lint, all sandbox/dataops tests, and git diff --check pass.
  • The handoff records the exact base, changed-path inventory, binary diff hash, source-template hash, focused-test hash, and resolved compact Sponsor service-policy hash/size; an independent IAM/security reviewer accepts that exact fingerprint.
  • The accepted aws-infra patch is committed/integrated to current main with Refs DataTalksClub/dataops#129 and no closing keyword; source integration alone does not authorize or perform AWS/application deployment.
  • [HUMAN/AWS] After run 30591398299 reaches a terminal rollback state, a credentialed operator creates and independently reviews a dataops-github-actions CloudFormation change set with CAPABILITY_NAMED_IAM. It must show only the expected in-place Sponsor service managed-policy document update, with no deploy-role replacement, trust/attachment change, or unrelated resource change; apply it sequentially, require terminal UPDATE_COMPLETE, and verify the effective resolved document/fingerprint and stack drift without manual IAM edits.
  • [HUMAN] Rerun the normal DataOps main GitHub Actions deployment with Sponsor sending disabled. The full application checks must remain green and CloudFormation must pass configuration-set tagging plus the default-off Roll out and cut over the Telegram conversational MVP #128/Use one Berlin business date in #106 scheduling and default-SPA E2E #134 deployment path. Use normal CI only—no manual app/Lambda deploy—and record sanitized On-Call evidence before closing Repair DataOps OIDC SES configuration-set tag-on-create permission #129.

Verification

sam validate --lint --template-file sandbox/dataops/template.github-actions.yaml
uv run --with pytest python -m pytest sandbox/dataops/tests -q
git diff --check

Also resolve and compact-serialize the Sponsor service policy under deterministic pseudo/default values, compare it to the effective Gate A baseline, and run mutations for wildcard resource/action, SES send families, UntagResource, extra tag keys, missing/non-limiting tag-key conditions when the condition path is used, inline-policy relocation, trust/attachment drift, and policy quota regression.

Out of Scope for this follow-up

SES sending, identities, templates, runtime Lambda permissions, Sponsor content/secrets, application SAM/runtime changes, manual IAM/app deployment, speculative tagging actions, and the separate #128 HUMAN canaries. #134 application calendar checks are already green in the triggering run and are not modified here.

Historical original scope and rationale

The detailed original #129 design below records the already integrated/effective Gate A work. It is context, not authorization to broaden this exact tag-on-create repair.

Update the shared DataTalksClub/aws-infra source-of-truth template at:

sandbox/dataops/template.github-actions.yaml

so the existing dataops-github-actions-deploy OIDC role can let CloudFormation create/update/roll back the exact reviewed sponsor-communication resources declared by #114. Change no DataOps runtime code and apply no AWS stack during agent implementation/review.

The implementation must start in an isolated aws-infra worktree/branch from the latest origin/main, not the current dirty sibling checkout. At grooming time, origin/main is cc96704 and includes merged PR #18 / commit 691666a; the engineer must fetch again and use the then-current remote head. Suggested ownership:

worktree: /home/alexey/git/.worktrees/aws-infra-129
branch: issue-129-sponsor-communications-deploy-role

Do not stash, reset, overwrite, or merge the dirty ../aws-infra checkout. Preserve unrelated Datamailer/bookkeeping/auth work. The final source change belongs in DataTalksClub/aws-infra, not copied into dataops.

Authoritative resource graph

The current #114 generated SAM handoff declares these deployment resources:

  • AWS::Serverless::Function logical IDs:
    • SponsorSendWorkerFunction
    • SponsorSesEventFunction
    • SponsorPrivateArchiveFunction
  • generated Lambda execution roles:
    • SponsorSendWorkerFunctionRole
    • SponsorSesEventFunctionRole
    • SponsorPrivateArchiveFunctionRole
  • corresponding /aws/lambda/<physical-function-name> log groups;
  • one DynamoDB Stream event-source mapping for SponsorSendWorkerFunction;
  • one Lambda EventInvokeConfig for SponsorSesEventFunction;
  • SponsorCommunicationFailureQueue, retained and encrypted with alias/aws/sqs;
  • SponsorSesConfigurationSet and SponsorSesEventDestination;
  • stack-scoped scheduled/EventBridge rules, including SponsorSesEventRule;
  • SponsorCommunicationPrivateArchiveBucket, retained/versioned/private/KMS-encrypted with ownership controls;
  • SponsorCommunicationPrivateArchiveKey, retained with rotation.

Before Tester/PM acceptance, compare the proposed policy to the final accepted #114 infra/template.full.yaml and a fresh sam build/generated resource inventory. Remove permissions for resources that #114 drops; add nothing for speculative resources. If a final #114 logical ID changes, update exact patterns/tests before merge.

Preserve #124 / PR #18

PR #18 is already merged into aws-infra/origin/main but has not yet been applied to AWS. Preserve all of its effective permissions and tests:

  • ConversationalExecutionWorkerFunction function/role/log patterns;
  • ConversationalExecutionFailureQueue exact queue lifecycle;
  • worker-conditioned Lambda event-source mapping create/read/update/delete;
  • mapping tag lifecycle;
  • existing stack-scoped EventBridge recovery-rule permissions.

#124 is not an implementation dependency for #129. A credentialed operator may apply PR #18 before #129 finishes. If the operator waits, the later #129 stack update must include and verify PR #18 as part of the same effective role update. #129 must not regress or silently relocate those permissions without an explicit equivalent-policy test.

Policy-size design

Do not append sponsor permissions to the existing inline role policy.

At grooming time, resolving origin/main with default parameters/account/region and compact-serializing dataops-v1-deploy produces approximately 9,341 non-whitespace characters, only 899 below IAM’s aggregate 10,240-character inline-policy quota for a role. Splitting into more inline policies would not help because the quota is aggregate across a role’s inline policies.

Add two stack-owned customer-managed policies attached only to GitHubActionsDeployRole:

  1. SponsorCommunicationsComputeDeployPolicy — Lambda functions/roles/logs, Sponsor worker event-source mapping, Sponsor SES-event async invoke config, and failure queue lifecycle.
  2. SponsorCommunicationsServiceDeployPolicy — exact SES configuration set/event-destination lifecycle and only the proven S3 ownership-control gap.

Use generated physical names for the managed policies (no globally fixed ManagedPolicyName). The credentialed operator, not the OIDC role, manages this dataops-github-actions infrastructure stack.

Add deterministic policy-size tests which resolve default parameters/pseudo-values and compact JSON:

  • existing aggregate role inline policies remain at or below 9,472 non-whitespace characters, leaving at least 768 characters below 10,240;
  • each new customer-managed policy remains at or below 5,120 non-whitespace characters, leaving at least 1,024 below the 6,144-character managed-policy limit;
  • total attached customer-managed-policy count remains below the role quota with explicit headroom;
  • sponsor additions appear only in the two new managed policies, not the nearly-full inline policy.

A source-text line count is not sufficient; tests must serialize the actual policy documents with deterministic substitutions. If current upstream changes already violate these headroom thresholds, stop and redesign/split rather than weakening the assertions.

AWS quota reference: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html

Exact compute deployment permissions

Lambda functions

Grant only the CloudFormation lifecycle/read/tag/permission actions needed for the three generated functions. Reuse the proven action set from LambdaDataOpsFunction, excluding Function URL operations because none of these functions has a Function URL, and excluding invoke unless a generated/deploy verification step demonstrably requires it.

Resource patterns must include unqualified and qualified forms for exactly:

arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorSendWorkerFunction-*
arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorSendWorkerFunction-*:* 
arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorSesEventFunction-*
arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorSesEventFunction-*:* 
arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorPrivateArchiveFunction-*
arn:${AWS::Partition}:lambda:${FullDocsRegion}:${AWS::AccountId}:function:${FullDocsStackName}-SponsorPrivateArchiveFunction-*:* 

Whitespace above is illustrative only; tests must assert exact ARN strings without trailing spaces. Do not add ${FullDocsStackName}-Sponsor*, ${FullDocsStackName}-*, or account-wide Lambda resources.

Generated execution roles and PassRole

Grant the existing proven IAM role lifecycle plus iam:PassRole only on:

arn:${AWS::Partition}:iam::${AWS::AccountId}:role/${FullDocsStackName}-SponsorSendWorkerFunctionRole-*
arn:${AWS::Partition}:iam::${AWS::AccountId}:role/${FullDocsStackName}-SponsorSesEventFunctionRole-*
arn:${AWS::Partition}:iam::${AWS::AccountId}:role/${FullDocsStackName}-SponsorPrivateArchiveFunctionRole-*

Do not grant iam:*, wildcard-role PassRole, user/group/policy-version administration, or PassRole for unrelated services/roles. The managed policies themselves are created/attached only when a credentialed operator applies the shared infra stack; the OIDC role does not receive permission to manage its own policy attachments.

Log groups

Grant the same proven log-group infrastructure lifecycle actions as existing functions for exactly both ARN forms of:

/aws/lambda/${FullDocsStackName}-SponsorSendWorkerFunction-*
/aws/lambda/${FullDocsStackName}-SponsorSesEventFunction-*
/aws/lambda/${FullDocsStackName}-SponsorPrivateArchiveFunction-*

Include base log-group and :* forms. Do not grant log-event read/write, arbitrary log groups, or retention changes outside these patterns.

Sponsor worker event-source mapping

Add lambda:CreateEventSourceMapping on Resource: "*" only because create has no resource ARN, with the supported lambda:FunctionArn condition restricted to the exact SponsorSendWorkerFunction-* ARN. Add get/update/delete on region/account event-source-mapping ARNs with the same worker-function condition.

Preserve PR #18’s separate Conversational worker conditions. Do not combine them into a broad *WorkerFunction* pattern. Existing mapping tag actions are already scoped to region/account mapping ARNs and need no new action/resource unless final generated behavior proves otherwise.

Sponsor SES-event async invoke config

Grant exactly:

  • lambda:GetFunctionEventInvokeConfig
  • lambda:PutFunctionEventInvokeConfig
  • lambda:UpdateFunctionEventInvokeConfig
  • lambda:DeleteFunctionEventInvokeConfig

on the unqualified/qualified SponsorSesEventFunction-* function ARN patterns only. Do not grant these actions on Backend, worker, archive, conversational, or wildcard functions.

Failure queue lifecycle

Grant exactly the CloudFormation queue lifecycle used and already proven for #124:

  • sqs:CreateQueue
  • sqs:DeleteQueue
  • sqs:GetQueueAttributes
  • sqs:GetQueueUrl
  • sqs:ListQueueTags
  • sqs:SetQueueAttributes
  • sqs:TagQueue
  • sqs:UntagQueue

on only:

arn:${AWS::Partition}:sqs:${FullDocsRegion}:${AWS::AccountId}:${FullDocsStackName}-SponsorCommunicationFailureQueue-*

Do not grant SendMessage, receive/delete/purge messages, sqs:*, or other queues to the deploy role. Runtime SendMessage belongs only to the generated Lambda roles managed through their exact role policy.

The queue uses AWS-managed SQS encryption (alias/aws/sqs), so this queue adds no deploy-role KMS permission. Tests must preserve DeletionPolicy: Retain, UpdateReplacePolicy: Retain, encryption, retention period, and sanitized-payload expectations in #114; #129 does not alter queue runtime configuration.

Exact SES configuration lifecycle

Add a SponsorSesConfigurationSetName parameter to the shared deploy-role template with a safe default matching #114 and constraints consistent with the application parameter. Bind the policy to exactly:

arn:${AWS::Partition}:ses:${FullDocsRegion}:${AWS::AccountId}:configuration-set/${SponsorSesConfigurationSetName}

Grant only the actions required by the two generated CloudFormation resource types:

  • ses:CreateConfigurationSet
  • ses:DeleteConfigurationSet
  • ses:GetConfigurationSet
  • ses:PutConfigurationSetSendingOptions
  • ses:CreateConfigurationSetEventDestination
  • ses:DeleteConfigurationSetEventDestination
  • ses:GetConfigurationSetEventDestinations
  • ses:UpdateConfigurationSetEventDestination

Do not grant ses:SendEmail, SendBulkEmail, SendRawEmail, identity creation/verification, template/contact-list operations, account-level SES changes, suppression-list operations, or ses:*. Add tag actions only if the final generated resource demonstrably emits SES tags and tests name the exact need; the current #114 resources declare none.

Service-authorization/resource references:

EventBridge boundaries

The existing inline EventBridgeDataOpsRules statement already grants the stack-scoped rule lifecycle on:

arn:${AWS::Partition}:events:${FullDocsRegion}:${AWS::AccountId}:rule/${FullDocsStackName}-*

This covers the generated recovery schedule and explicit SponsorSesEventRule. The Lambda function policy covers its exact AWS::Lambda::Permission; the SES event destination is covered through SES configuration-set actions.

Add no new EventBridge action, event bus write, archive/replay permission, wildcard rule resource, or rule DLQ permission unless a final generated-template diff proves a missing operation. Tests must assert the #114 rule physical-name family matches the existing resource boundary.

Proven KMS and S3 handling

KMS

The existing deploy policy already contains the KMS key lifecycle actions required by the same retained/rotating key shape used elsewhere. kms:CreateKey necessarily uses Resource: "*"; do not add a second KMS statement, crypto data-plane action, grant administration, alias, or broader action. Tests must prove #114’s archive key needs no action absent from the existing KMS set.

S3

The existing DataOpsExportArchiveBucket statement already covers generated ${FullDocsStackName}-* bucket create/delete, encryption, versioning, lifecycle, public-access-block, tagging, and location/list operations. The only generated Sponsor private-archive property not covered there is OwnershipControls.

Add only:

  • s3:GetBucketOwnershipControls
  • s3:PutBucketOwnershipControls

on the lowercase generated Sponsor archive bucket family:

arn:${AWS::Partition}:s3:::${FullDocsStackName}-*sponsorcommunicationprivatearchive*

Do not add object read/write/delete, bucket-policy actions, s3:*, arbitrary buckets, or KMS crypto actions to the deploy role. Runtime object/KMS access remains in the generated SponsorPrivateArchiveFunctionRole, not OIDC deployment policy.

If the final #114 SAM diff changes the bucket properties, prove each additional action from that exact resource before adding it.

Tests and independent review

Add/extend sandbox/dataops/tests/test_template_github_actions.py with parsed structural assertions, not only substring checks. Required automated evidence:

  • every exact function/qualified-function, generated role, and log-group pattern above exists once and broad near-matches fail tests;
  • function actions omit URL/bulk/unneeded invocation capabilities;
  • event-source create’s sole wildcard is condition-bound to SponsorSendWorker; read/update/delete have region/account mapping resources and exact function condition;
  • async invoke-config actions are exact and SponsorSesEvent-only;
  • failure queue actions/resource are exact and exclude data-plane/purge/wildcards;
  • SES actions and exact configuration-set ARN are exact; sending, identity, template, account, bulk/raw, and wildcard capabilities are rejected;
  • no new EventBridge or KMS permissions are introduced;
  • S3 addition is exactly ownership-controls on the Sponsor archive bucket family;
  • existing PR Import DTC Operations content directory #18 Conversational worker/queue/mapping effective permissions remain present;
  • inline and each managed-policy compact resolved sizes satisfy the headroom thresholds;
  • the OIDC trust policy remains restricted to DataTalksClub/dataops, branch main, and aud=sts.amazonaws.com;
  • only the deploy-role template and its focused test file change, unless a narrowly required public-safe README note is justified.

Run:

sam validate --lint --template-file sandbox/dataops/template.github-actions.yaml
uv run --with pytest python -m pytest sandbox/dataops/tests -q
git diff --check

Also parse the YAML, render/measure policy documents with deterministic default substitutions, and compare policy resource patterns against a fresh final #114 SAM build/resource inventory.

A separate security-focused Tester or Architect who did not implement the patch must review:

  • every action against the generated resource lifecycle;
  • every wildcard and supported condition key;
  • IAM inline/managed policy quotas and attachment count;
  • preservation of PR Import DTC Operations content directory #18;
  • absence of runtime send/object/secret access;
  • clean isolated diff and test evidence.

No live AWS simulation/apply is an agent test. Any IAM policy simulation is a post-apply read-only human/on-call check.

Historical original acceptance record

  • The change is based on latest aws-infra/origin/main in an isolated clean worktree/branch and preserves the dirty sibling checkout and merged PR Import DTC Operations content directory #18.
  • Two generated-name customer-managed policies, attached only to GitHubActionsDeployRole, hold all Sponsor additions; the existing inline policy receives no Sponsor permissions and retains at least 768 compact non-whitespace characters of headroom below 10,240.
  • Each new managed policy stays at least 1,024 compact non-whitespace characters below 6,144, and attachment-count tests preserve quota headroom.
  • Lambda lifecycle patterns cover exactly the three Sponsor function logical-ID families, their generated execution roles/PassRole, and their log groups—qualified forms where required—with no broad Sponsor/function/role/log wildcard.
  • Event-source mapping permissions cover only SponsorSendWorkerFunction through exact supported function conditions while PR Import DTC Operations content directory #18’s Conversational worker mapping behavior remains intact.
  • EventInvokeConfig lifecycle covers only SponsorSesEventFunction; existing stack-scoped EventBridge permissions cover Sponsor rules with no new EventBridge grant.
  • Failure queue lifecycle actions/resource are exact, exclude queue data-plane/purge access, and require no KMS addition because the queue uses alias/aws/sqs.
  • SES configuration-set/event-destination lifecycle actions are exact and scoped to the configured configuration-set ARN; the deploy role has no SES send, identity, template, bulk/raw, contact-list, suppression-list, or account-wide permission.
  • Existing KMS permissions are reused unchanged; the only S3 addition is Get/Put ownership controls on the generated Sponsor private-archive bucket family, with no object/bucket-policy/broad access.
  • Parsed negative tests reject every prohibited broadening above, verify exact OIDC trust, compare to the final Automate reviewed sponsor communications from booking milestones #114 resource inventory, and pass SAM lint, all sandbox/dataops tests, and git diff --check.
  • An independent security review confirms least privilege, policy-size/attachment headroom, PR Import DTC Operations content directory #18 preservation, and the isolated diff before source merge.
  • The accepted patch is committed/pushed from the isolated branch and merged through a reviewed DataTalksClub/aws-infra PR into main; the commit/PR uses Refs DataTalksClub/dataops#129, not a closing keyword because human application remains.
  • [HUMAN] A credentialed operator reviews the CloudFormation change set, applies dataops-github-actions with CAPABILITY_NAMED_IAM, and confirms terminal UPDATE_COMPLETE without exposing credentials or private configuration.
  • [HUMAN] On-Call reruns the DataOps deployment with Sponsor sending disabled, verifies Add transactional approval and durable execution recovery #124 and Automate reviewed sponsor communications from booking milestones #114 resource creation/effective IAM/event mappings/queue/SES/archive/log boundaries and sanitized defaults, then records evidence before Repair DataOps OIDC SES configuration-set tag-on-create permission #129 closes.

Test Scenarios

Exact generated resources

Given the final #114 SAM template/build and latest PR #18-based deploy-role template
When policy tests enumerate generated function/role/log/mapping/queue/SES/EventBridge/S3/KMS resources
Then every required lifecycle action matches exactly and speculative/broad/runtime capabilities are absent

Policy size and attachment headroom

Given resolved default/pseudo parameter values
When inline and managed policy documents are compact JSON serialized
Then aggregate inline size is at most 9,472, each new managed policy is at most 5,120, attachment count has headroom, and adding Sponsor statements to inline fails the test

#124 regression protection

Given merged PR #18 and its exact Conversational worker/queue/mapping conditions
When #129 is applied to the source template
Then all effective #124 capabilities remain and no condition is broadened to arbitrary workers/functions

Negative least privilege

Given mutations that add ses:SendEmail, SQS data-plane/purge, S3 object access, KMS crypto/grants, arbitrary PassRole/functions/logs, EventBridge bus writes, or unconditioned event-source create
When focused policy tests run
Then each mutation fails

Clean source integration

Given the dirty sibling checkout and current remote main
When the engineer implements/rebases/reviews #129
Then only the isolated worktree contains the focused diff, final branch includes current origin/main/PR #18, and the shared dirty checkout is untouched

Credentialed apply and rollback

Given a reviewed merged aws-infra commit and the current deployed dataops-github-actions stack
When a credentialed operator previews/applies the change
Then it reaches UPDATE_COMPLETE; if it fails, CloudFormation reaches a terminal rollback state and the operator can reapply the recorded pre-#129 template without manual deletion or policy edits

On-Call deployment verification

Given the deploy-role stack is updated and Sponsor send remains false
When the DataOps main deployment is rerun
Then CloudFormation creates/updates #124 and accepted #114 resources, generated role/function/log/mapping/queue/config-set/archive boundaries match source, no email is sent, logs/failure payloads remain sanitized, and On-Call records exact workflow/stack evidence

Source merge and human operator sequence

After Software Engineer, independent security Tester, and PM acceptance:

  1. Commit in the isolated aws-infra branch with Refs DataTalksClub/dataops#129; do not push from or merge into the dirty sibling checkout.

  2. Push the isolated branch and open a focused DataTalksClub/aws-infra PR to current main, following PR Import DTC Operations content directory #18’s shared-infrastructure precedent.

  3. Require the independent security review and green SAM lint/tests before merge.

  4. Merge to aws-infra/main. Source merge alone does not change AWS and does not unblock deployment.

  5. Add human to Repair DataOps OIDC SES configuration-set tag-on-create permission #129 and leave it open.

  6. [HUMAN] Record the exact pre-Repair DataOps OIDC SES configuration-set tag-on-create permission #129 aws-infra/main commit/template, inspect the CloudFormation change set, then run from aws-infra/sandbox/dataops:

    aws cloudformation deploy \
      --stack-name dataops-github-actions \
      --template-file template.github-actions.yaml \
      --region eu-west-1 \
      --capabilities CAPABILITY_NAMED_IAM
  7. Confirm UPDATE_COMPLETE and inspect the effective role’s inline/attached policy names and size boundaries. If apply fails, confirm terminal rollback; if a completed update must be reverted, deploy the recorded pre-Repair DataOps OIDC SES configuration-set tag-on-create permission #129 template with the same named-IAM capability. Do not manually delete roles/policies or DataOps application resources.

  8. Rerun the current DataOps deployment first with SponsorCommunicationSendEnabled=false. No template/HMAC secret value or real sponsor address is part of this issue.

  9. Relaunch On-Call. Verify actual function/role/log patterns, Sponsor worker mapping/recovery rule, SES-event async config, retained encrypted failure queue, exact SES configuration set/event destination, private archive bucket/key, default-off sending, effective deploy/runtime IAM, and redacted logs/failure inputs. Also verify Add transactional approval and durable execution recovery #124’s previously blocked worker/queue/mapping resources now deploy.

  10. Close Repair DataOps OIDC SES configuration-set tag-on-create permission #129 only after operator apply and On-Call evidence pass. Automate reviewed sponsor communications from booking milestones #114’s separate private-secret configuration and controlled test-inbox send remain its own human gates.

Rollback

The supported rollback is CloudFormation, not manual IAM editing:

  • before apply, keep the exact prior aws-infra/main template/commit and review the change set;
  • a failed update must reach UPDATE_ROLLBACK_COMPLETE or another explained terminal state before retry;
  • after a successful but unacceptable update, redeploy the prior template with CAPABILITY_NAMED_IAM, confirm the two Sponsor managed policies detach/delete and PR Import DTC Operations content directory #18 remains effective, then rerun read-only policy checks;
  • do not roll back/remove PR Import DTC Operations content directory #18 while Add transactional approval and durable execution recovery #124 resources exist;
  • do not delete the retained Sponsor failure queue/archive/key or alter the DataOps app stack from this infrastructure issue;
  • Sponsor sending remains disabled throughout apply/rollback verification.

Dependencies and boundaries

Out of Scope

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1ImportantbugSomething is brokenhumanCode done or issue blocked on human verificationinfraDeployment and infrastructuretestingTests and QA

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions