You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Extend the DataOps OIDC deploy role for reviewed sponsor communications
Status: in progress — PM accepted exact source; awaiting commit/integration and HUMAN apply
Tags: bug, infra, testing, P1
Depends on: None — implementation may run alongside #114; acceptance must recompare the final agent-approved #114 SAM resource graph
Blocks: #128 default-off deployment/HUMAN rollout; #134 On-Call closure after its application checks passed
Current evidence: run 30591398299 passed the complete app check job, then CloudFormation was denied ses:TagResource on the exact dataops-sponsor-reviewed-send-disabled configuration-set ARN and entered rollback.
Scope
Repair only the newly proven CloudFormation tag-on-create gap in the existing SponsorCommunicationsServiceDeployPolicy in DataTalksClub/aws-infra. The denied resource is exactly:
The observed deployed name is dataops-sponsor-reviewed-send-disabled. Add no SES sending or runtime capability and no wildcard SES resource. Preserve the already effective Gate A/Gate B policies, OIDC trust, attachments, and unrelated dirty ../aws-infra checkout. Implementation must use a fresh isolated worktree from current aws-infra/origin/main (currently ddbaf2893bf2985d32e5a33310fe57e2f4ac9452; fetch and recheck before work).
Acceptance Criteria
A fresh isolated aws-infra worktree starts from current origin/main; the dirty sibling checkout is untouched, and the candidate changes only sandbox/dataops/template.github-actions.yaml plus its focused policy test unless a narrowly justified public-safe fixture is required.
The existing SponsorCommunicationsServiceDeployPolicy adds only ses:TagResource for the exact parameterized configuration-set ARN above. No Sponsor permission moves to the inline policy, no new managed policy/attachment is created, and Gate A/Gate B trust and documents remain otherwise unchanged.
The implementation proves SES/CloudFormation tag-condition semantics. If aws:TagKeys safely applies, require a present key set limited by ForAllValues:StringEquals to exactly aws:cloudformation:logical-id, aws:cloudformation:stack-id, and aws:cloudformation:stack-name. If that condition would reject legitimate CloudFormation system tagging or is unsupported, record authoritative evidence and use only the exact-action/exact-ARN fallback—never a broader resource/action.
The deploy role gains no ses:UntagResource, ses:*, wildcard SES ARN, send/bulk/raw-send, identity, template, contact-list, suppression-list, account, or runtime permission. Existing SES lifecycle actions remain exact.
Parsed structural and negative-mutation tests prove the chosen tag statement/action/resource/condition, reject every prohibited broadening, preserve OIDC trust and all prior Gate A/Gate B capabilities, and keep policy-size plus attachment headroom. SAM lint, all sandbox/dataops tests, and git diff --check pass.
The handoff records the exact base, changed-path inventory, binary diff hash, source-template hash, focused-test hash, and resolved compact Sponsor service-policy hash/size; an independent IAM/security reviewer accepts that exact fingerprint.
The accepted aws-infra patch is committed/integrated to current main with Refs DataTalksClub/dataops#129 and no closing keyword; source integration alone does not authorize or perform AWS/application deployment.
[HUMAN/AWS] After run 30591398299 reaches a terminal rollback state, a credentialed operator creates and independently reviews a dataops-github-actions CloudFormation change set with CAPABILITY_NAMED_IAM. It must show only the expected in-place Sponsor service managed-policy document update, with no deploy-role replacement, trust/attachment change, or unrelated resource change; apply it sequentially, require terminal UPDATE_COMPLETE, and verify the effective resolved document/fingerprint and stack drift without manual IAM edits.
sam validate --lint --template-file sandbox/dataops/template.github-actions.yaml
uv run --with pytest python -m pytest sandbox/dataops/tests -q
git diff --check
Also resolve and compact-serialize the Sponsor service policy under deterministic pseudo/default values, compare it to the effective Gate A baseline, and run mutations for wildcard resource/action, SES send families, UntagResource, extra tag keys, missing/non-limiting tag-key conditions when the condition path is used, inline-policy relocation, trust/attachment drift, and policy quota regression.
Out of Scope for this follow-up
SES sending, identities, templates, runtime Lambda permissions, Sponsor content/secrets, application SAM/runtime changes, manual IAM/app deployment, speculative tagging actions, and the separate #128 HUMAN canaries. #134 application calendar checks are already green in the triggering run and are not modified here.
Historical original scope and rationale
The detailed original #129 design below records the already integrated/effective Gate A work. It is context, not authorization to broaden this exact tag-on-create repair.
Update the shared DataTalksClub/aws-infra source-of-truth template at:
sandbox/dataops/template.github-actions.yaml
so the existing dataops-github-actions-deploy OIDC role can let CloudFormation create/update/roll back the exact reviewed sponsor-communication resources declared by #114. Change no DataOps runtime code and apply no AWS stack during agent implementation/review.
The implementation must start in an isolated aws-infra worktree/branch from the latest origin/main, not the current dirty sibling checkout. At grooming time, origin/main is cc96704 and includes merged PR #18 / commit 691666a; the engineer must fetch again and use the then-current remote head. Suggested ownership:
Do not stash, reset, overwrite, or merge the dirty ../aws-infra checkout. Preserve unrelated Datamailer/bookkeeping/auth work. The final source change belongs in DataTalksClub/aws-infra, not copied into dataops.
Authoritative resource graph
The current #114 generated SAM handoff declares these deployment resources:
one DynamoDB Stream event-source mapping for SponsorSendWorkerFunction;
one Lambda EventInvokeConfig for SponsorSesEventFunction;
SponsorCommunicationFailureQueue, retained and encrypted with alias/aws/sqs;
SponsorSesConfigurationSet and SponsorSesEventDestination;
stack-scoped scheduled/EventBridge rules, including SponsorSesEventRule;
SponsorCommunicationPrivateArchiveBucket, retained/versioned/private/KMS-encrypted with ownership controls;
SponsorCommunicationPrivateArchiveKey, retained with rotation.
Before Tester/PM acceptance, compare the proposed policy to the final accepted #114infra/template.full.yaml and a fresh sam build/generated resource inventory. Remove permissions for resources that #114 drops; add nothing for speculative resources. If a final #114 logical ID changes, update exact patterns/tests before merge.
#124 is not an implementation dependency for #129. A credentialed operator may apply PR #18 before #129 finishes. If the operator waits, the later #129 stack update must include and verify PR #18 as part of the same effective role update. #129 must not regress or silently relocate those permissions without an explicit equivalent-policy test.
Policy-size design
Do not append sponsor permissions to the existing inline role policy.
At grooming time, resolving origin/main with default parameters/account/region and compact-serializing dataops-v1-deploy produces approximately 9,341 non-whitespace characters, only 899 below IAM’s aggregate 10,240-character inline-policy quota for a role. Splitting into more inline policies would not help because the quota is aggregate across a role’s inline policies.
Add two stack-owned customer-managed policies attached only to GitHubActionsDeployRole:
SponsorCommunicationsServiceDeployPolicy — exact SES configuration set/event-destination lifecycle and only the proven S3 ownership-control gap.
Use generated physical names for the managed policies (no globally fixed ManagedPolicyName). The credentialed operator, not the OIDC role, manages this dataops-github-actions infrastructure stack.
Add deterministic policy-size tests which resolve default parameters/pseudo-values and compact JSON:
existing aggregate role inline policies remain at or below 9,472 non-whitespace characters, leaving at least 768 characters below 10,240;
each new customer-managed policy remains at or below 5,120 non-whitespace characters, leaving at least 1,024 below the 6,144-character managed-policy limit;
total attached customer-managed-policy count remains below the role quota with explicit headroom;
sponsor additions appear only in the two new managed policies, not the nearly-full inline policy.
A source-text line count is not sufficient; tests must serialize the actual policy documents with deterministic substitutions. If current upstream changes already violate these headroom thresholds, stop and redesign/split rather than weakening the assertions.
Grant only the CloudFormation lifecycle/read/tag/permission actions needed for the three generated functions. Reuse the proven action set from LambdaDataOpsFunction, excluding Function URL operations because none of these functions has a Function URL, and excluding invoke unless a generated/deploy verification step demonstrably requires it.
Resource patterns must include unqualified and qualified forms for exactly:
Whitespace above is illustrative only; tests must assert exact ARN strings without trailing spaces. Do not add ${FullDocsStackName}-Sponsor*, ${FullDocsStackName}-*, or account-wide Lambda resources.
Generated execution roles and PassRole
Grant the existing proven IAM role lifecycle plus iam:PassRole only on:
Do not grant iam:*, wildcard-role PassRole, user/group/policy-version administration, or PassRole for unrelated services/roles. The managed policies themselves are created/attached only when a credentialed operator applies the shared infra stack; the OIDC role does not receive permission to manage its own policy attachments.
Log groups
Grant the same proven log-group infrastructure lifecycle actions as existing functions for exactly both ARN forms of:
Include base log-group and :* forms. Do not grant log-event read/write, arbitrary log groups, or retention changes outside these patterns.
Sponsor worker event-source mapping
Add lambda:CreateEventSourceMapping on Resource: "*" only because create has no resource ARN, with the supported lambda:FunctionArn condition restricted to the exact SponsorSendWorkerFunction-* ARN. Add get/update/delete on region/account event-source-mapping ARNs with the same worker-function condition.
Preserve PR #18’s separate Conversational worker conditions. Do not combine them into a broad *WorkerFunction* pattern. Existing mapping tag actions are already scoped to region/account mapping ARNs and need no new action/resource unless final generated behavior proves otherwise.
Sponsor SES-event async invoke config
Grant exactly:
lambda:GetFunctionEventInvokeConfig
lambda:PutFunctionEventInvokeConfig
lambda:UpdateFunctionEventInvokeConfig
lambda:DeleteFunctionEventInvokeConfig
on the unqualified/qualified SponsorSesEventFunction-* function ARN patterns only. Do not grant these actions on Backend, worker, archive, conversational, or wildcard functions.
Failure queue lifecycle
Grant exactly the CloudFormation queue lifecycle used and already proven for #124:
Do not grant SendMessage, receive/delete/purge messages, sqs:*, or other queues to the deploy role. Runtime SendMessage belongs only to the generated Lambda roles managed through their exact role policy.
The queue uses AWS-managed SQS encryption (alias/aws/sqs), so this queue adds no deploy-role KMS permission. Tests must preserve DeletionPolicy: Retain, UpdateReplacePolicy: Retain, encryption, retention period, and sanitized-payload expectations in #114; #129 does not alter queue runtime configuration.
Exact SES configuration lifecycle
Add a SponsorSesConfigurationSetName parameter to the shared deploy-role template with a safe default matching #114 and constraints consistent with the application parameter. Bind the policy to exactly:
Grant only the actions required by the two generated CloudFormation resource types:
ses:CreateConfigurationSet
ses:DeleteConfigurationSet
ses:GetConfigurationSet
ses:PutConfigurationSetSendingOptions
ses:CreateConfigurationSetEventDestination
ses:DeleteConfigurationSetEventDestination
ses:GetConfigurationSetEventDestinations
ses:UpdateConfigurationSetEventDestination
Do not grant ses:SendEmail, SendBulkEmail, SendRawEmail, identity creation/verification, template/contact-list operations, account-level SES changes, suppression-list operations, or ses:*. Add tag actions only if the final generated resource demonstrably emits SES tags and tests name the exact need; the current #114 resources declare none.
This covers the generated recovery schedule and explicit SponsorSesEventRule. The Lambda function policy covers its exact AWS::Lambda::Permission; the SES event destination is covered through SES configuration-set actions.
Add no new EventBridge action, event bus write, archive/replay permission, wildcard rule resource, or rule DLQ permission unless a final generated-template diff proves a missing operation. Tests must assert the #114 rule physical-name family matches the existing resource boundary.
Proven KMS and S3 handling
KMS
The existing deploy policy already contains the KMS key lifecycle actions required by the same retained/rotating key shape used elsewhere. kms:CreateKey necessarily uses Resource: "*"; do not add a second KMS statement, crypto data-plane action, grant administration, alias, or broader action. Tests must prove #114’s archive key needs no action absent from the existing KMS set.
S3
The existing DataOpsExportArchiveBucket statement already covers generated ${FullDocsStackName}-* bucket create/delete, encryption, versioning, lifecycle, public-access-block, tagging, and location/list operations. The only generated Sponsor private-archive property not covered there is OwnershipControls.
Add only:
s3:GetBucketOwnershipControls
s3:PutBucketOwnershipControls
on the lowercase generated Sponsor archive bucket family:
Do not add object read/write/delete, bucket-policy actions, s3:*, arbitrary buckets, or KMS crypto actions to the deploy role. Runtime object/KMS access remains in the generated SponsorPrivateArchiveFunctionRole, not OIDC deployment policy.
If the final #114 SAM diff changes the bucket properties, prove each additional action from that exact resource before adding it.
Tests and independent review
Add/extend sandbox/dataops/tests/test_template_github_actions.py with parsed structural assertions, not only substring checks. Required automated evidence:
every exact function/qualified-function, generated role, and log-group pattern above exists once and broad near-matches fail tests;
function actions omit URL/bulk/unneeded invocation capabilities;
event-source create’s sole wildcard is condition-bound to SponsorSendWorker; read/update/delete have region/account mapping resources and exact function condition;
async invoke-config actions are exact and SponsorSesEvent-only;
failure queue actions/resource are exact and exclude data-plane/purge/wildcards;
SES actions and exact configuration-set ARN are exact; sending, identity, template, account, bulk/raw, and wildcard capabilities are rejected;
no new EventBridge or KMS permissions are introduced;
S3 addition is exactly ownership-controls on the Sponsor archive bucket family;
inline and each managed-policy compact resolved sizes satisfy the headroom thresholds;
the OIDC trust policy remains restricted to DataTalksClub/dataops, branch main, and aud=sts.amazonaws.com;
only the deploy-role template and its focused test file change, unless a narrowly required public-safe README note is justified.
Run:
sam validate --lint --template-file sandbox/dataops/template.github-actions.yaml
uv run --with pytest python -m pytest sandbox/dataops/tests -q
git diff --check
Also parse the YAML, render/measure policy documents with deterministic default substitutions, and compare policy resource patterns against a fresh final #114 SAM build/resource inventory.
A separate security-focused Tester or Architect who did not implement the patch must review:
every action against the generated resource lifecycle;
every wildcard and supported condition key;
IAM inline/managed policy quotas and attachment count;
No live AWS simulation/apply is an agent test. Any IAM policy simulation is a post-apply read-only human/on-call check.
Historical original acceptance record
The change is based on latest aws-infra/origin/main in an isolated clean worktree/branch and preserves the dirty sibling checkout and merged PR Import DTC Operations content directory #18.
Two generated-name customer-managed policies, attached only to GitHubActionsDeployRole, hold all Sponsor additions; the existing inline policy receives no Sponsor permissions and retains at least 768 compact non-whitespace characters of headroom below 10,240.
Each new managed policy stays at least 1,024 compact non-whitespace characters below 6,144, and attachment-count tests preserve quota headroom.
Lambda lifecycle patterns cover exactly the three Sponsor function logical-ID families, their generated execution roles/PassRole, and their log groups—qualified forms where required—with no broad Sponsor/function/role/log wildcard.
Event-source mapping permissions cover only SponsorSendWorkerFunction through exact supported function conditions while PR Import DTC Operations content directory #18’s Conversational worker mapping behavior remains intact.
EventInvokeConfig lifecycle covers only SponsorSesEventFunction; existing stack-scoped EventBridge permissions cover Sponsor rules with no new EventBridge grant.
Failure queue lifecycle actions/resource are exact, exclude queue data-plane/purge access, and require no KMS addition because the queue uses alias/aws/sqs.
SES configuration-set/event-destination lifecycle actions are exact and scoped to the configured configuration-set ARN; the deploy role has no SES send, identity, template, bulk/raw, contact-list, suppression-list, or account-wide permission.
Existing KMS permissions are reused unchanged; the only S3 addition is Get/Put ownership controls on the generated Sponsor private-archive bucket family, with no object/bucket-policy/broad access.
An independent security review confirms least privilege, policy-size/attachment headroom, PR Import DTC Operations content directory #18 preservation, and the isolated diff before source merge.
The accepted patch is committed/pushed from the isolated branch and merged through a reviewed DataTalksClub/aws-infra PR into main; the commit/PR uses Refs DataTalksClub/dataops#129, not a closing keyword because human application remains.
[HUMAN] A credentialed operator reviews the CloudFormation change set, applies dataops-github-actions with CAPABILITY_NAMED_IAM, and confirms terminal UPDATE_COMPLETE without exposing credentials or private configuration.
Given the final #114 SAM template/build and latest PR #18-based deploy-role template
When policy tests enumerate generated function/role/log/mapping/queue/SES/EventBridge/S3/KMS resources
Then every required lifecycle action matches exactly and speculative/broad/runtime capabilities are absent
Policy size and attachment headroom
Given resolved default/pseudo parameter values
When inline and managed policy documents are compact JSON serialized
Then aggregate inline size is at most 9,472, each new managed policy is at most 5,120, attachment count has headroom, and adding Sponsor statements to inline fails the test
Given merged PR #18 and its exact Conversational worker/queue/mapping conditions
When #129 is applied to the source template
Then all effective #124 capabilities remain and no condition is broadened to arbitrary workers/functions
Negative least privilege
Given mutations that add ses:SendEmail, SQS data-plane/purge, S3 object access, KMS crypto/grants, arbitrary PassRole/functions/logs, EventBridge bus writes, or unconditioned event-source create
When focused policy tests run
Then each mutation fails
Clean source integration
Given the dirty sibling checkout and current remote main
When the engineer implements/rebases/reviews #129
Then only the isolated worktree contains the focused diff, final branch includes current origin/main/PR #18, and the shared dirty checkout is untouched
Credentialed apply and rollback
Given a reviewed merged aws-infra commit and the current deployed dataops-github-actions stack
When a credentialed operator previews/applies the change
Then it reaches UPDATE_COMPLETE; if it fails, CloudFormation reaches a terminal rollback state and the operator can reapply the recorded pre-#129 template without manual deletion or policy edits
On-Call deployment verification
Given the deploy-role stack is updated and Sponsor send remains false
When the DataOps main deployment is rerun
Then CloudFormation creates/updates #124 and accepted #114 resources, generated role/function/log/mapping/queue/config-set/archive boundaries match source, no email is sent, logs/failure payloads remain sanitized, and On-Call records exact workflow/stack evidence
Source merge and human operator sequence
After Software Engineer, independent security Tester, and PM acceptance:
Commit in the isolated aws-infra branch with Refs DataTalksClub/dataops#129; do not push from or merge into the dirty sibling checkout.
Push the isolated branch and open a focused DataTalksClub/aws-infra PR to current main, following PR Import DTC Operations content directory #18’s shared-infrastructure precedent.
Require the independent security review and green SAM lint/tests before merge.
Merge to aws-infra/main. Source merge alone does not change AWS and does not unblock deployment.
Confirm UPDATE_COMPLETE and inspect the effective role’s inline/attached policy names and size boundaries. If apply fails, confirm terminal rollback; if a completed update must be reverted, deploy the recorded pre-Repair DataOps OIDC SES configuration-set tag-on-create permission #129 template with the same named-IAM capability. Do not manually delete roles/policies or DataOps application resources.
Rerun the current DataOps deployment first with SponsorCommunicationSendEnabled=false. No template/HMAC secret value or real sponsor address is part of this issue.
Relaunch On-Call. Verify actual function/role/log patterns, Sponsor worker mapping/recovery rule, SES-event async config, retained encrypted failure queue, exact SES configuration set/event destination, private archive bucket/key, default-off sending, effective deploy/runtime IAM, and redacted logs/failure inputs. Also verify Add transactional approval and durable execution recovery #124’s previously blocked worker/queue/mapping resources now deploy.
The supported rollback is CloudFormation, not manual IAM editing:
before apply, keep the exact prior aws-infra/main template/commit and review the change set;
a failed update must reach UPDATE_ROLLBACK_COMPLETE or another explained terminal state before retry;
after a successful but unacceptable update, redeploy the prior template with CAPABILITY_NAMED_IAM, confirm the two Sponsor managed policies detach/delete and PR Import DTC Operations content directory #18 remains effective, then rerun read-only policy checks;
Do not add broad permissions “in case CloudFormation needs them.” A real denied action after the exact patch is routed back through a new evidence-backed issue/change, not preemptive wildcarding.
Out of Scope
Applying AWS from an agent session, manually editing live IAM, deleting retained resources, or deploying the DataOps application outside GitHub Actions.
Reworking unrelated OIDC role permissions, dirty Datamailer/bookkeeping/auth changes, other AWS stacks/repos, or the current public/private knowledge boundary.
Moving all existing inline permissions to managed policies, redesigning the entire deploy role, creating a permission boundary, or adding speculative services/actions.
Extend the DataOps OIDC deploy role for reviewed sponsor communications
Status: in progress — PM accepted exact source; awaiting commit/integration and HUMAN apply
Tags:
bug,infra,testing,P1Depends on: None — implementation may run alongside #114; acceptance must recompare the final agent-approved #114 SAM resource graph
Blocks: #128 default-off deployment/HUMAN rollout; #134 On-Call closure after its application checks passed
Current evidence: run 30591398299 passed the complete app check job, then CloudFormation was denied
ses:TagResourceon the exactdataops-sponsor-reviewed-send-disabledconfiguration-set ARN and entered rollback.Scope
Repair only the newly proven CloudFormation tag-on-create gap in the existing
SponsorCommunicationsServiceDeployPolicyinDataTalksClub/aws-infra. The denied resource is exactly:The observed deployed name is
dataops-sponsor-reviewed-send-disabled. Add no SES sending or runtime capability and no wildcard SES resource. Preserve the already effective Gate A/Gate B policies, OIDC trust, attachments, and unrelated dirty../aws-infracheckout. Implementation must use a fresh isolated worktree from currentaws-infra/origin/main(currentlyddbaf2893bf2985d32e5a33310fe57e2f4ac9452; fetch and recheck before work).Acceptance Criteria
aws-infraworktree starts from currentorigin/main; the dirty sibling checkout is untouched, and the candidate changes onlysandbox/dataops/template.github-actions.yamlplus its focused policy test unless a narrowly justified public-safe fixture is required.SponsorCommunicationsServiceDeployPolicyadds onlyses:TagResourcefor the exact parameterized configuration-set ARN above. No Sponsor permission moves to the inline policy, no new managed policy/attachment is created, and Gate A/Gate B trust and documents remain otherwise unchanged.aws:TagKeyssafely applies, require a present key set limited byForAllValues:StringEqualsto exactlyaws:cloudformation:logical-id,aws:cloudformation:stack-id, andaws:cloudformation:stack-name. If that condition would reject legitimate CloudFormation system tagging or is unsupported, record authoritative evidence and use only the exact-action/exact-ARN fallback—never a broader resource/action.ses:UntagResource,ses:*, wildcard SES ARN, send/bulk/raw-send, identity, template, contact-list, suppression-list, account, or runtime permission. Existing SES lifecycle actions remain exact.sandbox/dataopstests, andgit diff --checkpass.aws-infrapatch is committed/integrated to currentmainwithRefs DataTalksClub/dataops#129and no closing keyword; source integration alone does not authorize or perform AWS/application deployment.[HUMAN/AWS]After run 30591398299 reaches a terminal rollback state, a credentialed operator creates and independently reviews adataops-github-actionsCloudFormation change set withCAPABILITY_NAMED_IAM. It must show only the expected in-place Sponsor service managed-policy document update, with no deploy-role replacement, trust/attachment change, or unrelated resource change; apply it sequentially, require terminalUPDATE_COMPLETE, and verify the effective resolved document/fingerprint and stack drift without manual IAM edits.[HUMAN]Rerun the normal DataOpsmainGitHub Actions deployment with Sponsor sending disabled. The full application checks must remain green and CloudFormation must pass configuration-set tagging plus the default-off Roll out and cut over the Telegram conversational MVP #128/Use one Berlin business date in #106 scheduling and default-SPA E2E #134 deployment path. Use normal CI only—no manual app/Lambda deploy—and record sanitized On-Call evidence before closing Repair DataOps OIDC SES configuration-set tag-on-create permission #129.Verification
Also resolve and compact-serialize the Sponsor service policy under deterministic pseudo/default values, compare it to the effective Gate A baseline, and run mutations for wildcard resource/action, SES send families,
UntagResource, extra tag keys, missing/non-limiting tag-key conditions when the condition path is used, inline-policy relocation, trust/attachment drift, and policy quota regression.Out of Scope for this follow-up
SES sending, identities, templates, runtime Lambda permissions, Sponsor content/secrets, application SAM/runtime changes, manual IAM/app deployment, speculative tagging actions, and the separate #128 HUMAN canaries. #134 application calendar checks are already green in the triggering run and are not modified here.
Historical original scope and rationale
The detailed original #129 design below records the already integrated/effective Gate A work. It is context, not authorization to broaden this exact tag-on-create repair.
Update the shared
DataTalksClub/aws-infrasource-of-truth template at:so the existing
dataops-github-actions-deployOIDC role can let CloudFormation create/update/roll back the exact reviewed sponsor-communication resources declared by #114. Change no DataOps runtime code and apply no AWS stack during agent implementation/review.The implementation must start in an isolated
aws-infraworktree/branch from the latestorigin/main, not the current dirty sibling checkout. At grooming time,origin/mainiscc96704and includes merged PR #18 / commit691666a; the engineer must fetch again and use the then-current remote head. Suggested ownership:Do not stash, reset, overwrite, or merge the dirty
../aws-infracheckout. Preserve unrelated Datamailer/bookkeeping/auth work. The final source change belongs inDataTalksClub/aws-infra, not copied intodataops.Authoritative resource graph
The current #114 generated SAM handoff declares these deployment resources:
AWS::Serverless::Functionlogical IDs:SponsorSendWorkerFunctionSponsorSesEventFunctionSponsorPrivateArchiveFunctionSponsorSendWorkerFunctionRoleSponsorSesEventFunctionRoleSponsorPrivateArchiveFunctionRole/aws/lambda/<physical-function-name>log groups;SponsorSendWorkerFunction;EventInvokeConfigforSponsorSesEventFunction;SponsorCommunicationFailureQueue, retained and encrypted withalias/aws/sqs;SponsorSesConfigurationSetandSponsorSesEventDestination;SponsorSesEventRule;SponsorCommunicationPrivateArchiveBucket, retained/versioned/private/KMS-encrypted with ownership controls;SponsorCommunicationPrivateArchiveKey, retained with rotation.Before Tester/PM acceptance, compare the proposed policy to the final accepted #114
infra/template.full.yamland a freshsam build/generated resource inventory. Remove permissions for resources that #114 drops; add nothing for speculative resources. If a final #114 logical ID changes, update exact patterns/tests before merge.Preserve #124 / PR #18
PR #18 is already merged into
aws-infra/origin/mainbut has not yet been applied to AWS. Preserve all of its effective permissions and tests:ConversationalExecutionWorkerFunctionfunction/role/log patterns;ConversationalExecutionFailureQueueexact queue lifecycle;#124 is not an implementation dependency for #129. A credentialed operator may apply PR #18 before #129 finishes. If the operator waits, the later #129 stack update must include and verify PR #18 as part of the same effective role update. #129 must not regress or silently relocate those permissions without an explicit equivalent-policy test.
Policy-size design
Do not append sponsor permissions to the existing inline role policy.
At grooming time, resolving
origin/mainwith default parameters/account/region and compact-serializingdataops-v1-deployproduces approximately 9,341 non-whitespace characters, only 899 below IAM’s aggregate 10,240-character inline-policy quota for a role. Splitting into more inline policies would not help because the quota is aggregate across a role’s inline policies.Add two stack-owned customer-managed policies attached only to
GitHubActionsDeployRole:SponsorCommunicationsComputeDeployPolicy— Lambda functions/roles/logs, Sponsor worker event-source mapping, Sponsor SES-event async invoke config, and failure queue lifecycle.SponsorCommunicationsServiceDeployPolicy— exact SES configuration set/event-destination lifecycle and only the proven S3 ownership-control gap.Use generated physical names for the managed policies (no globally fixed
ManagedPolicyName). The credentialed operator, not the OIDC role, manages thisdataops-github-actionsinfrastructure stack.Add deterministic policy-size tests which resolve default parameters/pseudo-values and compact JSON:
A source-text line count is not sufficient; tests must serialize the actual policy documents with deterministic substitutions. If current upstream changes already violate these headroom thresholds, stop and redesign/split rather than weakening the assertions.
AWS quota reference: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html
Exact compute deployment permissions
Lambda functions
Grant only the CloudFormation lifecycle/read/tag/permission actions needed for the three generated functions. Reuse the proven action set from
LambdaDataOpsFunction, excluding Function URL operations because none of these functions has a Function URL, and excluding invoke unless a generated/deploy verification step demonstrably requires it.Resource patterns must include unqualified and qualified forms for exactly:
Whitespace above is illustrative only; tests must assert exact ARN strings without trailing spaces. Do not add
${FullDocsStackName}-Sponsor*,${FullDocsStackName}-*, or account-wide Lambda resources.Generated execution roles and PassRole
Grant the existing proven IAM role lifecycle plus
iam:PassRoleonly on:Do not grant
iam:*, wildcard-role PassRole, user/group/policy-version administration, or PassRole for unrelated services/roles. The managed policies themselves are created/attached only when a credentialed operator applies the shared infra stack; the OIDC role does not receive permission to manage its own policy attachments.Log groups
Grant the same proven log-group infrastructure lifecycle actions as existing functions for exactly both ARN forms of:
Include base log-group and
:*forms. Do not grant log-event read/write, arbitrary log groups, or retention changes outside these patterns.Sponsor worker event-source mapping
Add
lambda:CreateEventSourceMappingonResource: "*"only because create has no resource ARN, with the supportedlambda:FunctionArncondition restricted to the exactSponsorSendWorkerFunction-*ARN. Add get/update/delete on region/account event-source-mapping ARNs with the same worker-function condition.Preserve PR #18’s separate Conversational worker conditions. Do not combine them into a broad
*WorkerFunction*pattern. Existing mapping tag actions are already scoped to region/account mapping ARNs and need no new action/resource unless final generated behavior proves otherwise.Sponsor SES-event async invoke config
Grant exactly:
lambda:GetFunctionEventInvokeConfiglambda:PutFunctionEventInvokeConfiglambda:UpdateFunctionEventInvokeConfiglambda:DeleteFunctionEventInvokeConfigon the unqualified/qualified
SponsorSesEventFunction-*function ARN patterns only. Do not grant these actions on Backend, worker, archive, conversational, or wildcard functions.Failure queue lifecycle
Grant exactly the CloudFormation queue lifecycle used and already proven for #124:
sqs:CreateQueuesqs:DeleteQueuesqs:GetQueueAttributessqs:GetQueueUrlsqs:ListQueueTagssqs:SetQueueAttributessqs:TagQueuesqs:UntagQueueon only:
Do not grant
SendMessage, receive/delete/purge messages,sqs:*, or other queues to the deploy role. RuntimeSendMessagebelongs only to the generated Lambda roles managed through their exact role policy.The queue uses AWS-managed SQS encryption (
alias/aws/sqs), so this queue adds no deploy-role KMS permission. Tests must preserveDeletionPolicy: Retain,UpdateReplacePolicy: Retain, encryption, retention period, and sanitized-payload expectations in #114; #129 does not alter queue runtime configuration.Exact SES configuration lifecycle
Add a
SponsorSesConfigurationSetNameparameter to the shared deploy-role template with a safe default matching #114 and constraints consistent with the application parameter. Bind the policy to exactly:Grant only the actions required by the two generated CloudFormation resource types:
ses:CreateConfigurationSetses:DeleteConfigurationSetses:GetConfigurationSetses:PutConfigurationSetSendingOptionsses:CreateConfigurationSetEventDestinationses:DeleteConfigurationSetEventDestinationses:GetConfigurationSetEventDestinationsses:UpdateConfigurationSetEventDestinationDo not grant
ses:SendEmail,SendBulkEmail,SendRawEmail, identity creation/verification, template/contact-list operations, account-level SES changes, suppression-list operations, orses:*. Add tag actions only if the final generated resource demonstrably emits SES tags and tests name the exact need; the current #114 resources declare none.Service-authorization/resource references:
EventBridge boundaries
The existing inline
EventBridgeDataOpsRulesstatement already grants the stack-scoped rule lifecycle on:This covers the generated recovery schedule and explicit
SponsorSesEventRule. The Lambda function policy covers its exactAWS::Lambda::Permission; the SES event destination is covered through SES configuration-set actions.Add no new EventBridge action, event bus write, archive/replay permission, wildcard rule resource, or rule DLQ permission unless a final generated-template diff proves a missing operation. Tests must assert the #114 rule physical-name family matches the existing resource boundary.
Proven KMS and S3 handling
KMS
The existing deploy policy already contains the KMS key lifecycle actions required by the same retained/rotating key shape used elsewhere.
kms:CreateKeynecessarily usesResource: "*"; do not add a second KMS statement, crypto data-plane action, grant administration, alias, or broader action. Tests must prove #114’s archive key needs no action absent from the existing KMS set.S3
The existing
DataOpsExportArchiveBucketstatement already covers generated${FullDocsStackName}-*bucket create/delete, encryption, versioning, lifecycle, public-access-block, tagging, and location/list operations. The only generated Sponsor private-archive property not covered there isOwnershipControls.Add only:
s3:GetBucketOwnershipControlss3:PutBucketOwnershipControlson the lowercase generated Sponsor archive bucket family:
Do not add object read/write/delete, bucket-policy actions,
s3:*, arbitrary buckets, or KMS crypto actions to the deploy role. Runtime object/KMS access remains in the generatedSponsorPrivateArchiveFunctionRole, not OIDC deployment policy.If the final #114 SAM diff changes the bucket properties, prove each additional action from that exact resource before adding it.
Tests and independent review
Add/extend
sandbox/dataops/tests/test_template_github_actions.pywith parsed structural assertions, not only substring checks. Required automated evidence:DataTalksClub/dataops, branchmain, andaud=sts.amazonaws.com;Run:
Also parse the YAML, render/measure policy documents with deterministic default substitutions, and compare policy resource patterns against a fresh final #114 SAM build/resource inventory.
A separate security-focused Tester or Architect who did not implement the patch must review:
No live AWS simulation/apply is an agent test. Any IAM policy simulation is a post-apply read-only human/on-call check.
Historical original acceptance record
aws-infra/origin/mainin an isolated clean worktree/branch and preserves the dirty sibling checkout and merged PR Import DTC Operations content directory #18.GitHubActionsDeployRole, hold all Sponsor additions; the existing inline policy receives no Sponsor permissions and retains at least 768 compact non-whitespace characters of headroom below 10,240.SponsorSendWorkerFunctionthrough exact supported function conditions while PR Import DTC Operations content directory #18’s Conversational worker mapping behavior remains intact.SponsorSesEventFunction; existing stack-scoped EventBridge permissions cover Sponsor rules with no new EventBridge grant.alias/aws/sqs.sandbox/dataopstests, andgit diff --check.DataTalksClub/aws-infraPR intomain; the commit/PR usesRefs DataTalksClub/dataops#129, not a closing keyword because human application remains.[HUMAN]A credentialed operator reviews the CloudFormation change set, appliesdataops-github-actionswithCAPABILITY_NAMED_IAM, and confirms terminalUPDATE_COMPLETEwithout exposing credentials or private configuration.[HUMAN]On-Call reruns the DataOps deployment with Sponsor sending disabled, verifies Add transactional approval and durable execution recovery #124 and Automate reviewed sponsor communications from booking milestones #114 resource creation/effective IAM/event mappings/queue/SES/archive/log boundaries and sanitized defaults, then records evidence before Repair DataOps OIDC SES configuration-set tag-on-create permission #129 closes.Test Scenarios
Exact generated resources
Given the final #114 SAM template/build and latest PR #18-based deploy-role template
When policy tests enumerate generated function/role/log/mapping/queue/SES/EventBridge/S3/KMS resources
Then every required lifecycle action matches exactly and speculative/broad/runtime capabilities are absent
Policy size and attachment headroom
Given resolved default/pseudo parameter values
When inline and managed policy documents are compact JSON serialized
Then aggregate inline size is at most 9,472, each new managed policy is at most 5,120, attachment count has headroom, and adding Sponsor statements to inline fails the test
#124 regression protection
Given merged PR #18 and its exact Conversational worker/queue/mapping conditions
When #129 is applied to the source template
Then all effective #124 capabilities remain and no condition is broadened to arbitrary workers/functions
Negative least privilege
Given mutations that add
ses:SendEmail, SQS data-plane/purge, S3 object access, KMS crypto/grants, arbitrary PassRole/functions/logs, EventBridge bus writes, or unconditioned event-source createWhen focused policy tests run
Then each mutation fails
Clean source integration
Given the dirty sibling checkout and current remote main
When the engineer implements/rebases/reviews #129
Then only the isolated worktree contains the focused diff, final branch includes current
origin/main/PR #18, and the shared dirty checkout is untouchedCredentialed apply and rollback
Given a reviewed merged aws-infra commit and the current deployed
dataops-github-actionsstackWhen a credentialed operator previews/applies the change
Then it reaches
UPDATE_COMPLETE; if it fails, CloudFormation reaches a terminal rollback state and the operator can reapply the recorded pre-#129 template without manual deletion or policy editsOn-Call deployment verification
Given the deploy-role stack is updated and Sponsor send remains false
When the DataOps main deployment is rerun
Then CloudFormation creates/updates #124 and accepted #114 resources, generated role/function/log/mapping/queue/config-set/archive boundaries match source, no email is sent, logs/failure payloads remain sanitized, and On-Call records exact workflow/stack evidence
Source merge and human operator sequence
After Software Engineer, independent security Tester, and PM acceptance:
Commit in the isolated
aws-infrabranch withRefs DataTalksClub/dataops#129; do not push from or merge into the dirty sibling checkout.Push the isolated branch and open a focused
DataTalksClub/aws-infraPR to currentmain, following PR Import DTC Operations content directory #18’s shared-infrastructure precedent.Require the independent security review and green SAM lint/tests before merge.
Merge to
aws-infra/main. Source merge alone does not change AWS and does not unblock deployment.Add
humanto Repair DataOps OIDC SES configuration-set tag-on-create permission #129 and leave it open.[HUMAN]Record the exact pre-Repair DataOps OIDC SES configuration-set tag-on-create permission #129aws-infra/maincommit/template, inspect the CloudFormation change set, then run fromaws-infra/sandbox/dataops:Confirm
UPDATE_COMPLETEand inspect the effective role’s inline/attached policy names and size boundaries. If apply fails, confirm terminal rollback; if a completed update must be reverted, deploy the recorded pre-Repair DataOps OIDC SES configuration-set tag-on-create permission #129 template with the same named-IAM capability. Do not manually delete roles/policies or DataOps application resources.Rerun the current DataOps deployment first with
SponsorCommunicationSendEnabled=false. No template/HMAC secret value or real sponsor address is part of this issue.Relaunch On-Call. Verify actual function/role/log patterns, Sponsor worker mapping/recovery rule, SES-event async config, retained encrypted failure queue, exact SES configuration set/event destination, private archive bucket/key, default-off sending, effective deploy/runtime IAM, and redacted logs/failure inputs. Also verify Add transactional approval and durable execution recovery #124’s previously blocked worker/queue/mapping resources now deploy.
Close Repair DataOps OIDC SES configuration-set tag-on-create permission #129 only after operator apply and On-Call evidence pass. Automate reviewed sponsor communications from booking milestones #114’s separate private-secret configuration and controlled test-inbox send remain its own human gates.
Rollback
The supported rollback is CloudFormation, not manual IAM editing:
aws-infra/maintemplate/commit and review the change set;UPDATE_ROLLBACK_COMPLETEor another explained terminal state before retry;CAPABILITY_NAMED_IAM, confirm the two Sponsor managed policies detach/delete and PR Import DTC Operations content directory #18 remains effective, then rerun read-only policy checks;Dependencies and boundaries
aws-infra/main; Repair DataOps OIDC SES configuration-set tag-on-create permission #129 must preserve that fix, and a later combined apply may deliver both.Out of Scope