Skip to content

Allow OIDC deployment of Telegram rollout alarms and notifications #131

Description

@alexeygrigorev

Allow OIDC deployment of Telegram rollout alarms and health schedules

Status: corrected Gate B source integrated/applied/effective; production dark 17-alarm graph verification blocked by #136 (through #140 and #141)
Tags: enhancement, human, assistant, infra, testing, P1
Depends on: corrected #128 source/transform graph satisfied; successful normal CI deployment after #136 (which depends on #140, which depends on #141)
Blocks: #128 production/HUMAN closure only — source integration and IAM apply are complete
Related: #124, #129, #130

Current review state

The blocked bbb0841653603ee0830747fbb0df43979df0a0eb candidate was superseded by corrected Gate B source ddbaf2893bf2985d32e5a33310fe57e2f4ac9452. Independent Architecture, IAM/security, Tester, and PM reviews passed; the corrected source was integrated, applied through its reviewed CloudFormation change set, verified effective, and found IN_SYNC. No source correction or IAM apply remains.

Normal-CI attempt 2 produced no CloudWatch, SNS, or log IAM denial, but downstream rollback removed the rollout topic before the 17-alarm and health-schedule graph could remain live. The only remaining criterion is the normal-CI dark-deployment observation after #136#140#141 clears.

Historical source boundary

The accepted corrected change was limited to:

sandbox/dataops/template.github-actions.yaml
sandbox/dataops/tests/test_template_github_actions.py
sandbox/dataops/tests/fixtures/issue_128_observability.synthetic.yaml

That source stage is complete. It preserved #124/#129/#130 and introduced no DataOps application-source or CloudFormation service-role change.

Replacement #128 graph contract

Final review uses fresh replacement #128 source/build/transforms, not the old
fixture.

The observability family remains:

  • one ConversationalRolloutAlarmTopic, exact name
    ${FullDocsStackName}-conversational-rollout-alarms, alias/aws/sns, ARN
    output only, no subscription/topic policy/contact;
  • exactly three retained 30-day custom log groups:
    backend, execution, and result-dispatcher;
  • exactly 17 simple AWS::CloudWatch::Alarm resources with exact
    ${FullDocsStackName}-conversational-<suffix> names and the one topic action;
  • heartbeat suffixes execution-worker-heartbeat-stale,
    recovery-heartbeat-stale, and result-dispatcher-heartbeat-stale use
    TreatMissingData: breaching;
  • the other 14 alarms retain TreatMissingData: notBreaching;
  • no subscription, topic policy, dashboard, composite, anomaly, canary,
    metric/filter/subscription, or additional observability family.

Keep the same 17 suffixes already enumerated by bbb0841:

execution-outcome-unknown
execution-failure-queue-visible
recovery-oldest-due-age
execution-worker-heartbeat-stale
recovery-heartbeat-stale
result-delivery-outcome-unknown
result-delivery-oldest-pending
result-dispatcher-heartbeat-stale
result-delivery-failed
telegram-webhook-failures
model-failures
voice-failures
photo-failures
typefully-draft-failed
media-cleanup-failed
retention-cleanup-failed
execution-failed-safe

#128 adds one execution-conditioned two-minute health-pulse Schedule event to
the existing execution worker. The SAM transform therefore adds one
stack-prefixed AWS::Events::Rule and one AWS::Lambda::Permission targeting
ConversationalExecutionWorkerFunction, in addition to the existing recovery
and result-dispatcher schedule resources. Update the fixture/final graph test
with the exact generated logical IDs, state/condition, input, target, and
permission ownership.

Do not add EventBridge or Lambda actions to the observability managed policy.
The accepted Gate A deploy policy already covers stack-prefixed EventBridge
rule lifecycle/tagging and Add/RemovePermission on the exact worker function
family. Structural tests must prove the new transformed resources fit those
existing statements without changing them.

Managed-policy placement and quotas

Retain one generated-name
ConversationalRolloutObservabilityDeployPolicy, attached only to
GitHubActionsDeployRole. Do not set ManagedPolicyName, grow the nearly full
inline policy, mix domains into Sponsor/dispatcher policies, or grant policy
self-administration.

The corrected compact observability policy must be <= 5,632 characters,
leaving at least 512 below IAM's 6,144-character managed-policy limit. Preserve
all prior policy ceilings and the four-policy steady-state/eight-policy
replacement attachment model. Recompute from resolved documents.

Exact corrected OIDC permissions

CloudWatch alarm lifecycle and tag-on-create

Retain cloudwatch:PutMetricAlarm on only the exact 17 alarm ARNs, with:

  • a non-null cloudwatch:AlarmActions; and
  • every action equal to the one exact rollout topic ARN.

Retain cloudwatch:DescribeAlarms and cloudwatch:DeleteAlarms on those exact
17 ARNs.

Add cloudwatch:TagResource on only those same 17 exact alarm ARNs. It is
required because CloudFormation automatically supplies system tags, the alarm
resource supports tags, and AWS maps tagged alarm creation to
cloudwatch:TagResource in addition to PutMetricAlarm.

Do not place TagResource in the alarm-action-conditioned Put statement:
that condition is not present on the separate tag call. Keep it in a separate
statement or with the unconditioned exact-ARN lifecycle statement.

Do not condition CloudWatch tag permission on the three aws: system tag keys.
Although CloudWatch exposes generic request-tag condition keys, its alarm
tagging guidance explicitly says tag names beginning aws: cannot be used for
tag-based authorization control. The exact 17 ARNs are the reliable boundary.

Add no UntagResource, ListTagsForResource, metric data plane, state forcing,
composite/dashboard/anomaly/canary, or wildcard alarm capability.

Authoritative references:

SNS topic lifecycle and tag-on-create

Retain exactly:

sns:CreateTopic
sns:DeleteTopic
sns:GetTopicAttributes
sns:SetTopicAttributes

on the one exact topic ARN.

Add a separate sns:TagResource statement on that same exact topic ARN.
Require a non-null aws:TagKeys request and use
ForAllValues:StringEquals (or an independently proven equivalent) limiting
the request to:

aws:cloudformation:logical-id
aws:cloudformation:stack-id
aws:cloudformation:stack-name

The condition must reject a missing key set and any customer/extra tag key.
SNS documents aws:TagKeys for TagResource; the official CloudFormation SNS
provider supplies system tags on create.

Do not add ListTagsForResource, ListSubscriptionsByTopic,
GetDataProtectionPolicy, UntagResource, publish, subscription,
confirmation, topic-policy, endpoint, KMS, wildcard, or other optional action.
The provider deliberately tolerates denial of the optional reads; they are not
proven deployment blockers.

Authoritative references:

Retained log groups

Retain the accepted exact policy:

  • logs:CreateLogGroup and logs:PutRetentionPolicy on the base and :*
    forms for only the three custom /dataops/${FullDocsStackName}/conversational
    groups;
  • logs:DescribeLogGroups alone on Resource: "*";
  • no delete, stream/event data plane, query/export/filter/subscription/policy,
    KMS, delivery, or additional wildcard.

Acceptance criteria

  • Changes remain limited to the deploy-role template, focused tests, and
    corrected public-safe fixture.
  • One generated observability policy attaches only to the deploy role and
    preserves Add transactional approval and durable execution recovery #124/Repair DataOps OIDC SES configuration-set tag-on-create permission #129/Repair dispatcher generated-role IAM prefix for CloudFormation rollback #130 policies unchanged.
  • Put/read/delete permissions cover only the 17 exact alarm ARNs and retain
    the exact alarm-action condition.
  • cloudwatch:TagResource covers only the same 17 exact alarm ARNs, has no
    misleading system-tag condition, and no CloudWatch list/untag action is
    added.
  • SNS lifecycle covers only the exact topic; separate
    sns:TagResource is exact-topic and accepts only the three non-null
    CloudFormation system tag keys.
  • No optional tolerated SNS read, publish/subscription, topic-policy, KMS,
    wildcard, or other action is added.
  • Log-group policy remains exact and unchanged except for size bookkeeping
    if statement ordering changes.
  • Corrected compact policy is <=5,632 characters and all prior quota/
    attachment headroom remains safe.
  • Final fixture matches Roll out and cut over the Telegram conversational MVP #128's 17 alarms with exactly three
    missing-as-breaching heartbeat alarms and 14 not-breaching alarms.
  • Fixture/transform assertions include the new worker health EventBridge
    rule/Lambda permission and prove existing Gate A IAM covers them.
  • Mutation tests reject missing/broad tag actions, extra tag keys, absent
    aws:TagKeys, broad resources, optional reads, data plane, self-admin,
    existing-policy drift, and every prior adversarial mutation.
  • Fresh final Roll out and cut over the Telegram conversational MVP #128 source/build/transforms and all sandbox/dataops tests,
    SAM lint, resolved-size checks, and diff checks pass.
  • Independent IAM/security and PM review accept the corrected commit range
    and exact final Roll out and cut over the Telegram conversational MVP #128 graph before source integration.
  • [HUMAN/AWS] Corrected Gate B source is integrated; credentialed apply reached terminal success; the effective policy matches the accepted permissions and is in sync.
  • [HUMAN] A normal-CI Roll out and cut over the Telegram conversational MVP #128 dark deployment leaves the exact 17-alarm, topic, and health-schedule graph live without authorization denial, unexpected paging, or leakage.

Required tests

Run:

uv run --with pytest python -m pytest sandbox/dataops/tests -q
sam validate --lint --template-file sandbox/dataops/template.github-actions.yaml
git diff --check 2a1b95f...HEAD

Parsed structural/mutation tests must prove:

Fresh-build #128 into an isolated directory and translate dark, execution,
ingress, voice, photo, Typefully-external, and fully staged states. Compare
actual logical IDs, names, actions, conditions, schedules, permissions,
retention/retain policies, and prohibited families. Do not use stale
.aws-sam output or self-compare an in-test constant.

Sequencing and apply

  1. Roll out and cut over the Telegram conversational MVP #128 SWE produces the replacement runtime/SAM candidate.
  2. Assistant and Architect accept its heartbeat/maintenance behavior and fresh
    graph.
  3. Allow OIDC deployment of Telegram rollout alarms and notifications #131 SWE corrects the existing isolated IAM candidate and fixture.
  4. Independent IAM/security Tester and PM accept the exact corrected range.
  5. Integrate accepted source into aws-infra/main with
    Refs DataTalksClub/dataops#131; add human.
  6. A credentialed operator reviews change sets/live attachments and applies
    dataops-github-actions with CAPABILITY_NAMED_IAM.
  7. Verify terminal stack state and effective policy before Roll out and cut over the Telegram conversational MVP #128 commit/push.
  8. On-Call verifies the default-off app deployment creates the exact graph and
    no alarm pages. Roll out and cut over the Telegram conversational MVP #128 owns subscription confirmation and alarm canaries.

Rollback uses CloudFormation and the prior accepted template, never manual IAM
editing or destructive resource cleanup.

Out of scope

#128 runtime implementation, extra alarms/metrics, subscription/contact
management, notifications, provider/Telegram calls, secrets, application
deployment outside CI, manual IAM edits, dashboards/composites/anomalies/
canaries, log data plane, optional provider reads, and any other repository or
stack.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1ImportantassistantAssistant modules and jobsenhancementNew or improved functionalityhumanCode done or issue blocked on human verificationinfraDeployment and infrastructuretestingTests and QA

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions