Skip to content

feat(cloudformation): Add DevOps Agent alarm investigations template - #38

Open
LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn
Open

feat(cloudformation): Add DevOps Agent alarm investigations template#38
LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn

Conversation

@LorenzoBoccaccia

Copy link
Copy Markdown

Description

Add a CloudFormation addon that forwards a single CloudWatch alarm to a DevOps Agent generic (HMAC) webhook to open an investigation, plus a combined cloudformation/ README documenting it alongside the existing skill-policies template.

The signing Lambda hardens the forwarding path: layered SSRF egress checks with connect-to-validated-IP pinning, HMAC-SHA256 request signing, alarm-ARN match to reject spoofed/mismatched events, retry-stable incidentId for dedup, staleness cutoff matching the retry window, and least-privilege IAM. Reserved concurrency and log retention are configurable; the webhook HMAC key is read from a Secrets Manager secret (SecretString or SecretBinary).

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change

Testing

This change is a CloudFormation automation add-on (not a skill), so Agent Skill
Eval does not apply
. It was validated at four levels — template, unit, live AWS,
and a real DevOps Agent webhook.

1. Template validation

  • aws cloudformation validate-template — passed (6 parameters recognized,
    CAPABILITY_IAM).
  • Full YAML parse with CloudFormation intrinsic tags — well-formed.

2. Local unit / adversarial tests of the inline signing Lambda (python3.12, the
target runtime) — 24/24 passing, covering:

  • SSRF egress classifier: blocks RFC1918, loopback, link-local, 169.254.169.254
    (IMDS), CGNAT 100.64/10, TEST-NET/reserved, and IPv4-embedded-in-IPv6
    (::ffff:, NAT64 64:ff9b::, 6to4); allows public v4/v6.
  • Log-injection: control chars / newlines / NUL stripped and ARN capped at 2048.
  • Secret parsing: raw string, JSON key fields, and UTF-8 SecretBinary; rejects
    empty / missing-field / non-string.
  • Staleness cutoff (8h) and ALARM-state gating.
  • Alarm-ARN match: mismatched / empty / missing resources[0] rejected and logged.
  • Retry-stable incidentId (event id, then alarmArn+state-timestamp; never
    wall-clock).
  • HMAC signature: base64(HMAC-SHA256(key, "timestamp:payload")).
  • End-to-end against a local TLS mock that recomputed the HMAC and validated the
    JSON schema through the real _post / pinned-TLS connect.

3. Live AWS deploy-and-teardown (throwaway dev account, all resources tagged,
fully torn down afterwards):

  • Stack reached CREATE_COMPLETE (IAM role, inline-packaged Lambda, EventBridge
    rule, log group, conditional reserved concurrency).
  • Invoked the deployed Lambda with a synthetic ALARM event: exercised Secrets
    Manager fetch → HMAC sign → SSRF resolve → TLS POST and handled the endpoint
    response.
  • Spoofed event (wrong alarm ARN) → skipped: alarm ARN mismatch.
  • Confirmed DeletionPolicy: Delete behavior; stack, log group, and secret removed.

4. Real DevOps Agent webhook contract (manual) — posted one signed request to a
live generic (HMAC) webhook: HTTP 200, confirming the signing scheme
(x-amzn-event-timestamp / x-amzn-event-signature) and the payload schema
(eventType/incidentId/action/priority/title/description/timestamp/
data) are accepted as-is.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from d636396 to 6189121 Compare August 18, 2026 09:08
AllowedPattern: '^arn:aws[a-zA-Z-]*:secretsmanager:.+'
ConstraintDescription: Must be a Secrets Manager secret ARN.

AgentName:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: AgentName parameter: rename + make optional

AgentName is required but purely cosmetic — it only sets the DevOpsAgent= tag and, per its own description, "does not affect routing." Two small issues:

Misleading name. "Agent name" reads as if it selects/routes to a specific agent, so a deployer may think it must match their real agent or that a wrong value breaks delivery. Suggest AgentNameTag to signal it's metadata.
Needlessly required. Forcing a value that has no functional effect adds friction.
Proposed:

Rename AgentName → AgentNameTag (param, the four !Ref tag blocks, and the ParameterLabels/ParameterGroups entries).
Make it optional: Default: !Ref AWS::StackName, drop MinLength: 1.
Trim description to: "Value applied as the DevOpsAgent tag on created resources (identification / cost allocation only; does not affect routing)."

@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from 6189121 to 90b32a0 Compare August 18, 2026 15:03
Forward a single CloudWatch alarm to a DevOps Agent generic webhook so
the alarm opens an investigation. One stack per alarm: an EventBridge
rule scoped to the alarm ARN, an input transformer that builds the
incident payload, an API destination and connection that POST it with
the webhook API key, and a role limited to invoking that destination.
No Lambda function and no code.

incidentId is the EventBridge event id, which is constant across
retries, so redeliveries deduplicate instead of opening a second
investigation. The retry policy caps delivery at 32 attempts over eight
hours and stops retrying events older than that.

Give each template its own directory named after the template file,
holding the template and a README.md, and add a README for the existing
skill-policies template. Update the references to the skill-policies
template path that the move would otherwise break, and correct the
repository name in the two contributor-facing convention docs.
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from 90b32a0 to 9239bc0 Compare August 19, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants