Skip to content

Improve observability of GitHub workflows - #2332

Open
whi-tw wants to merge 3 commits into
mainfrom
observe-workflow-failures
Open

whi-tw wants to merge 3 commits into
mainfrom
observe-workflow-failures

Conversation

@whi-tw

@whi-tw whi-tw commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this pull request solve?

Trello card: https://trello.com/c/jPeqHHPu/772-improve-the-observability-of-our-github-workflows

If a workflow fails on main or in the merge queue, nobody is told unless they happen to look. This adds a reusable workflow that posts to #govuk-forms-alert when that happens. Failures on pull requests are not reported, because the author already sees them. notify-failure.yml lists the workflows it watches by name, so a new workflow that runs on main or in the merge queue needs adding to that list.

It also sets timeout-minutes on every job, so a hung job fails fast rather than running for up to six hours, and adds a check to the shared workflow lint that fails if a job has none. forms-deploy was not running that lint on itself, so it now calls it.

The limits are deliberately generous (10 to 30 minutes) against p95 durations of roughly two minutes or less.

Before merging

  • An org-level SLACK_WEBHOOK_GOVUK_FORMS_ALERT secret exists and is available to the forms-* repos.
  • Run "Notify failure" manually (workflow_dispatch) to check a test message arrives.
  • Open the equivalent PRs in forms-admin, forms-runner, forms-product-page and forms-e2e-tests straight after merging. They run this repo's lint at @main, so their workflow changes will fail it until they have timeouts.

Things to consider when reviewing

  • Do the timeouts look sensible for each job?
  • Is the workflow_run approach, with a list of watched workflows, acceptable compared with adding a notify job to each workflow?

Copilot AI balanced review requested due to automatic review settings October 20, 2026 12:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The notification watch list omits the new workflow lint, leaving its merge-queue failures unreported.

1 open issue
What changed in this PR

Improves GitHub Actions observability with Slack failure alerts and explicit job timeouts.

Changes:

  • Adds reusable Slack notifications for watched workflows failing on main or in the merge queue.
  • Sets job timeouts across existing workflows.
  • Enforces timeouts through shared workflow linting and enables that lint locally.
File Description
.github/​workflows/​update-provider-locks.yml Adds a 10-minute timeout.
.github/​workflows/​trello-poster.yml Adds a 10-minute timeout.
.github/​workflows/​terraform-ci.yml Adds 20-minute job timeouts.
.github/​workflows/​team-review-requests.yml Adds a 10-minute timeout.
.github/​workflows/​shell-ci.yml Adds a 10-minute timeout.
.github/​workflows/​ruby-ci.yml Adds 15-minute job timeouts.
.github/​workflows/​reusable-review_apps_on_pr_close.yml Limits cleanup to 20 minutes.
.github/​workflows/​reusable-review_apps_on_pr_change.yml Limits updates to 30 minutes.
.github/​workflows/​reusable-notify-failure.yml Adds reusable Slack message delivery.
.github/​workflows/​reusable-lint-workflows.yml Enforces job timeouts and adds its own.
.github/​workflows/​reusable-build-image.yml Limits builds to 30 minutes.
.github/​workflows/​require-pinned-github-actions.yml Adds a 10-minute timeout.
.github/​workflows/​pre-commit.yml Adds 10-minute job timeouts.
.github/​workflows/​pipeline-visualiser.yml Adds 20-minute job timeouts.
.github/​workflows/​notify-failure.yml Watches failures and supports test notifications.
.github/​workflows/​lint-workflows.yml Runs shared lint for pull requests and merge groups.
.github/​workflows/​guardrail_unique_job_names.yml Adds a 10-minute timeout.
.github/​workflows/​guardrail_tf_version_upgrade.yml Adds a 10-minute timeout.
.github/​workflows/​guardrail_matching_pentester_email_addresses.yml Adds a 10-minute timeout.
.github/​workflows/​guardrail_matching_pentester_cidr_blocks.yml Adds a 10-minute timeout.
.github/​workflows/​forms-cli.yml Adds a 15-minute timeout.

Review effort: Balanced

Comment thread .github/workflows/notify-failure.yml

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Slack delivery, secret availability, and coordinated rollout to repositories consuming the shared lint require human confirmation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 75725af to 07b3c9b Compare October 20, 2026 12:56
Copilot AI balanced review requested due to automatic review settings October 20, 2026 12:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Slack delivery and the coordinated rollout of stricter shared linting to consuming repositories require human verification.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 20, 2026 13:02
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 07b3c9b to c3d9974 Compare October 20, 2026 13:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Slack delivery needs manual verification, and shared timeout enforcement requires a coordinated rollout across consuming repositories.

No open issues 🎉

Review effort: Balanced

Failures on main or in the merge queue happen where nobody is watching
the run, so a broken main can go unnoticed. Failures on pull requests
are left alone because the author already sees them.

The sending logic is a reusable workflow so the other forms-* repos can
call it from their own workflow_run trigger.
Without timeout-minutes a hung job runs for up to six hours before
GitHub fails it, holding a runner and delaying the failure signal.
Limits are deliberately generous (10 to 30 minutes) against p95
durations of roughly two minutes or less, so they only catch hangs.
Adds a check to the shared workflow lint so the other forms-* repos get
it too, and calls that lint from this repo, which did not run it. The
lint runs in the merge queue, so add it to the failure notifications.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 13:16
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from c3d9974 to b5f7b6a Compare October 20, 2026 13:16
@github-actions

Copy link
Copy Markdown

Warning

You are changing the Terraform version.

Before you merge this PR, you must apply the forms/pipelines
Terraform root from this branch in each of the forms environments.

make ENV forms/pipelines apply

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The notification secret references must match the provisioning checklist for Slack delivery to work under the documented setup.

1 open issue

Review effort: Balanced

Comment thread .github/workflows/notify-failure.yml
@whi-tw
whi-tw requested review from theseanything and a balanced review from Copilot October 20, 2026 13:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Slack delivery, secret availability, and the coordinated rollout to shared-lint consumers still require human verification.

1 open issue

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants