Skip to content

Improve observability of GitHub workflows - #3231

Draft
whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures
Draft

whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures

Conversation

@whi-tw

@whi-tw whi-tw commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this pull request solve?

Trello card: https://trello.com/c/jPeqHHPu/772-improve-the-observability-of-our-github-workflows

There is no view of how often CI fails or slows down over time. This adds a weekly "CI digest" workflow: how often this repo's workflows failed in the last week compared with the previous 12 weeks, which jobs and specs failed, and how long runs take. It calls the shared workflow added in govuk-forms/forms-deploy#2365, where the digest is described in full.

It is a dry run for now. Each run writes the digest to its job summary and nothing is posted to Slack until we have seen a few weeks of output. Going live is a one-line change in forms-deploy, with nothing to change here.

It also sets timeout-minutes on every job that runs on a runner, so a hung job fails fast instead of running for up to six hours. The limits are generous and well above the observed run times, so they only catch hangs. Jobs that only call a reusable workflow cannot set one; their limits come from forms-deploy. Once govuk-forms/forms-deploy#2332 merges, the shared workflow lint fails any job that has none.

Earlier commits on this branch called the shared workflows from a forms-deploy branch, and those runs passed; they can be found in this branch's Actions history. The digest only runs from the default branch, so its first run in Actions will be after merge.

Before merging

Posting to Slack needs the org secret SLACK_WEBHOOK_GOVUK_FORMS_DEPLOYMENTS to be available to this repo. That is only needed once the digest goes live, not to merge this.

Things to consider when reviewing

  • Ensure that you consider the wider context.
  • Is it clear what the code is doing?
  • Do the commit messages explain why the changes were made?
  • Has all relevant documentation been updated?

@whi-tw
whi-tw force-pushed the observe-workflow-failures branch 2 times, most recently from 71a1ead to 9b2682d Compare October 20, 2026 13:16
@whi-tw
whi-tw requested a review from theseanything October 20, 2026 13:21
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 9b2682d to d3a59cc Compare October 20, 2026 13:29
There is no view of how often CI fails or slows down over time, so this
runs a weekly summary by calling the shared workflow in forms-deploy. It
is a dry run for now: the digest goes to the job summary, not Slack.
Jobs default to a six hour timeout, so a hung job ties up a runner and
delays the merge queue. Limits are well above normal run times.
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from d3a59cc to 14df8bb Compare October 29, 2026 13:38
@github-actions

Copy link
Copy Markdown

🎉 A review copy of this PR has been deployed! You can reach it at: https://pr-3231.admin.review.forms.service.gov.uk/

It may take 5 minutes or so for the application to be fully deployed and working. If it still isn't ready
after 5 minutes, there may be something wrong with the ECS task. You will need to go to the integration AWS account
to debug, or otherwise ask an infrastructure person.

For the sign in details and more information, see the review apps wiki page.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant