Repository navigation
Make CI failures reach Telegram, and fix stale CI facts #557
Description
Part of #554, change set A. Rulings: #554 rulings 1, 5, 12, 15, 16.
Summary
The runner watchdog and chatroom E2E fail silently. A Sentry major keeps failing CI, two job timeouts are too long, and five CI facts are wrong.
This issue sets three repo secrets, makes the watchdog warn, alerts on red chatroom E2E pushes and parks the Sentry major. It cuts two timeouts to 15 min and fixes the facts. It removes no pipeline time.
Depends on / Blocks
- Depends on: nothing. Blocks: nothing.
- Change set A: Make CI failures reach Telegram, and fix stale CI facts #557 is its own commit, after Roll back only after the deploy has started rolling services #555.
- Shares files with: Start the extension and webapp unit tests without waiting for backend validation #558 and Run only the CI suites a pull request or push can affect #563 (
prod.docs.plus.yml), and Make the webapp Docker installs fast and cacheable #559 and Prune Docker build cache toward a free-disk target and keep rollback images #564 (apps/webapp/docker/Dockerfile.bun, AGENTS.md §Docker). If one of them lands first, Step 0 finds the moved lines.
Current behaviour
Links are pinned to d0a53756f8676556358b6adc6c2397d2ed7c64c8.
-
The notifier.
scripts/ci/notify-telegram.sh:1-19takes<env-file|->and a text. With-, it readsTELEGRAM_BOT_TOKENandTELEGRAM_CHAT_IDfrom the environment. It never fails the caller. With empty credentials it prints::warning::Telegram credentials or message text unavailable; skipping notificationand exits 0.runner-watchdog.ymlcalls it with-(lines 39 and 43).- The prod deploy and observability jobs run on the
prod.docs.plusrunner and pass a host env file (prod.docs.plus.yml:965,observability.docs.plus.yml:142). A GitHub-hosted job cannot read that file, so hosted jobs need the repo secrets.
-
The watchdog exits silently (
runner-watchdog.yml:26-35):if [ -z "${GH_TOKEN}" ] || [ -z "${TELEGRAM_BOT_TOKEN}" ] || [ -z "${TELEGRAM_CHAT_ID}" ]; then exit 0; fi
- The header at lines 1-5 says "Silent no-op until all three secrets exist."
RUNNER_WATCHDOG_TOKENis a fine-grained PAT with repoAdministration: read, because the workflowGITHUB_TOKENcannot read the runners API.- The 100 runs from 2026-10-01 up to run 39388630259 (2026-10-20) all ended
success. In run 39388630259, the step log showsGH_TOKEN,TELEGRAM_BOT_TOKENandTELEGRAM_CHAT_IDas empty. A set secret prints as***. So every run took the silentexit 0. - On 2026-10-20,
gh secret list --repo docs-plus/docs.pluslists none of the three names. The two self-hosted runners,prod.docs.plusandstage.docs.plus, were bothonline. The watchdog alerts on either one.
-
Chatroom E2E has no failure alert (
chatroom-e2e.yml:38-73). The last step isRetain results and failure screenshotsat lines 62-73. The job runs onubuntu-latest. It was red on 5mainpushes in a row, and all four(build):deploys among them shipped:Chatroom run Commit subject Deploy run of the same push 37431580968 (build): front back37431581531, success 37440315062 (build): front37440315600, success 37904017847 refactor(supabase): deslop comments in the new migrations and scriptsnone 37906965146 (build): front back37906965365, success on attempt 2 37976785336 (build): back front observability37976785981, success -
Dependabot has no Sentry ignore (
.github/dependabot.yml:39-63). The parked majors list ends with@babel/*at lines 62-63.apps/webapp/next.config.js:3readsconst { withSentryConfig } = require('@sentry/nextjs').- Three
CI/CD Productionruns failed withwithSentryConfig is not a function: 36905204833, 37907231973 and 39275329280. The failing jobs were🧪 Webapp Unit Tests: 110514623896, 113744098640 and 118138638728. - PR chore(deps): bump @sentry/nextjs from 10.76.2 to 11.8.0 #539 (
@sentry/nextjs10.76.2 to 11.8.0) is still open. @sentry/nextjsis^10.76.2inapps/webappandapps/admin-dashboard.@sentry/bunis^10.76.2inapps/hocuspocus.server.
-
The timeouts in
prod.docs.plus.yml. Sample: the 85 runs from 2026-09-01 up to run 39414945889 (2026-10-20). That is 53 Build Verification jobs and 245 Ext jobs that were not skipped, cancelled jobs included. None hit a limit.extension-tests:timeout-minutes: 25at line 288. Median 151 s, max 289 s: job 113733096854,🧪 Ext (extension-hypermultimedia).build:timeout-minutes: 35at line 354. Median 191 s, max 232 s: job 113734704565.deploy:timeout-minutes: 45at line 426. Its comment at lines 423-425 explains why. It stays.
-
The stale facts.
- 6a. Extension suite time.
prod.docs.plus.yml:117-118says the suite takes "~14 min" (Step 5.4 quotes it). Measured: median 151 s, max 289 s per suite. - 6b. Webapp install cache.
AGENTS.md:290ends: "They pay from the next(build):deploy: both webapp installs should printCACHED, andnext buildshould fall below 277 s." Measured in six deploy jobs: 0 of the 12 webappbun installsteps printedCACHED. Thenext buildstep took 185.4–262.5 s, so that half is true. Jobs: 118549257854, 118217497479, 118142985559, 115281929767, 114780192678, 114262648443. - 6c. Deploy gap.
AGENTS.md:297: "Real deploy gaps are median 2.2 h with 1 of 9 over 24 h, so a mount would be warm on roughly eight deploys in nine." The same claim is inapps/webapp/docker/Dockerfile.bun:214-218. Measured: 20 runs have a🚀 Deploy Productionjob that endedsuccess, from run 33632421079 (2026-09-02) to run 39414945889 (2026-10-20). Of the 19 gaps between them, 10 are over 24 h. The median gap is 24.6 h. To list the 20 runs, run this and ignore any run after 39414945889:gh run list --repo docs-plus/docs.plus --workflow prod.docs.plus.yml \ --created 2026-09-01..2026-10-20 --limit 200 --json databaseId,createdAt \ --jq '.[] | "\(.databaseId) \(.createdAt)"' | sort -k2 | while read -r id at; do c=$(gh api "repos/docs-plus/docs.plus/actions/runs/${id}/jobs?per_page=100" \ --jq '.jobs[] | select(.name == "🚀 Deploy Production") | .conclusion') [ "${c}" = success ] && echo "${id} ${at}" done
- 6d. Production protection.
observability.docs.plus.yml:47-48claims a required-reviewer protection (Step 6 quotes it). On 2026-10-20,gh api repos/docs-plus/docs.plus/environments/production --jq .protection_rulesreturns[]. - 6e. Workflow docs.
.github/WORKFLOWS.md:38-42says the watchdog is a "No-op until theTELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_IDrepo secrets are set". It omitsRUNNER_WATCHDOG_TOKEN. Line 98 lists the notifier callers withoutchatroom-e2e.
- 6a. Extension suite time.
Target behaviour
- Secrets.
gh secret list --repo docs-plus/docs.plusshowsRUNNER_WATCHDOG_TOKEN,TELEGRAM_BOT_TOKENandTELEGRAM_CHAT_ID. - Watchdog, secrets set. Every run lists the runners. It prints
✅ All 2 self-hosted runners online, or it sends the existing Telegram alert. - Watchdog, a secret missing. The run stays green. It shows one
::warning::annotation that names each missing secret. It warns; it does not fail. - Chatroom E2E. A failed run on a
pushtomainsends one Telegram alert with the short SHA and the run URL. It goes to the same chat as the deploy-failure alert. A pull request, aworkflow_dispatchor a cancelled run sends nothing.- A run that hits the job's
timeout-minutes: 25sends no alert. GitHub cancels a job at its timeout, so the alert step skips. The run does not end green.
- A run that hits the job's
- Dependabot. It opens no new PR for a
@sentry/*semver-major update. Minor and patch updates still arrive. - Timeouts.
buildis 15 min,extension-testsis 15 min, anddeployis still 45 min. - Facts. Each of 6a to 6e states the measured value with its date or source.
- Time to live. It does not change. No step on the deploy path changes.
Implementation steps
"Comment here" means a comment on this GitHub issue, never a code comment.
Step 0. Read and check the base
- Read every file in Read first. Run
git log --oneline -1andgit status. Work onmainin the current working tree. Do not create a branch or a worktree. - Confirm that the lines this issue changes still match the quotes in Current behaviour and the Before blocks in Steps 5 and 6:
sed -n '1,5p;26,35p' .github/workflows/runner-watchdog.yml sed -n '38,73p' .github/workflows/chatroom-e2e.yml sed -n '53,63p' .github/dependabot.yml sed -n '117,118p;288p;354p;426p' .github/workflows/prod.docs.plus.yml sed -n '47,48p' .github/workflows/observability.docs.plus.yml grep -n -E 'should print .CACHED|eight deploys in nine' AGENTS.md sed -n '214,218p' apps/webapp/docker/Dockerfile.bun sed -n '38,42p;98p' .github/WORKFLOWS.md grep -n 'builder prune' .github/workflows/prod.docs.plus.yml
- If a sibling issue moved a line, find it with
grep -non the quoted text. If the text itself changed, or a quoted fact is already fixed, stop and comment here with the new text.- At the base, the
builder pruneline isdocker builder prune -af --filter "until=24h" 2>/dev/null || true. If that line is gone, Prune Docker build cache toward a free-disk target and keep rollback images #564 has landed, as [Tracking] Faster, change-aware CI/CD pipeline #554 ruling 15 allows. Write Step 7.2 as written. Skip Step 7.3, because Prune Docker build cache toward a free-disk target and keep rollback images #564 already replaced that Dockerfile sentence. At the Review checkpoint, expect seven files, withoutapps/webapp/docker/Dockerfile.bun.
- At the base, the
- Confirm the tools:
actionlint -versionandshellcheck --version. actionlint runs shellcheck onrun:blocks when it is installed.
Step 1. Maintainer: set the three repo secrets
The maintainer does this step after the Review checkpoint push and after Verification step 0. The code is safe without the secrets: both new paths warn and exit 0. The builder does not set secrets and never sees their values.
- Run
gh api repos/docs-plus/docs.plus/actions/runners --jq '.runners[] | [.name, .status] | @tsv'. Expectprod.docs.plusandstage.docs.plus, bothonline. If one is offline, every watchdog run will send an alert. Bring it online or remove it first, and comment here. - In Settings → Secrets and variables → Actions → Repository secrets, add:
TELEGRAM_BOT_TOKEN: the same bot the deploy-failure alert already uses.TELEGRAM_CHAT_ID: the same chat the deploy-failure alert already uses.RUNNER_WATCHDOG_TOKEN: a new fine-grained PAT. Set Resource owner todocs-plus, or the token cannot see the repo. Select only thedocs-plus/docs.plusrepository. Grant Administration: Read-only and nothing else. Note its expiry date.- If the org requires approval for fine-grained PATs, approve the token in the org settings. An unapproved token fails the runner list, and the
Could not list runnerspath only warns.
- Check the names, not the values:
Expect three lines. Comment here with the PAT expiry date. Do not post any value.
gh secret list --repo docs-plus/docs.plus | grep -E '^(RUNNER_WATCHDOG_TOKEN|TELEGRAM_BOT_TOKEN|TELEGRAM_CHAT_ID)\b'
Step 2. .github/workflows/runner-watchdog.yml: warn on missing secrets
- Replace the header lines 1-5 with these four lines:
# Pages Telegram when a self-hosted runner drops offline OR none is registered # (GitHub auto-deregisters runners dead >14 days — zero registered is terminal). # Needs RUNNER_WATCHDOG_TOKEN (PAT, repo Administration:read; GITHUB_TOKEN cannot # read runners). Without all three secrets, each run warns and checks nothing.
- In
🩺 Probe runner status, replace the one line that startsif [ -z "${GH_TOKEN}" ]. Indent each new line by 10 spaces, like the rest of therun: |body:MISSING="" [ -n "${GH_TOKEN}" ] || MISSING="${MISSING} RUNNER_WATCHDOG_TOKEN" [ -n "${TELEGRAM_BOT_TOKEN}" ] || MISSING="${MISSING} TELEGRAM_BOT_TOKEN" [ -n "${TELEGRAM_CHAT_ID}" ] || MISSING="${MISSING} TELEGRAM_CHAT_ID" if [ -n "${MISSING}" ]; then echo "::warning::Runner watchdog checked nothing. Missing repo secrets:${MISSING}" exit 0 fi
- Leave the rest of the
run:block as it is. TheCould not list runnersline already warns when the token cannot list runners. - Check:
actionlint -config-file .github/actionlint.yaml .github/workflows/runner-watchdog.yml. Expect no output.
Step 3. .github/workflows/chatroom-e2e.yml: notify on a failed push
Add this step after Retain results and failure screenshots (after line 73), at the same indent:
- name: 📣 Notify chatroom E2E failure
if: failure() && github.event_name == 'push'
env:
TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
run: |
bash scripts/ci/notify-telegram.sh - \
"🧪 Chatroom E2E FAILED on main (${GITHUB_SHA::9}): ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"failure()is false for a cancelled run. Thechatroom-e2e-${{ github.ref }}group cancels a superseded push, so a cancel sends nothing.- The workflow
pushtrigger ismainonly (line 19), so the event check is enough. - The step runs after the artifact upload, so an alert problem never loses the screenshots.
- Do not change
permissions, triggers, the jobifortimeout-minutes: 25. - Check:
actionlint -config-file .github/actionlint.yaml .github/workflows/chatroom-e2e.yml. Expect no output.
Step 4. Skip: do not park the Sentry major
Commit 7c3d7d26c (2026-10-24) took @sentry/nextjs and @sentry/bun 11.9.0. next.config.js now loads withSentryConfig from @sentry/nextjs/config, so the major no longer fails CI. Do not add the @sentry/* ignore below. Skip the Dependabot acceptance item and Verification item 3. At the Review checkpoint, expect seven files, without .github/dependabot.yml.
Append under ignore:, after the @babel/* entry (after line 63), at the same indent:
# Sentry 11 fails next.config.js with `withSentryConfig is not a function`
# (runs 36905204833, 37907231973, 39275329280). Remove this entry in the
# Sentry 11 migration.
- dependency-name: '@sentry/*'
update-types: ['version-update:semver-major']- The glob covers
@sentry/nextjsand@sentry/bun. - Check:
bunx prettier --check .github/dependabot.yml. Expect a pass.
Step 5. .github/workflows/prod.docs.plus.yml: timeouts and the suite-time comment
Edit from the bottom up, so the line numbers below stay true. If they moved, find each line by its job id.
- In
build, change line 354:# Max 232 s, median 191 s (85 runs to 2026-10-20). 15 min still kills a hang. timeout-minutes: 15
- In
extension-tests, change line 288:# Max 289 s, median 151 s per suite (85 runs to 2026-10-20). 15 min still kills a hang. timeout-minutes: 15
- Do not change
deploy(timeout-minutes: 45, line 426) or any other job's timeout. - Replace lines 117-118 only. Keep the rest of that comment block:
- Before:
# The clean-room extension suite (~14 min) is the pipeline's long pole and is # a hard deploy gate. It only needs to run when something that affects an
- After:
# Each clean-room extension suite takes a median of 151 s (max 289 s, 2026-10-20) # and is a hard deploy gate. It only needs to run when something that affects an
- Before:
- Check:
actionlint -config-file .github/actionlint.yaml .github/workflows/prod.docs.plus.yml. Expect no output.
Step 6. .github/workflows/observability.docs.plus.yml: the protection comment
Replace lines 47-48 only:
- Before:
# No `environment: production` — it would inherit the production environment's # required-reviewer protection and gate every monitoring deploy on manual approval.
- After:
# No `environment: production`. It has no protection rules today (2026-10-20), # but any rule added to it later would also gate every monitoring deploy.
Check: actionlint -config-file .github/actionlint.yaml .github/workflows/observability.docs.plus.yml. Expect no output.
Step 7. AGENTS.md §Docker and Dockerfile.bun: the cache facts
Change only the clauses quoted here. Do not rewrite any other text in AGENTS.md. New text keeps to at most 25 words per sentence.
- AGENTS.md:290. Replace the last sentence, quoted in 6b, with:
Measured 2026-10-20 on deploy jobs `118549257854`, `118217497479`, `118142985559`, `115281929767`, `114780192678` and `114262648443`: `next build` fell to 185–262 s. But 0 of the 12 webapp `bun install` steps printed `CACHED`. #559 owns that fix.
- AGENTS.md:297. Replace the sentence quoted in 6c with:
Keep "Measured 2026-08-28 …" and every other sentence of that bullet unchanged.
Deploy gaps were longer than that first read. Of the 19 gaps between successful deploys from 2026-09-02 to 2026-10-20, 10 were over 24 h (median 24.6 h). So under the `until=24h` prune, a mount was warm on only about half of deploys.
apps/webapp/docker/Dockerfile.bun:214-218. Replace the five comment lines that start# The cache mount is the only lever on this stepwith four lines:# The cache mount is the only lever on this step: the inlined git hash means the # layer can never be reused. Up to 2026-10-20 the deploy pruned build cache with # `until=24h`, and 10 of 19 deploy gaps were longer, so the mount was warm on # about half of deploys. The runner copies only .next/standalone and .next/static.
- Find the lines by that first comment line, not by the
RUNline below them. Measure and trim the Next.js build-trace step in the webapp build #568 rewrites thatRUNline (line 219 at the base). - Do not change the
RUNline or any instruction. A comment does not enter a BuildKit cache key.
- Find the lines by that first comment line, not by the
- In AGENTS.md, write
#559as a link,[#559](https://github.com/docs-plus/docs.plus/issues/559), like the#437links in.github/WORKFLOWS.md. - Check:
Expect a pass. The diff must show exactly two changed lines in AGENTS.md and the comment lines in
bun run check:agent-docs bunx prettier --check AGENTS.md git diff -U0 -- AGENTS.md apps/webapp/docker/Dockerfile.bun
Dockerfile.bun.
Step 8. .github/WORKFLOWS.md: the workflow entries
- Replace line 40 with:
- Needs three repo secrets: `RUNNER_WATCHDOG_TOKEN` (fine-grained PAT, repo Administration: read), `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID`. Without them each run warns with the missing names and checks nothing
- Under the
workflows/chatroom-e2e.ymlentry (after line 42), add:- A failed `push` run sends a Telegram alert through `scripts/ci/notify-telegram.sh`, with the `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID` repo secrets. It is not a deploy gate
- On line 98, change "shared by prod deploy, observability, and runner-watchdog failure steps" to "shared by prod deploy, observability, runner-watchdog, and chatroom-e2e failure steps".
- Check:
bunx prettier --check .github/WORKFLOWS.md. Expect a pass.
Review checkpoint
Do not commit, push, stash or amend inside these steps. Show the maintainer:
git status
git diff --stat
git diff -- .github AGENTS.md apps/webapp/docker/Dockerfile.bunExpect changes in exactly these eight files: .github/workflows/runner-watchdog.yml, .github/workflows/chatroom-e2e.yml, .github/workflows/prod.docs.plus.yml, .github/workflows/observability.docs.plus.yml, .github/dependabot.yml, .github/WORKFLOWS.md, AGENTS.md and apps/webapp/docker/Dockerfile.bun.
The maintainer commits and pushes per SP2 (#554), as its own commit, with the subject ci: alert on silent CI failures and fix stale CI facts. That subject does not deploy. Run SP1 (#554) before the push.
Acceptance criteria
-
gh secret list --repo docs-plus/docs.plusshowsRUNNER_WATCHDOG_TOKEN,TELEGRAM_BOT_TOKENandTELEGRAM_CHAT_ID. - After merge, a
workflow_dispatchofrunner-watchdog.ymlprints✅ All 2 self-hosted runners online. - With a secret missing, the step log shows
##[warning]Runner watchdog checked nothing. Missing repo secrets:with each missing name, and the run is green. -
chatroom-e2e.ymlhas📣 Notify chatroom E2E failurewithif: failure() && github.event_name == 'push', after the artifact upload. -
.github/dependabot.ymlignores@sentry/*withupdate-types: ['version-update:semver-major']. - In
prod.docs.plus.yml,buildandextension-testshavetimeout-minutes: 15, anddeploystill hastimeout-minutes: 45. -
grep -rn -E '~14 min|required-reviewer|2\.2 h|eight deploys in nine|should print .CACHED' .github AGENTS.md apps/webapp/dockerprints nothing. -
actionlint -config-file .github/actionlint.yamlpasses on the four changed workflows. -
bun run check:agent-docsandbunx prettier --checkon the changed Markdown and YAML files pass. - The first deploy after merge ends
success, and its M1 is at most 1510 s, the baseline maximum. A lower M1 also passes. - A
Measured, <date>comment is on this issue (see Accountability).
Verification
Before the commit:
actionlint -config-file .github/actionlint.yaml \
.github/workflows/prod.docs.plus.yml \
.github/workflows/chatroom-e2e.yml \
.github/workflows/runner-watchdog.yml \
.github/workflows/observability.docs.plus.yml
bunx prettier --check .github/dependabot.yml .github/WORKFLOWS.md AGENTS.md
bun run check:agent-docs- Not needed:
bash .github/scripts/parse-build-trigger.test.sh. Triage does not change. - Not needed: a
workflow_dispatchwithdry_run_subject. No jobif,needsor gate changes. Only twotimeout-minutesvalues change.
After the push, the maintainer or the builder runs:
-
Watchdog, secrets missing. Before Step 1, run
gh workflow run runner-watchdog.yml --repo docs-plus/docs.plus --ref main. Then rungh run view <run-id> --repo docs-plus/docs.plus --log | grep '##\[warning\]'. Expect one line that namesRUNNER_WATCHDOG_TOKEN,TELEGRAM_BOT_TOKENandTELEGRAM_CHAT_ID, and a green run. Record the run id. Then do Step 1, then step 1 below. -
Watchdog, secrets set.
gh workflow run runner-watchdog.yml --repo docs-plus/docs.plus --ref main gh run list --repo docs-plus/docs.plus --workflow runner-watchdog.yml --limit 1 gh run view <run-id> --repo docs-plus/docs.plus --log | grep -E 'All [0-9]+ self-hosted runners online|##\[warning\]'
Expect
✅ All 2 self-hosted runners onlineand no warning. The log also prints the script text, so match##[warning], the rendered annotation, not::warning::. -
Telegram path. Do not force a failure on
mainto test the chatroom alert. On the next chatroom-e2e push run, rungh run view <run-id> --repo docs-plus/docs.plus --json jobs --jq '.jobs[].steps[].name'. It must list📣 Notify chatroom E2E failure, skipped on a green run. Delivery is proven at the next real red push. -
Dependabot. After the next monthly Dependabot run,
gh pr list --repo docs-plus/docs.plus --search "@sentry in:title" --state openshows no new semver-major PR. PR chore(deps): bump @sentry/nextjs from 10.76.2 to 11.8.0 #539 predates the ignore. The maintainer closes it by hand. -
Benchmark metrics. This issue moves no metric. Read them from the first
(build):deploy run after merge. Measure the CI/CD after benchmark for v2.3 #569 defines each metric.Metric Read from Baseline (14 deploys, 2026-09-22 to 2026-10-20) Expected M1 time to live that run median 1090 s, range 777–1510 s at most 1510 s, no change from this issue M2g that run 🚀 Deploy Productionstart median 417 s, which is M2g plus 2–3 sno change M3 that run median 546 s no change M5 that run 57–70 s no change 🏗️ Build Verificationand every🧪 Ext (…)job durationthat run max 232 s and 289 s under 900 s, and none cancelledat the limit
Accountability
- The maintainer (@HMarzban) sets the three secrets in Step 1, confirms the PAT scope, and records its expiry date in a comment here.
- The builder runs the Verification steps after the first push and the first deploy after the merge. The builder records per SP3 ([Tracking] Faster, change-aware CI/CD pipeline #554), with the metric rows above. The same comment adds:
- the watchdog run id that shows the missing-secret warning;
- the watchdog dispatch run id and its
✅ All N self-hosted runners onlineline; - the longest
🏗️ Build Verificationand🧪 Ext (…)job durations in that run; - the first chatroom-e2e push run id that shows the new step.
- At the next red chatroom-e2e push, the maintainer confirms that the Telegram alert arrived and adds the run id to the same thread.
- Two weeks after merge, the maintainer checks the latest watchdog run log for the
✅line, then rules on a chatroom gate. - Before the PAT expires, the maintainer rotates
RUNNER_WATCHDOG_TOKEN. An expired PAT makes theCould not list runnerswarning fire, not an alert. - Stop conditions:
- Any
🏗️ Build Verificationor🧪 Ext (…)job is cancelled at the 15-minute limit. Revert that onetimeout-minutesline, and comment here with the job id. - The watchdog sends a false offline alert, while
gh api repos/docs-plus/docs.plus/actions/runnersshows every runner online. Rungh workflow disable runner-watchdog.yml --repo docs-plus/docs.plus, and comment here. Reverting Step 2 does not help, because the alert lines below the new check are unchanged. - The chatroom alert fires on a pull request or a cancelled run. Revert Step 3.
- Any
Rollback
- Revert per SP4 ([Tracking] Faster, change-aware CI/CD pipeline #554). First run
git show --stat <commit>. It must list only this issue's eight files. If it lists any other file, stop and comment here. - If only one step must go, revert only that file's hunk. The steps do not depend on each other.
- The secrets may stay. With the code reverted, the watchdog uses them as before, and nothing else reads them. To remove them, the maintainer deletes them in Settings → Secrets and variables → Actions and revokes the PAT in their GitHub account settings.
- Reverting Step 4 lets Dependabot open
@sentry/*majors again.
Docs to update
.github/WORKFLOWS.md§Active workflows, theworkflows/runner-watchdog.ymlandworkflows/chatroom-e2e.ymlentries (lines 38-42), and §Related files, the Telegram notifier line (line 98). Step 8 has the text.AGENTS.md§Docker, two sentences (lines 290 and 297). Step 7 has the text.- No other docs.
docs/self-hosting/configuration.mddocuments the host Telegram values for Grafana, not these repo secrets.
Out of scope
- Do not gate deploys on chatroom E2E. The tracker lists it under Not doing.
- Do not harden the runners. That is Restrict the production self-hosted runner to main deploys #552.
- Do not change what a sibling owns. Step 7 corrects only the gap facts.
- Prune Docker build cache toward a free-disk target and keep rollback images #564: the
until=24hprune, the disk guard and the cache mount. - Make the webapp Docker installs fast and cacheable #559: the webapp install cache.
- Write a timing table to every CI/CD run summary #556: a timing table or
reportjob. - Roll back only after the deploy has started rolling services #555: the rollback and the deploy-failure alert.
- Start the extension and webapp unit tests without waiting for backend validation #558 and Run only the CI suites a pull request or push can affect #563: the
back-validationneeds and the change scopes.
- Prune Docker build cache toward a free-disk target and keep rollback images #564: the
- Do not migrate to Sentry 11. Do not bump or pin
@sentry/*in anypackage.json. - Do not change
deploytimeout-minutes: 45, the chatroom-e2etimeout-minutes: 25, or any other job's timeout. - Do not alert on a chatroom job timeout. If one happens, record its run id here.
- Do not add
environment: productionto the observability job. The comment fix is text only. - Do not add another alert channel, metrics stack or dashboard ([Tracking] Faster, change-aware CI/CD pipeline #554 shared rule "No new platform"). Telegram and
scripts/ci/notify-telegram.shalready exist. - Do not add tests. No step adds logic that a test would pin.
- Do not edit the Dockerfile instructions. Only the comment changes. The settled Docker rules in [Tracking] Faster, change-aware CI/CD pipeline #554 §Shared rules still apply.
- Do not change the Bun version or the metascraper pin. The Bun floor is 1.4.0.
Read first
AGENTS.md, in full, then §Docker and §Simplified English Mandate..github/WORKFLOWS.md.github/workflows/runner-watchdog.yml.github/workflows/chatroom-e2e.yml.github/workflows/prod.docs.plus.yml, thechanges,extension-tests,buildanddeployjob headers..github/workflows/observability.docs.plus.yml, lines 30-60..github/dependabot.ymlscripts/ci/notify-telegram.shapps/webapp/docker/Dockerfile.bun, lines 195-219..github/actionlint.yaml.cursor/skills/tech-writer/SKILL.md§Simplified English (house standard), for the new AGENTS.md text.
Activity