Skip to content

ci: upload test reports when a watchdog-dumped job times out - #880

Merged
rkennke merged 2 commits into
mainfrom
ci/watchdog-dump-on-timeout
Oct 22, 2026
Merged

rkennke merged 2 commits into
mainfrom
ci/watchdog-dump-on-timeout

Conversation

@rkennke

@rkennke rkennke commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What

The Upload test reports step in the glibc amd64 and glibc aarch64 test jobs now also runs when the job was cancelled and the gdb watchdog left a thread dump (build/logs/gdb-watchdog.log):

if: failure() || (cancelled() && hashFiles('build/logs/gdb-watchdog.log') != '')

Why

The gdb watchdog (#760) dumps the native threads of a hung test JVM after 20 minutes, so that the dump "lands in the test-reports artifact" before the 180-minute job timeout. But when the timeout fires, GitHub ends the job as cancelled, not failed. prepare_reports.sh (an always() step) still copies the dump into test-reports/, but the upload step is if: failure(), so it is skipped and the dump is lost. That is exactly the case the watchdog exists for.

Seen on #868: test-linux-glibc-amd64 (8-ibm, debug, regular) hung in the regular suite, most likely the IBM SDK 8 JVMTI deadlock from #760, which so far had been seen only in the slow suite. The watchdog fired at 20 minutes, the job timed out at 180 minutes, and no artifact was uploaded. The job log confirms that always() steps run after the timeout: Prepare reports ran, and Upload CI outcome ran but found no file.

The watchdog itself could not attach

This PR's own CI run hit the same 8-ibm hang, and the new upload worked. But the uploaded gdb-watchdog.log held only ptrace: Inappropriate ioctl for device for both JVMs. Ubuntu's yama ptrace_scope=1 only lets a process attach to its own descendants, so the watchdog has never produced a dump on these runners. The second commit fixes that:

  • It runs gdb with sudo, which these runners allow without a password (the ASan path already uses sudo sysctl).
  • Before attaching, it sends the JVM SIGQUIT for Java-level stacks and lock owners. HotSpot prints them to stdout; J9 writes a javacore file, which prepare_reports.sh already collects.

What the uploaded test-raw.log did show: the hang is in DatadogSettingsTest.testRecordDatadogSetting (cpu=1ms, IBM SDK 8, CPU engine asgct), between the profiler start (6 libraries patched) and the stop's Restore libraries. That's the same test that hung on #868.

Scope

  • Only the two jobs with a gdb watchdog: the amd64 debug watchdog and the aarch64 ASan one, for both the upload condition and the watchdog commands.
  • Gating on the dump file keeps runs cancelled for other reasons, e.g. superseded by a newer push under ci.yml's cancel-in-progress, from uploading anything, unless the watchdog had already fired.
  • Nothing else changes: the step conditions for successful and failed runs are the same as before, and the watchdog still only fires after 20 (amd64) or 25 (aarch64 ASan) minutes.

🤖 Generated with Claude Code

A test JVM that hangs runs the job into its 180-minute timeout, which ends
the job as cancelled, not failed. The gdb watchdog's thread dump is then
copied into test-reports by prepare_reports.sh (an always() step), but the
"Upload test reports" step only runs on failure(), so the dump is lost in
exactly the case the watchdog exists for.

Also upload the test reports on cancellation when the watchdog left a dump
(build/logs/gdb-watchdog.log), in the two jobs that have a watchdog. Runs
cancelled for other reasons (e.g. superseded by a newer push) still upload
nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rkennke
rkennke requested a review from a team as a code owner October 21, 2026 14:14
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T14:16:50.596990Z 80f1cde PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The upload conditions preserve failure uploads and allow cancelled jobs to upload reports when a watchdog dump exists. Report preparation already runs after cancellation, so the collected dump remains available for upload.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 80f1cde

@dd-octo-sts

dd-octo-sts Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 600c4730

@dd-octo-sts

dd-octo-sts Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #39615128483 | Commit: 8bbc44f | Duration: 3h 0m 23s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - 🚫 - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 31 | Failed: 0 | Cancelled: 1


Updated: 2026-10-21 17:17:20 UTC

The watchdog's gdb has never produced a dump on the GitHub runners: Ubuntu's
yama ptrace_scope=1 only allows attaching to descendants, so every attach
failed with "ptrace: Inappropriate ioctl for device". Run it with sudo, as
the runners allow without a password.

Before attaching, send the JVM SIGQUIT as well, for the Java-level picture
(stacks, monitors and their owners): HotSpot prints it to stdout, J9 writes
a javacore file, which prepare_reports.sh already collects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-datadog-prod-us1

This comment has been minimized.

@rkennke
rkennke merged commit 8c7d432 into main Oct 22, 2026
1268 of 1316 checks passed
@rkennke
rkennke deleted the ci/watchdog-dump-on-timeout branch October 22, 2026 13:48
@github-actions github-actions Bot added this to the 1.52.0 milestone Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant