E2E status report — 2026-09-10 (gh-aw matrix)

[dsyme]

Run: https://github.com/githubnext/gh-aw-test/actions/runs/34432420259  ·  Trigger: schedule  ·  Generated: 2026-09-10T05:04:20Z  ·  Outcome: failure

Matrix summary

❌ Matrix incomplete: 3 failing test entries (57 failed test run(s)), 0 infrastructure error(s), 0 entries with no tests executed

entry	ref	mode	samples	exit code	passed	failed	status
main	main	source	false	0	1	0	✅ pass
main	main	source	true	1	107	16	❌ fail
release	v0.88.7	extension	true	1	109	15	❌ fail
pre-release	v0.89.0	extension	true	1	90	26	❌ fail

❌ Test errors — samples mode

test	v0.88.7	v0.89.0	main
test-copilot-call-workflow	✅	✅	❌
test-copilot-nosandbox-add-comment	❌	❌	❌
test-copilot-nosandbox-add-discussion-comment	❌	❌	❌
test-copilot-nosandbox-add-labels	❌	❌	❌
test-copilot-nosandbox-command	❌	❌	❌
test-copilot-nosandbox-create-discussion	❌	❌	❌
test-copilot-nosandbox-create-issue	❌	❌	❌
test-copilot-nosandbox-create-pull-request	❌	❌	❌
test-copilot-nosandbox-create-pull-request-review-comment	❌	❌	❌
test-copilot-nosandbox-create-repository-code-scanning-alert	❌	❌	❌
test-copilot-nosandbox-create-two-pull-requests	❌	❌	❌
test-copilot-nosandbox-custom-safe-outputs	❌	❌	❌
test-copilot-nosandbox-mcp	❌	❌	❌
test-copilot-nosandbox-push-to-pull-request-branch	❌	❌	❌
test-copilot-nosandbox-update-issue	❌	❌	❌
test-copilot-replace-label	❌	❌	❌
test-copilot-siderepo-add-comment	✅	❌	✅
test-copilot-siderepo-add-discussion-comment	✅	❌	✅
test-copilot-siderepo-add-labels	✅	❌	✅
test-copilot-siderepo-create-pull-request-review-comment	✅	❌	✅
test-copilot-siderepo-create-two-pull-requests	✅	❌	✅
test-copilot-siderepo-link-sub-issue	✅	❌	✅
test-copilot-siderepo-mcp	✅	❌	✅
test-copilot-siderepo-push-to-pull-request-branch-using-dispatch	✅	❌	✅
test-copilot-siderepo-sparse-push-to-pull-request-branch-using-dispatch	✅	❌	✅
test-copilot-siderepo-subdir-create-pull-request	✅	❌	✅
test-copilot-siderepo-update-issue	✅	❌	✅

✅ Test successes — samples mode

test	v0.88.7	v0.89.0	main
test-claude-create-issue	✅	✅	✅
test-codex-create-issue	✅	✅	✅
test-copilot-add-comment	✅	✅	✅
test-copilot-add-discussion-comment	✅	✅	✅
test-copilot-add-labels	✅	✅	✅
test-copilot-add-reviewer	✅	✅	✅
test-copilot-assign-milestone	✅	✅	✅
test-copilot-assign-to-agent	✅	✅	✅
test-copilot-assign-to-user	✅	✅	✅
test-copilot-close-discussion	✅	✅	✅
test-copilot-close-issue	✅	✅	✅
test-copilot-close-pull-request	✅	✅	✅
test-copilot-command	✅	✅	✅
test-copilot-create-check-run	✅	✅	✅
test-copilot-create-discussion	✅	✅	✅
test-copilot-create-issue	✅	✅	✅
test-copilot-create-pull-request	✅	✅	✅
test-copilot-create-pull-request-review-comment	✅	✅	✅
test-copilot-create-repository-code-scanning-alert	✅	✅	✅
test-copilot-create-two-pull-requests	✅	✅	✅
test-copilot-custom-safe-outputs	✅	✅	✅
test-copilot-dismiss-pull-request-review	✅	✅	✅
test-copilot-dispatch-workflow	✅	✅	✅
test-copilot-gh-steps	✅	✅	✅
test-copilot-hide-comment	✅	✅	✅
test-copilot-inline-sub-agents	✅	✅	✅
test-copilot-issue-intents	✅	✅	✅
test-copilot-link-sub-issue	✅	✅	✅
test-copilot-mark-pull-request-as-ready-for-review	✅	✅	✅
test-copilot-mcp	✅	✅	✅
test-copilot-mcp-code-quality	✅	✅	✅
test-copilot-missing-data	✅	✅	✅
test-copilot-missing-tool	✅	✅	✅
test-copilot-network-isolation	✅	✅	✅
test-copilot-noop	✅	✅	✅
test-copilot-push-to-pull-request-branch-using-dispatch	✅	✅	✅
test-copilot-push-to-pull-request-branch-using-slash-command	✅	✅	✅
test-copilot-remove-labels	✅	✅	✅
test-copilot-reply-to-pull-request-review-comment	✅	✅	✅
test-copilot-report-incomplete	✅	✅	✅
test-copilot-resolve-pull-request-review-thread	✅	✅	✅
test-copilot-restore-memory-custom-job	✅	✅	✅
test-copilot-set-issue-field	✅	✅	✅
test-copilot-set-issue-field-builtin-rejection	✅	✅	✅
test-copilot-set-issue-type	✅	✅	✅
test-copilot-siderepo-create-discussion	✅	✅	✅
test-copilot-siderepo-create-issue	✅	✅	✅
test-copilot-siderepo-create-pull-request	✅	✅	✅
test-copilot-siderepo-sparse-create-pull-request	✅	✅	✅
test-copilot-skills-frontmatter	✅	✅	✅
test-copilot-submit-pull-request-review	✅	✅	✅
test-copilot-submit-pull-request-review-locked	✅	✅	✅
test-copilot-unassign-from-user	✅	✅	✅
test-copilot-update-discussion	✅	✅	✅
test-copilot-update-issue	✅	✅	✅
test-copilot-update-pull-request	✅	✅	✅
test-copilot-update-release	✅	✅	✅
test-copilot-upload-asset	✅	✅	✅
test-copilot-upload-code-coverage	✅	✅	✅

✅ Test successes — inference mode

test	v0.88.7	v0.89.0	main
test-copilot-create-issue	—	—	✅

🤖 Agent triage prompt

Copy the block below into a coding agent (e.g. Copilot) to investigate the failures:

You are triaging failures from the gh-aw-test E2E suite.
Run: https://github.com/githubnext/gh-aw-test/actions/runs/34432420259
Repository under test: github/gh-aw (the gh-aw CLI/compiler).
Test harness repository: githubnext/gh-aw-test (this repo; runner is e2e.sh).

Focus: gh-aw main (commit 099efdda60fe), sampling = true (--use-samples; AI engine was NOT called).

Failed tests (main, samples=true):
  - test-copilot-replace-label  run: https://github.com/githubnext/gh-aw-test/actions/runs/34433328610
  - test-copilot-call-workflow  run: https://github.com/githubnext/gh-aw-test/actions/runs/34433934732
  - test-copilot-nosandbox-create-issue  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186471947
  - test-copilot-nosandbox-create-discussion  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472253
  - test-copilot-nosandbox-create-pull-request  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472249
  - test-copilot-nosandbox-create-two-pull-requests  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472420
  - test-copilot-nosandbox-mcp  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472322
  - test-copilot-nosandbox-create-repository-code-scanning-alert  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186471647
  - test-copilot-nosandbox-custom-safe-outputs  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472369
  - test-copilot-nosandbox-add-comment  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186473087
  - test-copilot-nosandbox-add-discussion-comment  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472910
  - test-copilot-nosandbox-add-labels  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186473244
  - test-copilot-nosandbox-update-issue  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186472750
  - test-copilot-nosandbox-command  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186601099
  - test-copilot-nosandbox-push-to-pull-request-branch  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186601160
  - test-copilot-nosandbox-create-pull-request-review-comment  run: https://github.com/githubnext/gh-aw-test/actions/runs/34186601135

Goal: for EACH failed test above, access the GitHub
Actions logs for the run above (and the per-entry artifacts e2e-<label>-samples-<bool>,
which contain e2e-test-*.log, e2e-output.log and fails.txt), determine the root
cause, and categorize the failure as exactly one of:

  1. TRANSIENT — flaky/infra/network/rate-limit/timing; not a real defect.
     Action: note it and recommend a re-run (./e2e.sh rerun).
  2. TEST-FRAMEWORK BUG — a defect in this repo's harness (e2e.sh), a workflow
     source file (.github/workflows/test-*.md), a sample, or CI config.
     Action: propose a concrete fix (file + change) in githubnext/gh-aw-test.
  3. GH-AW BUG — a defect in github/gh-aw itself (compiler output, runtime
     engine behaviour, safe-output handling, etc.).
     Action: open an issue in github/gh-aw with a minimal repro, the failing
     test name, the gh-aw ref/mode/samples combination, and links to the
     relevant log lines. Check for an existing open issue first and link it
     instead of filing a duplicate.

Steps:
  - Use 'gh run view <run-id> --log' and 'gh run download <run-id>' to fetch logs/artifacts.
  - Read AGENTS.md in githubnext/gh-aw-test for harness conventions before proposing fixes.
  - Group failures by suspected root cause; the same gh-aw bug may explain several.
  - Produce a table: test | category | root cause | recommended action | issue/PR link.
  - Only open github/gh-aw issues for category 3, and only after confirming no duplicate exists.

Previous status reports (closed by this run)

• E2E status report — 2026-09-09 (gh-aw matrix) #14677

[dsyme]

Superseded by #14990 — #14990