feat(harness): distinguish long-run vs startup Copilot CLI failures via token-count classification

[Copilot]

The Copilot harness treats two structurally different 0-turn failures identically: startup crashes (negligible cost) and long-run exits where the CLI wrapper exits non-zero after the agent completed substantial work (30+ min, 119–210 AIC lost with no recovery path).

Changes

copilot_harness.cjs

• Added LONG_RUN_TOKEN_THRESHOLD = 10000 constant
• Added extractTokenCountFromOutput(output) — sums "total_tokens" fields from CLI stdout/stderr (emitted in debug JSON response blocks); returns 0 when no token data is present (startup failures)
• classifyCopilotFailure() now accepts optional tokenCount; returns "long_run_exit" when hasOutput=true && tokenCount > 10000. All named error classes (auth, quota, etc.) continue to outrank this
• Failure log now includes tokenCount=N attemptDurationMs=N for audit visibility
• Extended terminal safe-output suppression to cover "long_run_exit" in addition to "partial_execution"

copilot_harness.test.cjs

• 15 new unit tests covering extractTokenCountFromOutput edge cases and long_run_exit classification boundary/priority

Classification logic

// Before
return detection.hasOutput ? "partial_execution" : "no_output";

// After
if (detection.hasOutput && (detection.tokenCount ?? 0) > LONG_RUN_TOKEN_THRESHOLD)
  return "long_run_exit";
return detection.hasOutput ? "partial_execution" : "no_output";

Retry behavior is unchanged — long_run_exit follows the same retry path as partial_execution. The classification distinction is surfaced in run logs and enables audit tooling to report wasted AIC by failure class.

• Fixes [Code Quality] Distinguish pre-agent vs long-run 0-turn copilot CLI failures for smarter retry #46506

---

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• pi.dev

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "pi.dev"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · 12.4 AIC · ⌖ 10.3 AIC · ⊞ 6K · ◷
_{Comment /souschef to run again}

[Copilot]

Copilot wasn't able to review any files in this pull request.

[pelikhan]

@copilot apply changes

[Copilot]

@copilot apply changes

Applied all changes described in the PR. Commits:

1. feat(harness): distinguish long-run vs startup Copilot CLI failures via token-count classification — adds LONG_RUN_TOKEN_THRESHOLD, extractTokenCountFromOutput(), updates classifyCopilotFailure() with long_run_exit classification, includes tokenCount/attemptDurationMs in failure logs, extends terminal safe-output suppression to long_run_exit, and adds 15 unit tests (241 passing).
2. fix: rename attemptStartTime to attemptStart — minor naming consistency fix from code review.

[pelikhan]

@copilot fix https://github.com/github/gh-aw/actions/runs/29682304721/job/88180508768?pr=46557

[Copilot]

@copilot fix https://github.com/github/gh-aw/actions/runs/29682304721/job/88180508768?pr=46557

The CI failure is a TypeScript error in add_labels.cjs (introduced via the merge from main): the JSDoc type cast at line 115 was missing parentheses around the expression, so TypeScript didn't recognize it as a cast:

- const metadataLabel = /** @type {{name?: string}} */ l;
+ const metadataLabel = /** @type {{name?: string}} */ (l);

Fix committed in 8b9d775. Typecheck now passes cleanly.

[gh-aw-bot]

@copilot please run the pr-finisher skill and rerun checks once the branch is up to date.

Failed checks:

• js-typecheck
• impacted-js-tests

Run: https://github.com/github/gh-aw/actions/runs/29683215651

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• pi.dev

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "pi.dev"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · 12.4 AIC · ⌖ 10.3 AIC · ⊞ 6K · ◷
_{Comment /souschef to run again}

[pelikhan]

@copilot make the 10000 threshold configurable through an env var

[Copilot]

@copilot make the 10000 threshold configurable through an env var

Done. LONG_RUN_TOKEN_THRESHOLD is now configurable via GH_AW_HARNESS_LONG_RUN_TOKEN_THRESHOLD. It follows the same resolver pattern as GH_AW_HARNESS_WATCHDOG_TIMEOUT_MS: defaults to 10000 when unset/invalid, falls back to default for negative values, and accepts 0 to effectively disable the long_run_exit classification. 5 new unit tests added (246 total passing).

[Copilot]

@copilot make the 10000 threshold configurable through an env var

Done in 7d49887. LONG_RUN_TOKEN_THRESHOLD is now configurable via GH_AW_HARNESS_LONG_RUN_TOKEN_THRESHOLD. Defaults to 10000; falls back to the default for negative/invalid values; 0 effectively disables long_run_exit classification.

[gh-aw-bot]

@copilot please run the pr-finisher skill, investigate and fix the failed checks below, refresh the branch if needed, and rerun checks.

Failed checks:

• lint-js

Run: https://github.com/github/gh-aw/actions/runs/29684195741

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• pi.dev

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "pi.dev"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · 16.2 AIC · ⌖ 9.8 AIC · ⊞ 7.5K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot please run the pr-finisher skill, investigate and fix the failed checks below, refresh the branch if needed, and rerun checks.

Failed checks:...

Fixed. The lint-js failure was caused by parse_copilot_log.test.cjs needing prettier formatting — a fix for this was committed to main (#46592) after our last merge. Merged main into the branch (86149a1) to include the formatting fix; lint-js should now pass.