Skip to content

Add deterministic AI reporter qualification gates - #5999

Open
Sean Larkin (TheLarkInn) wants to merge 221 commits into
mainfrom
copilot/reporter-r8a-ai-gates
Open

Add deterministic AI reporter qualification gates#5999
Sean Larkin (TheLarkInn) wants to merge 221 commits into
mainfrom
copilot/reporter-r8a-ai-gates

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • add a deterministic, network-free corpus for bootstrap/version, configuration, input, dependency-tool, operation, cache, network/auth, plugin, cancellation, and internal failures, plus successful controls
  • compare AI, detailed plaintext, legacy, and full-detail file output with machine-readable blocking results and actionable per-case failures
  • preserve stable diagnostic codes, root-cause ordering, classified context, remediation, payload-only stdout, warning suppression, and correlated absolute full-log references
  • expose a qualification decision helper without wiring environment-based automatic reporter selection

Part of #5981.

Stack

The main-relative diff remains exactly 17 R8 paths. This PR does not copy or modify the telemetry projection from #5990; it applies the corresponding producer-identity privacy boundary to presented reporter output and artifacts. Merge remains gated on the privacy prerequisite, current-head CI and independent approval.

Corpus and gates

The current corpus has 11 failure cases and two successful controls. These are the measured Linux Node 22 results using the normal OS temporary directory (/tmp), not the obsolete initial normalized-size percentages:

Gate Blocking threshold Measured result
corpus.failure-cases >= 10 11
corpus.control-cases >= 2 2
actionability >= 100% 100
size.absolute <= 65536 bytes 953
size.compact-case <= 2048 bytes when both baselines are below 1024 bytes 945
size.per-case-vs-legacy <= 100% when baseline >= 1024 bytes 66.62
size.per-case-vs-plaintext <= 100% when baseline >= 1024 bytes 57.90
size.vs-legacy <= 50% 45.86
size.vs-plaintext <= 50% 43.45
determinism >= 100% 100
privacy >= 100% 100
full-log >= 100% 100
stdout-contract >= 100% 100
warning-contract >= 100% 100
size.invocation-boundary <= 65536 emitted UTF-8 bytes, including delimiters, with a valid final result and preserved supplied log reference 34100

Size gates count actual emitted UTF-8 bytes, including delimiters, without path normalization. Determinism alone compares normalized output hashes across three runs. The invocation-boundary gate includes a valid final result and preservation of the supplied full-log reference.

Existing path-sensitivity limitation: a long external session TMPDIR fails the unchanged 50% legacy-size gate at 51.08%; the published pre-alignment head and this head produce identical results in that environment. Normal OS temp passes at 45.86%. The additional 96 emitted path bytes per case are counted, not stripped. Both failing comparisons and the normal-environment control are retained; no threshold, corpus, timeout or byte-accounting rule was relaxed. This is not a universal pass for arbitrary temporary-directory lengths.

Corpus cases:

bootstrap-unsupported-node, configuration-invalid-json, input-unknown-project, dependency-package-manager, operation-build-failure, cache-restore-failure, network-auth-unauthorized, plugin-api-incompatible, logical-cancellation, internal-unexpected-error, fallback-mixed-privacy, success-no-warning, success-warning-only.

External-service-shaped failures use stable canonical event fixtures. Returned results contain case metadata, byte counts, booleans and normalized hashes; absolute paths, private identities and classified values are not stored in the qualification result.

Reproduce

cd libraries/reporter
rushx build
node scripts/runAiReporterQualification.js

The script prints the versioned machine-readable result and exits nonzero when any blocking gate fails.

Selection boundary

getQualifiedAiReporterDecision() validates built-in COPILOT_CLI detection and configured agentEnvironmentVariables, rejects absent or failed qualification results, and keeps RUSH_REPORTER=legacy authoritative.

It is intentionally not consumed by the Rush frontend in this PR. No environment variable automatically selects the AI reporter; the pre-major live path remains explicit --reporter or repository-experiment opt-in, and no-opt-in behavior remains legacy.

Current landing qualification (September 11, 2026)

Published 0a9534920dbc6c623c740bd39f5e46ea264379f0 preserves the former R8 head and landed main@130daa8 as its two parents. Existing source/API behavior, worker isolation, raw-size accounting and privacy gates remain unchanged. Consumer build/types/lint/API and 274 focused Linux tests passed. All 15 canonical gates passed under normal OS temp, with the long-temp-path limitation disclosed above. New hosted CI, independent approval and privacy #5990 landing are still required.

Earlier validation (pre-landing snapshot)

  • rush build --to @microsoft/rush
  • rushx _phase:test in libraries/reporter (314 tests)
  • deterministic corpus, including 3 repeated normalized runs
  • rushx _phase:test in libraries/rush-lib
  • rushx _phase:test in apps/rush
  • direct AI invalid-project failure: exit 1, payload-only NDJSON, actionable parser diagnostic
  • no-opt-in COPILOT_CLI=1 legacy behavior and RUSH_REPORTER=legacy override
  • rush check
  • rush change --verify --no-fetch

Non-goals

  • enabling environment-based automatic reporter selection
  • changing Rush 6 or daemon-aligned defaults
  • Heft child-reporter work
  • bootstrap handoff integration
  • telemetry projection changes

@TheLarkInn
Add Rush reporter repository configuration
8c6b049 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Add Rush reporter frontend controls
b4f0fc1 
Create the authoritative frontend reporter host before version selection, register global reporter controls, and preserve legacy output unless a non-legacy reporter is explicitly selected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Fix reporter frontend integration
2450d5f 
Consume the repository experiment before Rush version selection, keep agent detection out of pre-major defaults, strip frontend-only controls before engine handoff, and preserve legacy verbosity compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Emit shadow Rush lifecycle events
c6b0349 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Add feature-flagged operation event adapter
b80a277 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Fix operation reporter stream edge cases
481b4f5 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Harden reporter demo integration
cbff953 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@github-project-automation github-project-automation Bot moved this to Needs triage in Bug Triage Aug 28, 2026
@TheLarkInn Sean Larkin (TheLarkInn) mentioned this pull request Aug 28, 2026
Open
25 tasks
@TheLarkInn
Harden AI reporter qualification gates
8a64282 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn

Sean Larkin (TheLarkInn) commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

Independent /pr-agentmerge review completed; fixes are in ac3fd75.

Findings fixed

  • Full-log qualification previously inspected only the file reporter artifact. It now validates the AI-emitted absolute path, exact artifact correlation, format/completeness, existence, owner-only mode, required lifecycle content, external evidence, and failure/session/diagnostic correlation.
  • Actionability now checks exact public context values and required privacy markers instead of key presence.
  • The uniform 4 KiB synthetic blob was replaced by scenario-specific deterministic evidence. Aggregate ratios are supplemented by compact-case and per-case baseline gates so one oversized fixture cannot hide a regression.
  • Local-sensitive producer identity is retained in the owner-only full log, while secret producer identity/values remain redacted. Secret diagnostics preserve honest aggregate counts and truncation without exposing code/category/detail or suppressing fallback errors.
  • Future eligibility now requires both a passing qualification result and an explicit accepted privacy prerequisite. COPILOT_CLI/configured detection alone stays ineligible; RUSH_REPORTER=legacy remains authoritative. No telemetry implementation or Rush 6/default selection behavior was added.
  • Added an adversarial reporter regression test: dropping artifactAvailable drives full-log to 0% with all affected case names in the CI diagnostic.

Corpus evidence

  • 12 cases: 10 failures plus success and warning-only controls covering bootstrap/environment, configuration, input, dependency tool, operation/build, cache, network/auth, plugin, cancellation, internal, and success paths.
  • Three independent standalone corpus executions were byte-identical across every normalized SHA-256.
  • Gates: actionability/privacy/full-log/stdout/warnings/determinism all 100%; max AI case 976 B; compact max 969/2048 B; worst comparable per-case 68.47% of legacy and 72.00% of plaintext; aggregate 43.77% and 46.51%.
  • Machine results were scanned for temp paths, fixture secrets, secret producer identity, and private plugin identity; none were present.

Validation

  • Reporter build/lint/API Extractor and 319 tests passed.
  • rush test --to @microsoft/rush passed the reporter, rush-lib, apps/rush, and dependency chain.
  • rush check and stacked rush change --verify --target-branch origin/copilot/reporter-r5b-demo-reporters --no-fetch passed.

State / blockers

This was referenced Aug 28, 2026
Open
Merged
@TheLarkInn

Sean Larkin (TheLarkInn) commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

Combined deep review of current head ac3fd75530.

  1. HIGH local-sensitive messageEmitted text can be copied verbatim into AI stdout fallback output. Only public text may enter AI output. Use a redacted fallback and protected log reference otherwise.
  2. LOW the oversized-record marker relabels non-public envelopes as public while retaining source and scope. Preserve privacy or remove those fields.

@TheLarkInn
Fix reporter frontend argument and close lifecycle
0ad2e30 
Stop reporter control scans at the pass-through separator and add an exactly-once frontend close contract across success, failure, and termination paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Preserve Rush CLI reporter compatibility
4cb2448 
Keep reporter controls out of ts-command-line globals, gate incompatible engines before initialization, and enforce bounded signal and close-error behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Refine reporter flag ownership
00f16e1 
Preserve unsupported custom reporter values until frontend ownership is unambiguous, and narrow emergency legacy stripping to the reporter selection flag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Tolerate value-less custom reporter flags
a5f1f7c 
Probe reporter ownership without requiring a value, then enforce strict reporter parsing only after frontend ownership is established.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Fix operation stream review findings
e83fa59 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Fix reporter demo review findings
5aa1d2c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Sean Larkin (TheLarkInn) force-pushed the copilot/reporter-r5b-demo-reporters branch from e30aa36 to 7159901 Compare August 28, 2026 20:08
@TheLarkInn
Fix reporter watch and lock handling
965e0db 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Fix overlapping reporter watch cycles
ae5c475 
Track explicit operation graph iteration identities across lifecycle events and isolate reporter totals, diagnostics, and output spools by watch cycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Merge corrected review parent into R8 side
eb2f9df 
Recover the persisted, conflict-resolved merge index into an isolated workspace after previous agent handles became unavailable. Preserve both exact parents and all prior side-specific changes; validation follows on this recovered tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Refresh R2B reporter controls onto native-private trunk
c857a9b 
Preserve the exact published R2B slice and review corrections while reconciling native private members and replacing unbranded parser test objects with real execution paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Preserve rollback flags and primary file detail defaults
c4c05af 
Share separated-value recognition with stripping so valueless controls cannot consume legacy flags, and use debug only as the unrequested primary file log-level default.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Respect command ownership when consuming reporter controls
55e4328 
Consume --verbose only for known actions that do not define it and parse repository opt-in value controls only when they are not command-owned. Preserve native aliases, declared custom values, unresolved plugin namespaces, and pass-through arguments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Refresh R3A scoped producers onto native-private trunk
d71c2ff 
Retain the exact scoped producer API and WeakMap-backed plugin facades while preserving native-private parser/plugin members and real launch-boundary coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Clarify the frontend reporter channel identity contract
47d4f36 
Document both the typed event sink and the frontend-assigned sessionId in the cross-version handoff without changing its shape.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Refresh R3B shadow lifecycle onto native-private trunk
2fc47f5 
Preserve published early-failure, late-telemetry and operation-callback corrections; reconcile native lifecycle fields and telemetry references, with real branded parser regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Fix shadow lifecycle error correlation and final registration
852b04d 
Keep immutable errors intact, capture original pre-execution parser failures without changing legacy rendering, and observe final configured operation silence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge post-R2A native R3B into reviewed R3C parity
188e673 
Preserve the published R3C slice and review fixes while incorporating the authoritative server-rebased immediate parent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge reconciled R3C into published R5A operation adapter
313e8ff 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge reconciled R5A into published R5B reporter demos
f47e0ce 
Retain the exact reviewed R5B slice, including unfiltered canonical full-detail logs and declaration-aware help.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Normalize Rush cwd before analyzing repository inputs
1a13d64 
Resolve physical cwd at parser entry so native Windows short names and directory aliases match Git repository paths. Keep real watch cancellation coverage and add symlink/junction regressions without mocking input analysis or watcher behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Use native realpath to expand Windows short directory names
1feb3ba 
Native Node 24 and 26 validation showed that generic realpathSync and FileSystem.getRealPath retain 8.3 names. Use the existing native-realpath pattern to resolve the physical directory before configuration discovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge native physical-cwd watch fix into R5A
af6ef7d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge native physical-cwd watch fix into R5B
0548584 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge post-R2A native-cwd R5B parent into published R8
485dd44 
Retain the exact 17-path AI qualification slice and APIs, including original privacy, corpus, worker isolation, and unchanged gate budgets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Capture successful Git setup diagnostics in watch regression tests
d672867 
Explicit pipes prevent Git line-ending notices from being mirrored onto the parent test stderr. Real setup failures still throw with the original captured error text. Reproduced the actual Rush production gate with process-local core.autocrlf=true and core.safecrlf=warn: unchanged tests exited with warnings before the fix and passed cleanly after it, without changing CI warning policy or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge captured watch-fixture Git setup into R5A
db9a2fc 
Preserve the existing slice and current parent histories; only capture successful fixture setup diagnostics, without changing production behavior, warning policy, or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge captured watch-fixture Git setup into R5B
6a7180d 
Preserve the existing slice and current parent histories; only capture successful fixture setup diagnostics, without changing production behavior, warning policy, or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge canonical watch fixture Git stdio parent into R8
570625c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Compare physical dependency targets in package manager tests
08cd910 
Resolve both expected and actual link locations using native-backed realpath before comparing them. Add a real directory-alias regression that still rejects wrong and missing targets, and run it alongside the unchanged npm and Yarn integration workflows. Reproduced the previous lexical mismatch before the fix; the regression and complete suite pass under an invocation-owned aliased temporary root. No production code, dependency versions, or CI gates changed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge physical package-link assertions into R5A
69e2d82 
Preserve the existing slice and parent histories while carrying the validated test-helper correction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge physical package-link assertions into R5B
74a269d 
Preserve the existing slice and parent histories while carrying the validated test-helper correction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Merge canonical physical package-link test parent into R8
51f6e1c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
An error occurred while trying to automatically change base from copilot/reporter-r5b-demo-reporters to copilot/reporter-r5a-operation-adapter September 11, 2026 05:30
@TheLarkInn
Merge pinned landed core into R8 AI qualification
0a95349 
Retain the published qualification, worker isolation and privacy prerequisites while inheriting main 130daa8. Resolve squash-ancestry conflicts against source-equivalent R5B and retain both R10 guidance and the R8 qualification section. Do not import the R7 watch test delta.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
@TheLarkInn
Sean Larkin (TheLarkInn) changed the base branch from copilot/reporter-r5b-demo-reporters to main September 11, 2026 05:58
@mojaza
Mo Jazayeri (mojaza) requested a balanced review from Copilot September 11, 2026 18:12
Copilot AI reviewed Sep 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The privacy gate misses a JSON identity leak, and flushing can leave AI output buffered when no log exists.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 2 High severity · 1 Medium severity

New issues introduced by this change (3)
Severity Finding
High severity libraries/​reporter/​src/​qualification/​AiReporterQualificationCorpus.ts — The privacy gate does not check jsonOutput for the private producer identity. In the plugin case,… View comment
High severity libraries/​reporter/​src/​reporters/​ReporterRedaction.ts — Secret diagnostic values can still be emitted through aliases here: the event source is copied… View comment
Medium severity libraries/​reporter/​src/​reporters/​AiReporter.ts — flushAsync() now leaves all pending output buffered whenever the file reporter never publishes a… View comment
What changed in this PR

Adds deterministic, network-free qualification gates for AI reporter output, including size, privacy, actionability, determinism, and log validation.

Changes:

  • Adds the qualification corpus, evaluator, CLI runner, and selection decision helper.
  • Improves AI output budgeting, diagnostic context, ordering, and redaction tests.
  • Adds worker-isolated regression tests and public API documentation.
File Description
libraries/​reporter/​src/​test/​JsonAiReporter.test.ts Expands privacy, budgeting, and ordering tests.
libraries/​reporter/​src/​test/​helpers/​AiQualificationWorker.ts Runs isolated qualification mutations.
libraries/​reporter/​src/​test/​helpers/​AiQualificationTestSession.ts Manages worker lifecycle and cleanup.
libraries/​reporter/​src/​test/​FileReporter.test.ts Tests producer identity redaction.
libraries/​reporter/​src/​test/​AiReporterQualification.test.ts Tests gates and selection decisions.
libraries/​reporter/​src/​reporters/​ReporterRedaction.ts Extends diagnostic redaction handling.
libraries/​reporter/​src/​reporters/​AiReporter.ts Adds invocation-wide budgeting and richer diagnostics.
libraries/​reporter/​src/​qualification/​AiReporterQualificationCorpus.ts Defines and executes the deterministic corpus.
libraries/​reporter/​src/​qualification/​AiReporterQualification.ts Defines thresholds, evaluation, and decisions.
libraries/​reporter/​src/​index.ts Exports qualification APIs.
libraries/​reporter/​scripts/​runAiReporterQualification.js Adds the qualification CLI.
libraries/​reporter/​README.md Documents qualification behavior.
common/​reviews/​api/​rush-reporter.api.md Updates the API report.
common/​changes/​@rushstack/​rush-reporter/​review-r8-qualification_2026-09-09-13-00.json Records output-budget changes.
common/​changes/​@rushstack/​rush-reporter/​r8-qualification-test-isolation_2026-09-10.json Records test isolation changes.
common/​changes/​@rushstack/​rush-reporter/​r8-native-windows-qualification_2026-09-09.json Records status coalescing changes.
common/​changes/​@rushstack/​rush-reporter/​copilot-reporter-r8a-ai-gates_2026-08-28-08-45.json Records the qualification feature.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libraries/reporter/src/qualification/AiReporterQualificationCorpus.ts
Comment on lines +1027 to +1030
!aiOutput.includes(PRIVATE_PRODUCER) &&
!aiOutput.includes(PRIVATE_COMPONENT) &&
!humanPresentedOutput.includes(PRIVATE_PRODUCER) &&
!humanPresentedOutput.includes(PRIVATE_COMPONENT);
Comment thread libraries/reporter/src/reporters/ReporterRedaction.ts
Comment on lines +40 to +44
const source: IReporterEventEnvelope<unknown>['source'] = event.source;
if (event.type === 'diagnosticEmitted') {
const diagnostic: { readonly parameters?: Readonly<Record<string, IClassifiedValue>> } =
event.payload as {
readonly parameters?: Readonly<Record<string, IClassifiedValue>>;
};
const diagnostic: {
readonly parameters?: Readonly<Record<string, IClassifiedValue>>;
readonly source?: unknown;
Comment thread libraries/reporter/src/reporters/AiReporter.ts
Comment on lines 381 to 385
public async flushAsync(): Promise<void> {
/* no-op */
if (this._logPath !== undefined) {
this._flushPendingProgress();
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Balanced
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@iclanton Ian Clanton-Thuon (iclanton) Awaiting requested review from iclanton iclanton is a code owner
@octogonz Pete Gonzalez (octogonz) Awaiting requested review from octogonz octogonz is a code owner
@apostolisms Apostolis Haitalis (apostolisms) Awaiting requested review from apostolisms apostolisms is a code owner
@dmichon-msft David Michon (dmichon-msft) Awaiting requested review from dmichon-msft dmichon-msft is a code owner
@jxanthony Jonathon Anthony (jxanthony) Awaiting requested review from jxanthony jxanthony is a code owner
@bmiddha Bharat Middha (bmiddha) Awaiting requested review from bmiddha bmiddha is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

Bug Triage
Status: Needs triage

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@TheLarkInn