Run public AL issue triage with a provisioned Copilot agent - #8308
Open
Thaddeus Loke (thloke) wants to merge 5 commits into
Open
Run public AL issue triage with a provisioned Copilot agent#8308Thaddeus Loke (thloke) wants to merge 5 commits into
Thaddeus Loke (thloke) wants to merge 5 commits into
Conversation
Adds a GitHub Actions workflow and PowerShell scripts that triage issues opened in this public repository entirely natively - no private GHE/ADO calls, no private credentials, no private agents. - Deterministic scope/template prefilter (ScopePrefilter.psm1): rule-table classification into in_scope / out_of_scope / needs_human, covering runtime, application, event/function requests, feature suggestions, support questions, missing-template/repro, and UI-only editor-host issues. Hard-filters the 'accepted' label as defense in depth - acceptance stays human-only. - Safe, structured fixture extraction from inline AL only (FixtureExtractor.psm1): never clones or executes a linked repository/script; bounds fixture size/count. - Runtime safety gate (SafetyGuard.psm1): refuses to execute (but still allows compiling) fixtures using DotNet interop, control add-ins, HttpClient/network, file-system, or process integration. - Tier 1 server-free reproduction (Tier1Reproduction.psm1) using pinned, published Microsoft.Dynamics.BusinessCentral.Development.Tools (ALTools/altool) NuGet package versions. - Optional Tier 2 disposable stock BC container reproduction (Tier2ContainerReproduction.psm1), opt-in only via workflow_dispatch, gated by the safety guard, with guaranteed teardown. - Structured report + idempotent Markdown comment builder (ReportBuilder.psm1) using a hidden marker so re-triage updates rather than duplicates a comment. - Orchestrator (Invoke-IssueTriage.ps1) using only the public github.com REST API and the workflow-scoped GITHUB_TOKEN. - Workflow (al-issue-triage.yml) with least-privilege permissions (contents: read, issues: write), per-issue concurrency, and a low-frequency stale-reconciliation schedule. - Pester unit tests plus a labeled eval corpus (in-scope, runtime, application, question, suggestion, duplicate, missing-repro, UI-only, unsafe-code, prompt-injection) - 44 tests, all passing; PSScriptAnalyzer clean (0 findings). Never closes issues and never applies 'accepted'; a human retains the acceptance decision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/duplicate reconciliation
Mandatory correction pass on public AL issue triage automation, addressing 13 review findings:
1. Fixed Tier1 compile invocation: al compile forwards args verbatim to alc.exe, which needs
colon-attached /project:/out:/packagecachepath: syntax, not --project. Verified by actually
running the pinned ALTools package (17.0.34.45391).
2. Fixtures are now dependency-free by default (no �pplication manifest dependency), so Tier1
only needs System symbols, never Application/Base Application. Fixtures whose only errors are
AL1021/AL1022 (missing package cache/symbols) are classified requiresContainer=true /
reproduction=inconclusive, never "reproduced" from a bare compile failure.
3. Added DiagnosticMatcher.psm1: parses real alc.exe diagnostics, extracts an explicit AL####
signature (or Expected/Actual text) from the issue, and only returns reproduced when a
non-environmental diagnostic matches what the issue cites. Compile success/failure alone is
never evidence; CLI-usage/package-restore/manifest errors are explicitly excluded.
4. Tier1 no longer treats the VS Code marketplace "AL Extension Version" as a NuGet package
version. A "reported" version is only tested when explicitly framed as an ALTools/CLI package
version and confirmed to exist via a real NuGet.org lookup (Test-NuGetPackageVersionExist).
testedVersions keys are now accurately labeled (stable/preview/reportedAlToolsPackage).
5. Tier2: Continue='Stop' and terminating errors inside the container job; a
successful container/compile/publish is never "reproduced" by itself - runtime verification
only runs when a safe, deterministic [Test] procedure is found (Find-DeterministicTestSelector),
and its result is symptom-matched (Resolve-TestExecutionStatus). -skipVerification is kept only
with an explicit justification (ephemeral self-signed sandbox container, no external trust to
protect).
6. Hardened SafetyGuard: quoted-identifier DotNet declarations ("My Var": DotNet), WebClient/
WebRequest, the File data type, the virtual "File" system table, InStream/OutStream, Automation,
and Shell()/Process invocation, each with dedicated tests including a FileName false-positive
guard.
7. Wired duplicate detection: bounded candidate fetch + Find-PossibleDuplicateIssue, included as
informational report data; 'duplicate' is only auto-applied on an exact normalized title match
(Test-ExactDuplicateTitle).
8. Accepted issues are now a strict no-op (Test-IsAcceptedNoOp) - zero API calls that could mutate
the issue once 'accepted' is present.
9. Added label reconciliation (Get-LabelReconciliationPlan): adds newly desired managed labels and
removes stale ones (e.g. missing-repro -> complete/in-scope), never touching accepted or
component/human labels outside the managed set.
10. Added a pull_request-triggered, issues-permission-free �alidate job (parser check,
PSScriptAnalyzer, security grep, full Pester suite) and pinned actions/checkout and
actions/setup-dotnet to commit SHAs.
11. Fixed the git diff --check blank-EOF finding and resolved all PSScriptAnalyzer findings down
to zero (Warning/Error/Information), verified by an actual re-run rather than assumed.
12. Added OrchestratorLogic.psm1 (accepted no-op, label reconciliation, exact-duplicate check,
combined Tier1 status) plus new/expanded Pester coverage: DiagnosticMatcher, Tier1 real
end-to-end execution against the pinned package, Tier2 pure-logic and one real
container-start-failure execution test, and a static SecurityGuard test for the public-only
credential/endpoint boundary. 110 tests passing, 1 environment-conditional skip.
13. Corrected documentation to only claim what was actually executed; Tier1's exact pinned-package
command now runs for real inside the test suite, and Tier2's container path is explicitly
documented as unverified-by-execution (no Windows container runtime in this environment)
rather than claimed as validated.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: af3d5664-9cd1-494c-9ac9-289163c4c222
Thaddeus Loke (thloke)
changed the title
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3828f007-4f8d-494d-80d3-b83a70071398
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3828f007-4f8d-494d-80d3-b83a70071398
Steffen Balslev (SBalslev)
approved these changes
Aug 12, 2026
|
|
||
| | Report | Destination | | ||
| |---|---| | ||
| | First-party application extensibility | `microsoft/ALAppExtensions` | |
There was a problem hiding this comment.
This is very soon outdated. See "Important" note in https://github.com/microsoft/ALAppExtensions/blob/main/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Starts a read-only Copilot cloud-agent session to triage public
microsoft/ALissues.workflow_dispatchwith an issue number.Assessment, and Recommended next step sections.
Agent environment
.github/workflows/copilot-setup-steps.ymlprovisions a Windows agent environment with:Microsoft.Dynamics.BusinessCentral.Development.Tools;The triage agent must record the ALTool version, BC artifact/container state, every attempted
investigation/reproduction step, and the observed result. Environment failures are reported as
blockers and never treated as product reproduction.
Session start
.github/workflows/al-issue-triage.ymlassignscopilot-swe-agent[bot]with custom agental-issue-triager. Repository secretCOPILOT_ASSIGNMENT_TOKENmust be a user-to-server token withthe GitHub permissions required for Copilot issue assignment.
Verification
dotnet tool install --tool-path ... Microsoft.Dynamics.BusinessCentral.Development.Tools --prerelease18.0.39.10160-beta;al --versionreturned18.0.39.10160+...Get-BcArtifactUrl -Type Sandbox -Country w1 -Select Latest28.4.53241.53533/w1New-BcContainer/Get-BcContainerIdcommands and parametersissues.opened+workflow_dispatch; no schedule/backfill triggersOpen decisions for review
COPILOT_ASSIGNMENT_TOKENmust be configured before enabling the workflow.copilot-setup-steps.ymlis onthe default branch; run its manual validation once merged.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com