Skip to content

Knowledge: Community Copilot domain (System.AI features) and review skill - #229

Merged
Jesper Schulz-Wedde (JesperSchulz) merged 4 commits into
microsoft:mainfrom
demiliani:knowledge/copilot-development
Oct 20, 2026
Merged

Jesper Schulz-Wedde (JesperSchulz) merged 4 commits into
microsoft:mainfrom
demiliani:knowledge/copilot-development

Conversation

@demiliani

@demiliani Stefano Demiliani (demiliani) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

BCQuality covers the Agent SDK (Community agents), but nothing covers the much more common case of extensions that build Copilot features directly on the System.AI module: Copilot Capability, Azure OpenAI, AOAI Chat Messages, and PromptDialog pages. Today the corpus only mentions these incidentally (4 hits, none normative). An agent asked to "add Copilot" to an app regularly produces a raw OpenAI HTTP call, registers the capability only on install, uses the output without checking whether the call succeeded, writes proposals during Generate, or concatenates the user's prompt into the metaprompt. None of these are caught by the compiler or analyzers.

What

A new Community domain copilot, structured like Agents: knowledge plus a community leaf review skill that runs as a top-level peer and is not composed by the Microsoft al-code-review.

Article Mistake prevented
register-copilot-capability-on-install-and-upgrade Registration that exists only in OnInstallAppPerDatabase never runs for tenants upgrading to the version that adds the capability, and an unregistered capability raises an error.
call-azure-openai-through-the-ai-module A raw HttpClient call to an LLM endpoint bypasses capability registration, admin deactivation, data-movement controls, billing, and telemetry. Calls through the module without SetCopilotCapability are also flagged.
check-aoai-operation-response-before-using-output A failed GenerateChatCompletion does not stop AL code, so GetLastMessage() without IsSuccess() stores or shows text that is not a completion.
save-promptdialog-proposals-only-when-the-user-keeps-them Writing real records in Generate/Regenerate leaves data behind after Discard; saving in OnQueryClosePage must test CloseAction.
keep-user-input-out-of-the-system-message User or record text concatenated into SetPrimarySystemMessage/AddSystemMessage gains system authority (prompt injection).

Each article has good/bad samples, false-positive carve-outs, and Microsoft Learn references. Where a statement is an inference rather than documented platform behavior, the article says so. Two cases: raw HTTP bypassing the module's controls, and the risk of mixing message roles.

Wiring

  • community/skills/review/al-copilot-review.md: new leaf (inputs pr-diff/file-path/folder-path, display domain Copilot), with a token list and one targeted cue per article. Agent SDK code stays with the Agents review.
  • evaluation/review-fixtures.json: a copilot override listing all five paired articles.
  • docs/using-bcquality.md (example prompt, coverage paragraph, domain table) and docs/troubleshooting.md (the domain is absent from a broad review).

I chose the Community layer for the same reason as Agents: a new domain with its own community-owned skill. It can be promoted together with its skill later if maintainers want that.

Validation

  • tools/Test-ReviewFixtures.ps1 (including -ChangedPathsFile and -PrepareDirectory): PASSED, 21 leaf domains, deterministic ranking selects each new article.
  • tools/Test-ReviewContract.ps1, .github/scripts/Test-SkillIndex.ps1, .github/scripts/Test-KnowledgeIndex.ps1 (419 articles): PASSED
  • validate_frontmatter.py was not run locally (no Python on this machine); CI will confirm.
  • AL samples are demonstrations, not compiled.

Independent of #228.

Add a Community-layer `copilot` domain for Copilot features built on the
System.AI module (non-agent), with five articles and good/bad samples:

- register the Copilot capability on install and on upgrade
- call Azure OpenAI through the AI module, not a raw HttpClient request
- check AOAI Operation Response before using generated output
- save PromptDialog proposals only when the user keeps them
- keep user input and record text out of the system message

Add the community leaf `al-copilot-review` (top-level peer, like
al-agents-review), register all five paired articles in the review-fixtures
overrides, and document the domain in using-bcquality and troubleshooting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@demiliani

Copy link
Copy Markdown
Contributor Author

Hi Jesper Schulz-Wedde (@JesperSchulz), could you take a look at this one, or route it to whoever should review it?

It only touches community/, docs/, and evaluation/. None of those paths has an owner in CODEOWNERS, so GitHub didn't request any reviewers automatically.

Quick summary: it adds a Community copilot domain for Copilot features built on the System.AI module (non-agent): capability registration, AI module vs raw HTTP, AOAI Operation Response checks, PromptDialog Keep it/Discard, and prompt injection via the system message. It also adds a leaf skill al-copilot-review, which runs as a top-level peer like al-agents-review.

Two questions where your input would help:

  1. Layer: I used Community for the same reason as Agents (a new domain with its own skill). Would you rather have it in the Microsoft layer from the start?
  2. CODEOWNERS: should /community/ get a default owner, so community PRs don't sit without a reviewer?

Thanks!

Align the Relevance section with the Microsoft leaves: when the orchestrator
supplies no target version, use the major version of the app.json
application field, so the [23..] Copilot articles filter instead of
staying conditionally applicable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JesperSchulz

Copy link
Copy Markdown
Contributor

I am on a short fall break, but let me try to find someone to take a look :-) Next week I'm back with full focus!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 21166c489b7e5acba411fa52b0e8ca6453e54e23. Domain/layer discovery, resolver/catalog/sample retrieval, independent Community-leaf integration and all five deterministic pairs are wired correctly, and the existing 19 Microsoft leaves remain intact. Two canonical-sample blockers remain:

  1. community/knowledge/copilot/call-azure-openai-through-the-ai-module.good.al:22 uses two-argument SetManagedResourceAuthorization although the article is applicable to BC23+. Microsoft's BC24.0 source has no managed-authorization overload; BC27.0 exposes only four/five-argument forms; the two-argument form appears in BC28.0. The response-checking samples repeat it. Consequently the recommended fix cannot compile for supported advertised versions. Provide version-qualified authorization examples, use an older-compatible setup where appropriate, or explicitly raise the affected modern example/article's minimum version without unnecessarily excluding older-compatible guidance.
  2. The good sample's codeunit name Item Description Generator Good is 31 characters, exceeding AL's 30-character object-name limit (AL0305). Neutralization replaces Good with Eval and does not fix the length. Shorten it to at most 30 characters.

Exact-head frontmatter, skill/schema, index/retrieval, review contract and neutral fixture checks pass (262 cases, 21 leaf domains), but those checks do not compile AL. The blockers were verified by platform-source signature comparison and identifier-length checks; no AL compile/runtime test is claimed.

Use own-resource authorization with caller-supplied SecretText credentials, document version-specific managed authorization, and shorten sample object names including neutralized fixtures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@demiliani

Copy link
Copy Markdown
Contributor Author

Addressed both blockers from Jesper Schulz-Wedde (@JesperSchulz)'s review in commit 7bc5580.

  • Replaced the BC28-only two-argument managed authorization call in the AI-module good sample and both operation-response samples with the older-compatible four-argument SetAuthorization own-resource setup. Endpoint, deployment, and the SecretText API key are provided by the caller; removed the AOAI Deployments dependency.
  • Documented version-qualified managed authorization guidance without raising the articles' minimum version: BC24 has no managed authorization overload, BC27 has four/five-argument forms, and the two-argument shorthand requires BC28+. Compared these signatures against Microsoft's release sources.
  • Shortened both item-description codeunit names to remain within 30 characters, including after Good/Bad is neutralized to Eval.

Validation passed:

  • 262 review fixtures across 21 leaf domains, including neutral fixture preparation and executable review-contract tests.
  • Knowledge-index determinism/full coverage and retrieval round-trip (419 articles, 725 samples), run from a temporary local copy because the retrieval validator rejects the OneDrive root as linked.
  • Explicit length checks for all 14 Copilot sample object names and their neutralized forms.
  • git diff --check.

No AL compilation or runtime execution is claimed; the samples still rely on their documented demonstration context.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 7bc55804319ea14452fb2186e6475c7c5dc7566c. Both previously reported issues are fixed: the revised own-resource SetAuthorization overload exists in the older target sources, credentials are caller-supplied SecretText, the managed shorthand version boundary is documented, and the renamed object remains within 30 characters after neutralization.

Checking the revised companions as independently prepared inputs exposed another sample-integrity blocker. check-aoai-operation-response-before-using-output.good.al:10,17 and its bad companion reference Record "Reminder Draft Demo" and capability Draft Payment Reminder, but the exact-head tree declares neither that table nor that enum value. The good AI-module companion likewise uses Draft Item Description while its enumextension exists only in a different article's companion, not in its own prepared input. The fixture harness supplies each companion independently; it does not assemble the registration sample with these files.

Please include the minimal Reminder Draft Demo table (including Body Text) and required custom Copilot Capability enumextensions in each independent companion that uses them, or replace these dependencies with genuinely available declarations. Runtime registration may remain a documented prerequisite; unresolved AL symbols may not. This self-containment gap was missed in the earlier review, but the changed canonical samples cannot be treated as valid clean controls until it is fixed. Existing structural/retrieval checks pass and do not compile AL.

Declare reminder draft tables and capability enumextensions in each independent companion. Document runtime registration prerequisites separately from AL symbol declarations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@demiliani

Copy link
Copy Markdown
Contributor Author

Addressed Jesper Schulz-Wedde (@JesperSchulz)'s latest sample-integrity review in commit 7d8bac1.

  • Both operation-response companions now independently declare the Reminder Draft Demo table, its primary key and Body Text field, plus the Draft Payment Reminder Copilot Capability enumextension.
  • The AI-module good companion now declares its own Draft Item Description capability enumextension instead of depending on another article's sample.
  • Documented runtime registration/activation prerequisites separately from compile-time declarations, including adapting the registration pattern rather than combining duplicate enum value declarations. Reminder samples require an existing draft record.

Validation passed:

  • 262 neutral fixtures across 21 leaf domains, including the actual PR changed-path coverage gate and executable review-contract checks.
  • Explicit checks on the exact three prepared Copilot inputs: both reminder tables and all required capability values are declared in their own companion; object names remain within 30 characters after neutralization.
  • Knowledge-index determinism/coverage and retrieval round-trip (419 articles, 725 samples), from a temporary local copy outside OneDrive.
  • Skill-index/schema checks preserving the existing 19 Microsoft review leaves, and git diff --check.

No AL compilation/runtime execution or model-score result is claimed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 7d8bac15bd5d18306171d864e188217b944f4d41 against the previous reviewed head. The remaining self-containment blocker is resolved: both response companions declare their own Reminder Draft Demo table/Body Text field and Draft Payment Reminder capability value; the AI-module good companion declares Draft Item Description locally. All added objects are top-level, and identifiers remain within the limit after neutralization. The articles correctly separate those AL symbol declarations from runtime registration/activation and an existing reminder record prerequisite, including how to adapt the separate registration example without duplicate enum values.

The earlier own-resource authorization/version-boundary fixes remain unchanged. Actual-changed-path preparation passes 262 cases across 21 leaf domains; frontmatter, contract, skill/schema and index/retrieval checks pass, with the existing 19 Microsoft review leaves preserved. These are native structural/retrieval checks, not an AL compiler/runtime claim. No remaining merge-critical issue found. Substantive GitHub validation workflows still await approval (action_required) and should run before merge.

@JesperSchulz
Jesper Schulz-Wedde (JesperSchulz) merged commit fb3b7eb into microsoft:main Oct 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants