Skip to content

Add device tests for CommunityToolkit.Maui, Apply Fixes for Bugs Found - #3344

Draft
TheCodeTraveler wants to merge 189 commits into
mainfrom
Add-device-tests-and-ui-tests
Draft

TheCodeTraveler wants to merge 189 commits into
mainfrom
Add-device-tests-and-ui-tests

Conversation

@TheCodeTraveler

@TheCodeTraveler TheCodeTraveler commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

This is a work in progress built upon @ne0rrmatrix’s work in #3277. It includes additional tests and additional bug fixes that the test discovered:

Partially Fixed (will be finished in a future PR): #3317, #2894, #2691, #3063, #2952, #1987, #2645, #3220, #3248, #2602

Partially Fixed (already closed, will be finished with #3317): #2383, #2950

Docs PR: MicrosoftDocs/CommunityToolkit#665

Copilot AI balanced review requested due to automatic review settings October 10, 2026 22:02
@TheCodeTraveler
TheCodeTraveler marked this pull request as draft October 10, 2026 22:04

This comment was marked as outdated.

@TheCodeTraveler TheCodeTraveler changed the title Add device tests for CommunityToolkit.Maui Add device tests for CommunityToolkit.Maui, Apply Fixes for Bugs Found Oct 10, 2026
TheCodeTraveler added a commit that referenced this pull request Oct 11, 2026
…evice tests

- CameraProvider (Windows): the camera refresh skips a camera that InitializeCameraForCameraView could not open, instead
  of reading the VideoDeviceController of a MediaCapture that never initialized (its getter throws, so one such camera
  made the whole refresh fail). New Windows device test:
  CameraProviderWindowsTests.RefreshWhileAnotherCaptureHoldsACameraListsExactlyTheCamerasThatCanBeOpened.
  (Copilot review on #3344)
- Device tests: the AttachAndRun tests of the Maui, MediaElement and Camera apps wait for the Loaded and Unloaded events
  they count. OnLoadedAsync returns once the view is loaded, which can come before MAUI raises Loaded on a slow
  emulator (Android API 28 in CI).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TheCodeTraveler
TheCodeTraveler requested a balanced review from Copilot October 11, 2026 02:56

This comment was marked as outdated.

@TheCodeTraveler

Copy link
Copy Markdown
Collaborator Author

We're now up to 89.5% Line Coverage and 82% Branch coverage across all unit tests and device tests. That's pretty great!

Code Coverage Report:
https://github.com/CommunityToolkit/Maui/actions/runs/38106699644/attempts/1#summary-114378095598

image

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

@TheCodeTraveler TheCodeTraveler added the breaking change This label is used for PRs that include a breaking change label Oct 11, 2026
ne0rrmatrix added a commit to ne0rrmatrix/EbpubReaderV2 that referenced this pull request Oct 14, 2026
- Unit tests for CalibreCoverUrl, OpdsEntryStagingCache, EpubPathUtilities,
  EpubArchive and PositionSyncService.
- tests/DisplayBook.DeviceTests: a MAUI app that runs xUnit on the device
  (DeviceRunners), covering code that calls MAUI APIs: BookStorageService,
  BookDatabase, OpdsServerRepository, OpdsCatalogCache, FirebaseAuthService
  (SecureStorage), GoogleBooksOptions (Preferences) and the value converters.
  Modelled on CommunityToolkit/Maui#3344.
- tools/coverage.ps1: unit tests + Windows device tests under dotnet-coverage,
  merged into artifacts/coverage/coverage.xml (Visual Studio XML).
- .github/workflows/build.yml: build, tests and coverage between the
  SonarQube Cloud scanner's begin/end.
- Android signing: the keystore and its passwords leave the repository. They
  now come from Secrets.local.props locally and from repository secrets in CI;
  without them Android builds sign with the debug key.
…the bugs they found

Platform code that the net10.0 unit tests cannot reach is now covered by
device tests that run inside real apps on Android, iOS, Mac Catalyst and
Windows. There is one device-test app per shipped feature package
(CommunityToolkit.Maui.DeviceTests for CommunityToolkit.Maui, .Core and
.Maps; CommunityToolkit.Maui.MediaElement.DeviceTests;
CommunityToolkit.Maui.Camera.DeviceTests), built like dotnet/maui's device
tests: xUnit, a vendored test runner and XHarness. eng/devices/android.ps1,
ios.ps1, catalyst.ps1 and windows.ps1 build and run them locally and in CI;
eng/devices/README.md and src/CommunityToolkit.Maui.DeviceTests/README.md
describe how to run and write them.

The Build workflow (dotnet-build.yml) calls the reusable Device Tests
workflow (device-tests.yml), which runs every app on Android API 28 and 36,
the iOS simulator, Mac Catalyst and Windows (packaged and unpackaged). Its
Code Coverage job merges the unit-test and device-test coverage into one
report. Tool versions shared by the workflows live in
.github/workflows/shared.env.

The tests found bugs in the libraries, which this change fixes: #3307,
#2827, #1818, #1826, #2358, #1845, #3091, #2320, #2153, #1879, #2078,
#2570, #3079, #3113, #2900, #3212, #2519, #3171, #3050, #2322 and #2208,
and in part #3317, #2894, #3081, #2538, #2691, #3063, #2952, #1987, #2645,
#3220, #3248, #3066 and #3247.

Rebased onto main as one commit. Conflicts:
- StatusBar.android.cs: keeps #3189's window flags (the decor never fits
  the system windows, API < 30 also clears LayoutNoLimits, nothing depends
  on transparency) with this change's FindChildWithTag lookups. This
  change's alpha-0 check is no longer needed. The Android StatusBar device
  tests now expect #3189's flags.
- Popup.shared.cs, PopupPage.shared.cs: keeps #3345's
  DefaultPopupSettings.BackgroundColorDark and OnInitializing, and this
  change's Style support. A value or binding set on the Popup, like the
  AppThemeBinding that #3345 set in the constructor, takes precedence over
  every Style (7 of 8 PopupStyleTests fail with it), so Popup.BackgroundColor
  stays null by default and the popup's Border shows
  DefaultPopupSettings.BackgroundColor, or BackgroundColorDark in the dark
  theme, following theme changes. #3345's two default-color tests now
  expect Popup.BackgroundColor to be null and check the Border.
Also: actions/upload-artifact and download-artifact pinned to main's 7.0.2
and 8.0.2 in the device-test and coverage jobs; the empty property pattern
(is { }), which CONTRIBUTING.md now asks to avoid, replaced by type
patterns; the comments of DialogFragmentService and its tests no longer
say that StatusBar.SetColor treats colors without alpha as transparent; and
the Sample's popups that set black text but no background color
(UpdatingPopup, CollectionViewPopup, ComplexPopup, CsharpBindingPopup and
the Popup Sizing Issues popup) now show white text in the dark theme, on
#3345's black default background (DefaultPopupSettings.BackgroundColorDark).

Co-authored-by: James Crutchley <ne0rmatrix@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TheCodeTraveler
TheCodeTraveler force-pushed the Add-device-tests-and-ui-tests branch from 2cda4d3 to 3f57379 Compare October 16, 2026 04:57
…and agents

CONTRIBUTING.md gains a Comments section: names say what the code does;
comments only explain why (a non-intuitive decision, a platform quirk or
workaround, an ordering constraint, code that looks out of scope).
.github/copilot-instructions.md and .github/instructions/codestyle.instructions.md
carry the same rule and the existing CONTRIBUTING.md rule against the empty
property pattern `is { }`, and the range-indexer example no longer teaches
comments that restate the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This comment was marked as outdated.

On .NET 11, StatusBar.SetStyle and StatusBarBehavior.StatusBarStyle only set
Window.StatusBarTheme, which UIKit ignores when Info.plist sets
UIViewControllerBasedStatusBarAppearance to false: the setup that the
toolkit documents for StatusBarBehavior. The style stopped changing, while
the same app with the .NET 10 code (UIApplication.SetStatusBarStyle) shows
it. .NET 11 rc.1 builds against the iOS 26.5 SDK, and UIKit applies
UIApplication.SetStatusBarStyle with that key in every app built with an iOS
SDK before 27.0, with or without the UIScene life cycle; only apps built with
the iOS 27 SDK ignore it.

With the key set to false, the .NET 11 code now also sets the style of the
whole app with UIApplication.SetStatusBarStyle, as .NET 10 does, and writes
its "remove the key" trace message only when UIKit ignores that call too.
The remarks and comments no longer say that every .NET 11 app is scene-based
or that scene-based apps ignore UIApplication.SetStatusBarStyle: the iOS SDK
the app is built with decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TheCodeTraveler and others added 27 commits October 23, 2026 17:28
run.ps1 needs them to tell whether an emulator is running before it starts android.ps1. A second copy would drift from the one that android.ps1
uses to boot the AVD and to find its device, so the three functions move unchanged and both scripts use the shared ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…machine can run with one command

pwsh eng/devices/run.ps1 detects the operating system and what it can run: the iOS simulator and Mac Catalyst on a Mac with Xcode (the iOS
simulator also needs a simulator runtime), the packaged and the unpackaged app on Windows, and Android everywhere when an emulator is running or
an AVD that this machine's CPU can run exists. A physical device is never used unless -DeviceId names it, as the README promises. It prints the
plan first (-DryRun / -WhatIf stops there), builds and runs each platform with the existing scripts in this process, and ends with one table:
total, passed, failed and skipped tests per platform and app, the time, and the results folder. The exit code is the first non-zero exit code
of the platform scripts.

- -Platform, -Project (default All), -TestFilter and -Configuration choose what runs; -Target, -Results, -BinlogDir, -SkipCapabilities,
  -RequireCapabilities, -Timeout, -InfraRetries, -BuildArgs, -Avd, -EmulatorPort, -DeviceId, -Udid, -Xcode and -KeepDevice pass through under the
  names the platform scripts use.
- A 'Category=' filter without -Project runs only the apps that declare the category, because an app that runs 0 tests fails with 94.
- A platform that this machine cannot run is skipped with the reason; one that -Platform names is an error. When nothing can run the exit code is 71.
- The platform scripts run in this process, because exit ends only the script when it is called with &, and because -File would flatten the
  array parameters to strings. Their output is shown as it is printed, in its colors, while the ERROR lines are kept for the summary.
- gate.json and summary.md of an earlier run are deleted before a platform runs, so a script that ends before it writes its verdicts cannot
  show the numbers of the run before.

Checked without a device: the helpers (platform names, AVD choice, category lookup, verdicts) and the whole script against copies of the platform
scripts that keep the real param blocks, with fake xcrun, xcode-select and adb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide for contributors, next to the reference in README.md:
- the one command (run.ps1): what it runs on a Mac and on Windows, how to narrow it, what the summary and the exit code mean;
- running a device test app from Visual Studio, Rider and Visual Studio Code on Android, iOS, Mac Catalyst and Windows (packaged and unpackaged),
  with the terminal commands that the IDEs run, and how to use the visual runner (run all, filter, run the filtered tests, run one test again);
- running the tests of the RequiresUserInteraction category by hand, per platform and test class, with what each test needs from the person
  (taken from the tests and their "// Person:" comments);
- what a local run does differently from the CI jobs: the Mac Catalyst privacy prompts for the camera, the microphone and speech recognition,
  the grants that ios.ps1 does not make on a simulator named with -Udid, the camera and microphone consent, Developer Mode and the certificate
  of the packaged Windows run, the cameras of an Android emulator, Debug against Release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run.ps1 is the first row of the script table and the first command of the quick start. Its own section lists what it runs on each operating
system, the order in which Android picks a device (-DeviceId, -Avd, a running emulator, an AVD that this CPU can run), the parameters, the
summary and the exit codes. The section on using your own devices says that run.ps1 follows the same rule as the scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… point to both device test guides

CONTRIBUTING.md gets a "Device tests" section: what the apps test, the one command (pwsh eng/devices/run.ps1), how to narrow it, and links to the
local guide (eng/devices/LOCAL.md), the script and CI reference (eng/devices/README.md) and the guide for writing a test. README.md says in the
step of the pull request that platform-specific code needs device tests and links the same pages. The authoring guide of the apps and the
instructions for device tests name run.ps1 next to the platform scripts, and the guide of the apps links the section about the tests that
need a person.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lors is shown plain

The forwarding of the scripts' output runs inside the pipeline of the script that is being run, so an exception in it would have stopped the
tests of that platform. The line is now written as plain text instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first emulator of several could be in use for something else, and android.ps1 does not choose among several attached devices either.
With exactly one emulator running, run.ps1 still uses it; with several, Android is skipped (an error when -Platform names it), and the reason
tells to choose one with -DeviceId or to boot an AVD of one's own with -Avd. Found by a dry run on a Mac with six emulators running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…platform scripts run the main app

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l, and a missing emulator package is reported in the plan

An AVD folder copied to another user or machine keeps a path= that points nowhere; the emulator then falls back to path.rel, and so does the
AVD list now. Without the emulator package in the SDK the plan promised to boot an AVD and android.ps1 failed with a script error, so the
plan skips Android and says what to install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A platform script reports an error that ends it as one message with its stack trace, and the pattern that collected the ERROR lines for the
summary required the reason to end the message, so it never matched one of several lines. The summary then said "The script printed no
error" for every such failure, the untrusted packaged-app certificate among them, which is what running the packaged platform first is for.
Only the first line is taken now. A verdict's reasons can also quote a tool's output of several lines, such as simctl's, so each reason is
shown on one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ier run

The platform scripts write build-summary.md when they only build, and summary.md when they test. run.ps1 linked summary.md for every target
and deleted the old one only before a test run, so a failed build-only run linked the summary of an earlier test run as its details, or
none. It now deletes and links the file that the target writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…f it

Directory.Build.targets builds a *.android.cs, *.ios.cs, *.macos.cs, *.macios.cs or *.windows.cs file, and a file in a folder of that name,
for its platforms only, and most categories have their tests in such files. Without -Platform, `run.ps1 -TestFilter 'Category=X'` ran
every platform of the machine, so an Android-only category failed on iOS and Mac Catalyst with 94 after building both: the command that
CONTRIBUTING.md, LOCAL.md and the device test instructions give for checking one category. The plan now skips the platforms whose apps
have no test of the category, with the reason; a file that it cannot place counts for every platform, so a platform is never left out
wrongly. An explicit -Platform still runs what it names.

The apps compare category names case-sensitively, so the check of the declaring apps does too: a name in another case now ends with exit
code 3 at once instead of 94 after the builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The emulator that android.ps1 starts also finds its AVDs through ANDROID_EMULATOR_HOME and the deprecated ANDROID_SDK_HOME, which run.ps1
did not read. With one of them set, the plan found no AVD to boot, and an AVD that -Avd named was reported as missing (exit code 81)
although android.ps1 alone would boot it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…open the built Mac Catalyst app to answer its prompts

- Starting a Mac Catalyst app from the IDE builds it again, and the apps are signed ad hoc, so an answer given to that build may not hold
  for the testOnly run. The workflow now opens the bundle that catalyst.ps1 built.
- Only the FileSaver and FolderPicker tests have a "// Person:" comment, not each test that needs a person, and those tests do not all
  wait between 30 seconds and two minutes (the camera prompt waits without a limit, a language download up to ten minutes).
- Locally only iOS saves a screenshot after a failure; ios.ps1 with -Udid boots the simulator and installs the app, it only grants nothing;
  the Simulator app does not show UDIDs; android.ps1 boots the AVD, run.ps1 only calls it.
- The summary example shows the totals line and the exit code that follow the rows.
- Visual Studio installs the Android SDK where the scripts do not look, so Android is skipped until ANDROID_HOME points there.
- The duration of a run was said twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and the device test instructions point to LOCAL.md

Windows has Windows PowerShell 5.1 but not always PowerShell 7, so the one command that CONTRIBUTING.md gave did not run there; LOCAL.md
already gives both. Android runs on the one emulator that is running or on an AVD that run.ps1 boots, not whenever an emulator runs. The
device test instructions named only eng/devices/README.md for running the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The platform scripts create, boot or choose a device only to run the tests, but the plan applied the device policy to -Target buildOnly
too. With several emulators running, or none and no AVD, `run.ps1 -Target buildOnly` skipped Android, so the testOnly run that LOCAL.md
puts after it found no package (exit code 4); with -Platform android it was an error (81). The plan also said that a build-only run would
boot an AVD or create a simulator, which it never does. A build-only plan now builds every platform whose tools are there, and says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its own

run.ps1 builds the packaged and then the unpackaged app in one checkout, and LOCAL.md's two Windows blocks do the same. Both builds went
to bin\<configuration>\<framework>\<rid> and obj\<configuration>\<framework>\<rid>, and the incremental unpackaged build kept the
packaged resources.pri: the PRI targets compare timestamps only, and the Windows App SDK .pri files that only a self-contained build
merges are older than the packaged build's outputs. On the Windows VM the unpackaged app then crashed at startup (0xC000027B) in the
default run.ps1 run, in a test-only run after a rebuild and when LOCAL.md's unpackaged block started it, while the same code passes in a
checkout that only ever builds unpackaged. The unpackaged build now uses bin\unpackaged and obj\unpackaged (the restore stays in obj).
windows.ps1 looks for another runtime identifier's publish folder next to the evaluated one instead of in the packaged build's folder,
and LOCAL.md's unpackaged block looks in bin\unpackaged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d the app before they start it

The Run target of the .NET SDK depends only on ComputeRunArguments, and the iOS and Mac Catalyst SDKs add no build to it, so
`dotnet build -t:Run` started the app as it was last built: in the verification run the Mac Catalyst command took 0.2 s and the iOS one
printed no build output, and a change since the last build would not be in the app, which defeats debugging a test. In a fresh checkout
there is no app to start. Android's Run builds and installs the app itself. The two Apple commands now use -t:Build,Run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es with it

The Windows consent section said that a Windows virtual machine usually has no camera. The Parallels VM that ran the
verification has camera sharing on (SharedCamera Enabled=1 in its config.pvs), and Parallels attaches a Virtual Video Camera,
fed by the Mac's cameras, to its USB bus: Windows lists three Camera and Image devices there. A contributor in such a VM would
expect no camera, while the camera tests open the shared camera, or skip until the app is allowed to use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…that runs

-Results is the root of a folder per platform, so a job that must find its results in an exact folder could not say so: the Android jobs
of the CI run upload their coverage from artifacts/device-tests/android-api36/coverage, and coverlet names the report after that folder
(android-api36.cobertura.xml), which the Code Coverage job labels the report with. -ResultsFolder is the platform scripts' own -Results,
for the one platform that runs. It is an error together with -Results and for a run of several platforms, which would share the folder;
the plan counts the skipped platforms too, so that a command is valid on every machine or on none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The jobs ran the platform scripts, so the command that a developer runs on their machine (run.ps1) was not what CI ran, and a reader of the
workflow could not see how to run the device tests. Every build and test step now calls run.ps1 with its -Platform and the arguments it
passed before. run.ps1 runs the platform script in its own process, so the exit code, the job summary that the script appends to and its
::error annotations reach the step as before. The results land in the folders that the upload steps read: run.ps1's folder for a platform is
the platform script's default, and only the Android jobs name theirs with -ResultsFolder, because artifacts/device-tests/android-api36 is
what the coverage upload reads and the coverage report is named after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w the commands

The CI section named the platform scripts as what the jobs run. It now says that every job builds and tests with run.ps1, one -Platform
each, lists the platform and the results folder of every job, what reaches the step from the platform script (exit code, annotations, job
summary) and how to replay a job with its environment. The capability example uses run.ps1 as the jobs do, and the run.ps1 section says
which environment variables the platform scripts read, because run.ps1 passes on neither -Coverage nor the capability lists and a reader
could think that it cannot collect coverage or declare capabilities as the jobs do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide said that the Device Tests job builds and runs the tests, as if it ran something else than the command that the guide starts
with. It now says that the jobs run run.ps1, one platform each, and where the lines are, so that a developer who wants to know how CI
runs a platform reads the workflow and sees the command that they run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd the platform script runs in the process of run.ps1

README.md said that a step gets the exit code of the platform script, such as 94. A Windows step gets 1: the runner starts a
"shell: powershell" step as powershell -command ". '<script>'", which turns every non-zero exit code into 1. Job 120263269469 (Windows
unpackaged, Build run 39907681941, the old lines) logged "Exit code: 80 (APP_CRASH)" and then "Process completed with exit code 1", and
the Windows replays of run.ps1 end the same way; the code is in the annotation and in the last line that run.ps1 prints.

"run.ps1 runs the platform script in its own process" (README.md, and the comment of the Windows steps) reads as a process of the
script's own, while it is the process of run.ps1, which is why the Windows steps keep shell: powershell. The header comment of
device-tests.yml now says in one sentence why -Platform and -Project are explicit. The -ResultsFolder help, README row and comment say
that the skipped platforms count: whether a command is valid depends on the operating system, not on what the machine can run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests workflow runs

The Device Tests section told agents to use the platform scripts, so an agent never learned of the one command that CONTRIBUTING.md,
README.md and LOCAL.md start with and that every CI job now runs. It now names run.ps1 first, with the platform scripts for one platform,
and links LOCAL.md next to README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o the Android resource designer assembly is found

With -restore, `dotnet msbuild -getItem:ReferencePath` gave the Android resource designer assembly
(obj/Debug/net10.0-android/_Microsoft.Android.Resource.Designer.dll) a full path under the folder it ran in, the repository root,
where no such file exists. The coverage staging then failed to copy it, and the Android API 36 job tested without coverage while
the step stayed green. The query now runs from the project's folder (1 of 297 references was missing before, none after).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Broad cross-platform changes lack build/device validation, and the Android API 21–22 status-bar assertion remains unresolved; final human review is required.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change This label is used for PRs that include a breaking change

Projects

None yet

2 participants