Skip to content

Edge hardened and back on, failing grid sessions that fail, and one preparation for every driver, from #7 - #40

Merged
jwrosewell merged 3 commits into
rcfrom
fix/rest-of-7
Nov 16, 2026
Merged

jwrosewell merged 3 commits into
rcfrom
fix/rest-of-7

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The rest of #7, "FIX: Harden WebDriver creation so browser tests stop timing out", by James Rogers, after the three parts #37 brought in. With this, everything #7 proposes is in the suite except the three parts listed under "Still not taken from #7".

What is taken from #7

  1. One preparation for every driver. BrowserDrivers.Prepare runs every browser headless and accepting insecure certificates, as FIX: Harden WebDriver creation so browser tests stop timing out #7's PrepareOptions does, and gives Chrome and Edge --no-sandbox and --disable-dev-shm-usage. Each is added once, and a headless argument the caller chose, such as the 51Did harness's --headless=new, is kept. CreateChrome, CreateFirefox, CreateEdge and the 51Did harness call it. The copies of these settings go, 36 lines across 15 test files, the two Edge arguments DriverSmokeTests.cs added by hand, and the harness's own three.
  2. Edge hardened. CreateEdge gives Edge the two Chromium arguments through Prepare, a driver service as quiet as Chrome's, and SessionTimeoutSeconds for each command, which is 300 seconds. The remote helper adds Edge's two arguments only where they are missing.
  3. Edge back in use. EdgeTests, CacheTests.JavaScript_BrowserCache_Edge and DriverSmokeTests.Edge_RunsThePage are no longer ignored. FIX: Harden WebDriver creation so browser tests stop timing out #7's own reason for this is "Shipping the hardening while leaving Edge off would make it unfalsifiable". On ARM64 Linux, where Microsoft ships no Edge, all three skip with the reason CreateEdge gives. CacheTests.JavaScript_BrowserCache_Edge gained its own catch for this in commit 9e7bd32, as Edge_RunsThePage already had one. No test can show that skip without an ARM64 runner, because BrowserDrivers.IsLinuxArm64 is fixed by the machine, and this repository's ARM64 runners do not run the CloudInternal category.
  4. Grid failures that fail. ChromeTests, GeckoTests and EdgeTests report inconclusive only when no SELENIUM_URL is set, as FIX: Harden WebDriver creation so browser tests stop timing out #7 does, so a missing browser on a developer's machine is still inconclusive and a grid that cannot hand out a browser fails the class.

README.md says all of this under "How a run is wired".

Still not taken from #7, and why

Exit paths of the narrowed catch

  • No SELENIUM_URL, and the browser or driver is missing. CreateChrome, CreateFirefox or CreateEdge throws WebDriverException, and the class reports inconclusive as before.
  • No SELENIUM_URL, on ARM64 Linux, for Edge. CreateEdge throws WebDriverException with its ARM64 reason, and EdgeTests reports inconclusive with it.
  • SELENIUM_URL set, and the grid cannot be reached or answers with an error Selenium reports as WebDriverException. The class now fails with it.
  • SELENIUM_URL set, and the grid refuses the session. Selenium .NET reports "session not created" as InvalidOperationException, which the catch never caught, so the class fails, as it did before.
  • Any other exception was never caught and still is not.

How it was checked

Selenium/DriverSettingsTests.cs gains seven tests. The first four reach the new members by name, so on a commit without them they fail when they run rather than stopping the build.

  • Prepare_ChromiumOptions_GetEverySettingOnce, Prepare_CallersHeadless_IsKept, Prepare_FirefoxOptions_AcceptInsecureCertificates and EdgeService_IsQuiet.
  • BrowserTestClasses_RemoteThatCannotBeReached_Fail points SELENIUM_URL at a port nothing listens on and runs each class's ClassInit.
  • CreateEdge_HasTheArgumentsAndTheSessionLimit, in the Browser category, starts a local Edge and checks both arguments and the 300 second limit. It is inconclusive on ARM64 Linux with CreateEdge's reason.

Red, run 43879807037, on the tests alone (8974e2c).

  • build-and-test: the four by-name tests failed with "BrowserDrivers has no Prepare" and "has no EdgeService".
  • BrowserTestClasses_RemoteThatCannotBeReached_Fail failed with "Expected exception type WebDriverException. Actual exception type AssertInconclusiveException", which is the grid failure being hidden.
  • On ubuntu-latest, CreateEdge_HasTheArgumentsAndTheSessionLimit, whose Edge was given none of the settings, failed as Edge started, with "session not created: Chrome instance exited". Edge_RunsThePage was still ignored.

Green, run 43880867400, with the change (14eb543).

  • build-and-test ran 72 tests, of which 57 passed and 15 skipped, being the tests that need a browser, the cloud or keys, which that job leaves out.
  • On ubuntu-latest the browser job passed all 10, Edge_RunsThePage and CreateEdge_HasTheArgumentsAndTheSessionLimit included. This is the public evidence that headless Edge no longer hangs or fails on the Ubuntu runner. With the two arguments it starts and loads the page, where before it exited as it started.
  • On ubuntu-22.04-arm and ubuntu-24.04-arm it passed 8 and skipped the two Edge tests with "Edge cannot run on ARM64 Linux, because Microsoft publishes no Edge browser and no Edge driver for it".
  • Locally on Windows, with Chrome 157, Edge 157 and Firefox, the same two filters gave 57 passed and 15 skipped of 72, and 10 passed of 10.

The CloudInternal category holds EdgeTests and CacheTests.JavaScript_BrowserCache_Edge, and needs a cloud, so this repository's own checks do not run it, and that run is not linked here. Run on ubuntu-latest against a cloud on the same machine with commit 14eb543, it gave 48 passed and 1 skipped of 49. The skip is LocationClientSideOverrideTests.JavaScript_ClientSideOverrides, ignored for its own reason. Without this change the same category skips 14 of the 49, so the 13 Edge tests now run, and each passed, with no hang on its first page. The Browser51Did category in the same run passed 27 and skipped 1 of 28, the same as before.

What remains

… and grid failures that fail

DriverSettingsTests checks that BrowserDrivers.Prepare gives Chrome and
Edge options headless, insecure certificates accepted and both Chromium
arguments, each once, keeps a headless argument the caller chose, and
gives Firefox options insecure certificates accepted. It checks that the
Edge driver service is as quiet as Chrome's, that a local Edge session has
the Chromium arguments and the session limit, and that the Chrome, Firefox
and Edge browser test classes fail, rather than report inconclusive, when
SELENIUM_URL names a Selenium that cannot be reached. Here the last fails
with AssertInconclusiveException and the others find no Prepare or
EdgeService.
…hand out a browser, and prepare every driver in one place

The rest of #7, by James Rogers, apart from WebDriverFactory.cs and
--disable-gpu.

- BrowserDrivers.Prepare runs every browser headless and accepting insecure
  certificates, and gives Chrome and Edge --no-sandbox and
  --disable-dev-shm-usage, each once, keeping a headless argument the caller
  chose. CreateChrome, CreateFirefox, CreateEdge and the 51Did harness call
  it, and the copies of those settings in the tests go.
- CreateEdge gives Edge a quiet driver service and SessionTimeoutSeconds
  for each command, and the remote helper adds Edge's two arguments only
  where they are missing.
- EdgeTests, CacheTests.JavaScript_BrowserCache_Edge and
  DriverSmokeTests.Edge_RunsThePage are no longer ignored. On ARM64 Linux,
  where Microsoft ships no Edge, they still skip with CreateEdge's reason.
- ChromeTests, GeckoTests and EdgeTests report inconclusive only when no
  SELENIUM_URL is set, so a grid that cannot hand out a browser fails the
  class rather than hiding.
JavaScript_BrowserCache_Edge lost its Ignore without a catch around
CreateEdge, so on ARM64 Linux, where Microsoft ships no Edge, it would
fail on CreateEdge's WebDriverException rather than skip. It now reports
inconclusive with that reason there, as Edge_RunsThePage does.
@jwrosewell
jwrosewell marked this pull request as ready for review November 16, 2026 15:38
@jwrosewell
jwrosewell merged commit 68691f9 into rc Nov 16, 2026
5 checks passed
@jwrosewell jwrosewell mentioned this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant