FIX: Let the browser test result reach the job on ARM64 Linux

[jwrosewell]

What was wrong

On the Ubuntu_ARM_Java_17 job the shared browser suite failed every test and
the job was still reported as a success. Run
35190312090,
job Build and Test - 0 - Ubuntu_ARM_Java_17, says:

Failed! - Failed: 9, Passed: 0, Total: 9 - SeleniumApiTests.dll

and the run above it has a green tick. Two separate faults sat behind that.

The first is in the shared suite, where the helper Selenium ships to fetch a
driver had no ARM64 Linux build, so no browser was ever launched. That is fixed
in selenium-api-tests#11.

The second is here. dotnet test returns non-zero when tests fail, and
PowerShell carries straight on past a native command that returns non-zero
unless it is told not to. So the nine failures never reached the job result, and
nobody saw them for as long as nobody opened the log.

What changed

ci/run-integration-tests.ps1, two things.

• The exit code from dotnet test is checked, and the step stops with a message
  when it is not zero. Failing in a try means the existing catch still
  prints the example application's output first, which is what a reader needs.
• SELENIUM_TESTS_REF picks a branch of the shared suite, defaulting to main,
  so a change there can be tried here before it is merged. The branch it clones
  is printed, so a run says which suite it tested.

How this was checked

Run 35198427368,
dispatched on this branch with dryrun=true, with the suite temporarily pinned
to the branch carrying its fix and the matrix temporarily trimmed to the ARM64
Linux job. Both of those were reverted in the commit after, so this branch
carries only the two changes above. That run says:

Running Selenium tests...
Cloning the Selenium contract tests from 'fix/arm-linux-browser-drivers'
Passed! - Failed: 0, Passed: 9, Skipped: 0, Total: 9, Duration: 40 s

Nine tests that used to die in about 155 milliseconds without starting a browser
now run a real browser for forty seconds and pass, on the same ARM64 Linux
image.

Still outstanding

$status = $LASTEXITCODE on line 75 captures the result of the Maven example
run and is never read again, so a Maven failure is swallowed in the same way the
browser failures were. It is left alone here because fixing it could turn jobs
red for reasons that have nothing to do with the browser tests, and it deserves
its own change and its own look at what those reasons are.

This pull request should only be merged after
selenium-api-tests#11,
because until the suite is fixed on main this change would turn the ARM64
Linux job red rather than green. That is the right way round, and it is the
point of the change, but it should land in the right order.

State on 17 September 2026, and why this is still a draft

Two things stand between this branch and a green run.

First, the evidence above is not on the head. Run
35198427368
ran on cf39ce8, the temporary proof commit that pinned the suite to its fix
branch and trimmed the matrix to the single ARM64 Linux job, so it reports four
jobs rather than twenty. The next commit, 4c54153, reverts that, and the head
4c54153 has had no full run.

Second, a full run on the head would be red, and that is the change working as
intended. The shared suite's fix is in
selenium-api-tests#11,
which is still an open draft, so main of the suite still ships a helper with no
ARM64 Linux build. Today's run on another branch of this repository,
35190312090,
job Build and Test - 0 - Ubuntu_ARM_Java_17, shows what happens on the current
suite:

System.ComponentModel.Win32Exception: An error occurred trying to start process
'.../runtimes/linux/native/selenium-manager' ... Exec format error
Failed!  - Failed:     9, Passed:     0, Skipped:     0, Total:     9,
Duration: 155 ms - SeleniumApiTests.dll (net10.0)

and that run is still reported as a success, which is exactly the fault this
change fixes. Once the exit code is checked, those nine failures turn the job
red until the suite is fixed on its main.

So this stays a draft until selenium-api-tests#11 has merged, at which point a
dispatched run on 4c54153 is the thing to read. Only someone who can merge that
suite pull request can clear this.

One more thing for whoever merges these. This pull request and
#589 both rewrite
parts of ci/run-integration-tests.ps1 and they conflict with each other, in the
Selenium block. Whichever merges second needs main merged into it by hand and
the block resolved, rather than a straight merge.

[jwrosewell]

Real run on the current head

Run 35272958512, dispatched on fix/selenium-result-reaches-the-job with dryrun=true, is the first full run on the head 4c54153. It failed, and it failed for the reason this change was written to expose.

Build and Test - 0 - Ubuntu_ARM_Java_17 (job log) says:

Running Selenium tests...
Cloning the Selenium contract tests from 'main'
OpenQA.Selenium.WebDriverException: Error running Selenium Manager process:
  .../selenium-api-tests/bin/Release/net10.0/runtimes/linux/native/selenium-manager
Failed!  - Failed:     9, Passed:     0, Skipped:     0, Total:     9,
Duration: 124 ms - SeleniumApiTests.dll (net10.0)
The Selenium contract tests failed, returning 1. Their output is above.
##[error]Process completed with exit code 1.

Build and Test - 0 - Ubuntu_Java_17 passed on the same run, so the x64 Linux path is unaffected, and every other job was cancelled by fail fast.

Two things are proven by that. The branch selection works and defaults to main, which the "Cloning the Selenium contract tests from 'main'" line shows. The exit code check works, because nine failures that were reported as a success before now stop the job.

Still a draft, and what clears it

This stays a draft. It goes green once selenium-api-tests#11 has merged to that repository's main, and that pull request is still a draft targeted at the branch of selenium-api-tests#10 rather than main, so 10 and then 11 have to merge first. A dispatched run on 4c54153 after that is the thing to read.

The conflict with 589 is resolved

#589 now has this branch merged into it, at 08751d7, with the Selenium block of ci/run-integration-tests.ps1 resolved so both changes survive. Merge this one first and 589 second. A local trial of that order onto main merged both with no conflict, and the reverse order works too because 589 now contains these commits.