TEST: Run the Selenium contract tests against the on-premise web example too - #589
jwrosewell wants to merge 7 commits into
Conversation
…ple too The integration tests already ran the shared contract tests against the cloud web example. They now also start the on-premise web example on port 8098 with the TAC data file, because the Lite file lacks DeviceType and the JavaScript properties the tests need. A failing Selenium run now fails the job. Before, the exit code of dotnet test was overwritten by later steps, so a run where all nine tests failed on the Arm64 Linux runner still reported success. That runner is now skipped with a message, because the Selenium Manager binary the suite ships for Linux is x64 only. The wait for the example uses Invoke-WebRequest, because curl on Windows could not write to /dev/null and returned exit code 23. The on-premise example must first be able to take its port and data file from the environment, a change in device-detection-java-examples that has to merge before this one.
A mandatory parameter refuses an empty collection, so the first call failed before any test ran.
The exit code of the examples' mvn clean test was stored in $status and never read, so failing example tests left the job green. The code is now kept and reported, together with any Selenium failure, once all the tests have run.
A failing dotnet test returns non-zero, and PowerShell carries on, so the Selenium step could fail every case and the job was still reported as a success. On the ARM64 Linux job it did exactly that, with nine failures in the log and a green tick on the run. The exit code is now checked and the step stops. SELENIUM_TESTS_REF picks a branch of the shared browser suite, defaulting to main, so a change there can be tried here before it is merged.
Points the browser suite at the branch carrying the ARM64 fix and trims the matrix to the ARM64 Linux job, so one dispatched run shows whether the browser tests really run there.
This reverts the temporary pin and matrix trim used for run 35198427368, which proved the browser tests really run on the ARM64 Linux job. The matrix is back to the full list and the suite is read from main again.
…ium-onprem-demo Both branches rewrite the Selenium block of ci/run-integration-tests.ps1, and that block is the only place they conflicted. The resolution keeps this branch's structure, being the Invoke-ContractTests function that runs the contract tests against one example, the Arm64 Linux skip and the collected failures, and adds the SELENIUM_TESTS_REF branch selection from the other branch so a change to the shared contract test suite can be tried here before it is merged. The other branch also checks the exit code from dotnet test so a failing contract run reaches the job result. That behaviour already has an equivalent on this branch, inside Invoke-ContractTests, which records a failure when dotnet test returns non-zero, so both changes survive the merge.
The conflict with 592 is resolved here
The resolution keeps this branch's structure, being the Merge 592 first and this one second. A local trial of that order onto Real run on the merged headRun 35273266844, dispatched on this branch with
The Why this is still a draftBoth of those faults are on 663 is ready, mergeable, clean, green on its head |
1 participant
Problem
ci/run-integration-tests.ps1ran the shared Selenium contract tests from selenium-api-tests (categoryContract, 9 cases) against the cloud web example only. Every web example, cloud and on-premise, is expected to give the same client-side results, so the on-premise example should be tested the same way.Reading the script and the logs of run 34989242481 showed three further faults.
dotnet testwas never checked, and the performance tests that follow reset$LASTEXITCODE. In that run theUbuntu_ARM_Java_17job reported success withFailed! - Failed: 9, Passed: 0.Exec format errorwhen startingselenium-manager, because the Selenium Manager binary the suite ships for Linux is built for x64 only.curl -o /dev/null) ended withcurl: (23) client returned ERROR on write, so it did not really wait. The tests passed anyway because the cloud example happened to be up.Change
Invoke-ContractTests, that builds one example module, starts it withStart-Processon its own port with the environment it needs, waits up to three minutes withInvoke-WebRequest, runsdotnet test selenium-api-tests -c Release --filter TestCategory=ContractwithEXAMPLE_URLandEXAMPLE_LANG=java, prints the example's output on failure, and stops the process.web/getting-started.cloudon port 8099 (as before) and forweb/getting-started.onpremon port 8098 with51DEGREES_DD_PATHset to the TAC data file already copied into the examples repository. The Lite data file has neither DeviceType nor the JavaScript properties the contract tests need.mvn clean testwas stored in$statusand never read. It is now kept and reported together with any Selenium failure, once all the tests have run.Merge order
The companion change in device-detection-java-examples, device-detection-java-examples#663, must merge first. This script clones that repository's
main, and before that change the on-premise example ignoresPORTand51DEGREES_DD_PATHand waits for Enter, so the on-premise run would fail.Verification
The branch
test/selenium-onprem-demoheld this branch's commits plus one extra commit that cloned the examples branch instead ofmain. It has been deleted now that the runs are done. The "Pull Requests" workflow was dispatched on it withdryrun=true, which builds that branch without merging anything.Run 35157685031, on the head of this branch plus the test commit, concluded success. Contract result lines per job:
The earlier run 35153491220, before the example exit code commit, gave the same 9 of 9 for both examples in the same 15 jobs. The first attempt, run 35152629775, failed before any test ran, because a mandatory parameter refused the empty failure list. The second commit on this branch fixes that.
Outstanding
State on 17 September 2026, and why this is still a draft
The last full run on this branch's head
0ebdda9,35161390489,
failed in
Ubuntu_Java_17for the reason the merge order section above gives.The on-premise example was started on port 8098 and ignored it, listening on
8081 instead, then read the data file path it was given as a relative path it
could not use, and finally died waiting for Enter:
That is the old example, taken from
mainof the examples repository, becauseci/run-integration-tests.ps1callsclone-repo.ps1with no branch and thatstep defaults to
main. So this pull request cannot go green untildevice-detection-java-examples#663
has merged. That one is now out of draft, so the nightly can take it.
One more thing for whoever merges these. This pull request and
#592 both rewrite
parts of
ci/run-integration-tests.ps1and they conflict with each other, in theSelenium block. Whichever merges second needs
mainmerged into it by hand andthe block resolved, rather than a straight merge.