Repository navigation
Run the API and JSF tests from Maven Tests, with server-side coverage - #12783
Open
ashmanda-cs wants to merge 80 commits into
Open
ashmanda-cs wants to merge 80 commits into
ashmanda-cs wants to merge 80 commits into
Conversation
Loads the container images and exploded WAR handed off by a build job, starts the stack and applies the test configuration. The steps are the same as in the existing container test workflows.
Takes the artifact name as input so multiple jobs in one workflow run can upload logs without name collisions.
Builds the containers once and uploads the saved images together with target/dataverse, which docker-compose-dev.yml bind-mounts into the app container.
Each browser gets its own runner and fresh stack. Playwright projects chain the browsers, so --no-deps isolates one browser, with authentication run as a separate invocation first.
Superseded by container_tests.yml. Kept as .disabled for easy rollback.
Superseded by container_tests.yml. Kept as .disabled for easy rollback.
…2782 Runs the old workflow side by side with container_tests.yml on the same commit for a timing comparison. Revert this commit afterwards.
Runs the old workflow side by side with container_tests.yml on the same commit for a timing comparison. Revert this commit afterwards.
The side-by-side timing comparison is done. This reverts commit 4f40418.
The workflow runs Dataverse's test suites inside containers; it doesn't test the containers themselves. The concurrency group follows the name.
Replaced by containerized_tests.yml.
Replaced by containerized_tests.yml.
Lets developers download the exact images a failed run tested against and start them locally.
On pull requests the built code is the merge commit, not the PR head, so the artifact now says exactly which commit to check out when reproducing a failure locally.
ashmanda-cs
force-pushed
the
12782-normalize-testing-harness
branch
from
October 3, 2026 18:36
01298b5 to
f898373
Compare
ashmanda-cs
marked this pull request as ready for review
October 3, 2026 19:17
Draft
3 tasks
…QSS#12782 Besides target/site, the integration-test job now hands over the jacoco-*.exec files in target/coverage-reports/, so the coverage job can merge them again with the server-side coverage of Dataverse Tests.
Maven Tests now runs the API and JSF tests after its unit tests, as a reusable workflow (like push-app-img), reusing the unit-test job's compiled classes. That puts all test suites in one workflow run, so their coverage can be merged. On pushes it only runs for develop, as Dataverse Tests did before.
…rts IQSS#12782 The coverage job waits for Dataverse Tests as well, puts its jacoco-*.exec files next to the unit and integration data in target/coverage-reports/, and reruns the existing jacoco-merge-unit-and-it and jacoco-report executions before submitting to Coveralls. SonarCloud analyzes that merged report. When Dataverse Tests is skipped (branch pushes), the merge covers the unit and integration tests as before.
Dataverse Tests now only runs when Maven Tests calls it (or by hand), so pull requests don't run it twice. Maven Tests' paths gain the ones Dataverse Tests had (conf, src, modules except backports and src/maven, docker-compose-dev.yml, tests, scripts, and the workflows and actions), which also brings container-base and dataverse-spi back in: the API and JSF tests run on images built from them. The path list is shared between push and pull_request with a YAML anchor.
Maven Tests' coverage job now merges the API and JSF tests' coverage with the unit and integration data using the existing JaCoCo executions, and submits it to Coveralls and SonarCloud. The separate jacococli merge, per-suite reports and summary table are no longer needed.
…ged coverage IQSS#12782 Remove the Dataverse Tests status badge (a called workflow has no runs of its own; the Maven Tests badge now covers all suites) and relabel the Coveralls badge as test coverage. The testing guide, release checklist and release note say Maven Tests runs the API and JSF tests after its unit tests, from the same compiled classes, and that their server-side coverage goes to Coveralls and SonarCloud.
The dependency plugin copies the JaCoCo runtime agent, at the version the JaCoCo Maven plugin uses, into the image's new AGENTS_DIR (/opt/payara/agents). It isn't attached by default; the next commit adds a way to attach agents at runtime.
startInForeground.sh adds a -javaagent option to JVM_ARGS for each space-separated entry in JAVA_AGENTS, next to the existing ENABLE_DUMPS handling. An entry is a jar, optionally followed by =options; relative jar paths are resolved against AGENTS_DIR. The list is split with read, so agent options containing * (like JaCoCo's includes) are not expanded as file globs. Empty by default, so nothing changes unless it is set.
docker-compose-dev.yml hands JAVA_AGENTS to dev_dataverse and publishes port 6300 for a JaCoCo agent in tcpserver mode. Like the other compose variables it defaults to empty both as a Maven property in the ct profile (mvn -Pct docker:start) and in .env (docker compose), so no agent is attached unless it is set.
start-dataverse-stack gets a java-agents input that it passes to the compose file as the JAVA_AGENTS property. The test jobs set it to the agent shipped in the base image, defined once next to BASE_URL, instead of downloading the agent and editing docker-compose-dev.yml on the runner.
The JaCoCo agent now ships in the base image and is attached through JAVA_AGENTS, so downloading it and patching docker-compose-dev.yml on the runner is no longer needed.
Set max-parallel: 1 on the language matrix, so a push takes one runner instead of four from the organization's shared concurrency limit. Each language is still its own independent job (fail-fast: false), in the listed order. The run takes about 8 minutes instead of 5, still well under the API and JSF tests.
ashmanda-cs
changed the title
…QSS#12782 docker-compose-dev.yml now builds gdcc/keycloak from conf/keycloak instead of pulling a published image. The build job built it, but only saved the app and configbaker images, so the test jobs tried to pull gdcc/keycloak from Docker Hub and the stack failed to start. Save every gdcc/* image the build produced, so images added to the compose file later travel along too.
The dump and upload steps ran with always(), so when the stack failed to start they failed too (connection refused on port 6300), adding a second red step next to the real cause. Run them only if the stack step succeeded and the run wasn't cancelled; they still run after test failures, so coverage from failing runs is kept.
run-jsf-tests now checks out gdcc/dataverse-jsf-tests at the java branch (a ref input, so it can be pinned) and runs the Java port of the suite: install the browser's system libraries with Playwright's Java CLI, then mvn verify for that browser only. Node.js, npm ci and the separate authentication step are gone; Java and Maven come from the job's setup-maven. The JUnit reports and the per-test traces, videos and screenshots are uploaded as jsf-test-results-<browser>.
SKIP_PREFLIGHT and DOTENV_CONFIG_QUIET were only read by the Node suite. The Java suite reads the remaining variables (BASE_URL, LOGIN_ADAPTER, DV_USERNAME, DV_PASSWORD, DV_FULL_NAME, ROOT_DATAVERSE, LOCALLY_FAIR_ENABLED).
IQSS#12782 The JSF tests now run the Java port of gdcc/dataverse-jsf-tests (java branch) with Maven. Say so, how CI runs it per browser, and that running it locally needs Java and Maven but no Node.js.
The Java JSF suite writes JUnit reports, so each browser job now publishes them with the same action as the API job, as a check and PR comment named "JSF Test Results (<browser>)". A failing JSF test shows up by name on the pull request instead of only as a failed job. The JSF jobs get checks: write and pull-requests: write for this; Maven Tests already grants them to the call.
IQSS#12782 Also drop the claim that the suites finish at about the same time: the JSF chain (Chromium, then Firefox and WebKit) takes longer than the API tests.
Instead of a check and PR comment per browser, a JSF Test Results job runs after the Chromium, Firefox and WebKit jobs, downloads their JUnit reports and publishes them together. Each test counts once, with a run per browser; report_individual_runs reports a failure in a single browser on its own, pointing at that browser's report. Only this job needs write access to checks and pull requests, so the JSF jobs go back to read-only.
This was referenced Oct 16, 2026
The JSF jobs failed only in the suite's own failsafe verify step, which read all three browsers' summaries in a single-browser run. That is fixed in gdcc/dataverse-jsf-tests#15 (java branch), which the workflow checks out at run time.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
How it was before
What this PR introduces
javabranch: Playwright for Java and JUnit 5) with Maven, so they no longer need Node.js.JAVA_AGENTSsetting attaches it, or any other Java agent. The test jobs record which server code they exercise. Maven Tests merges that with the unit and integration test coverage, using the existing JaCoCo executions inpom.xml, and sends it to Coveralls and SonarCloud.How it helps
Business value
Which issue(s) this PR closes:
Special notes for your reviewer:
maven_unit_test.yml):container_app_push.yml.container-baseanddataverse-spi.AGENTS_DIR.startInForeground.shattaches the agents listed inJAVA_AGENTS.docker-compose-dev.ymlpassesJAVA_AGENTSthrough. It is empty by default, so nothing changes unless it is set.setup-mavenactionBASE_URLinstead of hard-coded URLsjavabranch of gdcc/dataverse-jsf-tests (Port the suite to Java (Playwright for Java + JUnit 5) gdcc/dataverse-jsf-tests#14). Moving it into this repository is left for a follow-up.Suggestions on how to test this:
jacoco-api.execandjacoco-jsf-*.execfiles with the unit and integration data.Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No. It only changes CI, the base image and the developer setup.
Is there a release notes update needed for this change?:
Yes, for developers:
doc/release-notes/12782-normalize-testing-harness.md.Additional documentation:
JAVA_AGENTSandAGENTS_DIR.