Repository navigation
fix(storybook-desktop): report all-skipped WDIO runs as incomplete instead of passed - #4369
Open
udit (UditDewan) wants to merge 1 commit into
Open
udit (UditDewan) wants to merge 1 commit into
udit (UditDewan) wants to merge 1 commit into
Conversation
A WDIO selection in which every case called skip(...) was written to run.json as "passed" and exited successfully, so it could satisfy a required test gate without exercising the product. runWdioStoryTests now records such runs as "incomplete" and throws a WdioStoryCoverageUnavailableError (distinct from WdioStoryTestRunError) that carries every result and skip reason. Smoke runs treat a single all-skipped story as non-fatal, but report the aggregate as incomplete and fail when no test across the run passed. Both reports now include selected/passed/skipped/failed counts. Mixed pass/skip runs still pass. Fixes microsoft#4342
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Platforms Impacted
Description of changes
Fixes #4342.
A WDIO selection in which every case called
skip(...)was written torun.jsonas"status": "passed"and the command exited successfully, so an all-skipped run (for examplecomponents-checkbox--defaultwithout physical pointer input) could satisfy a required test gate without exercising the product.src/testing/runWdioTests.tsWdioStoryCoverageUnavailableError, thrown when no selected case passed and none failed. It is deliberately not aWdioStoryTestRunError, so coverage unavailability stays distinguishable from an assertion failure. It carries every result, and its message lists the skip reasons.run.jsonstatus is nowfailed/incomplete/passed, and the report includescounts: { selected, passed, skipped, failed }. Per-testskipReasons are unchanged.yarn storybook testtherefore exits non-zero for an all-skipped selection.src/cli/smokeTests.tsincompleteand the run fails. The smokerun.jsonalso gainscounts.Existing zero-selection, failure, timeout, crash, evidence, and session-cleanup paths are untouched; the new checks only run after them and only when there are no failures.
Out of scope here: CI summary rendering and required-cohort promotion policy (#4349/#4345). They can key off the new
incompletestatus andcounts.Verification
yarn build,yarn lint,yarn testin@fluentui-react-native/storybook-desktop(163 tests);yarn format:check;yarn lage test-links.status: 'incomplete'with counts and skip reason{ selected: 2, passed: 1, skipped: 1 }fixture-button--skipnow rejects withWdioStoryCoverageUnavailableErrorandrun.jsonisincomplete. The existing pass, fail/timeout/crash, named, mixedrestrictedButton, and empty-selection contract cases are unchanged.mainand pass with this change.run.json"status": "passed", exit 0"status": "incomplete"with counts, non-zero exit, skip reasons in the errorPull request checklist
This PR has considered (when applicable):