Skip to content

fix(storybook-desktop): report all-skipped WDIO runs as incomplete instead of passed - #4369

Open
udit (UditDewan) wants to merge 1 commit into
microsoft:mainfrom
UditDewan:fix/wdio-all-skipped-incomplete
Open

udit (UditDewan) wants to merge 1 commit into
microsoft:mainfrom
UditDewan:fix/wdio-all-skipped-incomplete

Conversation

@UditDewan

Copy link
Copy Markdown

Platforms Impacted

  • iOS
  • macOS
  • win32 (Office)
  • windows
  • android

Description of changes

Fixes #4342.

A WDIO selection in which every case called skip(...) was written to run.json as "status": "passed" and the command exited successfully, so an all-skipped run (for example components-checkbox--default without physical pointer input) could satisfy a required test gate without exercising the product.

src/testing/runWdioTests.ts

  • New WdioStoryCoverageUnavailableError, thrown when no selected case passed and none failed. It is deliberately not a WdioStoryTestRunError, so coverage unavailability stays distinguishable from an assertion failure. It carries every result, and its message lists the skip reasons.
  • run.json status is now failed / incomplete / passed, and the report includes counts: { selected, passed, skipped, failed }. Per-test skipReasons are unchanged.
  • yarn storybook test therefore exits non-zero for an all-skipped selection.

src/cli/smokeTests.ts

  • A single story whose callbacks all skip is not fatal in grouped smoke (its results are kept), so mixed suites like the current 7-passed/22-skipped native runs keep passing.
  • If nothing across the whole smoke run passed (plans + WDIO), the aggregate report is written as incomplete and the run fails. The smoke run.json also gains counts.

Existing zero-selection, failure, timeout, crash, evidence, and session-cleanup paths are untouched; the new checks only run after them and only when there are no failures.

Out of scope here: CI summary rendering and required-cohort promotion policy (#4349/#4345). They can key off the new incomplete status and counts.

Verification

  • yarn build, yarn lint, yarn test in @fluentui-react-native/storybook-desktop (163 tests); yarn format:check; yarn lage test-links.
  • New/updated tests:
    • smoke: all-skipped run rejects and persists status: 'incomplete' with counts and skip reason
    • smoke: mixed run (one passing plan + one all-skipped WDIO story) passes with counts { selected: 2, passed: 1, skipped: 1 }
    • smoke: inline-only filtered selection still passes when the callback passes
    • WDIO contract (real worker): fixture-button--skip now rejects with WdioStoryCoverageUnavailableError and run.json is incomplete. The existing pass, fail/timeout/crash, named, mixed restrictedButton, and empty-selection contract cases are unchanged.
  • The new smoke tests and the contract fail on main and pass with this change.
Before After
All-skipped selection → run.json "status": "passed", exit 0 "status": "incomplete" with counts, non-zero exit, skip reasons in the error
Mixed pass/skip → passed Unchanged (passed, now with counts)

Pull request checklist

This PR has considered (when applicable):

  • Automated Tests
  • Documentation and examples
  • Keyboard Accessibility
  • Voiceover
  • Internationalization and Right-to-left Layouts

A WDIO selection in which every case called skip(...) was written to
run.json as "passed" and exited successfully, so it could satisfy a
required test gate without exercising the product.

runWdioStoryTests now records such runs as "incomplete" and throws a
WdioStoryCoverageUnavailableError (distinct from WdioStoryTestRunError)
that carries every result and skip reason. Smoke runs treat a single
all-skipped story as non-fatal, but report the aggregate as incomplete
and fail when no test across the run passed. Both reports now include
selected/passed/skipped/failed counts. Mixed pass/skip runs still pass.

Fixes microsoft#4342
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Infrastructure: Reject all-skipped WDIO runs as completed coverage

1 participant