Repository navigation
feat(provider-webdriver): add TestMu virtual-device cloud provider - #3112
gautam-jain-dev wants to merge 1 commit into
Conversation
Add `testmu` as a cloud WebDriver provider beside BrowserStack and AWS Device Farm. TestMu (formerly LambdaTest) fronts Android emulators and iOS simulators with the same Appium hub as its real devices, so the provider sets `isRealMobile: false` in `lt:options` and otherwise reuses the shared WebDriver runtime. - `connect testmu` verifies the device/OS pair against the public virtual-device catalog and the credentials plus `lt://` app against the authenticated uploaded-app listing, without creating a session. - Local app paths and public URLs are uploaded through the virtual-device upload API when the session is created; `appiumVersion: latest` is requested unless pinned so `mobile:` extensions are available. - Device-feature flags project onto `lt:options` (orientation upper-cased); BrowserStack-only network and resign flags are rejected by name. - `artifacts` reads video, Appium, device, network, and command log URLs from the session-details API and adds the dashboard link. - Credentials: LT_USERNAME / LT_ACCESS_KEY. Endpoints override via TESTMU_WEBDRIVER_ENDPOINT, TESTMU_APP_UPLOAD_ENDPOINT, TESTMU_API_ENDPOINT. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
12 issues found across 29 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/provider-webdriver/src/testmu.ts">
<violation number="1" location="packages/provider-webdriver/src/testmu.ts:111">
P2: `postTestMuUpload` parses the body before checking `response.ok`, so an HTML or empty error response throws `SyntaxError` instead of the typed `COMMAND_FAILED` with status. Catch body-parsing failures and preserve the upload error contract.</violation>
<violation number="2" location="packages/provider-webdriver/src/testmu.ts:196">
P2: `fetchTestMuSessionDetails` treats any object as valid session details without checking the JSend status or rejecting an invalid `data` shape, so provider errors can be reported as pending artifacts. Validate the envelope before mapping URLs.</violation>
<violation number="3" location="packages/provider-webdriver/src/testmu.ts:234">
P2: `readTestMuAppReference` accepts arbitrary `app_url` values even though this helper promises an `lt://` reference and sends it directly to the hub. Require `isTestMuAppReference(record.app_url)` so malformed upload responses fail before session creation.</violation>
</file>
<file name="packages/provider-webdriver/src/browserstack-connection-verification.ts">
<violation number="1" location="packages/provider-webdriver/src/browserstack-connection-verification.ts:113">
P2: This hint now covers HTTP failures as well as transport failures. A 500 or 429 response therefore tells users to check network access and omits the previous BrowserStack service-status guidance; add a separate HTTP-failure hint to the shared helper.</violation>
</file>
<file name="src/commands/schema/cli-help.ts">
<violation number="1" location="src/commands/schema/cli-help.ts:627">
P3: The new TestMu virtual-device flow block ends at `artifacts --json` and omits `agent-device disconnect`, while the topic's lifecycle (`connect -> install/open -> commands -> close -> disconnect`) and every other flow block (Cloud, BrowserStack, AWS Device Farm, Local, Script) end with disconnect. Add it so the example matches the documented lifecycle.</violation>
</file>
<file name="src/__tests__/cloud-connect-profile.test.ts">
<violation number="1" location="src/__tests__/cloud-connect-profile.test.ts:73">
P3: The testmu branch in this mock is unreachable: no test in the file invokes `connect testmu` (or calls `verifyConnection` with `provider: 'testmu'`), and the mock is file-scoped. Add a `connect testmu` test that mirrors the browserstack case (assert the options passed to the mock and the generated state), or the branch should be removed. This also leaves the CLI-side TestMu connect path exercised only by nothing.</violation>
</file>
<file name="src/cli/connection/cloud-webdriver-profile.ts">
<violation number="1" location="src/cli/connection/cloud-webdriver-profile.ts:128">
P2: TestMu’s profile builder accepts BrowserStack-only feature flags and defers rejection until session preparation. As a result, `connect testmu` can report a verified connection and persist state, then fail on the first session; reject these flags in `testMuProfileFields` before generating the profile while retaining the runtime guard for hand-authored profiles.</violation>
</file>
<file name="packages/provider-webdriver/src/testmu-connection-verification.ts">
<violation number="1" location="packages/provider-webdriver/src/testmu-connection-verification.ts:31">
P2: This probe always uses the bundled production URL. `connect testmu` therefore checks the wrong device catalog when `TESTMU_API_ENDPOINT` targets a staging or private deployment; pass the configured endpoint into verification.</violation>
<violation number="2" location="packages/provider-webdriver/src/testmu-connection-verification.ts:76">
P2: This URL construction breaks `appsEndpoint` overrides that already contain query parameters. Build a `URL` and set `searchParams` so existing parameters and the required TestMu filters are composed correctly.</violation>
</file>
<file name="packages/provider-webdriver/src/webdriver-utils.ts">
<violation number="1" location="packages/provider-webdriver/src/webdriver-utils.ts:81">
P2: This session-details request has no deadline, so a stalled provider response can hang artifact retrieval indefinitely. Add the same bounded timeout used by connection verification.</violation>
<violation number="2" location="packages/provider-webdriver/src/webdriver-utils.ts:123">
P2: Session-details transport and JSON parsing failures escape as raw `TypeError` or `SyntaxError`, especially for released-session artifact queries. Wrap the fetch/parse path in `AppError('COMMAND_FAILED', ...)` and preserve the cause.</violation>
<violation number="3" location="packages/provider-webdriver/src/webdriver-utils.ts:124">
P2: The record-shape check admits JSON arrays, so malformed session responses can be reported as pending or ready artifacts instead of failing. Reject arrays alongside other non-object bodies.</violation>
</file>
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
Re-trigger cubic
| function readTestMuAppReference(value: unknown): string | undefined { | ||
| if (!value || typeof value !== 'object') return undefined; | ||
| const record = value as { app_url?: unknown; app_id?: unknown }; | ||
| if (typeof record.app_url === 'string' && record.app_url.length > 0) return record.app_url; |
There was a problem hiding this comment.
P2: readTestMuAppReference accepts arbitrary app_url values even though this helper promises an lt:// reference and sends it directly to the hub. Require isTestMuAppReference(record.app_url) so malformed upload responses fail before session creation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/testmu.ts, line 234:
<comment>`readTestMuAppReference` accepts arbitrary `app_url` values even though this helper promises an `lt://` reference and sends it directly to the hub. Require `isTestMuAppReference(record.app_url)` so malformed upload responses fail before session creation.</comment>
<file context>
@@ -0,0 +1,239 @@
+function readTestMuAppReference(value: unknown): string | undefined {
+ if (!value || typeof value !== 'object') return undefined;
+ const record = value as { app_url?: unknown; app_id?: unknown };
+ if (typeof record.app_url === 'string' && record.app_url.length > 0) return record.app_url;
+ if (typeof record.app_id === 'string' && record.app_id.length > 0) {
+ return isTestMuAppReference(record.app_id) ? record.app_id : `lt://${record.app_id}`;
</file context>
| const details = (json as { data?: unknown }).data ?? json; | ||
| return details && typeof details === 'object' ? (details as Record<string, unknown>) : {}; |
There was a problem hiding this comment.
P2: fetchTestMuSessionDetails treats any object as valid session details without checking the JSend status or rejecting an invalid data shape, so provider errors can be reported as pending artifacts. Validate the envelope before mapping URLs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/testmu.ts, line 196:
<comment>`fetchTestMuSessionDetails` treats any object as valid session details without checking the JSend status or rejecting an invalid `data` shape, so provider errors can be reported as pending artifacts. Validate the envelope before mapping URLs.</comment>
<file context>
@@ -0,0 +1,239 @@
+ service: 'TestMu',
+ });
+ // The API wraps the session in a jsend envelope: `{ status, data: {...}, message }`.
+ const details = (json as { data?: unknown }).data ?? json;
+ return details && typeof details === 'object' ? (details as Record<string, unknown>) : {};
+}
</file context>
| const details = (json as { data?: unknown }).data ?? json; | |
| return details && typeof details === 'object' ? (details as Record<string, unknown>) : {}; | |
| const envelope = asRecord(json); | |
| if (envelope.status !== undefined && envelope.status !== 'success') { | |
| throw new AppError('COMMAND_FAILED', 'TestMu session details lookup failed.', { | |
| response: json, | |
| }); | |
| } | |
| const details = Object.prototype.hasOwnProperty.call(envelope, 'data') ? envelope.data : json; | |
| if (!details || typeof details !== 'object' || Array.isArray(details)) { | |
| throw new AppError('COMMAND_FAILED', 'TestMu session details response was not an object.', { | |
| response: json, | |
| }); | |
| } | |
| return details as Record<string, unknown>; |
| body: form, | ||
| signal, | ||
| }); | ||
| const json = (await response.json()) as unknown; |
There was a problem hiding this comment.
P2: postTestMuUpload parses the body before checking response.ok, so an HTML or empty error response throws SyntaxError instead of the typed COMMAND_FAILED with status. Catch body-parsing failures and preserve the upload error contract.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/testmu.ts, line 111:
<comment>`postTestMuUpload` parses the body before checking `response.ok`, so an HTML or empty error response throws `SyntaxError` instead of the typed `COMMAND_FAILED` with status. Catch body-parsing failures and preserve the upload error contract.</comment>
<file context>
@@ -0,0 +1,239 @@
+ body: form,
+ signal,
+ });
+ const json = (await response.json()) as unknown;
+ const appUrl = readTestMuAppReference(json);
+ if (!response.ok || !appUrl) {
</file context>
| const json = (await response.json()) as unknown; | |
| let json: unknown; | |
| try { | |
| json = (await response.json()) as unknown; | |
| } catch (error) { | |
| throw new AppError('COMMAND_FAILED', 'TestMu app upload failed.', { | |
| status: response.status, | |
| }, error); | |
| } |
| hints: { | ||
| service: 'BrowserStack', | ||
| unauthorizedHint: 'Check BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY.', | ||
| networkHint: 'Check network access to api-cloud.browserstack.com and retry connect.', |
There was a problem hiding this comment.
P2: This hint now covers HTTP failures as well as transport failures. A 500 or 429 response therefore tells users to check network access and omits the previous BrowserStack service-status guidance; add a separate HTTP-failure hint to the shared helper.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/browserstack-connection-verification.ts, line 113:
<comment>This hint now covers HTTP failures as well as transport failures. A 500 or 429 response therefore tells users to check network access and omits the previous BrowserStack service-status guidance; add a separate HTTP-failure hint to the shared helper.</comment>
<file context>
@@ -105,42 +104,15 @@ async function fetchBrowserStackJson(
+ hints: {
+ service: 'BrowserStack',
+ unauthorizedHint: 'Check BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY.',
+ networkHint: 'Check network access to api-cloud.browserstack.com and retry connect.',
+ },
+ });
</file context>
| env?: EnvMap; | ||
| cwd: string; | ||
| }): RemoteConfigProfile { | ||
| return hubProviderProfileFields(TESTMU_HUB_PROFILE, options); |
There was a problem hiding this comment.
P2: TestMu’s profile builder accepts BrowserStack-only feature flags and defers rejection until session preparation. As a result, connect testmu can report a verified connection and persist state, then fail on the first session; reject these flags in testMuProfileFields before generating the profile while retaining the runtime guard for hand-authored profiles.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/cli/connection/cloud-webdriver-profile.ts, line 128:
<comment>TestMu’s profile builder accepts BrowserStack-only feature flags and defers rejection until session preparation. As a result, `connect testmu` can report a verified connection and persist state, then fail on the first session; reject these flags in `testMuProfileFields` before generating the profile while retaining the runtime guard for hand-authored profiles.</comment>
<file context>
@@ -82,29 +86,67 @@ function requireConnectProfileBuilder(
+ env?: EnvMap;
+ cwd: string;
+}): RemoteConfigProfile {
+ return hubProviderProfileFields(TESTMU_HUB_PROFILE, options);
+}
+
</file context>
| const response = await fetch(endpoint, { | ||
| headers: { | ||
| ...agentDeviceRequestHeaders(options.clientVersion), | ||
| ...(options.auth ? { Authorization: basicAuthHeader(options.auth) } : {}), | ||
| }, | ||
| signal: AbortSignal.timeout(15_000), |
There was a problem hiding this comment.
P2: This session-details request has no deadline, so a stalled provider response can hang artifact retrieval indefinitely. Add the same bounded timeout used by connection verification.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/webdriver-utils.ts, line 81:
<comment>This session-details request has no deadline, so a stalled provider response can hang artifact retrieval indefinitely. Add the same bounded timeout used by connection verification.</comment>
<file context>
@@ -50,3 +55,97 @@ export function withTrailingSlash(url: URL): URL {
+): Promise<unknown> {
+ const { service, unauthorizedHint, networkHint } = options.hints;
+ try {
+ const response = await fetch(endpoint, {
+ headers: {
+ ...agentDeviceRequestHeaders(options.clientVersion),
</file context>
| const response = await fetch(endpoint, { | |
| headers: { | |
| ...agentDeviceRequestHeaders(options.clientVersion), | |
| ...(options.auth ? { Authorization: basicAuthHeader(options.auth) } : {}), | |
| }, | |
| signal: AbortSignal.timeout(15_000), | |
| const response = await fetch(endpoint, { | |
| headers: { | |
| ...agentDeviceRequestHeaders(options.clientVersion), | |
| Authorization: basicAuthHeader(options.auth), | |
| }, | |
| signal: AbortSignal.timeout(15_000), | |
| }); |
| Authorization: basicAuthHeader(options.auth), | ||
| }, | ||
| }); | ||
| const json = (await response.json()) as unknown; |
There was a problem hiding this comment.
P2: Session-details transport and JSON parsing failures escape as raw TypeError or SyntaxError, especially for released-session artifact queries. Wrap the fetch/parse path in AppError('COMMAND_FAILED', ...) and preserve the cause.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/webdriver-utils.ts, line 123:
<comment>Session-details transport and JSON parsing failures escape as raw `TypeError` or `SyntaxError`, especially for released-session artifact queries. Wrap the fetch/parse path in `AppError('COMMAND_FAILED', ...)` and preserve the cause.</comment>
<file context>
@@ -50,3 +55,97 @@ export function withTrailingSlash(url: URL): URL {
+ Authorization: basicAuthHeader(options.auth),
+ },
+ });
+ const json = (await response.json()) as unknown;
+ if (!response.ok || !json || typeof json !== 'object') {
+ throw new AppError('COMMAND_FAILED', `${options.service} session details lookup failed.`, {
</file context>
| const auth = { username: options.username, accessKey: options.accessKey }; | ||
| const catalog = await fetchTestMuJson( | ||
| options.devicesEndpoint ?? | ||
| `${trimTrailingSlash(TESTMU_API_ENDPOINT)}/capability/generator?isVirtualDevice=true`, |
There was a problem hiding this comment.
P2: This probe always uses the bundled production URL. connect testmu therefore checks the wrong device catalog when TESTMU_API_ENDPOINT targets a staging or private deployment; pass the configured endpoint into verification.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/provider-webdriver/src/testmu-connection-verification.ts, line 31:
<comment>This probe always uses the bundled production URL. `connect testmu` therefore checks the wrong device catalog when `TESTMU_API_ENDPOINT` targets a staging or private deployment; pass the configured endpoint into verification.</comment>
<file context>
@@ -0,0 +1,181 @@
+ const auth = { username: options.username, accessKey: options.accessKey };
+ const catalog = await fetchTestMuJson(
+ options.devicesEndpoint ??
+ `${trimTrailingSlash(TESTMU_API_ENDPOINT)}/capability/generator?isVirtualDevice=true`,
+ undefined,
+ clientVersion,
</file context>
| agent-device open com.example.app | ||
| agent-device snapshot -i | ||
| agent-device close | ||
| agent-device artifacts --json |
There was a problem hiding this comment.
P3: The new TestMu virtual-device flow block ends at artifacts --json and omits agent-device disconnect, while the topic's lifecycle (connect -> install/open -> commands -> close -> disconnect) and every other flow block (Cloud, BrowserStack, AWS Device Farm, Local, Script) end with disconnect. Add it so the example matches the documented lifecycle.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/commands/schema/cli-help.ts, line 627:
<comment>The new TestMu virtual-device flow block ends at `artifacts --json` and omits `agent-device disconnect`, while the topic's lifecycle (`connect -> install/open -> commands -> close -> disconnect`) and every other flow block (Cloud, BrowserStack, AWS Device Farm, Local, Script) end with disconnect. Add it so the example matches the documented lifecycle.</comment>
<file context>
@@ -617,6 +618,14 @@ Cloud profile flow:
+ agent-device open com.example.app
+ agent-device snapshot -i
+ agent-device close
+ agent-device artifacts --json
+
BrowserStack hosted-device flow:
</file context>
| agent-device artifacts --json | |
| agent-device artifacts --json | |
| agent-device disconnect |
| status: 'missing', | ||
| message: | ||
| 'No app upload is attached; AWS Device Farm does not support install after allocation.', | ||
| : options.provider === 'testmu' |
There was a problem hiding this comment.
P3: The testmu branch in this mock is unreachable: no test in the file invokes connect testmu (or calls verifyConnection with provider: 'testmu'), and the mock is file-scoped. Add a connect testmu test that mirrors the browserstack case (assert the options passed to the mock and the generated state), or the branch should be removed. This also leaves the CLI-side TestMu connect path exercised only by nothing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/__tests__/cloud-connect-profile.test.ts, line 73:
<comment>The testmu branch in this mock is unreachable: no test in the file invokes `connect testmu` (or calls `verifyConnection` with `provider: 'testmu'`), and the mock is file-scoped. Add a `connect testmu` test that mirrors the browserstack case (assert the options passed to the mock and the generated state), or the branch should be removed. This also leaves the CLI-side TestMu connect path exercised only by nothing.</comment>
<file context>
@@ -70,24 +70,37 @@ beforeEach(() => {
- status: 'missing',
- message:
- 'No app upload is attached; AWS Device Farm does not support install after allocation.',
+ : options.provider === 'testmu'
+ ? {
+ provider: 'testmu',
</file context>
|
At 44fc427,
Would it be simpler to treat BrowserStack and TestMu as two configs of one hosted-hub provider factory? A Not blocking, and you can take or leave it: no test drives The PR body states only an Android emulator I did not run the code or the local gates. I did not measure the eager closure or bundle size; I checked the lazy-loading claim by import grep only. I could not verify the TestMu API shapes (capability layout, The only check not passing is the third-party cubic review, which is still in progress. It runs no test route this diff could affect, and no repository job (unit, size, smoke) is listed yet. Before merge, |
2 participants
Summary
Adds
testmuas a cloud WebDriver provider beside BrowserStack and AWS Device Farm. TestMu (formerly LambdaTest) fronts Android emulators and iOS simulators with the same Appium hub as its real devices, so the provider setsisRealMobile: falseinlt:optionsand otherwise reuses the shared WebDriver runtime. First emulator/simulator vendor on this seam.connect testmuverifies the device/OS pair against the public virtual-device catalog and the credentials pluslt://app against the authenticated uploaded-app listing; no session is created.appiumVersion: latestis requested unless pinned somobile:extensions exist.lt:options; BrowserStack-only network/resign flags are refused by name.artifactsreturns video, Appium, device, network and command log URLs plus the dashboard link.29 files touched; docs page
website/docs/docs/testmu.md.Validation
Commit 44fc427:
pnpm typecheck,pnpm lint,pnpm check:fallow --base origin/main, andpnpm check:affected --run(449 files / 3454 tests) pass locally.Live, production TestMu, Android emulator Pixel 5 / 14:
connect→open→snapshot -i→click 'label="…"'(settle diff observed) →screenshot→back→close→artifacts(7 ready). Provider session idsbf384a9f-9af4-465d-9830-f3e59b923272,9d92f504-1dbc-43d6-88dc-4806ea6510e7. iOS simulator path verified throughconnectonly.Risk: no iOS simulator session has been opened yet; the WebDriver runtime path is shared with BrowserStack iOS.
🤖 Generated with Claude Code