fix(util): recognize *.bsstag.com hosts as BrowserStack (v8, SDK-6948)

.changeset/pr-183.md

@@ -0,0 +1,7 @@
+---
+"@wdio/browserstack-service": patch
+---
+
+- SDK-6948: `*.bsstag.com` hubs are now recognised as BrowserStack on named internal staging envs (`BROWSERSTACK_STAGING_ENV` / `BROWSERSTACK_ENV`), so with the CLI flow (default) session marking and TRA linking work there. The classic (non-CLI) flow on a named staging env is not supported; its session update is skipped at debug.
+- SDK-6948: `update_cli` failures fall back to `BROWSERSTACK_BINARY_URL`. A failed fallback download uses the cached binary.
+- SDK-6948: `_printSessionURL` and `_update` log a failed session lookup or update (HTTP error) at debug instead of throwing.

.changeset/sdk-6948-v8-binary-url-fallback.md

@@ -0,0 +1,7 @@
+---
+"@wdio/browserstack-service": patch
+---
+
+fix(cli): fall back to `BROWSERSTACK_BINARY_URL` when `update_cli` fails (SDK-6948)
+
+`update_cli` runs during bootstrap — before the binary is spawned and before GRR localizes the API hosts — so it always targets production `api.browserstack.com`. On an internal staging run (`BROWSERSTACK_STAGING_ENV`) the staging credentials are rejected there (`401`), which left the CLI binary path empty and crashed the launcher with `spawn('') … ERR_INVALID_ARG_VALUE: 'file' cannot be empty`. When `update_cli` fails and `BROWSERSTACK_BINARY_URL` is set, download the binary from that URL instead so the run can proceed; the same applies when a 2xx reply carries no download URL and no binary is cached. If that download fails, the cached binary is used when there is one, and both errors are logged. Opt-in only — with the variable unset the original error propagates exactly as before. Ports main/v9 #224 to the v8 line.

.changeset/sdk-6948-v8-bsstag-host-recognition.md

@@ -0,0 +1,7 @@
+---
+"@wdio/browserstack-service": patch
+---
+
+fix(util): recognize `*.bsstag.com` hosts as BrowserStack on internal staging envs (SDK-6948)
+
+`getCloudProvider` and `isBrowserstackInfra` only matched `browserstack.com`, so a session pointed at an internal staging env (`*.bsstag.com`) was classified as non-BrowserStack: session marking and Automate <-> TRA session linking did not work there. `*.bsstag.com` hosts are now treated as BrowserStack when `BROWSERSTACK_STAGING_ENV` is a bare env name (e.g. `k8s`, trimmed; blank means unset) or `BROWSERSTACK_ENV` is `staging`/`stag`/`preprod`/`pre-prod`. Hosts are matched case-insensitively. Production behaviour is unchanged. With the CLI flow (the default), session marking and Automate <-> TRA session linking now work on those envs, because the binary supplies the staging session REST host. The classic (non-CLI) flow on a named staging env is not supported: it has no staging session REST host, so the session name/status update is skipped at debug instead of failing the hook. `_printSessionURL` and `_update` log a failed session lookup or update (HTTP error reply) at debug instead of throwing. Ports main/v9 #275 to the v8 line.

packages/browserstack-service/src/cli/cliUtils.ts

@@ -187,22 +187,63 @@ export class CLIUtils {
             const nullDevice = platform() === 'win32' ? 'NUL' : '/dev/null'
             queryParams.cli_version = await this.runShellCommand(`${existingCliPath} version 2>${nullDevice}`)
         }
-        const response = await this.requestToUpdateCLI(queryParams, config)
-        if (nestedKeyValue(response, ['updated_cli_version'])) {
-            logger.debug(`Need to update binary, current binary version: ${queryParams.cli_version}`)
-
-            const browserStackBinaryUrl = process.env.BROWSERSTACK_BINARY_URL || null
-            if (!isNullOrEmpty(browserStackBinaryUrl)) {
-                logger.debug(`Using BROWSERSTACK_BINARY_URL: ${browserStackBinaryUrl}`)
-                response.url = browserStackBinaryUrl
+        const browserStackBinaryUrl = process.env.BROWSERSTACK_BINARY_URL || null
+
+        const fallbackToBinaryUrl = async (reason: unknown) => {
+            logger.warn(`update_cli request failed (${reason}); falling back to BROWSERSTACK_BINARY_URL`)
+            try {
+                return await this.downloadLatestBinary(browserStackBinaryUrl as string, cliDir)
+            } catch (downloadErr) {
+                // A cached binary is still better than failing the run; keep the download error visible.
+                if (isNullOrEmpty(existingCliPath)) {
+                    logger.error(`BROWSERSTACK_BINARY_URL download failed (${util.format(downloadErr)}) after update_cli failed (${reason})`)
+                    throw downloadErr
+                }
+                logger.warn(`BROWSERSTACK_BINARY_URL download failed (${util.format(downloadErr)}); using cached binary ${existingCliPath}`)
+                return existingCliPath
             }
+        }
 
-            const finalBinaryPath = await this.downloadLatestBinary(nestedKeyValue(response, ['url']), cliDir)
+        // Single close for the span on every path, including a rejected download.
+        try {
+            let response
+            try {
+                response = await this.requestToUpdateCLI(queryParams, config)
+            } catch (err) {
+                // The version-check call (update_cli) failed — e.g. an internal staging run where
+                // update_cli is served from a host the provided creds are not valid on (staging creds
+                // against the prod api host -> 401). If an explicit binary URL was supplied, use it so
+                // the run is not blocked on this call. Opt-in only: with no BROWSERSTACK_BINARY_URL the
+                // error propagates exactly as before, so production behaviour is unchanged.
+                if (!isNullOrEmpty(browserStackBinaryUrl)) {
+                    logger.debug(`update_cli error: ${util.format(err)}`)
+                    const status = (err as { response?: { statusCode?: number } })?.response?.statusCode ?? (err as Error)?.message
+                    return await fallbackToBinaryUrl(status)
+                }
+                throw err
+            }
+
+            // A 2xx reply with neither `updated_cli_version` nor `url` means there is nothing to update:
+            // keep a cached binary, and only fetch from the explicit binary URL when there is none.
+            if (!isNullOrEmpty(browserStackBinaryUrl) && isNullOrEmpty(existingCliPath) &&
+                !nestedKeyValue(response, ['updated_cli_version']) && !nestedKeyValue(response, ['url'])) {
+                return await fallbackToBinaryUrl(JSON.stringify(response))
+            }
+
+            if (nestedKeyValue(response, ['updated_cli_version'])) {
+                logger.debug(`Need to update binary, current binary version: ${queryParams.cli_version}`)
+
+                if (!isNullOrEmpty(browserStackBinaryUrl)) {
+                    logger.debug(`Using BROWSERSTACK_BINARY_URL: ${browserStackBinaryUrl}`)
+                    response.url = browserStackBinaryUrl
+                }
+
+                return await this.downloadLatestBinary(nestedKeyValue(response, ['url']), cliDir)
+            }
+            return existingCliPath
+        } finally {
             PerformanceTester.end(PerformanceEvents.SDK_CLI_CHECK_UPDATE)
-            return finalBinaryPath
         }
-        PerformanceTester.end(PerformanceEvents.SDK_CLI_CHECK_UPDATE)
-        return existingCliPath
     }
 
     static getCliDir() {

packages/browserstack-service/src/service.ts

@@ -794,18 +794,27 @@ export default class BrowserstackService implements Services.ServiceInstance {
         }
         const sessionUrl = `${this._sessionBaseUrl}/${sessionId}.json`
         BStackLogger.debug(`Updating Browserstack session at ${sessionUrl} with request body: `, requestBody)
+        // SDK-6948: an HTTP error reply (e.g. 401/404 from a named staging env's session REST host)
+        // must not fail the calling hook. got rejects on non-2xx; main/v9 uses fetch, which resolves,
+        // so only HTTP-status errors are swallowed here and network errors still propagate.
+        const onHttpError = (err: { response?: { statusCode?: number } }) => {
+            if (!err?.response) {
+                throw err
+            }
+            BStackLogger.debug(`Could not update Browserstack session (${err.response.statusCode}) at ${sessionUrl}`)
+        }
         if (this._turboScale) {
             return got.patch(sessionUrl, {
                 json: requestBody,
                 username: this._config.user,
                 password: this._config.key
-            })
+            }).catch(onHttpError)
         }
         return got.put(sessionUrl, {
             json: requestBody,
             username: this._config.user,
             password: this._config.key
-        })
+        }).catch(onHttpError)
     }
 
     @PerformanceTester.Measure(PERFORMANCE_SDK_EVENTS.AUTOMATE_EVENTS.PRINT_BUILDLINK)
@@ -824,12 +833,25 @@ export default class BrowserstackService implements Services.ServiceInstance {
                 responseType: 'json'
             }
 
-            if (this._turboScale) {
-                const response = await got<TurboScaleSessionResponse>(sessionUrl, reqOpts)
-                browserUrl = response.body.url
-            } else {
-                const response = await got<SessionResponse>(sessionUrl, reqOpts)
-                browserUrl = response.body.automation_session.browser_url
+            try {
+                if (this._turboScale) {
+                    const response = await got<TurboScaleSessionResponse>(sessionUrl, reqOpts)
+                    browserUrl = response.body.url
+                } else {
+                    const response = await got<SessionResponse>(sessionUrl, reqOpts)
+                    const session = response.body?.automation_session
+                    if (!session) {
+                        BStackLogger.debug(`Could not fetch Browserstack session URL (${response.statusCode}) at ${sessionUrl}`)
+                        return
+                    }
+                    browserUrl = session.browser_url
+                }
+            } catch (err) {
+                // SDK-6948: on a named staging env the session REST host may not serve this session
+                // (e.g. classic non-CLI flow). Printing the link is informational, so don't fail the hook.
+                const status = (err as { response?: { statusCode?: number } })?.response?.statusCode ?? (err as Error)?.message
+                BStackLogger.debug(`Could not fetch Browserstack session URL (${status}) at ${sessionUrl}`)
+                return
             }
 
             if (!this._browser) {

packages/browserstack-service/src/util.ts

@@ -1029,16 +1029,38 @@ export function getUniqueIdentifierForCucumber(world: ITestCaseHookParameter): s
     return world.pickle.uri + '_' + world.pickle.astNodeIds.join(',')
 }
 
+/**
+ * SDK-6948: a named internal staging env (`BROWSERSTACK_STAGING_ENV` set to a bare token such as
+ * `k8s`) serves its hub/cdp from `*.bsstag.com` (e.g. `hub-k8s.bsstag.com`). Treat those hosts as
+ * BrowserStack so session marking and Automate <-> TRA linking work there. A dotted value (full
+ * host) or an unset/blank variable leaves the production-only host checks unchanged. The env value is
+ * trimmed and the hostname compared case-insensitively (shared rule across SDKs).
+ * `BROWSERSTACK_ENV` (trimmed, lowercased) of staging/stag/preprod/pre-prod also selects the bsstag
+ * hosts, matching node-agent constants and the binary's getEnvironmentUrl.
+ */
+const BSTAG_BROWSERSTACK_ENVS = ['staging', 'stag', 'preprod', 'pre-prod']
+
+export function isStagingEnvBsstagHost(hostname: string): boolean {
+    const stagingEnv = process.env.BROWSERSTACK_STAGING_ENV?.trim()
+    const namedStagingEnv = Boolean(stagingEnv) && !stagingEnv!.includes('.')
+    const browserstackEnv = process.env.BROWSERSTACK_ENV?.trim().toLowerCase() || ''
+    if (!namedStagingEnv && !BSTAG_BROWSERSTACK_ENVS.includes(browserstackEnv)) {
+        return false
+    }
+    const host = hostname.toLowerCase()
+    return host === 'bsstag.com' || host.endsWith('.bsstag.com')
+}
+
 export function getCloudProvider(browser: WebdriverIO.Browser | WebdriverIO.MultiRemoteBrowser): string {
     if (browser && 'instances' in browser) {
         // Loop through all instances
         for (const instanceName of browser.instances) {
             const instance = (browser as any)[instanceName] as WebdriverIO.Browser
-            if (instance.options && instance.options.hostname && instance.options.hostname.includes('browserstack')) {
+            if (instance.options && instance.options.hostname && (instance.options.hostname.includes('browserstack') || isStagingEnvBsstagHost(instance.options.hostname))) {
                 return 'browserstack'
             }
         }
-    } else if (browser.options && browser.options.hostname && browser.options.hostname.includes('browserstack')) { // Single browser instance
+    } else if (browser.options && browser.options.hostname && (browser.options.hostname.includes('browserstack') || isStagingEnvBsstagHost(browser.options.hostname))) { // Single browser instance
         return 'browserstack'
     }
     return 'unknown_grid'
@@ -1168,7 +1190,7 @@ export function isBrowserstackInfra(config: BrowserstackConfig & Options.Testrun
     // a utility function to check if the hostname is browserstack
 
     const isBrowserstack = (str: string ): boolean => {
-        return str.includes('browserstack.com')
+        return str.includes('browserstack.com') || isStagingEnvBsstagHost(str)
     }
 
     if ((config.hostname) && !isBrowserstack(config.hostname)) {

packages/browserstack-service/tests/cli/cliUtils.test.ts

@@ -441,6 +441,104 @@ describe('CLIUtils', () => {
                 mockConfig
             )
         })
+
+        it('falls back to BROWSERSTACK_BINARY_URL when update_cli fails (SDK-6948)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            process.env.BROWSERSTACK_BINARY_URL = 'https://example.com/staging-binary.zip'
+            try {
+                const fallbackPath = '/mock/cli/dir/binary-fallback'
+                vi.spyOn(CLIUtils, 'runShellCommand').mockResolvedValue('1.0.0')
+                // update_cli fails (e.g. staging creds against the prod api host -> 401)
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockRejectedValue({ response: { statusCode: 401 } })
+                vi.spyOn(CLIUtils, 'downloadLatestBinary').mockResolvedValue(fallbackPath)
+
+                const result = await CLIUtils.checkAndUpdateCli(mockExistingPath, mockCliDir, mockConfig)
+
+                expect(result).toBe(fallbackPath)
+                expect(CLIUtils.downloadLatestBinary).toHaveBeenCalledWith('https://example.com/staging-binary.zip', mockCliDir)
+            } finally {
+                if (saved === undefined) { delete process.env.BROWSERSTACK_BINARY_URL } else { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
+
+        it('uses the cached binary when the BROWSERSTACK_BINARY_URL download fails (SDK-6948)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            process.env.BROWSERSTACK_BINARY_URL = 'https://example.com/staging-binary.zip'
+            try {
+                vi.spyOn(CLIUtils, 'runShellCommand').mockResolvedValue('1.0.0')
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockRejectedValue({ response: { statusCode: 401 } })
+                vi.spyOn(CLIUtils, 'downloadLatestBinary').mockRejectedValue(new Error('download failed'))
+
+                const result = await CLIUtils.checkAndUpdateCli(mockExistingPath, mockCliDir, mockConfig)
+
+                expect(result).toBe(mockExistingPath)
+            } finally {
+                if (saved === undefined) { delete process.env.BROWSERSTACK_BINARY_URL } else { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
+
+        it('re-throws the download error when the BROWSERSTACK_BINARY_URL download fails and no binary is cached (SDK-6948)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            process.env.BROWSERSTACK_BINARY_URL = 'https://example.com/staging-binary.zip'
+            try {
+                const downloadErr = new Error('download failed')
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockRejectedValue({ response: { statusCode: 401 } })
+                vi.spyOn(CLIUtils, 'downloadLatestBinary').mockRejectedValue(downloadErr)
+
+                await expect(CLIUtils.checkAndUpdateCli('', mockCliDir, mockConfig)).rejects.toBe(downloadErr)
+            } finally {
+                if (saved === undefined) { delete process.env.BROWSERSTACK_BINARY_URL } else { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
+
+        it('falls back to BROWSERSTACK_BINARY_URL when a 2xx reply has neither url nor updated_cli_version and no binary is cached (SDK-6948)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            process.env.BROWSERSTACK_BINARY_URL = 'https://example.com/staging-binary.zip'
+            try {
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockResolvedValue({ message: 'Unauthorized' })
+                vi.spyOn(CLIUtils, 'downloadLatestBinary').mockResolvedValue('/mock/cli/dir/binary-fallback')
+
+                const result = await CLIUtils.checkAndUpdateCli('', mockCliDir, mockConfig)
+
+                expect(result).toBe('/mock/cli/dir/binary-fallback')
+                expect(CLIUtils.downloadLatestBinary).toHaveBeenCalledWith('https://example.com/staging-binary.zip', mockCliDir)
+            } finally {
+                if (saved === undefined) { delete process.env.BROWSERSTACK_BINARY_URL } else { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
+
+        it('keeps the cached binary when a 2xx reply has neither url nor updated_cli_version (SDK-6948)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            process.env.BROWSERSTACK_BINARY_URL = 'https://example.com/staging-binary.zip'
+            try {
+                vi.spyOn(CLIUtils, 'runShellCommand').mockResolvedValue('1.0.0')
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockResolvedValue({ success: true })
+                const downloadSpy = vi.spyOn(CLIUtils, 'downloadLatestBinary').mockResolvedValue('/should/not/be/used')
+
+                const result = await CLIUtils.checkAndUpdateCli(mockExistingPath, mockCliDir, mockConfig)
+
+                expect(result).toBe(mockExistingPath)
+                expect(downloadSpy).not.toHaveBeenCalled()
+            } finally {
+                if (saved === undefined) { delete process.env.BROWSERSTACK_BINARY_URL } else { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
+
+        it('re-throws when update_cli fails and no BROWSERSTACK_BINARY_URL is set (unchanged behaviour)', async () => {
+            const saved = process.env.BROWSERSTACK_BINARY_URL
+            delete process.env.BROWSERSTACK_BINARY_URL
+            try {
+                const err = new Error('Unauthorized')
+                vi.spyOn(CLIUtils, 'runShellCommand').mockResolvedValue('1.0.0')
+                vi.spyOn(CLIUtils, 'requestToUpdateCLI').mockRejectedValue(err)
+                const downloadSpy = vi.spyOn(CLIUtils, 'downloadLatestBinary').mockResolvedValue('/should/not/be/used')
+
+                await expect(CLIUtils.checkAndUpdateCli(mockExistingPath, mockCliDir, mockConfig)).rejects.toBe(err)
+                expect(downloadSpy).not.toHaveBeenCalled()
+            } finally {
+                if (saved !== undefined) { process.env.BROWSERSTACK_BINARY_URL = saved }
+            }
+        })
     })
 
     describe('setupCliPath', () => {