Repository navigation
ci(react-native): add lint, build and UI testing with BrowserStack - #151
Merged
Teodor Ciuraru (teodorciuraru) merged 131 commits intoSep 30, 2025
Merged
Conversation
Teodor Ciuraru (teodorciuraru)
requested review from
a team and
Kristopher Johnson (kristopherjohnson)
as code owners
September 1, 2025 11:13
Teodor Ciuraru (teodorciuraru)
marked this pull request as draft
September 1, 2025 11:16
Teodor Ciuraru (teodorciuraru)
changed the title
Teodor Ciuraru (teodorciuraru)
changed the title
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive end-to-end testing infrastructure for React Native apps using Maestro and BrowserStack integration, providing real device testing across Android, iOS, and macOS platforms.
- Adds 9 comprehensive Maestro test flows covering complete user journeys for task management functionality
- Integrates BrowserStack App Automate for real device testing with automated Python execution script
- Implements GitHub Actions workflows for Android, iOS, and macOS testing with proper environment handling
Reviewed Changes
Copilot reviewed 39 out of 40 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/test-maestro-locally.sh |
Interactive local test runner for cross-platform Maestro testing |
react-native/.maestro/flows/*.yaml |
Core test flows for bare React Native app testing |
react-native-expo/.maestro/flows/*.yaml |
Test flows adapted for Expo React Native app |
.github/workflows/*-maestro-browserstack.yml |
CI workflows for automated testing on real devices |
.github/scripts/browserstack-maestro-test.py |
Python automation script for BrowserStack integration |
react-native-expo/components/*.tsx |
Added accessibility test IDs for Maestro automation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
…tion Add complete end-to-end testing infrastructure for React Native apps using Maestro and BrowserStack App Automate for real device testing across Android, iOS, and macOS. ## 🧪 Maestro Test Framework - **9 comprehensive test flows** covering complete user journeys - **3 failure test scenarios** for validation and debugging - **Cross-platform support** for Android, iOS, and macOS - **Proper accessibility IDs** added to React Native components ## ☁️ BrowserStack Integration - **Real device testing** on Android (Pixel 7, Galaxy S23, OnePlus 11) - **iOS device support** for iPhone 14, iPhone 15, iPad Pro - **Automated test execution** with Python script - **Environment variable handling** for Ditto credentials ## 🚀 GitHub Actions Workflows - **Android workflow**: Ubuntu runners with BrowserStack integration - **iOS workflow**: macOS runners with real device testing - **macOS workflow**: Native app testing on macOS runners - **Standardized naming** following existing conventions - **45-minute timeout** and proper caching strategies ## 🛠️ Local Development Support - **Interactive test runner** script with platform selection - **Tag-based test execution** (smoke, core, integration) - **Cross-platform support** (Android emulator, iOS simulator, macOS) - **Comprehensive error handling** and validation ## 📋 Test Coverage - App launch and smoke tests - Task CRUD operations (create, read, update, delete) - Sync toggle functionality - Data persistence across app restarts - External sync verification via Ditto Cloud API - Full end-to-end user workflows 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix command injection vulnerability in env variable parsing - Remove hardcoded local path from Python script - Fix shell variable scoping in test runner using process substitution - Fix head command syntax causing iOS build failure (head -n-1 -> head -n -1)
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
The head command on macOS/BSD does not support negative line counts. Replace with sed '$d' which removes the last line and works on both BSD and GNU systems. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Change build commands from 'yarn build:android' to direct Gradle assembleDebug - BrowserStack App Automate requires APK files, not AAB files - Use gradle assembleDebug task which generates APK files in android/app/build/outputs/apk/debug/ 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fix API base URL: api.browserstack.com -> api-cloud.browserstack.com
- Update endpoints to use v2 API:
- /maestro/upload-suite -> /maestro/v2/test-suite
- /maestro/build -> /maestro/v2/android/build
- /maestro/builds/{id} -> /maestro/v2/builds/{id}
- Resolves 404 errors when uploading Maestro test suites
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Try multiple possible response field names (test_url, suite_url, test_suite_url) - Add debug logging to show full API response - Resolves issue where suite_url was None causing 422 'Invalid testing suite' error 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…n API - Change 'suite' to 'testSuite' in execution payload - Matches BrowserStack Maestro v2 API specification - Should resolve 422 'Invalid testing suite' error 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Change from objects to simple strings as required by v2 API
- Use format 'Device Name-OS Version' instead of {device: ..., osVersion: ...}
- Should resolve 422 'Invalid value passed for devices parameter' error
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix device names to match BrowserStack catalog exactly: - 'Google Pixel 7' → 'Pixel 7' - 'Samsung Galaxy S23' → 'Galaxy S23' - 'OnePlus 11' → 'OnePlus 11R' (11R is available, 11 is not) - Should resolve 422 'BROWSERSTACK_INVALID_DEVICE' error 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Change from dash format 'Pixel 7-13.0' to comma format 'Pixel 7, 13.0' - Matches BrowserStack API specification exactly - Final fix for device specification format 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Change device names from 'Pixel 7, 13.0' to 'Google Pixel 7-13.0' - Update all device specifications to match BrowserStack API requirements - Addresses 422 error: BROWSERSTACK_NO_DEVICE_SPECIFIED 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add fallback parsing for build_id and id fields - Include debug output to show full response structure - Handles cases where BrowserStack returns different field names 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Search multiple build paths for .app files (Debug-iphonesimulator, build dirs) - Add comprehensive debugging output for failed app detection - Look for Build, DerivedData, and Products directories - Addresses app verification failures in iOS and macOS workflows 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
iOS/macOS App Generation: - Replace react-native build commands with proper xcodebuild - Use xcodebuild with derivedDataPath to generate actual .app files - Update verification to search in correct xcodebuild output locations BrowserStack Test Reliability: - Add simplified basic launch test for BrowserStack compatibility - Reduce device count to single device for initial testing - Improve permission handling in test flows 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…rrors Remove specific iPhone 14 destination which isn't available on all runners. Use generic iOS Simulator platform destination which works universally. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Use 'DittoReactNativeSampleApp-macOS' instead of 'DittoReactNativeSampleApp'. The actual scheme includes the -macOS suffix as found in xcschemes directory. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Use 'id=dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder' instead of generic platform destination. This matches the available destination listed by xcodebuild -list output. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace placeholder ID with actual iPhone 16 simulator ID (75860960-263C-416D-B8B8-60C5169ACF52) from the available destinations list. The placeholder ID was listed as available but not working. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add outputs section to seed-ditto-cloud job in both CI workflows - Add step ID to document seeding step for proper output reference - Ensures MAESTRO_TASK_TO_FIND gets the actual task title instead of empty string - Fixes BrowserStack tests receiving empty task title parameter 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Replace unique timestamped task IDs with fixed "Basic Test Task" - Both react-native-ci and react-native-expo-ci now create identical task - Add scrolling functionality to all Maestro test flows - Ensures consistent test target across all React Native workflows - Simplifies BrowserStack testing with predictable task name 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…ests - Add testID props to task components in both React Native and Expo apps - Update all Maestro flows to use reliable testID selectors instead of text - Replace fixed scroll loops with scrollUntilVisible for efficient element finding - Standardize test flows across iOS and Android platforms - Fix CI workflow task naming for consistency 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Revert CI workflows to use timestamp-based unique task IDs - This ensures Maestro tests look for the dynamically created task - Fixes mismatch where tests looked for hardcoded task that doesn't exist 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add sed command to replace ${MAESTRO_TASK_TO_FIND} with actual task name in Android test files
- Remove ineffective env sections from CI jobs (only setEnvVariables in API calls work)
- Keep iOS tests using setEnvVariables (works correctly for iOS)
- Add TODO comments documenting BrowserStack Android API bug
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Keep only the testID on Text element (used by Maestro tests) - Remove unused testID from View container and TaskDone component - Clean up TaskDone props to remove unused testID parameter
Contributor
Author
|
Vincent Ahrend (@pvditto), there are some new developments here:
PR should be ready to go. |
Teodor Ciuraru (teodorciuraru)
requested a review
from Vincent Ahrend (pvditto)
September 25, 2025 12:17
Teodor Ciuraru (teodorciuraru)
changed the title
- Add output declarations for build_id in test jobs - Add report-results job that posts BrowserStack dashboard links to PR - Provides direct links to Android and iOS test dashboards
Contributor
📱 React Native Expo - BrowserStack Maestro Test ResultsAndroid Tests✅ View Android Dashboard iOS Tests❌ iOS tests did not produce a valid build ID |
- Add status output to test jobs (success/failure) - Display pass/fail status in PR comments with ✅/❌ icons - Add overall test result summary - Show clear status for each platform (Android/iOS)
Contributor
📱 React Native - BrowserStack Maestro Test ResultsAndroid Tests✅ View Android Dashboard iOS Tests✅ View iOS Dashboard |
- Add unique HTML comment identifiers to find previous reports - Update existing comment instead of creating new ones - Each workflow updates its own dedicated comment - Reduces PR noise while keeping latest results visible
Contributor
📱 React Native - BrowserStack Maestro Test ResultsAndroid Tests❌ Status: Failed (No build created) iOS Tests❌ Status: Failed (No build created)
|
Contributor
📱 React Native Expo - BrowserStack Maestro Test ResultsAndroid Tests❌ Status: Failed (No build created) iOS Tests❌ Status: Failed (No build created)
|
- Replace PR comments with GitHub Step Summary for cleaner PR experience - Results now appear in Actions tab workflow summary - Zero PR comment spam while maintaining full visibility - BrowserStack links and status still easily accessible from workflow run
Contributor
📱 React Native Expo - BrowserStack Maestro Test ResultsAndroid Tests❌ Status: Failed (No build created) iOS Tests❌ Status: Failed (No build created)
|
Contributor
📱 React Native - BrowserStack Maestro Test ResultsAndroid Tests❌ Status: Failed (No build created) iOS Tests❌ Status: Failed (No build created)
|
Vincent Ahrend (pvditto)
approved these changes
Sep 29, 2025
Vincent Ahrend (pvditto)
left a comment
Vincent Ahrend (pvditto)
left a comment
Contributor
There was a problem hiding this comment.
LGTM
I think the iOS test doesn't click "Allow" often enough. In the recording I can still see one open dialog at the end of the test. See https://app-automate.browserstack.com/dashboard/v2/builds/6d42de9f7a64c0790e8a6d608974e67e8f72a2e3/sessions/tests/de2b78c744f040d96d1b00f30a25b90553e55c4a525880fe
I prefer the check annotation over a comment as well but now that these integration tests use a mix of both it gets confusing.
Removes accidental debug configuration that disabled DQL strict mode in the React Native Expo app. This restores symmetry with the React Native CLI app and ensures proper production configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added conditional handling for the Bluetooth permission dialog that appears when the Ditto SDK initializes on iOS. The test now taps "OK" when the Bluetooth permission alert is visible, preventing test failures. Applied to both React Native CLI and React Native Expo iOS test flows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
Solved. |
Teodor Ciuraru (teodorciuraru)
deleted the
teodor/add-rn-apps-tests-pipeline
branch
September 30, 2025 10:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add React Native E2E testing infrastructure using Maestro and BrowserStack App Automate for automated real device testing on Android and iOS.
Problem Solved
What This Adds
E2E Testing Infrastructure
BrowserStack Android Workaround
sedto replace test variables during CI executionsetEnvVariableswhen officially supportedReact Native App Improvements
IdentityOnlinePlaygroundto modernDittoConfigsyntaxtransportConfig.connect.websocketURLsfor proper transport setupValidation
Automated
Manual