Repository navigation
fix: align Duo W3C actions and scrolling - #1288
Conversation
DanMPerfecto
left a comment
DanMPerfecto
left a comment
There was a problem hiding this comment.
I checked several review findings on real hardware and simulators, all built with Xcode 27.2 beta 2 at ecb650b:
- iPhone Duo simulator, iOS 27.1, unfolded. The inner display is display 3 and the app runs in LandscapeLeft.
- iPhone 18 Pro simulator, iOS 27.2.
- Physical iPhone 12 Pro, iOS 26.7. It also exposes secondary displays 2 and 3.
The touch fixture measured where touches actually landed. The new FBDisplayGestureTests and the display tests in FBW3CTouchActionsIntegrationTestsPart1 pass 11/11 on the Duo and on the iPhone 12 Pro. The viewport correction for LandscapeLeft is accurate: the PR's offset of (0, 273) lands within 0.1 pt of the target. Inline comments cover the issues that reproduced.
Note: with Xcode 26.x, the IntegrationApp runs in a compatibility window on the iOS 27 Duo, and XCTest reports wrong element frames there. Build with Xcode 27.x to reproduce.
| } | ||
|
|
||
| if (nil != element) { | ||
| // Normalized element coordinates already resolve against their own display. |
* fix: anchor native Duo coordinates to application windows * test: cover native coordinates across displays and orientations * fix: restrict native coordinates to windows on the selected display * refactor: use guards for native gesture coordinate routing
|
I have added 1b82530#diff-1683161d6856ce0798e3fd7c257ff81c041a906b3b5a0e14c99331195161487c about compatibility issue for old SDK built app with Duo - this came from XCTest so not WDA specific |
|
appium/appium#22857 has my testing note on simulator/real devices |
DanMPerfecto
left a comment
DanMPerfecto
left a comment
There was a problem hiding this comment.
I tested these two on the iPhone Duo simulator (iOS 27.1, Xcode 27.2 beta 2), fully open, with the app in landscape on the inner display (display 3, window 951×669) and currentDisplayId=3. Touch positions came from the IntegrationApp coordinate probe. For comparison I ran the same requests on an iPhone 18 Pro simulator (iOS 27.2) and a physical iPhone 12 Pro (iOS 26.7).
| // element may report displayID == 0. | ||
| (void)window.frame; | ||
| CGRect frame = window.wdFrame; | ||
| if (window.screen.displayID == screen.displayID && CGRectContainsPoint(frame, point)) { |
There was a problem hiding this comment.
CGRectContainsPoint doesn't count points on the right or bottom edge as inside, and any point outside every window is rejected. So application-root coordinates behave differently on a secondary display than on the main one.
On the Duo inner display, POST /wda/tap:
x=950.99: acceptedx=951(window width),y=669(window height),x=956: all fail withNo application window on display 3 contains the requested coordinate
/wda/dragfromtoforduration ending at x=951 or x=981 fails the same way. On the iPhone simulator and the physical iPhone (main display), the same edge and off-screen taps and drags are accepted.
Edge swipes that end exactly at the screen edge are common in client code. Consider an inclusive check that clamps or tolerates edge points, or fall back to the window that is nearest to the point or contains it.
There was a problem hiding this comment.
Re-tested at 1f944b5 on the Duo simulator (inner display, currentDisplayId=3, Xcode 27.2 beta 2). /wda/tap at x=951, y=669, x=981 and x=-30 is now accepted, matching the main display. Taps near the canvas edges land within 0.3 pt. Main-display behavior on the iPhone 18 Pro simulator and a physical iPhone 12 Pro is unchanged. Fixed, thanks!
| // taps even in pure XCTest without WDA; the same app built with SDK 27.2 works. | ||
| // Do not apply the viewport offset as a workaround for this separate issue. | ||
| // See docs/multiple-displays.md#legacy-sdk-compatibility-windows for evidence. | ||
| self.usesDisplayCorrection = NO; |
There was a problem hiding this comment.
This turns off the display correction for every element origin, including the XCUIApplication root. The gestureCoordinateWithOffset comment says those coordinates are rotated using the main display on Duo.
A W3C tap at the same target point on the Duo inner display:
origin: viewport: lands at canvas (206.7, 284.3), expected (206.8, 284.2) ✅origin:the canvas element: (206.7, 284.3) ✅origin:the application element, offset from the window centre: no touch delivered ❌origin:the application element, offset from the app's reported rect centre: lands at x=479.7 instead of 206.8 ❌
The error is about 273 pt, which is the inner long side minus the main display's long side (951 − 678), so the application root needs the same correction as viewport origins. Separately, the app element reports its rect as 669×951 while its window is 951×669, so a client computing the centre offset from the app rect is also off by its swapped width and height. On the iPhone simulator and the physical iPhone in landscape, all origin types land on target.
There was a problem hiding this comment.
Thank you! Actually I missed to cover the case. 5f4fa77
There was a problem hiding this comment.
Re-tested at 1f944b5 on the Duo simulator with Xcode 27.2 beta 2. With an application-element origin and the offset taken from the returned app rect, the tap now lands at canvas (206.7, 284.3), the same as viewport and canvas-element origins. A following pointer-relative move and a canvas→app drag also end on target. Checked at 180° and 90° (inner display) and 0° (outer display). No change on the main display for the iPhone 18 Pro simulator or a physical iPhone 12 Pro, in portrait and landscape. Fixed, thanks!
|
|
||
| This gesture support builds on [#1269](https://github.com/appium/WebDriverAgent/pull/1269). | ||
|
|
||
| ## Legacy SDK compatibility windows |
There was a problem hiding this comment.
this AI generated document contains lots of nitty-gritty details, that are not really necessary and make it harder to read by humans. Make sure the document is concise and only contains useful information to the point
| } | ||
|
|
||
| if (nil != element) { | ||
| // Application origins are converted to viewport coordinates above and need |
There was a problem hiding this comment.
this comment could be shortened
There was a problem hiding this comment.
Yup, will shorten docs/comments once we confirm this doesn't have further issues to support Duo.
DanMPerfecto
left a comment
DanMPerfecto
left a comment
There was a problem hiding this comment.
LGTM. I verified the edge-coordinate and application-origin fixes on the iPhone Duo simulator (iOS 27.1) in closed, 90° and fully open poses, plus the iPhone 18 Pro simulator (iOS 27.2) and a physical iPhone 12 Pro (iOS 26.7), all with Xcode 27.2 beta 2. Touch positions were checked with the coordinate probe, and I saw no regressions on the main display.
|
🎉 This PR is included in version 16.14.2 🎉 The release is available on: Your semantic-release bot 📦🚀 |
4 participants
On Duo's inner display, element-origin W3C actions could land hundreds of points away from viewport-origin actions, and scrolling could drag outside the app. Ordinary iPhones also used portrait visible rectangles while scrolling in landscape. This change aligns gesture coordinates with the selected display and the application's orientation.
NOandNSErrorfrom scroll-to-visible when display lookup fails. Directional void APIs retain their explicit exception behavior.Targets
master, incorporating the gesture work from #1269 and the merged #1291/#1292 geometry and native-coordinate changes. Also corrects portrait screen metadata on physical iPads whose native screen bounds are landscape: a portrait iPad mini previously reported 1133 × 744 points and no status bar; it now reports 744 × 1133 points and its 32-point status bar.Validation
Xcode 27.2 beta 2, iPhone 18 Pro / iOS 27.2: the focused gesture/action suite passes, including both landscape directions, portrait scrolling, unavailable-display errors, and default-main-display coordinate parity. The fold-only test is skipped on this ordinary phone.
watchOS simulator runner build-for-testing passes with Xcode 27.2 beta 2.
Earlier Duo validation: 54/54 HTTP W3C touch cases and 9/9 direction-scroll/native-swipe/W3C-scroll checks passed across closed, book, and open poses.
Combined validation with fix: report geometry for the selected display #1291/fix: anchor native Duo coordinates to application windows #1292 covers iPhone 17 on iOS 26.5 and 27.0, iPhone 18 Pro on 27.2, iPhone SE (2nd generation) on 26.5, iPad A16 on iPadOS 26.5/27.0/27.2, and Duo on 27.1. Final applicable gesture, error, default-selection, and geometry cases pass; hardware-specific cases are skipped on other devices. Native landscape cases are checked independently from a settled portrait state, avoiding an existing UIKit race during an immediate 180-degree flip.
Latest validation:
docs/multiple-displays.md.No physical Duo is available. Arbitrary rotations during an active gesture are not supported. Physical Safari iframe validation is separate: the final driver #3015 code now passes portrait → landscape → portrait iframe taps on both iPad and SE.
Application-origin review follow-up
The HTTP reproduction used an explicit app bundle ID and a cached application element. Before the fix, viewport and canvas-element origins delivered the expected point, but the application origin landed 273 points to the right on Duo’s open inner display. After the fix, all three origins deliver the same point within one point of tolerance. The new regression tests resolve serialized element IDs through the HTTP API’s element cache; raw XCTest element objects could hide this discrepancy.
The added fold-state test fails on the original implementation with the 273-point mismatch and passes with the fix through closed → open → book → closed.
Ordinary-device simulator regression: 28/28 tests pass across iPhone and iPad on OS 26.5, 27.0, and 27.2, plus iPhone SE on 26.5. Coverage includes portrait, both landscape directions, mixed origins within one touch, and pointer-relative continuation.
This follow-up was validated on simulators; no physical Duo is available. The earlier physical-device results above are separate.
Native edge-coordinate review follow-up
Native tap/drag coordinates now use a non-empty window on the selected display only as a coordinate anchor. The previous containment check incorrectly rejected right/bottom edge points and off-window drag endpoints on Duo while accepting them on main-display devices. Removing that check preserves the requested coordinates without introducing nearest-window selection, clamping, or additional coordinate corrections. Missing windows on the selected display still produce an error.
W3C actions set the display ID on their synthesized event record; native XCUICoordinate operations obtain display affinity from the anchor element. The native path therefore retains its existing window anchor instead of applying the W3C numeric correction to an application anchor.
Full regression recheck at
1f944b57Rechecked the PR's earlier fixes after the application-origin and native edge-coordinate changes. No additional regression was found; no further code change was needed.
Integration total: 326 passed, 58 skipped, 0 failed. Coordinate/scroll math unit tests: 16 passed, 0 failed. Skips are capability-specific (ordinary-device vs foldable cases, pressure support) or the existing disabled long-press test; they are not counted as passes. Production and regression source files in the integrated test checkout were compared with this PR before execution.
This is new simulator coverage only. It does not replace the earlier physical-iPad portrait-geometry validation or claim a fix for the documented legacy-SDK XCTest compatibility limitation.
CI is tracked separately: the tvOS 18.5 functional job failed before test execution while installing its Runner (
Missing bundle ID) and was rerun once. This does not count as a successful CI run.