feat(browser): sharper computer use for the desktop browser agent - #8283
Conversation
A chord such as Control+Shift+Y was sent as one key-down with a modifier bitmask, so pages that track held keys never saw Control or Shift go down. Modifiers are now pressed in order before the main key and released in reverse, as a physical keyboard delivers them, and a bare modifier key can be pressed on its own.
Clicks released the button immediately, so press-and-hold controls could not be operated. A click now accepts holdMs (0 to 10000, single clicks only) and keeps the button down that long before release.
MUI hides every <body> child except a modal's mount node; a disablePortal modal mounts inside the app root it just hid, so every node had an aria-hidden ancestor and the snapshot came back empty while the dialog was on screen. Visibility checks now skip aria-hidden only on ancestors above the single topmost modal hidden that way; portaled modals and other hidden regions are unchanged.
…pens a Save dialog The save path was set only after an asynchronous non-conflicting-name lookup. When Electron's own path step won that race it opened a native Save dialog, so an agent download stalled forever at full size (and a user would see a surprise dialog). Downloads now write to a hidden staging file set during will-download and are renamed to the allocated name when they complete.
…s snapshot After an action the agent saw a fresh snapshot but no hint of what changed, so a popup, suggestion list, or validation message looked like any other line. The page now remembers which elements an earlier snapshot of the document listed, and later snapshots mark the rest new (the first snapshot marks nothing).
- Modifier aliases (Ctrl, Cmd, Command, Option) share one descriptor with their canonical key, so a bare alias sets its own flag; a bare modifier's key-up reports it released. - The framed-control fallback no longer treats a held click as a plain click, and a batch refuses press-and-hold so eight holds cannot outlast its watchdog. - installPageHelpers caches the modal lookup per invocation, and one serializePageCall builds the page expression for the driver and the tests. - Download completion drops a branch whose state was never published.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
github-actions
Bot
added
the
requires-mothership-merge
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…changes - A held right-click renews its agent context-menu marker before release, so Windows' release-time menu stays suppressed after a hold over one second. - A failed chord releases only the keys whose press was attempted. - A disk probe that resolves after a download completed can no longer cancel it. - A disablePortal modal nested inside another open modal is picked as topmost. - holdMs is an integer bounded to 0-10000 in the tool contract.
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 13 files
Confidence score: 5/5
- In
apps/desktop/src/main/browser-agent/page-functions.ts, standalone headings and landmarks can bypass the marker, so newly rendered sections may not be identified as new; apply the marker and record those elements in their emission paths.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/desktop/src/main/browser-agent/page-functions.ts">
<violation number="1" location="apps/desktop/src/main/browser-agent/page-functions.ts:581">
P3: Standalone heading and landmark lines bypass this marker, so newly rendered headings or sections are not identified as new. Apply the marker and record those elements in their emission paths too.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
- An aborted press-and-hold rejects and releases the button at once instead of staying held into the next action. - The disablePortal exemption applies only when a <body> child hides the modal (MUI's mechanism), so a dialog the app hid itself stays hidden, and it is looked up per document so same-origin iframes get it too. - A completed download's move retries transient EBUSY/EPERM/EACCES errors, as Chromium's own final rename does, instead of discarding the file. - Keyboard docs describe the separate modifier presses; the partial-release test covers a failure at the first modifier.
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.
Fix all with cubic | Re-trigger cubic
The allocated destination gets an empty placeholder created with O_EXCL, as Firefox does, so another program picking a name sees it taken and the final rename only ever replaces Sim's own placeholder. A name that something else grabbed first stops the download instead of being overwritten. Teardown and failures remove the placeholder once, so a late-settling download cannot delete a name a newer download has claimed.
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…loads - An already-aborted click presses nothing, and a held click marks its outcome pending before dispatch, so cancelling mid-hold reports an unknown outcome with doNotRetry instead of an error that invites a retry. - The disablePortal exemption resets when a visibility walk crosses from an iframe into its host page, where the host's own aria-hidden applies. - Teardown keeps a download's name reserved until its placeholder claim settles, so a newer download cannot be handed a name the claim then takes. - Downloads are paused before the staging path is set, as on staging, so a pause failure leaves no staging file behind.
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 13 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 13 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
This branch was previously deployed
1 participant
Summary
browser_clickacceptsholdMs(0–10,000 ms, single click only) for press-and-hold controls; batches refuse holds so they can't outlast the batch watchdogdisablePortalmodals whose app root isaria-hidden: the open modal is exempt from the hidden-ancestor walknew, so the agent can see what an action changed without diffingserializePageCall+installPageHelperspath for the driver and testsType of Change
Testing
check:audits, and docs manifest all passCompanion: simstudioai/mothership#524
Checklist