Skip to content

feat(studio): AI chat context is opt-in (last run, widget tree), redacted; backend for a selector fix - #467

Merged
pbertsch merged 5 commits into
mainfrom
feat/studio-ai-selector-fix
Oct 15, 2026
Merged

pbertsch merged 5 commits into
mainfrom
feat/studio-ai-selector-fix

Conversation

@pbertsch

@pbertsch pbertsch commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Summary

  • The Studio AI chat sends the open file with every message, as before. The failures of the last run and the device's widget tree are sent only when the workspace turns them on. Both switches are off by default. They live in probe.yaml under studio.ai_context (run_results, widget_tree) and in the Workspace settings overlay, under AI chat context.
  • The Go side reads the switches from the workspace's probe.yaml for every chat turn, and it fails closed. With no workspace, an unreadable file, or a switch that is off, nothing beyond the open file is sent, and the device is not asked for its tree.
  • While either switch is on, a line above the chat input says what is sent: "Sending the last run and the screen to the AI provider", or the one-part variant.
  • Redaction happens before anything is sent. An editable field that is obscured, or whose key names a secret (password, PIN, token, OTP, CVV, API key, credential), loses its key and is marked. Credential-looking text (sk-…, ghp_…, Bearer values, long hex and base64 runs) becomes [redacted] in the tree and in the run text. Each part is cut to its size limit and marked when it is cut.
  • SuggestSelectorFix (backend only) asks for one ProbeScript line that replaces the last failed step. It needs the run-results switch and refuses without it. Nothing is applied. The pane button, and the regenerated Wails bindings for it, are a follow-up.
  • The Workspace settings overlay now reads and writes the workspace root's probe.yaml, not the folder currently browsed in the file pane. The toggles and the chat line therefore use the same file as the Go side. The fields the overlay already managed are unchanged.

What the widget tree contains today (please review)

The agent's probe.dump_tree returns widget types and ValueKeys. It does not return the text typed into fields, so typed text is not in the tree today. The agent writes no "obscured" marker either, so the obscured rule has nothing to match yet; the secret-name and credential rules do apply. Writing an obscured marker into the agent's dump is the follow-up that would make the obscured rule effective.

Test plan

  • go test ./studio/ on the whole package, merged with origin/main, with the real frontend build embedded. The new tests cover: off by default (no probe.yaml, no key, no workspace); neither part is sent and the device is not asked when both are off; each switch sends only its own part; both on sends both, redacted; no secret reaches the prompt; SuggestSelectorFix needs the run switch; redaction of common credential shapes; redaction of obscured and secret-named editable fields; probe.yaml save and load keep the switches and the rest of the file, and switching both off removes the section; a studio key that is not a map is refused and the file is left alone
  • go vet ./studio/, and gofmt for the new Go files
  • npm run build (tsc and vite build) on the merged tree
  • Not run against the live Anthropic API. The tests use a fake provider and make no network calls
  • Not run in the Studio window: the settings overlay and the chat line were not exercised by hand

🤖 Generated with Claude Code

…gestSelectorFix returns one ProbeScript line

The chat's system prompt now carries the failures of the last run (tests and steps, with their lines and errors) and, when a device is connected, its widget tree, each cut to a size limit and marked when cut. The key still comes only from the keychain. SuggestSelectorFix asks for one ProbeScript line that replaces the last failed step; nothing is applied. The Go tests use a fake provider. The pane button is a follow-up.
@pbertsch
pbertsch requested a review from a team as a code owner October 15, 2026 15:21
…es, a visible line, redaction

The last run and the device's widget tree now reach the AI provider only when the workspace turns them on: two switches under studio.ai_context in probe.yaml, both off by default, shown in the Workspace settings overlay. While either is on, a line above the chat input says what is sent. The Go side reads the same file, fails closed when it cannot, and does not ask the device for its tree when the switch is off. The widget tree and the run text are redacted first: obscured or secret-looking editable fields lose their keys, and API keys, GitHub tokens, Bearer values and long hex or base64 runs are removed. The Workspace settings overlay now reads and writes the workspace root, not the folder being browsed. SuggestSelectorFix needs the run-results switch and returns one ProbeScript line; its button and Wails bindings follow in a separate change.
…or-fix

# Conflicts:
#	CHANGELOG.md
#	website/src/content/docs/advanced/configuration.md
@pbertsch pbertsch changed the title feat(studio): the AI chat sends the last run and the widget tree; backend for a selector fix feat(studio): AI chat context is opt-in (last run, widget tree), redacted; backend for a selector fix Oct 15, 2026
@pbertsch
pbertsch merged commit 7a28f35 into main Oct 15, 2026
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant