fix(ai-red-teaming): resolve bare task names across accessible orgs + steer off the CLI (ENG-8434) - #142
Merged
Conversation
… steer off the CLI (ENG-8434) Weaker models shell out to 'dreadnode env' and guess syntax, then hit a 404 because a bare task name resolves only within the caller's org. provision_environment now resolves the owning org on that 404 and retries qualified as <org>/<name>: - _resolve_task_owner: one list_tasks(caller_org, search, include_public=True) call covers the common case (public 'dreadnode/' catalog + caller org); only if that misses does it fan out to the user's other accessible orgs (covering a PRIVATE task in a member org), stopping at the first exact-name match. Ambiguous names prefer the public catalog, else bail so the caller qualifies explicitly. - Agent prompt: never shell out to 'dreadnode env'; always use provision_environment; bare public/bundled names work directly. Bumps capability 1.14.0 -> 1.15.0. 5 resolver tests (public hit / private-member fan-out / not-found / ambiguous / exact-name-only); full env suite 32 passed.
…ardrails (ENG-8434) Prod analysis of the aisf-learner-aug-2026 org (48h) showed the friction is concentrated in the provision -> run -> teardown lifecycle: 76% of provisions used bare task names (the ENG-8434 404), env-status was polled after teardown (top 4xx class), and ~30% of sandboxes were never explicitly torn down. The tool layer already resolves refs, picks endpoints, retries transients, and auto-tears-down; the gap is agent KNOWLEDGE, so this adds a loadable playbook rather than a new tool. - New skills/provisioning-and-lifecycle/SKILL.md: provisioning decision tree, bare-name resolution (never self-org-qualify a bundled task), endpoint-per-target map, teardown/billing, transient-vs-fatal interpretation (incl. 'a 404 on env-status = gone, stop polling'), ASR display, attribution. - error-troubleshooting: added Provisioning & Sandbox Errors subsections (task-404, env-status-404, endpoint mismatch, billing, Note: vs Error:). - agent md: register the skill in the lazy-load hint. - environments.py: teardown_note now states teardown fires once every planned attack is recorded (pass or fail) and to reap explicitly on abandon; surfaces AIRT_ENV_TEARDOWN_GRACE_SEC. Env test suite 32 passed.
…ill (ENG-8434) Parity with the CLI simplification on tiger #2452, after review + prod evidence showed a client-side task resolver cannot fix a real 404. The backend visibility rule already resolves a bare name to a task in the caller's org OR any public task, and the cross-org guard blocks a private task even when qualified as <org>/<name>. So the retry was dead code: a public task never 404s bare, and a private cross-org task 404s identically on the retry. Prod confirms bare-name env creates already succeed (aisf org: mnist 16, fraud 16, imdb 13, no errors). - Remove _resolve_task_owner (it was also wrongly wrapped in @safe_tool). - provision_environment no longer retries; on a task-not-found it raises a clear hint (bare = your org or any public task; else qualify as <org>/<name>). - Correct the provisioning-and-lifecycle skill Section 2 and the agent prompt so they no longer claim the tool resolves/retries internally. Tests: dropped 5 resolver unit tests; env suite 27 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (ENG-8434, reporter: Corch X)
With full traces on, the agent (weaker model, deepseek-v4-flash) shells out to the
dreadnode envCLI to provision and guesses syntax —dreadnode env provision …(no such verb) thendreadnode env create ml-extraction-fraud-tabular→ 404. Root cause: a bare task name resolves only within the caller's org, but bundled targets are owned by another org (dreadnode/public catalog), so from a learner workspace (aisf-learner-aug-2026) it 404s.Fix (capability — model-agnostic prevention)
provision_environmentnow resolves the owning org on a bare-name 404 and retries qualified as<org>/<name>, via_resolve_task_owner:list_tasks(caller_org, search=name, include_public=True)call covers the common case (public catalog + caller org) in a single request.list_user_organizations), stopping at the first exact-name match — so a private task in an org the user belongs to also resolves. (Per reviewer/Corch note about private, cross-org targets.)dreadnode env; always useprovision_environment; bare public/bundled names work directly.Bumps capability 1.14.0 → 1.15.0.
Tests
5 resolver tests (public hit / private-member fan-out / not-found / ambiguous→public / exact-name-only); full env suite 32 passed.
Companion PR
The
dreadnode envCLI itself gets theprovisionalias + the same public fallback in tiger #2452 (that fixes the exact CLI path in the report; this makes the agent prefer the tool and resolves across the user's accessible orgs).