Skip to content

fix: capture child and descriptor stdout in JSON envelopes - #420

Open
codeforester wants to merge 11 commits into
security/385-20261003-security-validate-trust-of-ancestor-discovered-project-confifrom
bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
Open

codeforester wants to merge 11 commits into
security/385-20261003-security-validate-trust-of-ancestor-discovered-project-confifrom
bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr

Conversation

@codeforester

@codeforester codeforester commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

JSON invocations now capture process descriptor 1 as well as Python stdout, so inherited subprocess output remains inside the single envelope. A concurrent drain avoids pipe deadlocks, the native path enforces the JSON capture limit, and descriptor restoration precedes envelope emission.

The guides state the boundaries: wait for children, flush native stdio before returning, and use NDJSON for large output. A child retaining stdout causes a bounded capture error.

Fixes #379.

Branch maintenance

Refs #426. Targets the branch for #419. Retarget and refresh after that parent is squash-merged; preserve the ordered stack.

The branch was refreshed without rewriting history to include main at a576cc279739eae5e4cfc33ffab2a7fb56de24de.

Current-head validation

At 0441cec3b3098425d8c87aa9c8eaeee6e5496969: uv lock freshness and baseline, runtime, strict typing, style, and contracts passed locally with all declared extras. Runtime result: 629 passed, 1 warning, 262 subtests passed in 9.37s.

Hosted checks: 7/7 required checks passed; 0 checks pending; 0 unsuccessful checks at 2026-10-04T14:19:40.760978+00:00. See the PR Checks tab and #426 for subsequent results.

…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr

@codeforester codeforester left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against #379's acceptance criteria. Replacing redirect_stdout with a real fd-1 redirect plus a draining thread is the right fix, and the overflow path keeps draining so children don't deadlock. Docs name the subprocess case, and the regression test runs a real child. CI is green.

I found one verified behavioral regression, inline. It also conflicts with the issue's non-goal "do not silently ... discard child-process output", since the command's own output is dropped. There is also one design note.

os.close(saved)
if write_fd != -1:
os.close(write_fd)
worker.join(timeout=2)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression (verified): successful commands become failures. Any child still holding the inherited stdout when the command returns turns the run into an error, even if that child writes nothing. Repro: a command does subprocess.Popen(["sleep", "5"]) (a deliberately detached helper, which is common for agents, watchers and servers) and then prints work done.

exit time envelope
main 0 ~1s success, stdout: "work done\n"
this PR 1 +2s error, code: "capture_limit", stdout: ""

There are three problems here:

  1. The side effects have already happened, but the exit code now reports failure, so automation will retry non-idempotent work.
  2. The command's own captured output is discarded, even though it's sitting in the spool.
  3. capture_limit is the wrong code; no limit was exceeded. A distinct code (e.g. capture_incomplete) would make this diagnosable.

Suggestion: after the bounded wait, emit the envelope with what was captured, plus a warning or a details.capture_incomplete: true flag, rather than failing the command. Or make the strict behavior opt-in. Either way, document that children should use stdout=subprocess.DEVNULL / start_new_session when detaching. Please also add a test for the detached-child case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2a0a9bb and 3b4fc3f: a child retaining stdout now returns capture_incomplete, preserves output drained before the timeout in the single error envelope, and documents DEVNULL/start_new_session guidance. The detached-child regression test passes.

writer: TextIO | None = None
try:
worker.start()
os.dup2(write_fd, 1)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design note: dup2 on fd 1 is process-global. For the duration of a JSON invocation, fd-1 writes from any thread are pulled into this command's envelope, which affects consumers embedding run_app in a long-lived multi-threaded host (a worker pool, a server, an in-process test runner). They also become subject to the 2s abandonment rule above. The new docs say "the invocation owns the process output boundary". Please state explicitly that JSON-mode run_app is not safe to call concurrently from multiple threads, or guard it with a module-level lock that fails fast on re-entry from another thread.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in 3b4fc3f: fd 1 capture is process-wide and run_app() already rejects concurrent same-process invocations; parallel CLI work must use separate processes or serialize calls.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant