fix: capture child and descriptor stdout in JSON envelopes - #420
Conversation
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
…or-discovered-project-confi' into bug/379-20261003-bug-json-single-envelope-stdout-contract-is-broken-by-any-wr
codeforester
left a comment
There was a problem hiding this comment.
Reviewed against #379's acceptance criteria. Replacing redirect_stdout with a real fd-1 redirect plus a draining thread is the right fix, and the overflow path keeps draining so children don't deadlock. Docs name the subprocess case, and the regression test runs a real child. CI is green.
I found one verified behavioral regression, inline. It also conflicts with the issue's non-goal "do not silently ... discard child-process output", since the command's own output is dropped. There is also one design note.
| os.close(saved) | ||
| if write_fd != -1: | ||
| os.close(write_fd) | ||
| worker.join(timeout=2) |
There was a problem hiding this comment.
Regression (verified): successful commands become failures. Any child still holding the inherited stdout when the command returns turns the run into an error, even if that child writes nothing. Repro: a command does subprocess.Popen(["sleep", "5"]) (a deliberately detached helper, which is common for agents, watchers and servers) and then prints work done.
| exit | time | envelope | |
|---|---|---|---|
| main | 0 | ~1s | success, stdout: "work done\n" |
| this PR | 1 | +2s | error, code: "capture_limit", stdout: "" |
There are three problems here:
- The side effects have already happened, but the exit code now reports failure, so automation will retry non-idempotent work.
- The command's own captured output is discarded, even though it's sitting in the spool.
capture_limitis the wrong code; no limit was exceeded. A distinct code (e.g.capture_incomplete) would make this diagnosable.
Suggestion: after the bounded wait, emit the envelope with what was captured, plus a warning or a details.capture_incomplete: true flag, rather than failing the command. Or make the strict behavior opt-in. Either way, document that children should use stdout=subprocess.DEVNULL / start_new_session when detaching. Please also add a test for the detached-child case.
There was a problem hiding this comment.
Fixed in 2a0a9bb and 3b4fc3f: a child retaining stdout now returns capture_incomplete, preserves output drained before the timeout in the single error envelope, and documents DEVNULL/start_new_session guidance. The detached-child regression test passes.
| writer: TextIO | None = None | ||
| try: | ||
| worker.start() | ||
| os.dup2(write_fd, 1) |
There was a problem hiding this comment.
Design note: dup2 on fd 1 is process-global. For the duration of a JSON invocation, fd-1 writes from any thread are pulled into this command's envelope, which affects consumers embedding run_app in a long-lived multi-threaded host (a worker pool, a server, an in-process test runner). They also become subject to the 2s abandonment rule above. The new docs say "the invocation owns the process output boundary". Please state explicitly that JSON-mode run_app is not safe to call concurrently from multiple threads, or guard it with a module-level lock that fails fast on re-entry from another thread.
There was a problem hiding this comment.
Documented in 3b4fc3f: fd 1 capture is process-wide and run_app() already rejects concurrent same-process invocations; parallel CLI work must use separate processes or serialize calls.
JSON invocations now capture process descriptor 1 as well as Python stdout, so inherited subprocess output remains inside the single envelope. A concurrent drain avoids pipe deadlocks, the native path enforces the JSON capture limit, and descriptor restoration precedes envelope emission.
The guides state the boundaries: wait for children, flush native stdio before returning, and use NDJSON for large output. A child retaining stdout causes a bounded capture error.
Fixes #379.
Branch maintenance
Refs #426. Targets the branch for #419. Retarget and refresh after that parent is squash-merged; preserve the ordered stack.
The branch was refreshed without rewriting history to include
mainata576cc279739eae5e4cfc33ffab2a7fb56de24de.Current-head validation
At
0441cec3b3098425d8c87aa9c8eaeee6e5496969: uv lock freshness and baseline, runtime, strict typing, style, and contracts passed locally with all declared extras. Runtime result: 629 passed, 1 warning, 262 subtests passed in 9.37s.Hosted checks: 7/7 required checks passed; 0 checks pending; 0 unsuccessful checks at 2026-10-04T14:19:40.760978+00:00. See the PR Checks tab and #426 for subsequent results.