Conversation
`Allocate::broadcast_peers` allocates a broadcast channel that delivers to every worker except the sender, while the sender's puller still receives what other workers push. The default implementation drops the sender's own pusher from `allocate`, and the zero-copy allocators share their existing broadcast paths, so remote processes and local peers still receive one serialization. `Pull::quiet` stops a puller from recording an event for its channel once it has been drained. By default, a drained channel that yielded messages schedules its recipient once more, which a recipient that finishes its work in the draining call does not need. The default implementation does nothing, and the counting and zero-copy pullers forward it to the pullers they wrap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A subgraph collects the progress its child scopes report while scheduling them, and previously applied it only on its next activation, which it requested for itself. A change deep in nested scopes therefore climbed one level per worker step, and each step scheduled every enclosing scope again. The subgraph now propagates these updates before it returns, so that the effect on its outputs reaches the parent in the same call. Children whose frontiers change as a result still run on the next activation, which the subgraph requests when any remain. Propagating early can drain the tracker while such children have yet to observe their new frontiers. A subgraph with pending children now reports itself incomplete, so that the worker does not drop the dataflow before a probe sees its final frontier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Progcaster` broadcast each batch of progress updates to all workers, including the sender. The sender's copy went through its own channel, and on arrival activated the scope and every scope around it on a later step. The progress channel now excludes the sender, and `Progcaster::send` adds the sender's copy to the caller's batch of received updates. Correctness needs each worker's updates to reach every worker in the order sent, and batches applied locally keep that order. `Subgraph::schedule` propagates the copy before it returns, together with the progress of its child scopes. Progress logging records the local delivery as a receive, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A counting puller records an event for its channel once it has been drained of messages, so that the channel's recipient runs once more. For progress channels the recipient is `Subgraph::schedule`, which applies everything it receives in the same call and activates itself when work remains. The extra activation scheduled the scope and all of its enclosing scopes to find nothing to do. The progress puller is now quiet. Data channels keep the event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Applying progress before a subgraph returns lets its effect reach the parent in the same call. A dataflow root has no parent, so propagating its own sent updates early only splits one propagation per step into two, one for its own updates and one for those that arrive from other workers. In a loop at the root, `barrier` with four workers took 8% longer than before this series. A dataflow root now leaves the updates for its next activation, which it requests, and applies them together with those it receives by then. Nested scopes still propagate before they return. The root counts unapplied updates as pending work when it reports whether it is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With nested regions, most scope schedules come from progress bookkeeping rather than operator work. Scheduling a scope also schedules every scope that contains it, so extra progress activations cost once per nesting level. This PR removes three sources of such activations:
Allocate::broadcast_peers.Progcaster::sendtakes the batch to add the local copy to.Pull::quiet). Data channels keep this behavior.Against master, medians of six runs on an x86 machine with 32 cores:
event_driven 100 10completes 20% more rounds per second with one worker, and 4% fewer with sixteen.barriertakes 8% longer with four workers, and is unchanged with one and sixteen.The
barrierregression comes mostly from applying progress early in the dataflow root, which has no parent to report to. Limiting it to nested scopes removes most of that regression but gives up part of the gain with four workers.Posted by Claude Code.
🤖 Generated with Claude Code