ci: benchmark every pull request against its base branch - #3775
Open
nigrosimone wants to merge 2 commits into
Open
nigrosimone wants to merge 2 commits into
nigrosimone wants to merge 2 commits into
Conversation
nigrosimone
marked this pull request as ready for review
September 13, 2026 09:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs an A/B benchmark on every pull request: the pg, pg-pool, pg-cursor and pg-query-stream of the PR against the same packages of the base branch, on the same runner in the same job, over the unix socket of the PostgreSQL that comes with the runner image.
Each arm runs in two processes, so every round also measures base against base and head against head. The same code on both sides gives the noise of that run on that machine, and a head/base ratio is marked only when it moved further than that noise, in every round.
The table goes to the job summary and to a comment on the PR. Only the ratio is comparable across runs, the absolute queries per second depend on the runner.
This is an example of the comment published on PRs
Benchmark:
cbb85dbagainst96830534 rounds of 3s per scenario, each round measuring base, head and a second process of each one after the other, in an order that changes every round. "Head / base" is the median of the per-round ratios and "Rounds" their range. "Noise" is how far base/base and head/head, the same code on both sides, got from 1 in this same run: that is what the machine did, so a row is marked only when the median moved further than that, at least 2%, and every round moved the same way: 👀 slower, 🏆 faster. Only the ratio is comparable across runs, the absolute q/s depend on the runner.
Node v26.8.2, AMD EPYC 7763 64-Core Processor, 4 cores.