Skip to content

ci: benchmark every pull request against its base branch - #3775

Open
nigrosimone wants to merge 2 commits into
brianc:masterfrom
nigrosimone:benchmark-ci
Open

nigrosimone wants to merge 2 commits into
brianc:masterfrom
nigrosimone:benchmark-ci

Conversation

@nigrosimone

@nigrosimone nigrosimone commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Runs an A/B benchmark on every pull request: the pg, pg-pool, pg-cursor and pg-query-stream of the PR against the same packages of the base branch, on the same runner in the same job, over the unix socket of the PostgreSQL that comes with the runner image.

Each arm runs in two processes, so every round also measures base against base and head against head. The same code on both sides gives the noise of that run on that machine, and a head/base ratio is marked only when it moved further than that noise, in every round.

The table goes to the job summary and to a comment on the PR. Only the ratio is comparable across runs, the absolute queries per second depend on the runner.

This is an example of the comment published on PRs


Benchmark: cbb85db against 9683053

Scenario Base q/s Head q/s Head / base Rounds Noise
select 1 row, all types, prepared 7635 7760 1.025x (+2.5%) 1.00 to 1.03 ±6.1%
select 100 rows, all types, prepared 1139 1138 0.997x (-0.3%) 0.99 to 1.01 ±5.7%
select 500 rows, all types, prepared 278 276 0.987x (-1.3%) 0.98 to 1.00 ±3.0%
select 500 rows, simple types, prepared 1602 1596 0.993x (-0.7%) 0.99 to 1.02 ±3.8%
select 500 rows, simple types, array mode 1481 1495 1.012x (+1.2%) 0.99 to 1.02 ±1.9%
select 500 rows, simple types, binary 826 818 0.989x (-1.1%) 0.99 to 1.02 ±2.7%
select by id, parameterized 5518 5457 1.006x (+0.6%) 0.94 to 1.04 ±3.9%
insert, all types, returning 4214 4233 1.005x (+0.5%) 0.96 to 1.04 ±5.3%
update, parameterized 6179 6069 0.974x (-2.6%) 0.95 to 1.01 ±5.3%
delete, parameterized 7214 7251 0.990x (-1.0%) 0.97 to 1.05 ±3.4%
transaction, 3 queries 11974 12210 1.025x (+2.5%) 0.92 to 1.05 ±7.1%
pool query, select 1 row 6938 7032 1.006x (+0.6%) 0.99 to 1.03 ±7.2%
cursor, 500 rows in 5 reads 211 211 1.000x (+0.0%) 0.99 to 1.04 ±3.8%
query stream, 500 rows 213 213 0.995x (-0.5%) 0.99 to 1.01 ±1.8%
select 100 rows, all types, pipelined x10 1175 1193 1.010x (+1.0%) 0.99 to 1.02 ±2.8%

4 rounds of 3s per scenario, each round measuring base, head and a second process of each one after the other, in an order that changes every round. "Head / base" is the median of the per-round ratios and "Rounds" their range. "Noise" is how far base/base and head/head, the same code on both sides, got from 1 in this same run: that is what the machine did, so a row is marked only when the median moved further than that, at least 2%, and every round moved the same way: 👀 slower, 🏆 faster. Only the ratio is comparable across runs, the absolute q/s depend on the runner.

Node v26.8.2, AMD EPYC 7763 64-Core Processor, 4 cores.

@nigrosimone
nigrosimone marked this pull request as ready for review September 13, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant