Skip to content

Back off GitHub server errors before exhausting job retries - #1508

Merged
gulkaran merged 1 commit into
mainfrom
gulkaran/backoff-github-server-errors
Oct 13, 2026
Merged

gulkaran merged 1 commit into
mainfrom
gulkaran/backoff-github-server-errors

Conversation

@gulkaran

@gulkaran gulkaran commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GitHub server errors currently exhaust BackgroundJob's default five executions in roughly 12 seconds, so longer outages spill into the queue adapter's retries and repeatedly report errors. Use Active Job's built-in polynomial backoff with ten total attempts: about 4.3 hours plus jitter before re-raising.

The shared policy covers deployment creation, deployment statuses, and check-run refreshes. Exhausted retries still raise to the queue adapter and existing error reporter; rate-limit and authentication handling retain their existing behavior.

Tests exercise all ten attempts through serialized jobs, recovery of a failed deployment status without recreating published records, check-run backoff, and failure propagation from CreateOnGithubJob.

Validation: reproduced the short-delay regression before the fix; 1,254 tests / 3,575 assertions pass with SQLite and Ruby 3.2.0 (bundle exec rake test); repository-wide bundle exec rubocop passes (328 files).

Operational tradeoff: persistent GitHub 5xx failures take longer to reach the error reporter. Production adoption requires releasing the engine, updating Shipit's dependency, and deploying Shipit.

Related: https://github.com/shop/issues/issues/94251

@gulkaran
gulkaran merged commit f69b3cc into main Oct 13, 2026
15 checks passed
@gulkaran
gulkaran deleted the gulkaran/backoff-github-server-errors branch October 13, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants