Skip to content

feat(stovepipe): [DRAFT] cool down admissions after failures - #709

Draft
mnoah1 wants to merge 1 commit into
mnoah1/stovepipe-admission-config-v2from
mnoah1/stovepipe-failure-cooldown
Draft

mnoah1 wants to merge 1 commit into
mnoah1/stovepipe-admission-config-v2from
mnoah1/stovepipe-failure-cooldown

Conversation

@mnoah1

@mnoah1 mnoah1 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • Avoid immediately consuming more build resources after a runner-reported failure.

Changes:

  • Persist the first terminal observation time on Build.
  • Advance the admission deadline while releasing the failed build slot in one queue CAS.
  • Add per-queue failure cooldown configuration to the YAML policy store.
  • Keep success, cancellation, and DLQ-forced failure outside the cooldown policy.

This change builds on the per-queue admission configuration introduced by the parent PR.

Test Plan

  • go test ./stovepipe/... ./service/stovepipe/server
  • ./tool/bazel test //stovepipe/... //service/stovepipe/server:go_default_test

Revert Plan

Revert this PR to disable failure-specific admission cooldowns while retaining general throttling.

Issues

@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-admission-throttle branch from fd1f6ce to fe94134 Compare September 15, 2026 17:37
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-cooldown branch from 9d96d6f to 8bff754 Compare September 15, 2026 17:37
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-admission-throttle branch from fe94134 to 6514bbe Compare September 15, 2026 17:51
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-cooldown branch from 8bff754 to 2e13236 Compare September 15, 2026 17:51
@mnoah1
mnoah1 added this pull request to stack #712 September 15, 2026 20:54
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-cooldown branch from 2e13236 to a4407f5 Compare September 15, 2026 20:59
@mnoah1
mnoah1 removed this pull request from stack #712 September 15, 2026 21:02
@mnoah1
mnoah1 changed the base branch from mnoah1/stovepipe-admission-throttle to mnoah1/stovepipe-admission-config-v2 September 15, 2026 21:03
@mnoah1
mnoah1 added this pull request to stack #715 September 15, 2026 21:03
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-cooldown branch from a4407f5 to d681ca1 Compare September 15, 2026 21:10
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-admission-config-v2 branch from aa3aa8b to c6cb600 Compare September 15, 2026 21:10
@mnoah1 mnoah1 changed the title feat(stovepipe): cool down admissions after failures feat(stovepipe): [DRAFT] cool down admissions after failures Sep 15, 2026
@mnoah1
mnoah1 removed this pull request from stack #715 September 15, 2026 21:20
Summary:
Intent:
- Avoid immediately consuming more build resources after a runner-reported failure.

Changes:
- Persist the first terminal observation time on Build.
- Advance the admission deadline while releasing the failed build slot in one queue CAS.
- Add per-queue failure cooldown configuration to the YAML policy store.
- Keep success, cancellation, and DLQ-forced failure outside the cooldown policy.

This change builds on the per-queue admission configuration introduced by the parent PR.
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-cooldown branch from d681ca1 to 7d99424 Compare September 15, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant