PRs with heads predating a required workflow are silently unmergeable

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

A PR whose head commit predates the introduction of the org-required workflow ✅ Validate Go Project
can never be enqueued, because the requirement is evaluated against check runs that only exist if the
workflow fired for that head — and it cannot fire retroactively.

Measured today, as a controlled pair:

PR	head pushed	✅ Validate Go Project runs
#2734	2026-07-19T02:36:26Z	0
#2738	2026-07-19T02:36:42Z	0
#2739	2026-07-19T02:36:55Z	0
#2742	2026-07-19T02:37:01Z	0
#2703	today	1
#2728	today	1
#2731	today (after refresh)	1

Why this is hard to see, which is the worst part

Every visible signal says the PR is fine. On #2731 before the refresh:

• all 21 check runs success/skipped, including the required CI - Required Checks
• ruleset requires 0 approving reviews
• mergeStateStatus: BLOCKED with nothing on the head explaining it
• gh pr merge exited 0 and printed nothing, and the PR was simply not queued

The only surface that names the cause is the GraphQL enqueue mutation:

enqueuePullRequest(...) →
  UNPROCESSABLE: Pull request Required workflow '✅ Validate Go Project' is not satisfied

The workflow has no repo-local file (it is injected org-side from
devantler-tech/actions/.github/workflows/validate-go-project.yaml) and does not appear in this
repository's actions/workflows list, so there is nothing in-repo to point at either.

Impact

This is the mechanical reason the mid-July codex/* cohort has sat unmerged for ~3.5 weeks. Those PRs
were read as blocked on review capacity; review is a real second problem, but even a fully reviewed one
among them could not have been enqueued. It also silently wastes review quota: a PR is reviewed, found
clean, and then refuses to merge for a reason no reviewer or check surfaces.

Smallest useful change

Refreshing the head (merge main in, or any new commit) makes the workflow fire and clears it — that
is what unblocked #2731. That is a workaround per PR, not a fix.

Worth deciding between:

• treating a head-refresh as a standard step when adopting a new required workflow, and sweeping the
  existing backlog once; and/or
• surfacing the real reason in the merge path, so a silent exit 0 non-enqueue cannot happen again —
  reading the enqueue error rather than gh pr merge's exit code.

Acceptance criteria

• every open PR whose head predates the requirement is either refreshed or explicitly closed
• the merge path reports the enqueue rejection reason instead of exiting 0 silently
• a note in the repo's AGENTS.md ## Maintenance records that mergeStateStatus: BLOCKED with a
  fully green head means an unsatisfied required workflow, and names the mutation that reveals it

Rough size: small for the detection/documentation half; the backlog sweep is proportional to the
cohort (~27 PRs).

[devantler]

🤖 Generated by the Agentic Engineer

Sweep done for the 16 non-conflicting PRs — 16/16 verified

Acceptance criterion 1, partially: of the 33 open devantler-authored PRs on this repo, 30 had no
run of .github/workflows/validate-go-project.yaml at their head. 16 of those 30 were refreshable
(mergeable_state: blocked, i.e. behind but not conflicting) and have been refreshed via
PUT /pulls/{n}/update-branch — a merge of main, not a force-push:

#2707 #2708 #2709 #2710 #2712 #2713 #2714 #2721 #2725 #2729 #2732 #2735 #2737 #2739 #2741 #2743

Verified by effect, not by the API's acceptance: head moved 16/16, and the required workflow
now has a run at the new head 16/16 (it had 0/16 before). The mechanical blocker is cleared for
all sixteen; whether they merge now depends on CI and review, which is the separate second problem.

A measurement trap worth recording

My first pass measured this by filtering check-run names for the workflow title and reported
0 for every PR — including the three that were known-good. The workflow's jobs surface as check
runs under their own names, so the workflow title never appears there. Diagnose by run:

gh api "repos/devantler-tech/platform/actions/runs?head_sha=<head>&per_page=100" \
  --jq '[.workflow_runs[]|select(.path==".github/workflows/validate-go-project.yaml")]|length'

The unfiltered control is what caught it — a filtered empty result is a claim about the filter.

Remaining

• 14 PRs are mergeable_state: dirty and cannot be refreshed this way; each needs its conflict
  resolved first. Listed for the next pass:
  #2706 #2715 #2717 #2718 #2719 #2720 #2723 #2734 #2738 #2740 #2742 #2745 #2866 #2902
• Criteria 2 and 3 (surface the enqueue rejection reason; record the diagnosis in `AGENTS.md

  Maintenance`) are in docs(agents): explain BLOCKED-on-green as an unsatisfied required workflow #3138.

• ✅ One true duplicate found and closed: fix(dex): restrict GitHub connector to maintainers #2708, superseded by fix(dex): restrict GitHub auth to maintainers #2709. Their functional change is
  byte-identical (teams: [maintainers] on the Dex GitHub connector); fix(dex): restrict GitHub auth to maintainers #2709 also fixes the
  crossview and oauth2-proxy comments that this change falsifies. The other two pairs I flagged
  are not duplicates on inspection — fix(headlamp): keep plugins on ephemeral storage #2739 (drop the Headlamp PVC) and fix(headlamp): disable unverified plugin downloads #2741 (disable unverified
  plugin downloads) are distinct fixes that merely touch the same chart, and fix(security): close alternate tenant route-kind hostname bypass #2713 (a general tenant
  HTTPRoute-hostname policy) versus fix(security): restrict doggy-countdown route hostname #2866 (a doggy-countdown-specific one, with tests) is a
  scope-overlap question for a human, not a duplicate.

---

Corrected after posting: the conflicting list first published here included #2703, which is not in that set — it is refreshed and satisfied. The 14 conflicting PRs are exactly the list above. The "duplicate pairs" note originally named three pairs; only #2708/#2709 survived inspection.

[devantler]

🤖 Generated by the Agentic Engineer

Post-refresh CI state — the unblocking revealed real failures, which is the point

Now that the gate actually runs on these heads, their true state is visible for the first time. Of the
16 refreshed (15 still open — #2708 was closed as a duplicate of #2709):

Outcome	Count	PRs
All checks green	6	#2710 #2712 #2713 #2721 #2732 #2735
🔐 Validate EKS Authorization fails	8	#2707 #2709 #2714 #2729 #2737 #2739 #2741 #2743
kubescape fails	1	#2725

The refresh did not cause these. I checked the obvious way it could have: the six green PRs
genuinely ran 🔐 Validate EKS Authorization and passed it — conclusion: success, not skipped —
so the job is not failing for everyone who merges main. Two PRs based on current main that I did
not touch (#2703, #3034) also pass it. The failure is specific to what those eight branches contain.

The failure itself is a batch of unresolved Flux substitution in authorization resource errors from
scripts/validate-eks-ci-role-policy, naming HelmReleases the PRs do not touch (oauth2-proxy,
openbao, reloader, velero, vertical-pod-autoscaler, …) — so it reads as those branches being
stale against a validation contract that tightened while they sat, rather than as eight independent
mistakes. Worth one diagnosis rather than eight.

What this changes for the next pass: these eight need a fix before they need a reviewer. Requesting
reviews on them now would spend metered review quota on PRs that cannot merge anyway — the same waste
this issue was opened about, one step further along.

[devantler]

🤖 Generated by the Agentic Engineer

Correction: those 8 failures are eight independent problems, not one

My previous comment guessed they looked like "one staleness problem, not eight" because the log tail
was full of unresolved Flux substitution in authorization resource lines naming HelmReleases the PRs
never touch. That guess was wrong, and I should have read the validator before publishing it.

Those substitution lines are explicitly diagnostic, not the failure —
scripts/validate-eks-ci-role-policy/main.go says so in a comment at the emit site, and only appends
them to problems alongside a real failure, to explain it. The actual assertion is:

--- FAIL: TestValidateAuthorizationAcceptsCommittedPolicy
    validateAuthorization() error = unapproved rendered authorization surface fingerprint: <sha>

The validator pins a fingerprint over the whole rendered authorization surface, so any PR that
changes an authorization-relevant resource moves it and must update the approved value. The decisive
test is whether the eight share a fingerprint. They do not:

PR	fingerprint
#2739	6e186a84d19e7c3b…
#2714	294acd5002069691…
#2707	e699196f66d81b6a…
#2741	f04f5e92b60c188d…

Four PRs, four distinct fingerprints. So this is the ordinary per-PR gate doing its job: each of those
eight changes the authorization surface and needs its own approved fingerprint regenerated. There is no
shared root cause to find, and the earlier framing would have sent someone hunting for one.

What stands from the previous comment: the refresh did not cause these (the six green PRs ran the same
job and passed), and they still need fixing before they need a reviewer.

[devantler]

🤖 Generated by the Agentic Engineer

Six refreshed PRs are still BLOCKED with every check green — the documented cause no longer explains them

Independently re-derived from the API this run (not carried over from a digest).

Still BLOCKED, 0 failing, 0 pending: #2710 #2712 #2713 #2721 #2732 #2735
Structurally identical but CLEAN: #2703 #2734

What I ruled out

Hypothesis	Test	Result
The required workflow never ran (this issue's documented cause)	actions/runs?head_sha=… filtered on .path == ".github/workflows/validate-go-project.yaml"	Refuted — completed/success on both BLOCKED (#2710 18:03:24Z, #2732 18:02:47Z) and CLEAN (#2703 16:11:33Z, #2734 18:37:41Z) heads
Missing/blocking review	reviewDecision	Refuted — empty on all 8
Draft vs ready difference	isDraft	Refuted — all 8 are drafts
Different base	baseRefName	Refuted — all main
Different author	author.login	Refuted — all devantler

So after the head refresh these PRs do satisfy the org-required workflow, and they are still
BLOCKED — while two PRs matching them on every attribute above are CLEAN.

What I am deliberately NOT claiming

I cannot name the cause from read-only state. Nothing on the head distinguishes the two groups, and
the only surface that reports why a PR cannot enqueue is the enqueuePullRequest mutation — a
write, so it is not a diagnostic I can run speculatively against six PRs someone else is actively
pushing to.

Flagging this rather than theorising: the last hypothesis I published on this issue
("one staleness problem, not eight") was wrong and had to be corrected, and the loudest available log
lines were the ones that meant nothing.

Suggested next step

Run the enqueue mutation against one of the six and read the rejection reason it returns. That
names the cause in a single call, where inference from the head has now failed twice.

⚠️ All six carry pushes from the codex/* lane within the last ~2h, so they are actively owned —
this is an observation for whoever holds them, not a takeover.

[devantler]

🤖 Generated by the Agentic Engineer

Verified 2026-09-26: all three criteria are met, closing

• Every PR whose head predated the requirement is refreshed or closed. The six this issue last listed as stuck (fix(crossplane): require SNI for registry egress #2710, fix(coroot): use dedicated postgres scrape credentials #2712, fix(security): close alternate tenant route-kind hostname bypass #2713, fix(kyverno): scope Flagger primary mutation RBAC #2721, fix(openbao): gate vault route behind SSO #2732, fix(velero): isolate kopia repository password #2735) are all merged. Every PR open today (fix(talos): stop the audit flood by selecting AppArmor correctly #4197, fix(ci): hold the failure heal until later merge groups resolve #4192, fix(deploy): repair a bad root-source verify before reconciling #4102, chore(deps): update dependency siderolabs/talos to v1.14.0 #4044, chore(deps): update dependency kubernetes/kubernetes to v1.37.1 #3534, security(kubescape): set fsGroup on flux-operator and longhorn-ui #4215) has a successful run of each workflow the ruleset currently requires at its current head.
• The merge path reports the rejection reason. AGENTS.md → Maintenance now says never to judge an enqueue by gh pr merge's exit code, to read the reason from the enqueuePullRequest mutation pinned with expectedHeadOid, and to confirm the effect with isInMergeQueue.
• The note is recorded. The same section documents the BLOCKED-with-green-head case and the mutation that reveals it.

One thing changed underneath this issue. Read live from the ruleset today, ✅ Validate Go Project is no longer a required workflow on main. The three required workflows are Dependency Review, Enable Auto-Merge and Scan for Workflow Vulnerabilities. The documentation still names the old workflow, so its example queries return 0 on every healthy head. That is filed as #4216 so this can close on what it asked for.