fix(deploy): fail the deploy when a kubelet runs without a declared setting

[devantler]

🤖 Generated by the Agentic Engineer

Why

Two production nodes once ran for weeks without the disk-pressure protection the platform declares, and nothing reported it. A check that reads every node after each deploy was added as a warning only, because it had never been compared with production.

What

That check has passed on all 25 production deploys since it was added, so a node that runs without a declared setting now fails the deploy instead of only warning. Nothing changes while the nodes match what is declared.

Fixes #3137

[devantler]

🤖 Generated by the Agentic Engineer

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository YAML (base), Organization UI (inherited)
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 24fa1593-495b-4993-978d-986747f25b2b

📥 Commits

Reviewing files that changed from the base of the PR and between 752a0f0 and 807bb8d.

📒 Files selected for processing (3)

• .github/actions/deploy-prod/action.yml
• scripts/tests/test-kubelet-config-readback.sh
• scripts/verify-kubelet-config-readback.sh

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details

🔇 Additional comments (3)

.github/actions/deploy-prod/action.yml (1)

476-479: LGTM!

scripts/tests/test-kubelet-config-readback.sh (1)

28-30: LGTM!

Also applies to: 33-33, 36-36

scripts/verify-kubelet-config-readback.sh (1)

32-33: LGTM!

Also applies to: 42-42

---

📝 Walkthrough

Walkthrough

The production deploy no longer sets KUBELET_READBACK_ENFORCE to "false" for the kubelet readback step. The deployment-step test now requires that the variable is unset. Comments in the verification script describe observe-only behavior when the cluster result is unknown and state that observe-only cannot fail a deploy.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 807bb

Production deployment will now stop if any node, including an autoscaler node, has kubelet drift. Prior checks reportedly passed, and no current mismatch is established; drift would intentionally block deployment.

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Linked Issues check	❓ Inconclusive	[ #3137 ] The reviewed code checks every node's live configz against declared kubelet settings, including autoscaler-provisioned nodes. The tests cover the reported autoscaler drift. This PR removes…	Provide a finding-free readback result, or equivalent live configz evidence, for all nine production nodes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The changes to the deploy action, its contract test, and readback comments all support [ #3137 ]: they make the existing all-node drift check block deployment and test that behavior. No unrelated chan…
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (1 skipped: 1 …
Title check	✅ Passed	The title clearly states the main change: kubelet setting read-back findings now fail the deployment.
Description check	✅ Passed	The description explains why the check is changing and that it will fail deployments when a node lacks a declared setting.

Full details: Linked Issues check

Explanation

[ #3137 ] The reviewed code checks every node's live configz against declared kubelet settings, including autoscaler-provisioned nodes. The tests cover the reported autoscaler drift. This PR removes observe-only mode from the deploy step, so a finding blocks deployment. However, the PR does not establish that all nine production nodes currently report mergeDefaultEvictionSettings: true and all five evictionHard signals. The reported 25 successful runs do not resolve this: the prior observe-only mode exits successfully even when it finds drift.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devantler]

🤖 Generated by the Agentic Engineer

Evaluation at 807bb8db1998d9d3fd62bf7726904b20eb3748ed.

• Production evidence for making it blocking: I read the deploy log of every merge-queue run since the check merged (feat(deploy): read back the declared kubelet settings on every node #4559, 2026-10-06 10:19Z). 25 of them deployed, and all 25 logged PASS: 7 node(s), 1 of them autoscaler-provisioned, run every declared kubelet setting … with no warning. That list includes mergeDefaultEvictionSettings and evictionHard, the two settings Two autoscaler nodes did not pick up the kubelet eviction fix #3137 found missing.
• A failing read-back does not strand the deploy: the steps after it that reassert credentials and the rollout gate carry always() or !cancelled() conditions, the same as for the blocking boot-image read-back directly above it.
• Guard: CI run 37607114384 at this commit ran the read-back test in 🧪 Validate Talos Machine Config and it passed. Locally, putting the observe-only switch back makes that test fail with "the deploy must run the kubelet readback once, blocking".
• Review: CodeRabbit reviewed 752a0f0b…807bb8db at 11:16Z with no actionable comments; 0 threads.

Promoting and adding to the merge queue pinned to this commit. After it merges I will read the next deploy's step to confirm it still passes.