Repository navigation
(WIP)CKS: reconcile node capacity after live resize - #14223
andrijapanicsb wants to merge 10 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14223 +/- ##
============================================
+ Coverage 20.45% 20.46% +0.01%
- Complexity 21415 21480 +65
============================================
Files 6487 6489 +2
Lines 589303 589540 +237
Branches 72245 72307 +62
============================================
+ Hits 120525 120655 +130
- Misses 455643 455706 +63
- Partials 13135 13179 +44
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andrijapanicsb
changed the title
|
@ACSHomeBot package G |
Packaging results:
Test packages are available at: |
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 25.51% |
| Branch coverage | 19.86% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
40b50b0 to
7e5a95a
Compare
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
Packaging results:
Package building did not start. No repository was published. Status: Infrastructure failure before package building started. |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
|
This branch has not been deployed
2 participants
Why this follow-up is needed
#13226 allowed CKS node compute offerings to change while a KVM cluster was running, building on the fixed-offering live resize support in #12975. The VM could gain CPU or RAM while Kubernetes continued advertising the old capacity. Operators still had to activate the guest resources where needed, restart
kubelet, and check the Node's capacity and allocatable values themselves.#14213 tracks that gap. This PR puts those steps into the CKS scale workflow so that success means the guest and Kubernetes have caught up with the new offering.
Package-based QA also exposed a pre-existing bug in cluster CPU/RAM totals, not a regression introduced by this PR. The faulty calculation and resize-before-accounting order were already present in the base
maincommit (af9764b87a). After a control node and worker each grew from 2 vCPU / 2048 MiB to 3 vCPU / 3072 MiB, their offerings and actual resources were correct, but the cluster API and UI still showed 4 CPU / 4096 MiB. The old calculation read the VM offering after resize and treated it as the previous offering, producing a zero difference. The total is now recalculated from each node type's configured offering and node count. This also avoids carrying an already stale total into the next scale operation and preserves separate control, worker, and etcd offerings when the worker count changes.The wider QA run found two more existing per-node-offering edge cases: a global offering change tried to look up dedicated etcd nodes even when there were none, and adding workers required the legacy global offering even when a worker offering was configured. Both paths now use the applicable node types and offerings.
QA also caught a gap in retry selection: looking at the first VM's current offering could skip the group after that VM had already resized. The decision now uses the cluster's committed node-type offering, so a retry can reconcile the partially resized node and continue with the untouched nodes.
What changes
For a CPU-count or RAM increase on a running, Cloud-managed KVM worker or control-plane node, CKS handles one node at a time:
MemTotal.kubeletwhen Kubernetes still has the old resources, then wait for the Node to be Ready with matching capacity and increased allocatable resources.The node SSH connection uses its actual port-forwarding rule on isolated/VPC networks, or its private address on direct-access networks. A failed post-resize verification leaves the CKS cordon in place and reports the failing phase. A previously operator-cordoned node remains cordoned. The workflow does not drain or evict resident Pods.
Guest/kubelet reconciliation applies to running managed KVM workers and control-plane nodes. Stopped clusters retain the existing VM lifecycle workflow. Dedicated etcd live resize remains rejected pending explicit etcd health/quorum handling.
Validation
2a404aa1ffRPMs were installed and tested first. Final QA uses locally built EL9 RPMs from978b29173ed70c695b50bce9ea7ecdf7436ae070on both management servers and both KVM hosts. Embedded revisions and the installed scale-worker class were checked against the build output.This QA7 run covers Cloud-managed KVM on an isolated network; it does not qualify VPC/direct-access networking or multi-control HA. Dedicated etcd live resize remains unsupported. The PR stays a draft for review and remaining coverage. The latest coverage workflow hit a Maven Central HTTP 502 before producing a Java report; its failed job has been requeued rather than treating the older coverage result as current.