Skip to content

feat: bake node-exporter into Azure Linux 3 Kata VHDs - #9810

Open
chmill-zz wants to merge 1 commit into
mainfrom
kataExporter
Open

chmill-zz wants to merge 1 commit into
mainfrom
kataExporter

Conversation

@chmill-zz

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Bakes Prometheus node-exporter into Azure Linux 3 Kata (Pod Sandboxing) VHDs, so Kata uses the same VHD-owned exporter as regular Azure Linux 3 instead of the VM extension's copy.

Kata was left out when node-exporter moved into the VHD (#7704) only to keep that PR's scope small. No technical reason was recorded.

Changes

  • components.json: adds an azurelinuxkata entry for node-exporter, mirroring azurelinux v3.0. Kata builds look up packages under azurelinuxkata, so before this no version resolved.
  • install-dependencies.sh / packer_source.sh: removes the Kata skips for the install and for the startup script and unit files. Both skips are removed together, since installing without the unit files fails the build.
  • Content test: Kata now has to have the baked exporter instead of having to lack it.
  • No CSE change: configureNodeExporter already starts the exporter whenever skip_vhd_node_exporter exists.

Expected behavior: no pressure (PSI) metrics on Kata

The Kata host kernel (kernel-mshv) is built without CONFIG_PSI, so /proc/pressure does not exist. node-exporter's pressure collector then returns no data: there are no node_pressure_* series and node_scrape_collector_success{collector="pressure"} is 0. The exporter keeps running and scrapes succeed.

This matches the extension-managed exporter on Kata today. PSI was enabled for kernel-mshv in microsoft/azurelinux#9720 and appears to have been dropped by a later config sync. Tracked in microsoft/azurelinux#19417.

E2E (ValidateMetrics) therefore:

  • still requires node_pressure_cpu_waiting_seconds_total on every non-Kata image, so losing PSI there remains a failure;
  • on Kata, requires it only when /proc/pressure/cpu exists. Otherwise it asserts that the pressure collector reports no data and that no PSI metric is exposed, so the scenario starts requiring the metric once the kernel is fixed.

Validation

  • PR-gate pipeline on this branch: all VHD builds succeeded, including the Kata VHD and its content test. All E2E scenarios passed. In AzureLinuxV3Gen2Kata, node-exporter validation ran and passed with PSI available=false.
  • Unit tests for the new metric contract; e2e go vet, pkg/agent tests, CUE validation, shellcheck, and the relevant ShellSpec suites (483 examples) pass.
  • Disposable Kata node pool from this branch's VHD (kernel 6.18.34.mshv3-1.azl3):
    • The baked node-exporter-kubernetes-1.12.1-10.azl3 started from CSE. CONFIG_PSI is not set, /proc/pressure is absent, and the pressure collector reports 0.
    • After applying AKSLinuxExtension 1.515: the extension honored skip_vhd_node_exporter. It installed no exporter of its own, and the baked exporter was not restarted. Extension-managed NPD loaded its node-exporter health monitor, which reports healthy. NPD conditions are published.
    • With three running Kata pods (guest kernel differs from the host): no extra node_filesystem_* series (still six host mounts) and no Kata/virtiofs/erofs mounts in the host namespace. The exporter stayed healthy (about 33 MB RSS).
    • After a reboot: the exporter came back active on the baked package, and the Kata pods returned to Running.
  • For comparison, an existing extension-managed Kata pool (kernel 6.6.137.mshv2-2.azl3) shows the same PSI behavior with the extension's exporter 1.9.1.

Which issue(s) this PR fixes:

Related: microsoft/azurelinux#19417

Kata was deferred when node-exporter moved into the VHD, not excluded for a
technical reason. Resolve the package through azurelinuxkata, copy the
startup and unit files, and require the baked exporter in content tests.

The Kata host kernel (kernel-mshv) is built without CONFIG_PSI, so the
pressure collector returns no data there. E2E requires PSI metrics on every
other image and, on Kata, requires them only when /proc/pressure exists;
otherwise it asserts the collector reports no data.
See microsoft/azurelinux#19417.
Copilot AI balanced review requested due to automatic review settings October 17, 2026 02:09
@chmill-zz
chmill-zz requested review from a team as code owners October 17, 2026 02:09
@github-actions github-actions Bot added the components This pull request updates cached components on Linux or Windows VHDs label Oct 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   18 suites   1m 13s ⏱️
556 tests 556 ✅ 0 💤 0 ❌
559 runs  559 ✅ 0 💤 0 ❌

Results for commit baaa790.

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
test 0% ❌
windows 27% ❌
staging_cse_windows Package 1 65% ➖
debug 0% ❌
provisioningscripts 2% ❌
parts_windows Package 1 74% ➖
Summary 38% (1649 / 6831) ❌

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 72% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 68% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 88% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 88% ✔
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 24% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 0% ❌
github.com/Azure/agentbaker/pkg/agent 76% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 76% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 87% ✔
Summary 74% (5621 / 7572) ➖

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 25% ❌
Summary 25% (3865 / 15527) ❌

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The coordinated installation and validation changes preserve existing exclusions and non-Kata requirements, with no blocking issues identified.

No open issues 🎉

What changed in this PR

Extends VHD-owned node-exporter to Azure Linux 3 Kata, using the existing provisioning startup path.

Changes:

  • Adds Kata package resolution and enables exporter installation and service assets.
  • Updates VHD content checks to require the exporter on Kata.
  • Adds PSI-aware validation and unit tests while preserving non-Kata pressure metric requirements.
File Description
vhdbuilder/​packer/​test/​linux-vhd-content-test.sh Requires baked node-exporter on Kata.
vhdbuilder/​packer/​packer_source.sh Includes exporter service assets on Kata.
vhdbuilder/​packer/​install-dependencies.sh Enables exporter installation on Kata.
parts/​common/​components.json Adds the Azure Linux 3 Kata package entry.
e2e/​scenario/​validators.go Detects Kata PSI availability for validation.
e2e/​nodeexporter/​metrics.go Makes pressure metric expectations conditional.
e2e/​nodeexporter/​metrics_test.go Tests metrics validation with and without PSI.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

components This pull request updates cached components on Linux or Windows VHDs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants