You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: use custom cloud init status reporting - #9022
What this PR does / why we need it:
This PR adds CustomDataOnlyProvisioning mode which removes the need for CSE by using a custom report_ready.py script to report ready status to Azure Fabric once AKS considered the node succuessfully provisioned. This is only possible by bypassing cloud-init's built-in report_ready (through custom cloud init pkg with experiemental skip ready report flag) as well as waagent's report_ready (by generating sentinel to skip ready report). The mode is enabled through passing in feature flag "USE_CUSTOM_DATA_ONLY_PROVISIONING=true".
…ript
Bake experimental_skip_ready_report into VHD via cloud.cfg.d to skip
cloud-init's built-in health ready report to Azure fabric. Add a
standalone Python script (report_ready.py) that can be invoked from
CSE to report ready at the appropriate time during node provisioning.
Depends on canonical/cloud-init#6771.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Supersede the single-file DataSourceAzure.py vendoring hack with the
actual patched cloud-init Jammy .deb built from
https://github.com/peytonr18/cloud-init/tree/probertson/jammy-runtime-ds-options-skip-ready
(base version 26.1-0ubuntu1~22.04.1), which carries both the
experimental_skip_ready_report patch (canonical/cloud-init#6771) and a
second patch allowing datasource options to be overridden at runtime
via custom data.
Installing the full package via dpkg instead of overwriting a single
module keeps cloud-init's schema, tests, and packaging metadata
consistent, and avoids the risk of the vendored file drifting from the
rest of the installed package.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9e623d45-e026-4e5c-b9e2-3107c1d96592
Run:https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=174679898 Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E TL;DR:Test_Ubuntu2204 failed in CSE provisioning: vmssCSE returned VMExtensionProvisioningError with CSE ExitCode 161 while running installSecureTLSBootstrapClient, reporting secure TLS bootstrapping enabled without a custom client download URL. Likely cause/signature: PR-change-caused CSE/provisioning regression; stable wiki signature linux-vhd-prgate-cse-secure-tls-bootstrap-no-client-url. Confidence: High for failure mode, medium-high for PR-change-caused attribution. Owner/action: PR author/Node Lifecycle should validate secure TLS bootstrap client URL/config propagation for the Ubuntu2204 E2E path and remove/avoid the temp forced E2E failure before rerunning through normal policy. Strongest alternative: infrastructure/agent flake, but less likely because the same Test_Ubuntu2204 vmssCSE ExitCode 161 repeated in build 174680846 on the same branch and the PR changes CSE/cloud-init/Packer paths. Evidence: timeline failed task log 612; failed test result run 555715746; PR #9022 changed parts/linux/cloud-init/artifacts/cse_install.sh, cse_start.sh, report_ready.py, and Packer/cloud-init files; adjacent manual branch build 174680846 reproduced the same signature. Wiki: https://dev.azure.com/msazure/CloudNativeCompute/_wiki/wikis/=%2FAKS%2FSIGs%20and%20Teams%2FAKS%20Components%2FSIG%3A%20Node%20Lifecycle%2FAI%20Agent%20Knowledge%2FAgentBaker%20Gate%20PR%20Pipeline%20Flakiness%2FNoisy%20Active%20Items
Detective summary: Ubuntu 26.04 Minimal VHD build completed and the content-test VM ran, but vhd-scanning.sh retried twice and failed with Azure Storage BlobNotFound for the expected scan artifact. Cleanup then completed and the task returned exit code 2.
Likely cause / signature:vhd-scanning-trivy-blobnotfound - recurring VHD scanning/trivy artifact upload or blob lookup flake after successful image/content validation.
Confidence: Medium-high for scanning infrastructure/artifact flake; timeline isolates the failure to Test/Scan/Cleanup, and the focused log shows content-test progress before repeated BlobNotFound in vhd-scanning.
Recommended owner/action: VHD validation/scanning owner should inspect trivy scan artifact upload/blob naming for vhd-scans; PR owner only needs to act if this PR touched scan inputs or Ubuntu 26.04 Minimal image content.
Strongest alternative: PR 9022 could alter VHD content through cloud-init/reporting changes, but the fatal error is storage BlobNotFound during scan artifact handling rather than a package/content assertion.
Evidence:build run, failed timeline records, focused Test/Scan/Cleanup log, build metadata, PR changed files, and wiki tracker.
Run:184471790 Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
Detective summary: E2E did not reach scenario execution; Go package build failed at pkg/agent/datamodel/types.go:1859 with syntax error unexpected name int at end of statement.
Likely cause / signature:e2e-go-build-datamodel-types-syntax-error - PR source syntax/compile regression in datamodel types.
Confidence: High: compiler error, package build failures, no test results, and PR change scope align.
Recommended owner/action: PR owner should fix the datamodel syntax and rerun the gate.
Strongest alternative: Transient Go dependency/download issue, but the fatal line is a source syntax error emitted by the compiler.
Evidence:build run, build report issues, focused failed job log, retry/task result signals, and wiki tracker.
Run:184480889 Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
Detective summary: After the PR's earlier compile failure was fixed, this E2E run reached scenario execution and failed many AzureLinuxV3 validations with 'Substring not found' while checking waagent output. The same run also hit known recurring signatures: CustomCA update_certs.service exit 161 and ArtifactStreaming referrer timeout.
Likely cause / signature: e2e-azurelinuxv3-waagent-log-substring-missing - PR-change-shaped waagent/log reporting validation regression across AzureLinuxV3 scenarios.
Confidence: Medium-high for PR-change-caused reporting/log-validation issue; build report, failed E2E log groups, repeated identical substring failures, and PR cloud-init/reporting scope align. Secondary recurring signatures are tracked separately in the wiki.
Recommended owner/action: PR owner should inspect the reporting/ready/waagent log expectation changes and reconcile E2E validators with the new AzureLinuxV3 output; CustomCA recurrence remains tracked by repair item #40042898.
Strongest alternative: WestUS3 host/waagent flake, but the breadth of identical AzureLinuxV3 substring failures on a reporting-focused PR makes a PR-side validator/log contract regression more likely.
Evidence:build run, failed timeline/job Run AgentBaker E2E, focused E2E log groups, build report warnings/errors, PR 9022 reporting/ready commits, and wiki tracker.
Run:184527609 Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
Detective summary: Several AzureLinuxV3 E2E scenarios failed VM log validation because waagent.log did not contain the expected Azure Linux Agent Version: 2.15.0.1 substring after HostGAPlugin/vmSettings fetch retries/timeouts.
Confidence: High: build report, focused E2E log, repeated scenario failures, and PR cloud-init/reporting scope align with the existing signature.
Recommended owner/action: PR owner should verify report-ready/waagent output expectations after recent reporting changes and update either emitted output or validators.
Strongest alternative: Host/waagent goal-state transient, but this repeats on PR 9022 after report-ready changes.
Evidence:build run, build report issues, focused failed job log, PR metadata, and wiki tracker.
Run:185816146 Failed job/stage/task: e2e / Run AgentBaker E2E / Ubuntu2604Minimal_DisableKubeletServingCertificateRotationWithTags_AlreadyDisabled_CustomKubeletConfig
Detective summary: One E2E scenario failed after provisioning because kernel validation detected a drm_atomic_helper_wait_for_vblanks call trace/panic in Ubuntu2604Minimal_DisableKubeletServingCertificateRotationWithTags_AlreadyDisabled_CustomKubeletConfig. Many adjacent scenarios completed with no critical kernel issues.
Confidence: High: failed test record, E2E task log, failed timeline, and existing repair-linked wiki row match prior occurrences; PR 9022 cloud-init reporting changes have weak direct linkage to this kernel call trace.
Recommended owner/action: Covered by repair #39828927; PR owner likely only needs to rerun after the gate flake clears unless cloud-init changes are suspected to affect kernel/VHD state.
Strongest alternative: PR 9022 caused kernel behavior change, but the exact drm_atomic signature is recurring across unrelated PRs and adjacent scenarios were clean.
🟡 Medium Risk — 🏗️ Architecture: This reports failures only after provision_start.sh starts. The generated command exits earlier if init-aks-cloud.sh fails (aks-node-controller/parser/templates/cse_cmd.sh.gtpl:3), and ANC can also fail while building the provisioning command. Its outer error handler writes local status files, while the launcher only logs the exit code. With CSE removed and built-in Ready reporting suppressed, these failures never reach Azure Fabric, leaving VM creation waiting for a provisioning timeout. Add failure reporting at the ANC/launcher error boundary, gated on ownership of reporting, including errors before this wrapper starts.
if config.IsCustomDataOnlyProvisioningEnabled() && (config.IsFlatcar() || config.IsACL() || config.IsAzureLinux()) {
config.CustomDataOnlyProvisioning = false
}
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lilypan26
changed the title
What this PR does / why we need it:
This PR adds
CustomDataOnlyProvisioningmode which removes the need for CSE by using a customreport_ready.pyscript to report ready status to Azure Fabric once AKS considered the node succuessfully provisioned. This is only possible by bypassing cloud-init's built-in report_ready (through custom cloud init pkg with experiemental skip ready report flag) as well as waagent's report_ready (by generating sentinel to skip ready report). The mode is enabled through passing in feature flag "USE_CUSTOM_DATA_ONLY_PROVISIONING=true".Cloud-init changes
By removing CSE for reporting status, we are able to save on average around ~50 seconds from VMSS creation:
Which issue(s) this PR fixes:
Fixes #