Repository navigation
feat: retry downloads and package installs, make monitoring_plugins and icinga2_agent idempotent, fix the Rocky security lane - #414
Merged
Conversation
NavidSassan
added this pull request to stack #415
October 2, 2026 08:20
markuslf
force-pushed
the
feat/retries-and-idempotency
branch
from
October 2, 2026 08:44
362769c to
6abe295
Compare
NavidSassan
approved these changes
Oct 2, 2026
A brief outage of a download source aborted the run, as seen in the lab with the Icinga mirror and dl.fedoraproject.org. Retry get_url, git clones, rpm_key with a URL and the GitHub release lookups up to three times, 10 seconds apart (83 tasks). Since ansible-core 2.16 `retries` without `until` repeats a task until it succeeds, verified with 2.16 and 2.18. Calls that change something, such as API POSTs, are not retried.
…every run The legacy dependency list was pushed with the times and owner of the controller's checkout, so every run reported a change; compare its content like the other transfers do. On Linux the role also stopped and started icinga2 around every deployment, taken over from Windows, where files in use cannot be replaced. Linux has no such lock: every check starts the plugin afresh and files are replaced by renaming, so the agent keeps running.
…ds it `icinga2 node setup` ran on every run, created a new key and certificate, requested a ticket and restarted the agent. Run it only if the agent certificate is missing, expired or not signed by the master's CA (`icinga2 pki verify`, checked against lib/cli/pkiverifycommand.cpp), the master presents a different certificate, or the setup parameters changed (hash in /var/lib/icinga2/linuxfabrik-node-setup.sha256). The Director's 422 "Trying to recreate" for a known host now counts as success instead of an ignored error. Verified on Rocky 9, Debian 13 and Fedora 44: the second run skips the node setup.
/usr/local/sbin is a symlink to bin there (filesystem package, verified on Fedora 44), so removing the pre-FHS /usr/local/bin/update-and-reboot deleted the script deployed a moment before, and the weekly update had no script to run. Skip that removal if /usr/local/sbin is a symlink.
A brief outage of a package mirror aborted the run, as seen in the lab with the Icinga repository on mirror-rhel9. Retry every package, apt, dnf and pip task that installs or updates, up to three times, 10 seconds apart (160 tasks). Removals need no repository and are not retried.
…x needs
The lane upgraded with only the security repo enabled, so a hot-fix that
needs a newer package from BaseOS or AppStream never got installed: on
Rocky 8.3 the kernel requires a newer linux-firmware ("nothing provides
linux-firmware >= 20220713"), and the lane failed on every run. List the
latest versions the security repo has updates for and upgrade exactly
these name-version-releases with all enabled repositories, so the
dependencies resolve and the lane still installs the hot-fixes only, not
newer versions from BaseOS. Verified on Rocky 8.3, 9.4 and 10.0: kernel
from @security, linux-firmware from @BasEOS on 8.3.
NavidSassan
force-pushed
the
feat/retries-and-idempotency
branch
from
October 2, 2026 12:13
6abe295 to
3f80dc7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #413 (base
fix/delegated-tasks-become), so this PR only shows its own six commits.Changes
get_url, git clones,rpm_keywith a URL and the GitHub release lookups (83 tasks). Since ansible-core 2.16retrieswithoutuntilrepeats a task until it succeeds. Calls that change something are not retried. CONTRIBUTING describes the rule.icinga2 node setup, the ticket request and the agent restart run only if the agent certificate is missing, expired or not signed by the master's CA (icinga2 pki verify), the master presents a different certificate, or the setup parameters changed (hash in/var/lib/icinga2/linuxfabrik-node-setup.sha256, delete it to force a new setup). The Director's 422 "Trying to recreate" for a known host counts as success.package,apt,dnfandpiptask that installs or updates (160 tasks), not removals.securityrepository has updates for, resolved with all enabled repositories. A hot-fix that needs a newer package from BaseOS or AppStream no longer fails: on Rocky 8.3 the kernel hot-fix requires a newerlinux-firmware, and--enablerepo=securityalone failed on every run./usr/local/sbinis a symlink tobin, so removing the pre-FHS/usr/local/bin/update-and-rebootdeleted the script just deployed. That removal is skipped there.Tests
retrieswithoutuntil: verified with ansible-core 2.16 and 2.18.setup_basicruns on fresh Rocky 9, Debian 13 and Fedora 44 clones (ansible-core 2.18, agent against a lab Icinga master): allfailed=0; the second run skips the node setup and no longer restarts icinga2; the third run reports no changes except the controller-side git clones, which only change in the lab because of a gitinsteadOfredirect to local checkouts (verified: against GitHub the clone reports no change).update-and-rebootis present after the run.setup_basicon a fresh Rocky 9 clonefailed=0; the EPEL key download hit a 404 once and passed on the retry.@securityeverywhere,linux-firmwarefrom@baseoson 8.3; the reboot is requested via the spool.icinga2 pki verifyexit codes checked againstlib/cli/pkiverifycommand.cpp.