Skip to content

feat: retry downloads and package installs, make monitoring_plugins and icinga2_agent idempotent, fix the Rocky security lane - #414

Merged
NavidSassan merged 8 commits into
mainfrom
feat/retries-and-idempotency
Oct 2, 2026
Merged

NavidSassan merged 8 commits into
mainfrom
feat/retries-and-idempotency

Conversation

@markuslf

@markuslf markuslf commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Stacked on #413 (base fix/delegated-tasks-become), so this PR only shows its own six commits.

Changes

  • roles: downloads from the network are retried up to three times, 10 s apart: get_url, git clones, rpm_key with a URL and the GitHub release lookups (83 tasks). Since ansible-core 2.16 retries without until repeats a task until it succeeds. Calls that change something are not retried. CONTRIBUTING describes the rule.
  • role:monitoring_plugins: the legacy dependency list is compared by content, so an unchanged host reports no change; on Linux the role no longer stops and starts icinga2 around every deployment (a Windows workaround for locked files).
  • role:icinga2_agent: icinga2 node setup, the ticket request and the agent restart run only if the agent certificate is missing, expired or not signed by the master's CA (icinga2 pki verify), the master presents a different certificate, or the setup parameters changed (hash in /var/lib/icinga2/linuxfabrik-node-setup.sha256, delete it to force a new setup). The Director's 422 "Trying to recreate" for a known host counts as success.
  • roles: package installations are retried as well: every package, apt, dnf and pip task that installs or updates (160 tasks), not removals.
  • role:system_update: the security lane installs exactly the latest versions the security repository has updates for, resolved with all enabled repositories. A hot-fix that needs a newer package from BaseOS or AppStream no longer fails: on Rocky 8.3 the kernel hot-fix requires a newer linux-firmware, and --enablerepo=security alone failed on every run.
  • role:system_update: on Fedora 42 and later /usr/local/sbin is a symlink to bin, so removing the pre-FHS /usr/local/bin/update-and-reboot deleted the script just deployed. That removal is skipped there.

Tests

  • retries without until: verified with ansible-core 2.16 and 2.18.
  • Three consecutive setup_basic runs on fresh Rocky 9, Debian 13 and Fedora 44 clones (ansible-core 2.18, agent against a lab Icinga master): all failed=0; the second run skips the node setup and no longer restarts icinga2; the third run reports no changes except the controller-side git clones, which only change in the lab because of a git insteadOf redirect to local checkouts (verified: against GitHub the clone reports no change).
  • Fedora 44: update-and-reboot is present after the run.
  • Package retries: setup_basic on a fresh Rocky 9 clone failed=0; the EPEL key download hit a 404 once and passed on the retry.
  • Security lane on fresh Rocky 8.3, 9.4 and 10.0 clones, started by hand: the kernel hot-fix comes from @security everywhere, linux-firmware from @baseos on 8.3; the reboot is requested via the spool.
  • icinga2 pki verify exit codes checked against lib/cli/pkiverifycommand.cpp.
  • ansible-lint: two findings fewer, no new ones. pre-commit green.

@NavidSassan
NavidSassan added this pull request to stack #415 October 2, 2026 08:20
@markuslf
markuslf force-pushed the feat/retries-and-idempotency branch from 362769c to 6abe295 Compare October 2, 2026 08:44
@markuslf markuslf changed the title feat: retry downloads, make monitoring_plugins and icinga2_agent idempotent feat: retry downloads and package installs, make monitoring_plugins and icinga2_agent idempotent, fix the Rocky security lane Oct 2, 2026
Base automatically changed from fix/delegated-tasks-become to main October 2, 2026 12:05
A brief outage of a download source aborted the run, as seen in the lab
with the Icinga mirror and dl.fedoraproject.org. Retry get_url, git
clones, rpm_key with a URL and the GitHub release lookups up to three
times, 10 seconds apart (83 tasks). Since ansible-core 2.16 `retries`
without `until` repeats a task until it succeeds, verified with 2.16 and
2.18. Calls that change something, such as API POSTs, are not retried.
…every run

The legacy dependency list was pushed with the times and owner of the
controller's checkout, so every run reported a change; compare its
content like the other transfers do. On Linux the role also stopped and
started icinga2 around every deployment, taken over from Windows, where
files in use cannot be replaced. Linux has no such lock: every check
starts the plugin afresh and files are replaced by renaming, so the
agent keeps running.
…ds it

`icinga2 node setup` ran on every run, created a new key and certificate,
requested a ticket and restarted the agent. Run it only if the agent
certificate is missing, expired or not signed by the master's CA
(`icinga2 pki verify`, checked against lib/cli/pkiverifycommand.cpp),
the master presents a different certificate, or the setup parameters
changed (hash in /var/lib/icinga2/linuxfabrik-node-setup.sha256). The
Director's 422 "Trying to recreate" for a known host now counts as
success instead of an ignored error. Verified on Rocky 9, Debian 13 and
Fedora 44: the second run skips the node setup.
/usr/local/sbin is a symlink to bin there (filesystem package, verified
on Fedora 44), so removing the pre-FHS /usr/local/bin/update-and-reboot
deleted the script deployed a moment before, and the weekly update had
no script to run. Skip that removal if /usr/local/sbin is a symlink.
A brief outage of a package mirror aborted the run, as seen in the lab
with the Icinga repository on mirror-rhel9. Retry every package, apt,
dnf and pip task that installs or updates, up to three times, 10 seconds
apart (160 tasks). Removals need no repository and are not retried.
…x needs

The lane upgraded with only the security repo enabled, so a hot-fix that
needs a newer package from BaseOS or AppStream never got installed: on
Rocky 8.3 the kernel requires a newer linux-firmware ("nothing provides
linux-firmware >= 20220713"), and the lane failed on every run. List the
latest versions the security repo has updates for and upgrade exactly
these name-version-releases with all enabled repositories, so the
dependencies resolve and the lane still installs the hot-fixes only, not
newer versions from BaseOS. Verified on Rocky 8.3, 9.4 and 10.0: kernel
from @security, linux-firmware from @BasEOS on 8.3.
@NavidSassan
NavidSassan force-pushed the feat/retries-and-idempotency branch from 6abe295 to 3f80dc7 Compare October 2, 2026 12:13
@NavidSassan
NavidSassan merged commit 26f5299 into main Oct 2, 2026
13 checks passed
@NavidSassan
NavidSassan deleted the feat/retries-and-idempotency branch October 2, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants