apache_httpd: OWASP CRS 4.30.0 and CRS/reload fixes on Debian and Ubuntu

CHANGELOG.md

@@ -64,6 +64,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 * Downloads from the Internet, such as release tarballs, GPG keys, git clones, GitHub release lookups and package installations, are retried up to three times, so a brief outage of a download source or package repository no longer aborts the run (all roles).
+* **role:apache_httpd**: Bump the OWASP Core Rule Set to 4.30.0.
+* **role:apache_httpd**: The OWASP Core Rule Set is downloaded on the Ansible controller, so the target no longer needs access to GitHub.
 * **role:monitoring_plugins**: On Linux, the role no longer stops the Icinga2 agent while it deploys the plugins, so a run no longer interrupts the monitoring of the host.
 * **role:icinga2_agent**: The role runs `icinga2 node setup` and restarts the agent only when the agent needs a new certificate or its settings changed, instead of on every run.
 * LFOps requires community.general 7.0.0 or newer (still below 9.0.0), which `ansible-galaxy collection install linuxfabrik.lfops` pulls in, while a manually maintained collection list has to be raised.
@@ -91,6 +93,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+* **role:apache_httpd**: On Debian and Ubuntu, the first run against a fresh host no longer fails reloading Apache, since the role points `PidFile` to where `apachectl` looks; a server started by an earlier version of the role is restarted once.
+* **role:apache_httpd**: Apache reloads after the OWASP Core Rule Set is updated, instead of applying the previous rules until its next reload.
+* **role:apache_httpd**: The OWASP Core Rule Set deploys on Debian and Ubuntu, where the run failed until now, and on Ubuntu 22.04, whose ModSecurity 2.9.5 is too old for it, the run aborts with a message saying so.
 * **playbook:redis, playbook:setup_mastodon**: On Debian and Ubuntu, the playbooks no longer abort with "'redis__skip_repo_redis' is undefined" or "'setup_mastodon__skip_repo_redis' is undefined" unless the skip variable is set in the inventory.
 * **role:grafana_grizzly, role:monitoring_plugins_grafana_dashboards**: On Grafana 13.1 and newer, dashboards that the roles add or update stay in their folder, instead of landing in the root, where viewers and the graphs embedded in IcingaWeb2 got "403". A run also moves back the dashboards that earlier runs left in the root.
 * **role:aide**: The AIDE check no longer fails on hosts with fwupd, where it reported `/etc/fwupd/fwupd.conf` as changed after the first start of the fwupd daemon.

COMPATIBILITY.md

@@ -12,7 +12,7 @@ Which Ansible role is proven to run on which OS?
 | aide                                  |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     |                                              |
 | alternatives                          |   x    |   x    |   x    |   x    |   (x)   |     x     |     x     |    (x)    |                                              |
 | ansible_init                          |        |        |        |        |         |           |           |           | Fedora 35+                                   |
-| apache_httpd                          |   x    |   x    |   x    |   x    |    x    |    (x)    |     x     |    (x)    |                                              |
+| apache_httpd                          |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     |                                              |
 | apache_solr                           |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
 | apache_tomcat                         |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
 | apps                                  |  (x)   |  (x)   |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    |                                              |

extensions/molecule/apache_httpd/inventory/group_vars/systems_under_test.yml

@@ -1,5 +1,23 @@
 apache_httpd__conf_server_admin: 'webmaster@example.com'
 
+# The OWASP Core Rule Set, included server-wide so that every vHost below runs behind it. verify.yml
+# sends an attack and expects the rules to block it, which proves that the release the role deployed
+# is the one the running Apache applies. The ordinary requests of the other checks pass through the
+# same rules. Ubuntu 22.04 ships a ModSecurity that is too old for the CRS (see host_vars).
+apache_httpd__conf__group_var:
+  - filename: 'z00-crs'
+    enabled: true
+    state: 'present'
+    template: 'raw'
+    raw: |-
+      SecRuleEngine On
+      SecRequestBodyAccess On
+      Include modsecurity.d/crs/crs-setup.conf
+      Include modsecurity.d/crs/rules/*.conf
+# Pinned here as well, because verify.yml compares the release in the log against it.
+apache_httpd__mod_security_coreruleset_version: '4.30.0'
+apache_httpd__skip_mod_security_coreruleset: false
+
 # The same user in two htpasswd files. The username and the path together identify an entry, so
 # both entries have to reach their own file instead of being merged into one. Each file protects
 # the location of h2.example.com named after it (see the raw block below), and verify.yml logs in
@@ -16,10 +34,15 @@ apache_httpd__htpasswd__group_var:
     password: 'linuxfabrik'
     path: '/var/www/htpasswd-b'
 
+# mod_security2 is off by default and needed for the Core Rule Set above.
 # The role enables the deflate conf, but leaves mod_deflate and mod_filter off, and the deflate conf
 # sits inside <IfModule> blocks for both. Without them nothing is compressed, so the compression
 # check in verify.yml needs them loaded, the way a reverse proxy that compresses has them.
 apache_httpd__mods__group_var:
+  - filename: 'security2'
+    enabled: true
+    state: 'present'
+    template: 'security2'
   - filename: 'deflate'
     enabled: true
     state: 'present'

extensions/molecule/apache_httpd/inventory/host_vars/ubuntu2204-vm.yml

@@ -0,0 +1,6 @@
+# Ubuntu 22.04 ships ModSecurity 2.9.5, which rejects the rules of the OWASP Core Rule Set 4, so the
+# role refuses to deploy them there. The host runs the rest of the scenario without them.
+apache_httpd__conf__host_var:
+  - filename: 'z00-crs'
+    state: 'absent'
+apache_httpd__skip_mod_security_coreruleset: true

extensions/molecule/apache_httpd/verify.yml

@@ -9,7 +9,8 @@
 #
 # The htpasswd files are checked the same way: by logging in through Apache, not by reading them.
 # So is the compression: by the Content-Encoding of a real response, not by the deflate conf.
-- name: 'Verify apache_httpd serves HTTP/2, basic auth and compression where it is configured to'
+# And the OWASP Core Rule Set: by an attack that the running Apache blocks, not by the rule files.
+- name: 'Verify apache_httpd serves HTTP/2, basic auth, compression and the Core Rule Set where it is configured to'
   hosts: 'systems_under_test'
   gather_facts: true
   become: true
@@ -237,3 +238,49 @@
           - '__molecule__apache_httpd_curl_deflate_result["stdout"] is search("^content-type: text/markdown", ignorecase=true, multiline=true)'
           - '__molecule__apache_httpd_curl_deflate_result["stdout"] is search("^content-encoding: gzip", ignorecase=true, multiline=true)'
         fail_msg: 'expected a gzip-compressed text/markdown response, got: {{ __molecule__apache_httpd_curl_deflate_result["stdout"] }}'
+
+    # The Core Rule Set is included server-wide (see group_vars), so the first vHost has to block a
+    # request carrying a shell command in a parameter (rule 932160, paranoia level 1). The 200 for
+    # every vHost above shows that ordinary requests still pass. A host where the rules are not
+    # loaded answers 200 here, and one that runs an older release than the inventory asks for logs
+    # a different version in the rule match, which the second assertion catches.
+    - name: 'curl https://<vhost>/?exec=/bin/bash' # noqa command-instead-of-module (ansible.builtin.uri cannot pin the name to 127.0.0.1 the way --resolve does)
+      ansible.builtin.command: >-
+        curl --silent --insecure
+        --resolve {{ __molecule__server_name }}:443:127.0.0.1
+        --output /dev/null
+        --write-out '%{http_code}'
+        https://{{ __molecule__server_name }}/?exec=/bin/bash
+      register: '__molecule__apache_httpd_curl_crs_result'
+      changed_when: false
+      vars:
+        __molecule__server_name: '{{ apache_httpd__vhosts__group_var[0]["conf_server_name"] }}'
+      when:
+        - 'not apache_httpd__skip_mod_security_coreruleset | bool'
+
+    - name: 'grep the release of the matching rule from <vhost>-error.log'
+      ansible.builtin.command: >-
+        grep --only-matching --max-count 1 --perl-regexp
+        '\[id "932160"\].*\[ver "\KOWASP_CRS/[^"]+'
+        {{ __molecule__error_log }}
+      register: '__molecule__apache_httpd_crs_version_result'
+      changed_when: false
+      failed_when: false
+      vars:
+        __molecule__error_log: >-
+          {{ (ansible_facts["os_family"] == "Debian") | ternary("/var/log/apache2", "/var/log/httpd") }}/{{
+          apache_httpd__vhosts__group_var[0]["conf_server_name"] }}-error.log
+      when:
+        - 'not apache_httpd__skip_mod_security_coreruleset | bool'
+
+    - name: 'Assert the Core Rule Set blocks the attack with the configured release'
+      ansible.builtin.assert:
+        that:
+          - '__molecule__apache_httpd_curl_crs_result["stdout"] == "403"'
+          - '__molecule__apache_httpd_crs_version_result["stdout"] == "OWASP_CRS/" ~ apache_httpd__mod_security_coreruleset_version'
+        fail_msg: >-
+          expected 403 from OWASP_CRS/{{ apache_httpd__mod_security_coreruleset_version }}, got
+          {{ __molecule__apache_httpd_curl_crs_result["stdout"] }} and
+          "{{ __molecule__apache_httpd_crs_version_result["stdout"] }}" in the error log
+      when:
+        - 'not apache_httpd__skip_mod_security_coreruleset | bool'

roles/apache_httpd/README.md

@@ -41,11 +41,18 @@ This role supports both Red Hat and Debian-based systems. The following paths an
 * User/Group: Red Hat `apache`/`apache`, Debian `www-data`/`www-data`
 * PHP-FPM socket: Red Hat `/run/php-fpm/www.sock`, Debian `/run/php/www.sock`
 
+The OWASP ModSecurity Core Rule Set (CRS) is downloaded on the Ansible controller and copied to the target, so only the controller needs access to GitHub. The role deploys the rules to `/etc/httpd/modsecurity.d/crs/`, but does not activate them: include `modsecurity.d/crs/crs-setup.conf` and `modsecurity.d/crs/rules/*.conf` in a vHost (see [EXAMPLES.md](https://github.com/Linuxfabrik/lfops/blob/main/roles/apache_httpd/EXAMPLES.md)). Each version is kept in its own directory, so a change of `apache_httpd__mod_security_coreruleset_version` switches the symlink and reloads Apache, and setting the previous version again rolls back. `crs-setup.conf` is reset to the shipped example on every run; put your CRS settings into the vHost.
+
 This role does NOT:
 
 * install PHP or PHP-FPM. It prefers PHP-FPM over mod_php, but installs neither.
 
 
+## Known Limitations
+
+* The OWASP Core Rule Set needs ModSecurity 2.9.6 or newer. Ubuntu 22.04 ships 2.9.5, so `apache_httpd__skip_mod_security_coreruleset` has to stay `true` there.
+
+
 ## Dependent Roles
 
 Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/README.md) that installs this role runs these for you. Optional ones can be disabled via the playbook's skip variables.
@@ -78,7 +85,7 @@ Manual steps:
 * Creates or updates the global Apache configuration (`httpd.conf`).
 * Removes rpmnew/rpmsave files (and Debian equivalents).
 * Removes, creates, disables, and enables conf-available configs.
-* Triggers: httpd.service reload.
+* Triggers: httpd.service reload, or a restart on Debian and Ubuntu where Apache still runs with the PID file of an earlier version of the role (`/run/apache2.pid`).
 
 `apache_httpd:htpasswd`
 
@@ -87,11 +94,12 @@ Manual steps:
 
 `apache_httpd:mod_security_coreruleset`
 
+* Aborts if the installed ModSecurity is older than 2.9.6, which the Core Rule Set 4 needs.
 * Installs `tar`.
-* Downloads and verifies the OWASP ModSecurity Core Rule Set (CRS).
-* Extracts the archive and creates a symlink to the CRS directory.
+* Downloads the OWASP ModSecurity Core Rule Set (CRS) on the Ansible controller.
+* Extracts the archive to `/etc/httpd/modsecurity.d/` and points the `modsecurity.d/crs` symlink at it.
 * Copies the default `crs-setup.conf.example` to `crs-setup.conf`.
-* Triggers: none.
+* Triggers: httpd.service reload.
 
 `apache_httpd:mods`
 
@@ -881,7 +889,7 @@ apache_httpd__mod_log_config_custom_log: 'logs/access.log combined'
 
 * The OWASP ModSecurity Core Rule Set (CRS) version number without "v".
 * Type: String.
-* Default: `'4.27.0'`
+* Default: `'4.30.0'`
 
 `apache_httpd__skip_mod_security_coreruleset`
 
@@ -893,7 +901,7 @@ Example:
 ```yaml
 # optional - mod_security
 apache_httpd__mod_security_coreruleset_url: 'https://github.com/coreruleset/coreruleset/archive'
-apache_httpd__mod_security_coreruleset_version: '4.27.0'
+apache_httpd__mod_security_coreruleset_version: '4.30.0'
 apache_httpd__skip_mod_security_coreruleset: true
 ```
 

roles/apache_httpd/defaults/main.yml

@@ -213,7 +213,7 @@ apache_httpd__mod_http2_window_size: 65535
 # get details from https://coreruleset.org/docs/deployment/install/
 # origin: https://github.com/coreruleset/coreruleset/releases
 apache_httpd__mod_security_coreruleset_url: 'https://github.com/coreruleset/coreruleset/archive'
-apache_httpd__mod_security_coreruleset_version: '4.27.0'
+apache_httpd__mod_security_coreruleset_version: '4.30.0'
 
 # mod_ssl
 apache_httpd__mod_ssl_ssl_use_stapling: 'off'

roles/apache_httpd/handlers/main.yml

@@ -16,6 +16,8 @@
     state: 'reloaded'
   when:
     - '__apache_httpd__service_state_result is not defined or __apache_httpd__service_state_result is not changed'
+    # a server with the PID file of an earlier role version cannot be reloaded, it gets restarted (see tasks)
+    - 'not (__apache_httpd__legacy_pid_file_result["stat"]["exists"] | d(false))'
     - 'apache_httpd__systemd_state != "stopped"'
 
 - name: 'apache_httpd: httpd -t; systemctl restart httpd'

roles/apache_httpd/meta/argument_specs.yml

@@ -279,7 +279,7 @@ argument_specs:
       apache_httpd__mod_security_coreruleset_version:
         type: 'str'
         required: false
-        default: '4.27.0'
+        default: '4.30.0'
         description: >-
           Version of the OWASP Core Rule Set to deploy.
 

roles/apache_httpd/tasks/main.yml

@@ -67,7 +67,7 @@
 
   - ansible.builtin.include_tasks: 'mod_security_coreruleset.yml'
   when:
-    - 'not apache_httpd__skip_mod_security_coreruleset'
+    - 'not apache_httpd__skip_mod_security_coreruleset | bool'
 
   tags:
     - 'apache_httpd'
@@ -77,6 +77,17 @@
 # Manage global Apache config
 - block:
 
+  # Before the role set PidFile, Apache on Debian and Ubuntu wrote its PID to the compiled-in
+  # default /run/apache2.pid, where apachectl no longer looks. `apachectl graceful` then takes
+  # the server for stopped, tries to start a second one and fails on the bound ports, so such a
+  # server has to be restarted once instead of reloaded.
+  - name: 'stat /run/apache2.pid'
+    ansible.builtin.stat:
+      path: '/run/apache2.pid'
+    register: '__apache_httpd__legacy_pid_file_result'
+    when:
+      - 'ansible_facts["os_family"] == "Debian"'
+
   - name: 'Create or update global Apache configuration {{ __apache_httpd__config_file }}'
     ansible.builtin.template:
       backup: true
@@ -88,6 +99,15 @@
     notify:
       - 'apache_httpd: httpd -t; systemctl reload httpd'
 
+  - name: 'Request a restart of {{ __apache_httpd__service_name }}, which runs with the PID file of an earlier role version'
+    ansible.builtin.debug:
+      msg: '/run/apache2.pid exists, so {{ __apache_httpd__service_name }} is restarted instead of reloaded'
+    changed_when: true # notify the restart handler; the reload handler skips this case
+    notify:
+      - 'apache_httpd: httpd -t; systemctl restart httpd'
+    when:
+      - '__apache_httpd__legacy_pid_file_result["stat"]["exists"] | d(false)'
+
   - name: 'Remove rpmnew / rpmsave (and Debian equivalents)'
     ansible.builtin.include_role:
       name: 'shared'