beats: resolve the output hosts once instead of in every template

CONTRIBUTING.md

@@ -89,6 +89,14 @@ Modules document themselves through the `DOCUMENTATION`, `EXAMPLES` and `RETURN`
 
 Shared code under `plugins/module_utils/` has no such block, because `ansible-doc` does not read it. Those functions are described in [plugins/module_utils/README.md](plugins/module_utils/README.md), please keep it current when you change them.
 
+## Tags
+
+Every tag listed in a role README has to work on its own. `--tags <tag>` must not fail because a task it depends on was skipped.
+
+* A `set_fact` task that only builds variables and cannot fail gets `tags: always`. The variables then exist in every run, and the task does not need to be touched when other tasks that read them get new tags.
+* A task that can stop the run, such as `assert` or `fail`, gets the same tags as the tasks it protects.
+* A task that changes the host and registers a result other tasks read gets the tags of those tasks.
+
 ## Testing
 
 Besides testing your change against a real setup, the repository has Molecule scenarios that exercise the whole stack, and it checks for ansible-lint and yamllint errors. To run the linters locally there is a `makefile`:

changelogs/fragments/beats_output_hosts.yml

@@ -0,0 +1,18 @@
+breaking_changes:
+  - The beats role no longer takes ``beats_target_hosts``. When ``elasticstack_full_stack`` is
+    false, set ``beats_elasticsearch_hosts`` or ``beats_logstash_hosts`` instead, depending on the
+    output the Beats use. Without them the run stops with a message instead of shipping to
+    ``localhost``.
+
+minor_changes:
+  - The beats role takes ``beats_elasticsearch_hosts`` and ``beats_logstash_hosts``, so the two
+    output types can point at different hosts. When ``elasticstack_full_stack`` is true they can be
+    set as well, and otherwise get the hosts of the inventory groups named in
+    ``elasticstack_elasticsearch_group_name`` and ``elasticstack_logstash_group_name``.
+
+trivial:
+  - The three Beats templates read the resolved host lists instead of working the hosts out
+    themselves, which removes nine inventory lookups and the ``elasticstack_full_stack`` branch
+    around each of them.
+  - CONTRIBUTING.md describes which tags a task gets, so that every tag listed in a role README
+    works on its own.

molecule/beats_default/converge.yml

@@ -10,6 +10,8 @@
   vars:
     elasticstack_ci: true
     elasticstack_full_stack: false
+    beats_logstash_hosts:
+      - localhost
     elasticsearch_jna_workaround: true
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"

molecule/beats_extended/converge.yml

@@ -25,6 +25,8 @@
     elasticsearch_jna_workaround: true
     elasticsearch_disable_systemcallfilterchecks: true
     elasticstack_full_stack: false
+    beats_logstash_hosts:
+      - localhost
     elasticstack_no_log: false
     beats_filebeat_mysql_slowlog_input: true
     beats_auditbeat: true

roles/beats/README.md

@@ -39,7 +39,8 @@ Run only parts of the role with `--tags`:
 | `beats_filebeat`<br>**Type**: `bool`<br>**Default**: `true` | Install and manage Filebeat. |
 | `beats_auditbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Auditbeat. |
 | `beats_metricbeat`<br>**Type**: `bool`<br>**Default**: `false` | Install and manage Metricbeat. |
-| `beats_target_hosts`<br>**Type**: `list` of `str`<br>**Default**: `['localhost']` | Hosts the Beats ship to. Only used when the role runs standalone; with the other Elastic Stack roles the targets are determined automatically. |
+| `beats_elasticsearch_hosts`<br>**Type**: `list` of `str` | Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise the run stops. |
+| `beats_logstash_hosts`<br>**Type**: `list` of `str` | Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_logstash_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the logstash output, otherwise the run stops. |
 | `beats_fields`<br>**Type**: `list` of `str` | Global fields added to the log and syslog (tcp/udp) inputs (not to the mysql, journald or docker inputs), given as a list of "key: value" strings. This is the global counterpart to the per-input fields. Unset by default. See the Filebeat inputs documentation. |
 | `beats_logging`<br>**Type**: `str`<br>**Default**: `"file"` | Where the Beats log. Set to "file" to log into beats_logpath; any other value leaves the Beats built-in logging. |
 | `beats_loglevel`<br>**Type**: `str`<br>**Default**: `"info"` | Log level for all Beats. |

roles/beats/defaults/main.yml

@@ -4,8 +4,6 @@ beats_security: false
 beats_filebeat: true
 beats_auditbeat: false
 beats_metricbeat: false
-beats_target_hosts:
-  - localhost
 beats_logging: file
 beats_logpath: /var/log/beats
 beats_loglevel: info

roles/beats/meta/argument_specs.yml

@@ -40,14 +40,25 @@ argument_specs:
         description: Install and manage Metricbeat.
 
       # ----- General -----
-      beats_target_hosts:
+      beats_elasticsearch_hosts:
+        type: list
+        elements: str
+        description: >-
+          Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and
+          this variable is not set, the role fills it with the hosts of the inventory group
+          named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is
+          false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise
+          the run stops.
+
+      beats_logstash_hosts:
         type: list
         elements: str
-        default:
-          - localhost
         description: >-
-          Hosts the Beats ship to. Only used when the role runs standalone; with
-          the other Elastic Stack roles the targets are determined automatically.
+          Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this
+          variable is not set, the role fills it with the hosts of the inventory group
+          named in elasticstack_logstash_group_name. When elasticstack_full_stack is false,
+          set it yourself as soon as a Beat uses the logstash output, otherwise the run
+          stops.
 
       beats_fields:
         type: list

roles/beats/tasks/main.yml

@@ -65,16 +65,38 @@
     - renew_ca
     - renew_beats_cert
 
-- name: Check that the inventory has the groups the Beats outputs need
+- name: Set the Elasticsearch hosts the Beats ship to
+  ansible.builtin.set_fact:
+    beats_elasticsearch_hosts: >-
+      {{ groups[elasticstack_elasticsearch_group_name] | default([])
+         if elasticstack_full_stack | bool else [] }}
+  when: beats_elasticsearch_hosts is undefined
+  tags: always
+
+- name: Set the Logstash hosts the Beats ship to
+  ansible.builtin.set_fact:
+    beats_logstash_hosts: >-
+      {{ groups[elasticstack_logstash_group_name] | default([])
+         if elasticstack_full_stack | bool else [] }}
+  when: beats_logstash_hosts is undefined
+  tags: always
+
+- name: Check that every enabled Beat has hosts to ship to
   ansible.builtin.assert:
-    that: groups[beats_output_group] | default([]) | length > 0
+    that: __beats_output_hosts | length > 0
     fail_msg: >-
-      {{ item.variable }} is set to "{{ item.output }}", so the hosts come from the inventory
-      group "{{ beats_output_group }}", which does not exist or is empty. Add that group to
-      your inventory.
+      {{ item.variable }} is set to "{{ item.output }}", but {{ __beats_output_hosts_variable }} is
+      empty. {{ __beats_output_source }}
     quiet: true
   vars:
-    beats_output_group: "{{ elasticstack_logstash_group_name if item.output == 'logstash' else elasticstack_elasticsearch_group_name }}"
+    __beats_output_hosts: "{{ beats_logstash_hosts if item.output == 'logstash' else beats_elasticsearch_hosts }}"
+    __beats_output_hosts_variable: "{{ 'beats_logstash_hosts' if item.output == 'logstash' else 'beats_elasticsearch_hosts' }}"
+    __beats_output_group: "{{ elasticstack_logstash_group_name if item.output == 'logstash' else elasticstack_elasticsearch_group_name }}"
+    __beats_output_source: >-
+      {{ 'elasticstack_full_stack is true, so it is filled from the inventory group "'
+         ~ __beats_output_group ~ '" unless you set it.'
+         if elasticstack_full_stack | bool
+         else 'elasticstack_full_stack is false, so you have to set it.' }}
   loop:
     - variable: beats_filebeat_output
       enabled: "{{ beats_filebeat }}"
@@ -88,9 +110,13 @@
   loop_control:
     label: "{{ item.variable }}"
   when:
-    - elasticstack_full_stack | bool
     - item.enabled | bool
     - item.output in ['logstash', 'elasticsearch']
+  tags:
+    - configuration
+    - beats_configuration
+    - beats_filebeat_configuration
+    - beats_auditbeat_configuration
 
 - name: Import Filebeat tasks
   ansible.builtin.import_tasks: filebeat.yml

roles/beats/templates/auditbeat.yml.j2

@@ -41,31 +41,19 @@ setup.kibana:
 {% if beats_auditbeat_output == "elasticsearch" %}
 output.elasticsearch:
 {% if beats_security | bool %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"
   ssl.enabled: true
   ssl.verification_mode: none
   ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"]
 {% else %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
 {% endif %}
 {% endif %}
 {% if beats_auditbeat_output == "logstash" %}
 output.logstash:
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   loadbalance: {{ beats_auditbeat_loadbalance }}
 {% if beats_security | bool %}
   ssl.enabled: true

roles/beats/templates/filebeat.yml.j2

@@ -119,31 +119,19 @@ setup.kibana:
 {% if beats_filebeat_output == "elasticsearch" %}
 output.elasticsearch:
 {% if beats_security | bool %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"
   ssl.enabled: true
   ssl.verification_mode: none
   ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"]
 {% else %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
 {% endif %}
 {% endif %}
 {% if beats_filebeat_output == "logstash" %}
 output.logstash:
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   loadbalance: {{ beats_filebeat_loadbalance }}
 {% if beats_security | bool %}
   ssl.enabled: true

roles/beats/templates/metricbeat.yml.j2

@@ -16,31 +16,19 @@ setup.kibana:
 {% if beats_metricbeat_output == "elasticsearch" %}
 output.elasticsearch:
 {% if beats_security | bool %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   username: "elastic"
   password: "{{ beats_writer_password.stdout }}"
   ssl.enabled: true
   ssl.verification_mode: none
   ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"]
 {% else %}
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
 {% endif %}
 {% endif %}
 {% if beats_metricbeat_output == "logstash" %}
 output.logstash:
-{% if elasticstack_full_stack | bool %}
-  hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% else %}
-  hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
-{% endif %}
+  hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}]
   loadbalance: {{ beats_metricbeat_loadbalance }}
 {% if beats_security | bool %}
   ssl.enabled: true