diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1dcd9b59..e04d918b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -89,6 +89,14 @@ Modules document themselves through the `DOCUMENTATION`, `EXAMPLES` and `RETURN` Shared code under `plugins/module_utils/` has no such block, because `ansible-doc` does not read it. Those functions are described in [plugins/module_utils/README.md](plugins/module_utils/README.md), please keep it current when you change them. +## Tags + +Every tag listed in a role README has to work on its own. `--tags ` must not fail because a task it depends on was skipped. + +* A `set_fact` task that only builds variables and cannot fail gets `tags: always`. The variables then exist in every run, and the task does not need to be touched when other tasks that read them get new tags. +* A task that can stop the run, such as `assert` or `fail`, gets the same tags as the tasks it protects. +* A task that changes the host and registers a result other tasks read gets the tags of those tasks. + ## Testing Besides testing your change against a real setup, the repository has Molecule scenarios that exercise the whole stack, and it checks for ansible-lint and yamllint errors. To run the linters locally there is a `makefile`: diff --git a/changelogs/fragments/beats_output_hosts.yml b/changelogs/fragments/beats_output_hosts.yml new file mode 100644 index 00000000..976c7da3 --- /dev/null +++ b/changelogs/fragments/beats_output_hosts.yml @@ -0,0 +1,18 @@ +breaking_changes: + - The beats role no longer takes ``beats_target_hosts``. When ``elasticstack_full_stack`` is + false, set ``beats_elasticsearch_hosts`` or ``beats_logstash_hosts`` instead, depending on the + output the Beats use. Without them the run stops with a message instead of shipping to + ``localhost``. + +minor_changes: + - The beats role takes ``beats_elasticsearch_hosts`` and ``beats_logstash_hosts``, so the two + output types can point at different hosts. When ``elasticstack_full_stack`` is true they can be + set as well, and otherwise get the hosts of the inventory groups named in + ``elasticstack_elasticsearch_group_name`` and ``elasticstack_logstash_group_name``. + +trivial: + - The three Beats templates read the resolved host lists instead of working the hosts out + themselves, which removes nine inventory lookups and the ``elasticstack_full_stack`` branch + around each of them. + - CONTRIBUTING.md describes which tags a task gets, so that every tag listed in a role README + works on its own. diff --git a/molecule/beats_default/converge.yml b/molecule/beats_default/converge.yml index 1d984bfd..9b412799 100644 --- a/molecule/beats_default/converge.yml +++ b/molecule/beats_default/converge.yml @@ -10,6 +10,8 @@ vars: elasticstack_ci: true elasticstack_full_stack: false + beats_logstash_hosts: + - localhost elasticsearch_jna_workaround: true elasticsearch_disable_systemcallfilterchecks: true elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}" diff --git a/molecule/beats_extended/converge.yml b/molecule/beats_extended/converge.yml index 449fbc5d..1ef8f67b 100644 --- a/molecule/beats_extended/converge.yml +++ b/molecule/beats_extended/converge.yml @@ -25,6 +25,8 @@ elasticsearch_jna_workaround: true elasticsearch_disable_systemcallfilterchecks: true elasticstack_full_stack: false + beats_logstash_hosts: + - localhost elasticstack_no_log: false beats_filebeat_mysql_slowlog_input: true beats_auditbeat: true diff --git a/roles/beats/README.md b/roles/beats/README.md index 18d43373..7287c99f 100644 --- a/roles/beats/README.md +++ b/roles/beats/README.md @@ -39,7 +39,8 @@ Run only parts of the role with `--tags`: | `beats_filebeat`
**Type**: `bool`
**Default**: `true` | Install and manage Filebeat. | | `beats_auditbeat`
**Type**: `bool`
**Default**: `false` | Install and manage Auditbeat. | | `beats_metricbeat`
**Type**: `bool`
**Default**: `false` | Install and manage Metricbeat. | -| `beats_target_hosts`
**Type**: `list` of `str`
**Default**: `['localhost']` | Hosts the Beats ship to. Only used when the role runs standalone; with the other Elastic Stack roles the targets are determined automatically. | +| `beats_elasticsearch_hosts`
**Type**: `list` of `str` | Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise the run stops. | +| `beats_logstash_hosts`
**Type**: `list` of `str` | Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this variable is not set, the role fills it with the hosts of the inventory group named in elasticstack_logstash_group_name. When elasticstack_full_stack is false, set it yourself as soon as a Beat uses the logstash output, otherwise the run stops. | | `beats_fields`
**Type**: `list` of `str` | Global fields added to the log and syslog (tcp/udp) inputs (not to the mysql, journald or docker inputs), given as a list of "key: value" strings. This is the global counterpart to the per-input fields. Unset by default. See the Filebeat inputs documentation. | | `beats_logging`
**Type**: `str`
**Default**: `"file"` | Where the Beats log. Set to "file" to log into beats_logpath; any other value leaves the Beats built-in logging. | | `beats_loglevel`
**Type**: `str`
**Default**: `"info"` | Log level for all Beats. | diff --git a/roles/beats/defaults/main.yml b/roles/beats/defaults/main.yml index 8d1bd6c7..f80ed9ea 100644 --- a/roles/beats/defaults/main.yml +++ b/roles/beats/defaults/main.yml @@ -4,8 +4,6 @@ beats_security: false beats_filebeat: true beats_auditbeat: false beats_metricbeat: false -beats_target_hosts: - - localhost beats_logging: file beats_logpath: /var/log/beats beats_loglevel: info diff --git a/roles/beats/meta/argument_specs.yml b/roles/beats/meta/argument_specs.yml index ab9988e6..dc2b960e 100644 --- a/roles/beats/meta/argument_specs.yml +++ b/roles/beats/meta/argument_specs.yml @@ -40,14 +40,25 @@ argument_specs: description: Install and manage Metricbeat. # ----- General ----- - beats_target_hosts: + beats_elasticsearch_hosts: + type: list + elements: str + description: >- + Elasticsearch hosts the Beats ship to. When elasticstack_full_stack is true and + this variable is not set, the role fills it with the hosts of the inventory group + named in elasticstack_elasticsearch_group_name. When elasticstack_full_stack is + false, set it yourself as soon as a Beat uses the elasticsearch output, otherwise + the run stops. + + beats_logstash_hosts: type: list elements: str - default: - - localhost description: >- - Hosts the Beats ship to. Only used when the role runs standalone; with - the other Elastic Stack roles the targets are determined automatically. + Logstash hosts the Beats ship to. When elasticstack_full_stack is true and this + variable is not set, the role fills it with the hosts of the inventory group + named in elasticstack_logstash_group_name. When elasticstack_full_stack is false, + set it yourself as soon as a Beat uses the logstash output, otherwise the run + stops. beats_fields: type: list diff --git a/roles/beats/tasks/main.yml b/roles/beats/tasks/main.yml index 1be84161..3f5c5717 100644 --- a/roles/beats/tasks/main.yml +++ b/roles/beats/tasks/main.yml @@ -65,16 +65,38 @@ - renew_ca - renew_beats_cert -- name: Check that the inventory has the groups the Beats outputs need +- name: Set the Elasticsearch hosts the Beats ship to + ansible.builtin.set_fact: + beats_elasticsearch_hosts: >- + {{ groups[elasticstack_elasticsearch_group_name] | default([]) + if elasticstack_full_stack | bool else [] }} + when: beats_elasticsearch_hosts is undefined + tags: always + +- name: Set the Logstash hosts the Beats ship to + ansible.builtin.set_fact: + beats_logstash_hosts: >- + {{ groups[elasticstack_logstash_group_name] | default([]) + if elasticstack_full_stack | bool else [] }} + when: beats_logstash_hosts is undefined + tags: always + +- name: Check that every enabled Beat has hosts to ship to ansible.builtin.assert: - that: groups[beats_output_group] | default([]) | length > 0 + that: __beats_output_hosts | length > 0 fail_msg: >- - {{ item.variable }} is set to "{{ item.output }}", so the hosts come from the inventory - group "{{ beats_output_group }}", which does not exist or is empty. Add that group to - your inventory. + {{ item.variable }} is set to "{{ item.output }}", but {{ __beats_output_hosts_variable }} is + empty. {{ __beats_output_source }} quiet: true vars: - beats_output_group: "{{ elasticstack_logstash_group_name if item.output == 'logstash' else elasticstack_elasticsearch_group_name }}" + __beats_output_hosts: "{{ beats_logstash_hosts if item.output == 'logstash' else beats_elasticsearch_hosts }}" + __beats_output_hosts_variable: "{{ 'beats_logstash_hosts' if item.output == 'logstash' else 'beats_elasticsearch_hosts' }}" + __beats_output_group: "{{ elasticstack_logstash_group_name if item.output == 'logstash' else elasticstack_elasticsearch_group_name }}" + __beats_output_source: >- + {{ 'elasticstack_full_stack is true, so it is filled from the inventory group "' + ~ __beats_output_group ~ '" unless you set it.' + if elasticstack_full_stack | bool + else 'elasticstack_full_stack is false, so you have to set it.' }} loop: - variable: beats_filebeat_output enabled: "{{ beats_filebeat }}" @@ -88,9 +110,13 @@ loop_control: label: "{{ item.variable }}" when: - - elasticstack_full_stack | bool - item.enabled | bool - item.output in ['logstash', 'elasticsearch'] + tags: + - configuration + - beats_configuration + - beats_filebeat_configuration + - beats_auditbeat_configuration - name: Import Filebeat tasks ansible.builtin.import_tasks: filebeat.yml diff --git a/roles/beats/templates/auditbeat.yml.j2 b/roles/beats/templates/auditbeat.yml.j2 index 2485e741..b2a688f1 100644 --- a/roles/beats/templates/auditbeat.yml.j2 +++ b/roles/beats/templates/auditbeat.yml.j2 @@ -41,31 +41,19 @@ setup.kibana: {% if beats_auditbeat_output == "elasticsearch" %} output.elasticsearch: {% if beats_security | bool %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] username: "elastic" password: "{{ beats_writer_password.stdout }}" ssl.enabled: true ssl.verification_mode: none ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"] {% else %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] {% endif %} {% endif %} {% if beats_auditbeat_output == "logstash" %} output.logstash: -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] loadbalance: {{ beats_auditbeat_loadbalance }} {% if beats_security | bool %} ssl.enabled: true diff --git a/roles/beats/templates/filebeat.yml.j2 b/roles/beats/templates/filebeat.yml.j2 index a7a89cfc..453d8def 100644 --- a/roles/beats/templates/filebeat.yml.j2 +++ b/roles/beats/templates/filebeat.yml.j2 @@ -119,31 +119,19 @@ setup.kibana: {% if beats_filebeat_output == "elasticsearch" %} output.elasticsearch: {% if beats_security | bool %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] username: "elastic" password: "{{ beats_writer_password.stdout }}" ssl.enabled: true ssl.verification_mode: none ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"] {% else %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] {% endif %} {% endif %} {% if beats_filebeat_output == "logstash" %} output.logstash: -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] loadbalance: {{ beats_filebeat_loadbalance }} {% if beats_security | bool %} ssl.enabled: true diff --git a/roles/beats/templates/metricbeat.yml.j2 b/roles/beats/templates/metricbeat.yml.j2 index 2218fd15..373146ab 100644 --- a/roles/beats/templates/metricbeat.yml.j2 +++ b/roles/beats/templates/metricbeat.yml.j2 @@ -16,31 +16,19 @@ setup.kibana: {% if beats_metricbeat_output == "elasticsearch" %} output.elasticsearch: {% if beats_security | bool %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"https://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] username: "elastic" password: "{{ beats_writer_password.stdout }}" ssl.enabled: true ssl.verification_mode: none ssl.certificate_authorities: ["/etc/beats/certs/ca.crt"] {% else %} -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_elasticsearch_group_name] %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_elasticsearch_hosts %}"http://{{ host }}:{{ elasticstack_elasticsearch_http_port }}"{% if not loop.last %},{% endif %}{% endfor %}] {% endif %} {% endif %} {% if beats_metricbeat_output == "logstash" %} output.logstash: -{% if elasticstack_full_stack | bool %} - hosts: [ {% for host in groups[elasticstack_logstash_group_name] %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% else %} - hosts: [ {% for host in beats_target_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] -{% endif %} + hosts: [ {% for host in beats_logstash_hosts %}"{{ host }}:{{ elasticstack_beats_port }}"{% if not loop.last %},{% endif %}{% endfor %}] loadbalance: {{ beats_metricbeat_loadbalance }} {% if beats_security | bool %} ssl.enabled: true