Prometheus Blackbox exporter - Configure and deploy

Prometheus blackbox exporter is a probing tool that can be used to monitor web endpoints (HTTP, SMTP etc) and determine their status.

The simplest way to use it is to deploy a Docker image with the desired configuration to run alongside with the rest of your Prometheus containers. Even simpler when Ansible is used to make sure the action is idempotent, and can be redeployed anytime without all the fuss of configuration from scratch.

We will use the new syntax for Ansible modules (from 2.10 on), however the example should be cross-compatible with older versions as well.

Required elements and tools for this task:

• Ansible

Files that need to be created:

• Ansible
  • files/blackbox.yml (contains the exporter modules’ configuration)
  • role/tasks/blackbox.yml (task to deploy the blackbox exporter container)
  • modify your prometheus.yml with the job configuration for blackbox

First, the exporter configuration. A base configuration for HTTP checks:

modules:
  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ipv4"

Then we need to deploy the blackbox exporter itself (Docker deployment, there are plenty of other ways to do it of course).

The Ansible task needed to complete this is:

- name: Deploy  blackbox-exporter container
  community.general.docker_container:
    name: blackbox-exporter
    container_default_behavior: compatibility
    image: prom/blackbox-exporter:latest
    volumes:
      - /files/blackbox.yml:/etc/prometheus/blackbox-exporter.yml:ro
    pull: yes
    restart: yes
    state: started
    recreate: yes
    restart_policy: "always"
    ports:
      - "9115:9115"
    networks:
      - name: prometheus-net # note to keep network configuration the same accross containers

The final step is to modify prometheus.yml and reload its configuration.

Add:

  - job_name: blackbox
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://aorfanos.com
          - https://blog.aorfanos.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

In the above example, note replacement: blackbox-exporter:9115. This is needed in order to scrape the right host in the Docker network (as of Docker 1.10 we can address containers by name).

If we were to deploy directly on a host, we would need a directive addressing the host the exporter runs in, like replacement: 127.0.0.1:9115.

Note: It is even better if prometheus.yml is templated so that targets are auto-generated. Makes adding new hosts to be monitored less messy as a process.

If you decide to go this route, prometheus.yml should now be in templates/prometheus.yml.j2, pulling variables from a list like:

prometheus:
  blackbox:
    http:
      targets:
        - "https://aorfanos.com"
        - "https://blog.aorfanos.com"

, and the blackbox segment in prometheus.yml should look like the Jinja2 below.

It is better to use a Jinja macro in the beginning of the file, so we can mass indent the targets:

{% macro render_blackbox_http_targets(target) %}
{% for target in prometheus.blackbox.http.targets %}
- {{ target }}
{% endfor %}
{% endmacro %}

- job_name: blackbox
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
{{ render_blackbox_http_targets(target)| indent(6, true) }}

Reload Prometheus by running (remember to enable the Lifecycle API): curl -X POST http://localhost:9090/-/reload

This is all. The Prometheus blackbox exporter should be up and running. Check by running:

alex@prometheus-vm:~# curl --silent localhost:9090/targets | grep 9115
              <a href="http://blackbox-exporter:9115/probe?module=http_2xx&amp;target=https%3A%2F%2Fblog.aorfanos.com">http://blackbox-exporter:9115/probe</a><br>