How to configure Prometheus Federation for Flex appliances

Description

This article briefly introduces how to configure an external Prometheus server to federate metrics data from Flex appliances. Please contact Prometheus for any issues concerning external Prometheus servers.

Prerequisites

• Latest version of Flex Appliance (5.0+)
• External Prometheus Server

Create metrics token on Flex Appliance

From the Flex Appliance Console, create a predefined API token of the type Metrics. See the Flex Appliance Getting Started and Administration Guide for the instructions.

Update Prometheus configuration file

Locate the configuration file on the external Prometheus server (for example, /etc/prometheus/prometheus.yml). Under scrape_configs, add a new job block for each appliance.

For example:

  - job_name: 'flex_federate_1'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/metric/federate'
    params:
      'match[]':
        - '{job="node"}'
    scheme: 'https'
    tls_config:
      insecure_skip_verify: true
    bearer_token: METRICS_TOKEN_1
    static_configs:
      - targets:
        - 'HOST_1:443

Note: You cannot group multiple federate configurations into one job block because different Flex appliances have different metrics tokens, while the bearer credentials are bound to job names.

Restart the external Prometheus Server with the following command:

systemctl restart prometheus

Verify in the Prometheus web UI

• Click Status > Targets and check if the corresponding endpoint is up.
• Click Graph and try to search for some appliance metrics.

Note: The metrics that the federate endpoint returns become "untyped" due to a well-known issue with Prometheus Federation.

Verify with curl

On a UNIX-like system where the curl command is available, open a terminal and run the following commands:

host=<Flex_Appliance_Address>
url="https://$host/metric/federate?match[]={job='node'}"
token=<Metrics_Token> 
curl -k --globoff -H "Authorization: Bearer $token" $url

These commands should be able to retrieve the metrics data successfully.