Monitoring Apache NiFi with Prometheus and Grafana

Apache NiFi is an open-source data processing and distribution system. Monitoring NiFi is important to ensure the stability and performance of your data flow. Prometheus and Grafana are popular open-source tools that can be used to monitor NiFi.

Here's a step-by-step guide to monitor NiFi using Prometheus and Grafana:

  1. Configure NiFi JMX metrics: NiFi exposes metrics through JMX (Java Management Extensions). To enable JMX metrics, you need to add the following properties to your NiFi configuration file (nifi.properties):

nifi.remote.input.host=0.0.0.0 nifi.remote.input.secure=false

  1. Install Prometheus: You can install Prometheus by downloading the pre-compiled binaries from the official website or by building from source. In this tutorial, I will use the pre-compiled binaries. Download the latest version of Prometheus from https://prometheus.io/download/.
  2. Start Prometheus: Once you have downloaded the Prometheus binary, you can start the Prometheus server using the following command:

./prometheus

By default, Prometheus will listen on port 9090, and you can access the Prometheus web interface by visiting http://localhost:9090 in your web browser.

  1. Configure Prometheus to scrape NiFi metrics: To configure Prometheus to scrape NiFi metrics, you need to add the following scrape configuration to your Prometheus configuration file (prometheus.yml):

scrape_configs:

- job_name: nifi

static_configs:

- targets: ['<NiFi hostname>:<NiFi JMX port>']

metrics_path: /jmx

params:

mbean:

name: "org.apache.nifi:type=Controller,name=Controller Status"

relabel_configs:

- source_labels: [__address__]

target_label: instance

- source_labels: [__param_mbean]

target_label: job

- source_labels: []

target_label: job

replacement: nifi

  1. Install Grafana: You can install Grafana by downloading the pre-compiled binaries from the official website or by building from source. In this tutorial, I will use the pre-compiled binaries. Download the latest version of Grafana from https://grafana.com/grafana/download.
  2. Start Grafana: Once you have downloaded the Grafana binary, you can start the Grafana server using the following command:

./grafana-server

By default, Grafana will listen on port 3000, and you can access the Grafana web interface by visiting http://localhost:3000 in your web browser.

  1. Add Prometheus as a data source in Grafana: To add Prometheus as a data source in Grafana, you need to log in to the Grafana web interface and follow these steps:
  • Click on the "Configuration" menu in the left-side panel.
  • Select "Data Sources."
  • Click the "Add data source" button.
  • Select "Prometheus" as the data source type.
  • Enter the URL for your Prometheus server (e.g., http://localhost:9090).

 

أحدث أقدم