Install and Setup Prometheus on Ubuntu 20.04

In this guide, we are going to learn how to install Prometheus on Ubuntu 20.04. Prometheus is an open-source systems and service monitoring system. It collects metrics from configured targets via HTTP calls at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some conditions are met.

Some of the main features of Prometheus include;

• a multi-dimensional data model with time series data identified by metric name and key/value pairs
• PromQL, a flexible query language to leverage this dimensionality
• no reliance on distributed storage; single server nodes are autonomous
• time series collection happens via a pull model over HTTP
• pushing time series is supported via an intermediary gateway
• targets are discovered via service discovery or static configuration
• multiple modes of graphing and dashboarding support

Installing Prometheus on Ubuntu 20.04

Prometheus is made up various components;

• The main Prometheus server which scrapes and stores time series data
• Client libraries for instrumenting application code
• Push gateway for supporting short-lived jobs
• Exporters for exporting existing metrics from third-party systems as Prometheus metrics in cases where it is not feasible to instrument a given system with Prometheus metrics directly for example in services like HAProxy, StatsD, Graphite, etc.
• Alertmanager to handle alerts.

Prometheus is available on the default Ubuntu 20.04 repos;

apt-cache policy prometheus

prometheus:
  Installed: (none)
  Candidate: 2.15.2+ds-2
  Version table:
     2.15.2+ds-2 500
        500 http://ke.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

However, as you might have noted, the Prometheus release versions provided by the default focal repos is not up-to-date. The version 2.18 is current stable release version of Prometheus.

Install Prometheus Using Pre-compiled Binaries on Ubuntu 20.04

To ensure that you are installing the latest release version of Prometheus, use the pre-compiled binaries which can be downloaded directly from Prometheus downloads section.

Prerequisites

Before you can go ahead and install Prometheus using pre-compiled binaries on Ubuntu 20.04;

Create Prometheus System User and Group

Run the command below to create Prometheus system user and group;

useradd -M -r -s /bin/false prometheus

Create Prometheus Directories

Next, you need to create the directories that will be used to store Prometheus configurations files and other data.

mkdir /etc/prometheus /var/lib/prometheus

Download Prometheus Binary

Next, navigate to Prometheus downloads section and grab the latest version of Prometheus. You simply use wget to download the Prometheus binary for Linux as shown below;

wget https://github.com/prometheus/prometheus/releases/download/v2.18.1/prometheus-2.18.1.linux-amd64.tar.gz

Verify the integrity of the binary file downloaded by calculating the SHA256 hash.

sha256sum prometheus-2.18.1.linux-amd64.tar.gz

5fcc35b78bd0a1b84afae6de94248a4bea3cdb4daf0d54a37b5491cb86b014d7 prometheus-2.18.1.linux-amd64.tar.gz

Compare the resulting hash with the hash provided on the downloads page. Ensure that they match.

Install Prometheus on Ubuntu 20.04

Extract the downloaded Prometheus binary;

tar xzf prometheus-2.18.1.linux-amd64.tar.gz

Copy the prometheus and promtool binaries under the extracted Prometheus archive folder to /usr/local/bin directory.

cp prometheus-2.18.1.linux-amd64/{prometheus,promtool} /usr/local/bin/

After copying, set the user and group ownership of these binaries to prometheus.

chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}

Next, copy the consoles and console_libraries directories to Prometheus configuration directory, /etc/prometheus.

cp -r prometheus-2.18.1.linux-amd64/{consoles,console_libraries} /etc/prometheus/

Create Prometheus Configuration file

A sample Prometheus configuration file is available on the extracted archive folder. To make our work easier, just copy it to Prometheus configuration directory.

cp prometheus-2.18.1.linux-amd64/prometheus.yml /etc/prometheus/

Modify the configuration file to suit your needs. In this case, we just go with the defaults.

vim /etc/prometheus/prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

In the default configuration there is only a single job, called prometheus , which scrapes the time series data exposed by the Prometheus server. The job contains a single, statically configured, target, the localhost on port 9090.

Next, set the user and group ownership of Prometheus configuration directory, /etc/prometheus to prometheus.

chown -R prometheus:prometheus /etc/prometheus

Once that is done, similarly set the user and group ownership of Prometheus data directory, /var/lib/prometheus/ to prometheus.

chown prometheus:prometheus /var/lib/prometheus

Running Prometheus

At the very least, Prometheus is now set and is ready to run. However, at this point we do not have the Prometheus service configuration file and hence, we can run it as shown below;

prometheus --config.file=/etc/prometheus/prometheus.yml

...
level=info ts=2020-05-29T08:32:47.317Z caller=main.go:678 msg="Starting TSDB …"
level=info ts=2020-05-29T08:32:47.320Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile"
level=info ts=2020-05-29T08:32:47.320Z caller=web.go:523 component=web msg="Start listening for connections" address=0.0.0.0:9090
...
level=info ts=2020-05-29T08:32:47.326Z caller=main.go:695 msg="TSDB started"
level=info ts=2020-05-29T08:32:47.326Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-05-29T08:32:47.731Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-05-29T08:32:47.731Z caller=main.go:646 msg="Server is ready to receive web requests."

Accessing Prometheus from Web Interface

Open Prometheus port on firewall (UFW) if it is running. By default, it listens on TCP port 9090.

ufw allow 9090/tcp

Prometheus is now ready to receive web requests. You can access it from the browser using the address, http://server-IP-or-Hostname:9090.

To check the status of your node, navigate to Status > Targets.

To view scraped metrics, navigate to, http://<server_IP>:9090/metrics

To check memory statistics, for example, free memory available, select the go_memstats_frees_total query and click execute and see the result on the console tab.

To view the graph for the memory statistics, click Graph tab.

Create Prometheus Systemd Service File

To run Prometheus as a service, you can create a systemd service configuration file as shown below;

vim /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Next, reload systemd configuration files and start and enable Prometheus to run on system boot.

systemctl daemon-reload
systemctl enable --now prometheus

To check the status of Prometheus service;

systemctl status prometheus

● prometheus.service - Prometheus Time Series Collection and Processing Server
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-05-29 08:56:32 UTC; 18s ago
   Main PID: 105588 (prometheus)
      Tasks: 6 (limit: 2281)
     Memory: 17.1M
     CGroup: /system.slice/prometheus.service
             └─105588 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/promethe>

May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.655Z caller=main.go:678 msg="Starting TSDB ..."
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.657Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may tak>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.657Z caller=web.go:523 component=web msg="Start listening for connectio>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.660Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.660Z caller=head.go:627 component=tsdb msg="WAL replay completed" durat>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.662Z caller=main.go:694 fs_type=EXT4_SUPER_MAGIC
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.662Z caller=main.go:695 msg="TSDB started"
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.662Z caller=main.go:799 msg="Loading configuration file" filename=/etc/>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.710Z caller=main.go:827 msg="Completed loading of configuration file" f>
May 29 08:56:33 freeradius.kifarunix-demo.com prometheus[105588]: level=info ts=2020-05-29T08:56:33.711Z caller=main.go:646 msg="Server is ready to receive web requests."

Further Reading;

Prometheus Getting Started

Other Related Tutorials

Monitor Linux System Metrics with Prometheus Node Exporter

Install Graylog 3.0 on CentOS 7

Monitor Squid Access Logs with Graylog Server

Monitor Squid logs with Grafana and Graylog