Monitor Linux System Metrics with ELK Stack

November 17, 2020

1734

Last updated on November 26th, 2020 at 11:51 pm

In this tutorial, you will learn how to monitor Linux system metrics with ELK Stack. ELK Stack, or if you like, Elastic stack can be used to monitor Linux system metrics with the aid of the Metricbeat. Metricbeat is a lightweight shipper for system metrics. It can collect system-level CPU usage, memory, file system, disk IO, and network IO statistics, as well as top-like statistics for every process running on your systems and ship them to Elasticsearch for indexing to enable visualization through the visualization tools like Kibana.

Monitor Linux System Metrics with ELK Stack

Install and Setup ELK Stack

In order to be able to monitor Linux system metrics with ELK Stack, you need to have an ELK stack already setup and running. The setup of ELK Stack is beyond the scope of this guide, however, you can follow the links below to install and setup ELK Stack

Install ELK Stack on Ubuntu 20.04

Installing ELK Stack on CentOS 8

Deploy a Single Node Elastic Stack Cluster on Docker Containers

Install Elastic Stack 7 on Fedora 30/Fedora 29/CentOS 7

Install Metricbeat in Linux Systems

In this tutorial we will be using a CentOS 8 and an Ubuntu 20.04 systems for demo.

Install Metricbeat on Ubuntu 20.04

You can install Metricbeat from the official Elastic repos or directly using the Metricbeat DEB binary. We use the former method in this setup.

Install Elastic stack repo Public Signing Key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Install Elastic stack repo on Ubuntu 20.04;

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Update package cache;

apt update

Install Metricbeat on Ubuntu 20.04;

apt install metricbeat

Install Metricbeat on CentOS 8

Similarly, you can install Metricbeat on CentOS 8 using the RPM binary or directly from the Elastic stack repos. For convenience sake, we install it from the Elastic stack repo, hence;

Install Elastic stack repo Public Signing Key:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Install Elastic stack repo on CentOS 8;

cat << EOL > /etc/yum.repos.d/elasticstack.repo 
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOL

Update package cache;

dnf update

Install Metricbeat on Ubuntu 20.04;

dnf install metricbeat

Configure Metricbeat to collect various system metrics

Once the installation of metric is done, configure it to collect various system metrics and ship them to either Elasticsearch or Logstash depending on your setup.

Configure Metricbeat in Linux

To begin with, configure Metricbeat output. In this setup, we are sending Metricbeat data directly to Elasticsearch running on a remote host. As such, edit the Metricbeat configuration file and define Elasticsearch output.

vim /etc/metricbeat/metricbeat.yml

...
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  hosts: ["192.168.57.30:9200"]

If you are sending the metric data to Logstash instead, comment the Elasticsearch output configs and enable Logstash output as follows;

...
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  #hosts: ["192.168.57.30:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: ["192.168.57.30:5044"]

Configure Kibana Endpoint

If you will want to use Metricbeat prebuilt Kibana dashboards and both Kibana and Elasticsearch are running on separate hosts, then you will need to load the dashboard manually. For this reason, you need to configure Kibana endpoint. Otherwise, if Kibana and Elasticsearch are running on the same host, you don’t need to configure Kibana endpoint and can safely ignore this configuration.

vim /etc/metricbeat/metricbeat.yml

...
# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  #host: "localhost:5601"
  host: "192.168.57.30:5601"

Configure Metricbeat logging

To ensure that Metricbeat is logging to its specific file instead of the syslog (/var/log/syslog, /var/log/messages), enter the following lines into the configuration file;

# ================================= Logging ====================================
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644

Save and exit the configuration.

Enable and configure metrics collection modules

Metricbeat ships with a dozen modules that enabled it to collect specific service metrics. You can list modules by running the command below;

metricbeat modules list

The command shows enabled and disabled modules;

Enabled:
system

Disabled:
activemq
aerospike
apache
...

To enable a module, simply run;

metricbeat modules enable <module-name>

Where a module-name you can obtain from the list above.

Similarly, to disable a module;

metricbeat modules disable <module-name>

In this setup, we are using the Metricbeat system module, which is already enabled. This module collects the following metricsets; cpu ,load ,memory ,network ,process ,process_summary ,socket_summary ,entropy ,core ,diskio ,socket ,service ,users.

If you want to modify the module, simply open the file, /etc/metricbeat/modules.d/system.yml, for editing.

Load Metric Kibana Dashboards/Elasticsearch Index Template

You can load the recommended index template for writing to Elasticsearch and deploy the sample dashboards for visualizing the Metricbeat data in Kibana by executing the command below;

metricbeat setup -e

If you are using Logstash or any other processing engines instead of using Elasticsearch as output, you need to load the index template manually into Elasticsearch. This requires a direct connection to Elasticsearch. The command below is used to load the index template manually to Elasticsearch (It temporarily disables Logstash Output).

metricbeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['192.168.57.30:9200'] -E setup.kibana.host=192.168.57.30:5601

Running Metricbeat in Linux System

Once the installation and configuration of Metricbeat is done, you can start and enable Metricbeat service to run on system boot;

systemctl enable --now metricbeat

To check the status;

systemctl status metricbeat

● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
     Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-11-26 20:03:35 UTC; 2s ago
       Docs: https://www.elastic.co/products/beats/metricbeat
   Main PID: 95897 (metricbeat)
      Tasks: 9 (limit: 2281)
     Memory: 23.6M
     CGroup: /system.slice/metricbeat.service
             └─95897 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metr>

Nov 26 20:03:35 solr.kifarunix-demo.com systemd[1]: Started Metricbeat is a lightweight shipper for metrics..

View Metricbeat Data in Kibana

Navigate to your Elastic stack dashboard and create Kibana Metricbeat index to enable you view and visualize the metric data. It should be already existing if you loaded with the metricbeat setup command above;

Metricbeat Raw Data

Under the Discover navigation, select metricbeat-* index to view the Metricbeat raw data. Adjust the time range accordingly.

Metricbeat System Overview Dashboard

In the side navigation, click Dashboard, then select the dashboard that you want to open. For example, [Metricbeat System] Host overview ECS.