How to Monitor Remote Linux Hosts and Services with Icinga 2 on Ubuntu 18.04

0
6417

In this very tutorial, you are going learn how to monitor hosts and services with Icinga 2 on Ubuntu 18.04 LTS. Well, Icinga 2 can be used to monitor the availability of hosts and services. Hosts and services can be virtually anything; Network services (HTTP, SMTP, SNMP, SSH, etc.),Printers, Switches or routers, Temperature sensors, or any other local or network-accessible services.

In our last tutorial, we learnt how to install Icinga 2 and Icinga Web 2 on Ubuntu 18.04 LTS.

Hosts and services can be monitored in two ways;

  • Use of Simple Host Monitoring
    This involves the use of services that are accessible across the local network or the Internet for example include HTTP, SMTP, SNMP, SSH, and ICMP etc. This method is fit for monitoring hosts on which monitoring agents cannot be installed.
  • Use of Monitoring Agents
    In this method, an agent is required in order to monitor host and services like CPU load, disk space, memory and the running process. In this configuration, Icinga 2 acts as the master node and the remote host acts as the client node.

Simple Host Monitoring

For simple host monitoring, we are going to learn how to monitor state of the host and services like HTTP and SSH.

Hosts can be in any one of the following states:

  • UP – which means the host is available.
  • DOWN – which means the host is not available.

Services can be in any one of the following states:

  • OK – The service is working properly.
  • WARNING – The service is experiencing some problems but is still considered to be in working condition.
  • CRITICAL – The service is in a critical state.
  • UNKNOWN – The check could not determine the service’s state.

To begin with, you can create a configuration file for your hosts, if you like, and add services to be monitored as shown below.

vim /etc/icinga2/conf.d/myhosts.conf
object Host "mydevapp.example.com" {
  import "generic-host"
  address = "192.168.43.214"
  vars.os = "Linux"

  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  }

import keyword is used to import the generic-host template which takes care of setting up the host check command to hostalive. If you require a different check command, you can override it in the object definition.

The vars attribute can be used to define custom attributes which are available for checks for example checking for ssh service as defined in the services.conf file for all systems whose OS is set to Linux.

vars.http_vhosts defines http vhost attributes for the http service apply rule defined in services.conf.

You can also define the services as shown below.

object Host "mydevapp.example.com" {
address = "192.168.43.214"
check_command = "hostalive"
}

object Service "http" {
host_name = "mydevapp.example.com"
check_command = "http"
}
object Service "ssh" {
host_name = "mydevapp.example.com"
check_command = "ssh"
}

Once you are done with host definition save the configuration file and run the following command to check for syntactical errors.

icinga2 daemon -C

If all is well, restart icinga2 service

systemctl restart icinga2

Now login to the Icinga Web 2 dashboard and should be able to see your services.

Monitor http, ssh and heartbeat for Linux Server

Agent-based Monitoring

Agent based monitoring becomes handy when remote services are not directly accessible through the network. While there are certain mechanisms to ensure a secure communication between all nodes (firewalls, policies, software hardening, etc.), Icinga 2 imposes mandatory SSL communication between nodes. The communication between the master and client nodes happens on port 5665 and thus, if firewall is running, this port has to be opened.

Our environment comprises of;

  • A single master node, icinga.example.com running Ubuntu 18.04 LTS
  • Two client nodes; server01.example.com, mydevapp.example.com running CentOS 7 and Ubuntu 18.04 respectively.

Set up Icinga 2 Master Node

Run the command below to setup a Icinga 2 master node.

icinga2 node wizard

This will prompt you if  it is a Satellite or Client setup. Choose n to select a master setup.

Press Enter to accept the defaults.

Icinga Master Node Setup Wizard

Once that is done, restart Icinga 2 daemon to finalize setup.

systemctl restart icinga2

If firewall is running, open port 5665.

ufw allow 5665

Generate a client ticket for client nodes

A client which sends a certificate signing request (CSR) must authenticate itself in a trusted way. The master generates a client ticket which is included in this request. That way the master can verify that the request matches the previously trusted ticket and sign the request.

Run the commands below on the master node to generate the tickets for your client nodes;

For client node, server01.example.com;

icinga2 pki ticket --cn server01.example.com
970a747db8d676fd2057e3ede2d0e9b5e6ae0dfd

For client node, mydevapp.example.com;

icinga2 pki ticket --cn mydevapp.example.com
99030d0af2c9cb27696cbe3ca3c40a6b043bd487

Save these keys as you will require them later while setting up the client nodes.

Configure Client Nodes

Note that before you can set up the client nodes, Icinga 2 on the master node must be running and accepting connections on port 5665.

ss -altnp | grep 5665
LISTEN 0 128 0.0.0.0:5665 0.0.0.0:* users:(("icinga2",pid=18872,fd=14))

Setting up Client Nodes

Login in to the client nodes and Install the Icinga 2 package and setup the required plugins as follows;

Add the Icinga repository to your package management configuration

For CentOS 7, server01.example.com

yum install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm -y

Setup EPEL Repository

yum install epel-release -y

For Ubuntu 18.04 LTS, mydevapp..example.com

wget -O - https://packages.icinga.com/icinga.key | apt-key add -
echo "deb http://packages.icinga.com/ubuntu icinga-bionic main" > /etc/apt/sources.list.d/icinga2.list
echo "deb-src http://packages.icinga.com/ubuntu icinga-bionic main" >> /etc/apt/sources.list.d/icinga2.list
apt update

Install Icinga2 and monitoring plugins

On CentOS 7;

yum install icinga2 nagios-plugins-all -y

On Ubuntu 18.04 LTS;

apt install icinga2 monitoring-plugins -y

Once the installation is done, start and enable icinga2 daemon to run on boot.

On CentOS 7:

systemctl start icinga2
systemctl enable icinga2

On Ubuntu 18.04, icinga 2 is started and enabled upon installation. You can check this by running;

systemctl status icinga2
systemctl is-enabled icinga2

Next, Set up the client nodes as shown below.

For CentOS 7 Client Node;

icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: y <- Enter y to setup client

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [server01.example.com]: ENTER

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): icinga.example.com

Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 192.168.43.138 < IP of the master node
Master/Satellite endpoint port [5665]: ENTER

Add more master/satellite endpoints? [y/N]: n
Parent certificate information:

Subject: CN = icinga.example.com
Issuer: CN = Icinga CA
Valid From: Oct 13 08:47:58 2018 GMT
Valid Until: Oct 9 08:47:58 2033 GMT
Fingerprint: 64 A7 AA F8 C0 F4 A8 A1 FE 95 9D 1E DC 86 07 3B E8 1F 97 41

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
(Hint: # icinga2 pki ticket --cn 'server01.example.com'): 970a747db8d676fd2057e3ede2d0e9b5e6ae0dfd  < ticket generated before
Please specify the API bind host/port (optional):
Bind Host []: ENTER
Bind Port []: ENTER

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Local zone name [server01.example.com]: ENTER
Parent zone name [master]: ENTER

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: n

Do you want to disable the inclusion of the conf.d directory [Y/n]: y
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

Repeat the same for the other client node. Be sure to replace the ticket with the respective one created above.

The setup wizard fetches the parent node’s certificate and ask you to verify that information. This is to prevent MITM attacks or any kind of untrusted parent relationship.

Once done, restart Icinga2 daemon as requested.

systemctl restart icinga2

If firewall is running, open port 5665.

On Ubuntu;

ufw allow 5665
ufw reload

On CentOS 7

firewall-cmd --add-port=5665/tcp --permanent
firewall-cmd --reload

You can verify that the master and the client node are now connected.

For client node, server01.example.com

ss | grep :5665
tcp ESTAB 0 0 192.168.43.63:33970 192.168.43.138:5665

For client node, mydevapp.example.com

ss | grep :5665
tcp ESTAB 0 0 192.168.43.214:43528 192.168.43.138:5665

Configure the Agent Monitoring on Icinga 2 Master

Now that the master nodes and the client nodes are connected, it is time to configure the master so it can execute a remote check on the client using the command endpoint.

Login to master node and create a zone directory where you can add the configuration files for the host and service objects you want to monitor.

mkdir /etc/icinga2/zones.d/master

Create the hosts configuration file for the clients.

vim /etc/icinga2/zones.d/master/hosts.conf
// for server01.example.com
object Endpoint "server01.example.com" {
}

object Zone "server01.example.com" {
endpoints = [ "server01.example.com" ]
parent = "master"
}
object Host "server01.example.com" {
check_command = "hostalive" //check is executed on the master
address = "192.168.43.63"

vars.client_endpoint = name //follows the convention that host name == endpoint name
}
// for mydevapp.example.com
object Endpoint "mydevapp.example.com" {
}

object Zone "mydevapp.example.com" {
endpoints = [ "mydevapp.example.com" ]
parent = "master"
}
object Host "mydevapp.example.com" {
check_command = "hostalive" 
address = "192.168.43.214"

vars.client_endpoint = name 
}

Create a services configuration file for your clients.

vim /etc/icinga2/zones.d/master/services.conf
// Check System Load
apply Service "System Load" {
  check_command = "load"
  command_endpoint = host.vars.client_endpoint // Check executed on client node
  assign where host.vars.client_endpoint
}

// Check number of running system Processes
apply Service "Process" {
  check_command = "procs"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

// Check number of Logged in Users
apply Service "Users" {
  check_command = "users"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

// Check System Disk Usage
apply Service "Disk" {
  check_command = "disk"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

// Check for SWAP memory Usage
apply Service "SWAP" {
  check_command = "swap"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

// SSH Service Check
apply Service "SSH Service" {
  check_command = "ssh"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint

}
apply Service "Ping" {
  check_command = "ping4"
  assign where host.address
}

Save the changes and validate the configuration on the master node:

icinga2 daemon -C

Restart the Icinga 2 daemon

systemctl restart icinga2

Login to Icinga Web 2 and confirm the checks.

server01.example.com checks
mydevapp.example.com checks

You have successfully configured Icinga 2 agent-based monitoring on Ubuntu 18.04. Happy monitoring!!.

Related Tutorials;

Install Icinga Web 2 on Debian 10 Buster

Install Icinga 2 on Debian 10 Buster

Install Nagios Core on Debian 10 Buster

Install Elastic Stack 7 on Fedora 30/Fedora 29/CentOS 7

Install Elasticsearch 7.x on Ubuntu 18.04/Debian 9.8

LEAVE A REPLY

Please enter your comment!
Please enter your name here