In this very tutorial, you are going learn how to monitor hosts and services with Icinga 2 on Ubuntu 18.04 LTS. Well, Icinga 2 can be used to monitor the availability of hosts and services. Hosts and services can be virtually anything; Network services (HTTP, SMTP, SNMP, SSH, etc.),Printers, Switches or routers, Temperature sensors, or any other local or network-accessible services.
In our last tutorial, we learnt how to install Icinga 2 and Icinga Web 2 on Ubuntu 18.04 LTS.
Hosts and services can be monitored in two ways;
- Use of Simple Host Monitoring
This involves the use of services that are accessible across the local network or the Internet for example include HTTP, SMTP, SNMP, SSH, and ICMP etc. This method is fit for monitoring hosts on which monitoring agents cannot be installed. - Use of Monitoring Agents
In this method, an agent is required in order to monitor host and services like CPU load, disk space, memory and the running process. In this configuration, Icinga 2 acts as the master node and the remote host acts as the client node.
Simple Host Monitoring
For simple host monitoring, we are going to learn how to monitor state of the host and services like HTTP and SSH.
Hosts can be in any one of the following states:
- UP – which means the host is available.
- DOWN – which means the host is not available.
Services can be in any one of the following states:
- OK – The service is working properly.
- WARNING – The service is experiencing some problems but is still considered to be in working condition.
- CRITICAL – The service is in a critical state.
- UNKNOWN – The check could not determine the service’s state.
To begin with, you can create a configuration file for your hosts, if you like, and add services to be monitored as shown below.
vim /etc/icinga2/conf.d/myhosts.conf
object Host "mydevapp.example.com" {
import "generic-host"
address = "192.168.43.214"
vars.os = "Linux"
vars.http_vhosts["http"] = {
http_uri = "/"
}
}
import keyword is used to import the generic-host template which takes care of setting up the host check command to hostalive. If you require a different check command, you can override it in the object definition.
The vars attribute can be used to define custom attributes which are available for checks for example checking for ssh service as defined in the services.conf file for all systems whose OS is set to Linux.
vars.http_vhosts defines http vhost attributes for the http service apply rule defined in services.conf.
You can also define the services as shown below.
object Host "mydevapp.example.com" {
address = "192.168.43.214"
check_command = "hostalive"
}
object Service "http" {
host_name = "mydevapp.example.com"
check_command = "http"
}
object Service "ssh" {
host_name = "mydevapp.example.com"
check_command = "ssh"
}
Once you are done with host definition save the configuration file and run the following command to check for syntactical errors.
icinga2 daemon -C
If all is well, restart icinga2 service
systemctl restart icinga2
Now login to the Icinga Web 2 dashboard and should be able to see your services.
Agent-based Monitoring
Agent based monitoring becomes handy when remote services are not directly accessible through the network. While there are certain mechanisms to ensure a secure communication between all nodes (firewalls, policies, software hardening, etc.), Icinga 2 imposes mandatory SSL communication between nodes. The communication between the master and client nodes happens on port 5665 and thus, if firewall is running, this port has to be opened.
Our environment comprises of;
- A single master node, icinga.example.com running Ubuntu 18.04 LTS
- Two client nodes; server01.example.com, mydevapp.example.com running CentOS 7 and Ubuntu 18.04 respectively.
Set up Icinga 2 Master Node
Run the command below to setup a Icinga 2 master node.
icinga2 node wizard
This will prompt you if it is a Satellite or Client setup. Choose n to select a master setup.
Press Enter to accept the defaults.
Once that is done, restart Icinga 2 daemon to finalize setup.
systemctl restart icinga2
If firewall is running, open port 5665.
ufw allow 5665
Generate a client ticket for client nodes
A client which sends a certificate signing request (CSR) must authenticate itself in a trusted way. The master generates a client ticket which is included in this request. That way the master can verify that the request matches the previously trusted ticket and sign the request.
Run the commands below on the master node to generate the tickets for your client nodes;
For client node, server01.example.com;
icinga2 pki ticket --cn server01.example.com 970a747db8d676fd2057e3ede2d0e9b5e6ae0dfd
For client node, mydevapp.example.com;
icinga2 pki ticket --cn mydevapp.example.com 99030d0af2c9cb27696cbe3ca3c40a6b043bd487
Save these keys as you will require them later while setting up the client nodes.
Configure Client Nodes
Note that before you can set up the client nodes, Icinga 2 on the master node must be running and accepting connections on port 5665.
ss -altnp | grep 5665
LISTEN 0 128 0.0.0.0:5665 0.0.0.0:* users:(("icinga2",pid=18872,fd=14))
Setting up Client Nodes
Login in to the client nodes and Install the Icinga 2 package and setup the required plugins as follows;
Add the Icinga repository to your package management configuration
For CentOS 7, server01.example.com
yum install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm -y
Setup EPEL Repository
yum install epel-release -y
For Ubuntu 18.04 LTS, mydevapp..example.com
wget -O - https://packages.icinga.com/icinga.key | apt-key add -
echo "deb http://packages.icinga.com/ubuntu icinga-bionic main" > /etc/apt/sources.list.d/icinga2.list
echo "deb-src http://packages.icinga.com/ubuntu icinga-bionic main" >> /etc/apt/sources.list.d/icinga2.list
apt update
Install Icinga2 and monitoring plugins
On CentOS 7;
yum install icinga2 nagios-plugins-all -y
On Ubuntu 18.04 LTS;
apt install icinga2 monitoring-plugins -y
Once the installation is done, start and enable icinga2 daemon to run on boot.
On CentOS 7:
systemctl start icinga2 systemctl enable icinga2
On Ubuntu 18.04, icinga 2 is started and enabled upon installation. You can check this by running;
systemctl status icinga2 systemctl is-enabled icinga2
Next, Set up the client nodes as shown below.
For CentOS 7 Client Node;
icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!
We will guide you through all required configuration details.
Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: y <- Enter y to setup client
Starting the Client/Satellite setup routine...
Please specify the common name (CN) [server01.example.com]: ENTER
Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): icinga.example.com
Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 192.168.43.138 < IP of the master node
Master/Satellite endpoint port [5665]: ENTER
Add more master/satellite endpoints? [y/N]: n
Parent certificate information:
Subject: CN = icinga.example.com
Issuer: CN = Icinga CA
Valid From: Oct 13 08:47:58 2018 GMT
Valid Until: Oct 9 08:47:58 2033 GMT
Fingerprint: 64 A7 AA F8 C0 F4 A8 A1 FE 95 9D 1E DC 86 07 3B E8 1F 97 41
Is this information correct? [y/N]: y
Please specify the request ticket generated on your Icinga 2 master (optional).
(Hint: # icinga2 pki ticket --cn 'server01.example.com'): 970a747db8d676fd2057e3ede2d0e9b5e6ae0dfd < ticket generated before
Please specify the API bind host/port (optional):
Bind Host []: ENTER
Bind Port []: ENTER
Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y
Reconfiguring Icinga...
Local zone name [server01.example.com]: ENTER
Parent zone name [master]: ENTER
Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: n
Do you want to disable the inclusion of the conf.d directory [Y/n]: y
Disabling the inclusion of the conf.d directory...
Done.
Now restart your Icinga 2 daemon to finish the installation!
Repeat the same for the other client node. Be sure to replace the ticket with the respective one created above.
The setup wizard fetches the parent node’s certificate and ask you to verify that information. This is to prevent MITM attacks or any kind of untrusted parent relationship.
Once done, restart Icinga2 daemon as requested.
systemctl restart icinga2
If firewall is running, open port 5665.
On Ubuntu;
ufw allow 5665 ufw reload
On CentOS 7
firewall-cmd --add-port=5665/tcp --permanent firewall-cmd --reload
You can verify that the master and the client node are now connected.
For client node, server01.example.com
ss | grep :5665
tcp ESTAB 0 0 192.168.43.63:33970 192.168.43.138:5665
For client node, mydevapp.example.com
ss | grep :5665
tcp ESTAB 0 0 192.168.43.214:43528 192.168.43.138:5665
Configure the Agent Monitoring on Icinga 2 Master
Now that the master nodes and the client nodes are connected, it is time to configure the master so it can execute a remote check on the client using the command endpoint.
Login to master node and create a zone directory where you can add the configuration files for the host and service objects you want to monitor.
mkdir /etc/icinga2/zones.d/master
Create the hosts configuration file for the clients.
vim /etc/icinga2/zones.d/master/hosts.conf
// for server01.example.com
object Endpoint "server01.example.com" {
}
object Zone "server01.example.com" {
endpoints = [ "server01.example.com" ]
parent = "master"
}
object Host "server01.example.com" {
check_command = "hostalive" //check is executed on the master
address = "192.168.43.63"
vars.client_endpoint = name //follows the convention that host name == endpoint name
}
// for mydevapp.example.com
object Endpoint "mydevapp.example.com" {
}
object Zone "mydevapp.example.com" {
endpoints = [ "mydevapp.example.com" ]
parent = "master"
}
object Host "mydevapp.example.com" {
check_command = "hostalive"
address = "192.168.43.214"
vars.client_endpoint = name
}
Create a services configuration file for your clients.
vim /etc/icinga2/zones.d/master/services.conf
// Check System Load
apply Service "System Load" {
check_command = "load"
command_endpoint = host.vars.client_endpoint // Check executed on client node
assign where host.vars.client_endpoint
}
// Check number of running system Processes
apply Service "Process" {
check_command = "procs"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Check number of Logged in Users
apply Service "Users" {
check_command = "users"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Check System Disk Usage
apply Service "Disk" {
check_command = "disk"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// Check for SWAP memory Usage
apply Service "SWAP" {
check_command = "swap"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
// SSH Service Check
apply Service "SSH Service" {
check_command = "ssh"
command_endpoint = host.vars.client_endpoint
assign where host.vars.client_endpoint
}
apply Service "Ping" {
check_command = "ping4"
assign where host.address
}
Save the changes and validate the configuration on the master node:
icinga2 daemon -C
Restart the Icinga 2 daemon
systemctl restart icinga2
Login to Icinga Web 2 and confirm the checks.
You have successfully configured Icinga 2 agent-based monitoring on Ubuntu 18.04. Happy monitoring!!.
Related Tutorials;
Install Icinga Web 2 on Debian 10 Buster
Install Icinga 2 on Debian 10 Buster
Install Nagios Core on Debian 10 Buster
