How do I make HAProxy high availability? In this guide, we are going to learn how to configure highly available HAProxy with Keepalived on Ubuntu 20.04. Keepalived can work with HAProxy to provide the failover services to a backup route in the event that a specific route becomes unavailable. This ensures a more robust and scalable high availability environment.
Table of Contents
Configuring Highly Available HAProxy with Keepalived
While Keepalived uses Linux virtual server (LVS) kernel module to perform load balancing and failover tasks on the active and passive routers, HAProxy performs load balancing and high-availability services to TCP and HTTP applications.
Keepalived utilizes Virtual Router Redundancy Protocol to send periodic advertisements between the master (active) and backup (passive) LVS routers (which in our case is the HAProxy servers since are load balancing web apps) to determine the state of each other. If a master server fails to advertise itself within a predefined period of time, Keepalived initiates failover and the backup server becomes the master.
All virtual servers are assigned a Virtual IP, also known as floating IP. This is a publicly routable IP/address. It is assigned dynamically to an active server at any given time.
This setup requires that you already have an HAProxy server setup and running. We have covered the installation and setup of HAProxy load balancer on various systems in our previous guides;
Install and Setup HAProxy on Ubuntu 20.04
Install and Setup HAProxy on CentOS 8
Setup HAProxy Load Balancer on Fedora 30/Fedora 29
Configure HAProxy Load Balancer with SSL on Ubuntu 18.04/Debian 10/9
In this tutorial, we will be using two HAProxy servers with Keepalived for high availability. Below is our deployment architecture.
Assuming that you already have HAProxy setup, proceed with installation and configuration of Keepalived on Ubuntu 20.04.
Install and Configure Keepalived on Ubuntu
Install Keepalived on HAProxy Servers
In our demo environment, we are running HAProxy servers on Ubuntu 20.04. Therefore, assuming your system package cache is up-to-date, run the command below install Keepalived on Ubuntu 20.04
On HAProxy Node 01;
apt install keepalivedSimilarly, install Keepalived on second HAProxy server.
apt install keepalivedConfigure IP forwarding and non-local binding
To enable Keepalived service to forward network packets to the backend servers, you need to enable IP forwarding. Run this command on both HAProxy servers;
sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.confSimilarly, you need to enable HAProxy and Keepalived to bind to non-local IP address, that is to bind to the failover IP address (Floating IP).
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.confReload sysctl settings;
sysctl -pConfigure Keepalived
The default configuration file for Keepalived should be /etc/keepalived/keepalived.conf. However, this configuration is not created by default. Create the configuration with the content below;
vim /etc/keepalived/keepalived.confKeepalived Configuration for Master Node (lb01)
# Global Settings for notifications
global_defs {
notification_email {
[email protected] # Email address for notifications
}
notification_email_from [email protected] # The from address for the notifications
smtp_server 127.0.0.1 # SMTP server address
smtp_connect_timeout 15
}
# Define the script used to check if haproxy is still working
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
# Configuration for Virtual Interface
vrrp_instance LB_VIP {
interface enp0s8
state MASTER # set to BACKUP on the peer machine
priority 101 # set to 99 on the peer machine
virtual_router_id 51
smtp_alert # Enable Notifications Via Email
authentication {
auth_type AH
auth_pass myP@ssword # Password for accessing vrrpd. Same on all devices
}
unicast_src_ip 192.168.57.7 # Private IP address of master
unicast_peer {
192.168.58.4 # Private IP address of the backup haproxy
}
# The virtual ip address shared between the two loadbalancers
virtual_ipaddress {
192.168.100.200
}
# Use the Defined Script to Check whether to initiate a fail over
track_script {
chk_haproxy
}
}
Keepalived Configuration for Backup Node (lb02)
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 15
}
vrrp_script chk_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance LB_VIP {
interface enp0s8
state BACKUP
priority 100
virtual_router_id 51
smtp_alert
authentication {
auth_type AH
auth_pass myP@ssword
}
unicast_src_ip 192.168.58.4 # Private IP address of the backup haproxy
unicast_peer {
192.168.57.7 # Private IP address of the master haproxy
}
virtual_ipaddress {
192.168.100.200
}
track_script {
chk_haproxy
}
}
Note that the notification section is optional. You can however follow the link below to install and configure Postfix to use Gmail SMTP as relay;
Configure Postfix to Use Gmail SMTP on Ubuntu 20.04
Read more about the configuration parameters used above on Keepalived man pages and Keepalived Configuration synopsis.
Running Keepalived on Ubuntu
You can now start and enable Keepalived to run on system boot on all nodes;
systemctl enable --now keepalivedCheck the status on Master Node;
systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-06-06 18:24:20 UTC; 6s ago
Main PID: 7097 (keepalived)
Tasks: 2 (limit: 2282)
Memory: 2.1M
CGroup: /system.slice/keepalived.service
├─7097 /usr/sbin/keepalived --dont-fork
└─7107 /usr/sbin/keepalived --dont-fork
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Initial state master is incompatible with AH authentication - clearing
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Registering gratuitous ARP shared channel
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: VRRP_Script(chk_haproxy) succeeded
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Changing effective priority from 100 to 102
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering MASTER STATE
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent
You can as well check the status on the slave node.
Check the IP address assigment;
On the master node;
ip --brief add
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s3 UP 10.0.2.15/24 fe80::a00:27ff:fe9d:888e/64
enp0s8 UP 192.168.100.81/24 192.168.100.200/32 fe80::a00:27ff:feba:9e8c/64
enp0s9 UP 192.168.57.7/24 fe80::a00:27ff:fe31:b7db/64
See the highlighted line, it is assigned a floating IP, 192.168.100.200/32.
On the slave;
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s3 UP 10.0.2.15/24 fe80::a00:27ff:fefe:fc06/64
enp0s8 UP 192.168.100.80/24 fe80::a00:27ff:fe75:9eae/64
enp0s9 UP 192.168.58.4/24 fe80::a00:27ff:fef4:41/64
Now, let us take down interface enp0s8, on the master node. Be sure to login from the console before you can take the interface down.
ip link set enp0s8 downCheck Keepalived status on the backup node;
systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2020-06-06 19:24:27 UTC; 26min ago
Main PID: 9948 (keepalived)
Tasks: 2 (limit: 2282)
Memory: 2.3M
CGroup: /system.slice/keepalived.service
├─9948 /usr/sbin/keepalived --dont-fork
└─9949 /usr/sbin/keepalived --dont-fork
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Registering gratuitous ARP shared channel
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: VRRP_Script(chk_haproxy) succeeded
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Changing effective priority from 99 to 101
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering MASTER STATE
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:50:40 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.
Configure HAProxy with Keepalived Virtual IP
The only change that we are going to make on our HAProxy configuration file is to adjust the frontend bind IP and the stats IP address to the VIP or the floating IP, which in our case is 192.168.100.200
vim /etc/haproxy/haproxy.cfg
...
frontend kifarunixlb
bind 192.168.100.200:443 ssl crt /etc/ssl/certs/haproxy.pem
default_backend webapps
option forwardfor
...
listen stats
bind 192.168.100.200:8443 ssl crt /etc/ssl/certs/haproxy.pem
stats enable
...
Save and exit the config file.
Restart HAProxy.
systemctl restart haproxyYou should now be able to access your web apps even when one of the HAProxy servers goes down via the floating IP. Sample screenshot for our setup is in below;
Further Reading
Other tutorials
How to Install and Configure Pound as Apache HTTP Load balancer on Ubuntu 16.04
