Configure Highly Available HAProxy with Keepalived on Ubuntu 20.04

0
2326

In this guide, we are going to learn how to configure highly available HAProxy with Keepalived on Ubuntu 20.04. Keepalived can work with HAProxy to provide the failover services to a backup route in the event that a specific route becomes unavailable. This ensures a more robust and scalable high availability environment. 

While Keepalived uses Linux virtual server (LVS) to perform load balancing and failover tasks on the active and passive routers, HAProxy performs load balancing and high-availability services to TCP and HTTP applications.

Keepalived utilizes Virtual Router Redundancy Protocol to send periodic advertisements between the master (active) and backup (passive) LVS routers (which in our case is the HAProxy servers since are load balancing web apps) to determine the state of each other. If a master server fails to advertise itself within a predefined period of time, Keepalived initiates failover and the backup server becomes the master.

All virtual servers are assigned a Virtual IP, also known as floating IP. This is a publicly routable IP/address. It is assigned dynamically to an active server at any given time.

Configure Highly Available HAProxy with Keepalived on Ubuntu 20.04

This setup requires that you already have an HAProxy server setup and running. We have covered the installation and setup of HAProxy load balancer on various systems in our previous guides;

Install and Setup HAProxy on Ubuntu 20.04

Install and Setup HAProxy on CentOS 8

Setup HAProxy Load Balancer on Fedora 30/Fedora 29

Configure HAProxy Load Balancer with SSL on Ubuntu 18.04/Debian 10/9

In this tutorial, we will be using two HAProxy servers with Keepalived for high availability. Below is our deployment architecture.

Assuming that you already have HAProxy setup, proceed with installation and configuration of Keepalived on Ubuntu 20.04.

Install and Configure Keepalived on Ubuntu 20.04

Install Keepalived on HAProxy Servers

In our demo environment, we are running HAProxy servers on Ubuntu 20.04. Therefore, assuming your system package cache is up-to-date, run the command below install Keepalived on Ubuntu 20.04

[email protected]:~# apt install keepalived

Similarly, install Keepalived on second HAProxy server.

[email protected]:~# apt install keepalived

Configure IP forwarding and non-local binding

To enable Keepalived service to forward network packets to the backend servers, you need to enable IP forwarding. Run this command on both HAProxy servers;

sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf

Similarly, you need to enable HAProxy and Keepalived to bind to non-local IP address, that is to bind to the failover IP address (Floating IP).

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

Reload sysctl settings;

sysctl -p

Configure Keepalived

The default configuration file for Keepalived should be /etc/keepalived/keepalived.conf. However, this configuration is not created by default. Create the configuration with the content below;

vim /etc/keepalived/keepalived.conf

Keepalived Configuration for Master Node (lb01)

# Global Settings for notifications
global_defs {
    notification_email {
        [email protected]     # Email address for notifications 
    }
    notification_email_from [email protected]	# The from address for the notifications
    smtp_server 127.0.0.1     			# SMTP server address
    smtp_connect_timeout 15
}
  
# Define the script used to check if haproxy is still working
vrrp_script chk_haproxy { 
    script "/usr/bin/killall -0 haproxy"
    interval 2 
    weight 2 
}
  
# Configuration for Virtual Interface
vrrp_instance LB_VIP {
    interface enp0s8
    state MASTER        # set to BACKUP on the peer machine
    priority 101        # set to  99 on the peer machine
    virtual_router_id 51
  
    smtp_alert          # Enable Notifications Via Email
  
    authentication {
        auth_type AH
        auth_pass [email protected]	# Password for accessing vrrpd. Same on all devices
    }
    unicast_src_ip 192.168.57.7 # Private IP address of master
    unicast_peer {
        192.168.58.4		# Private IP address of the backup haproxy
   }
  
    # The virtual ip address shared between the two loadbalancers
    virtual_ipaddress {
        192.168.100.200
    }
     
    # Use the Defined Script to Check whether to initiate a fail over
    track_script {
        chk_haproxy
    }
}

Keepalived Configuration for Backup Node (lb02)

global_defs {
    notification_email {
        [email protected]
    }
    notification_email_from [email protected]
    smtp_server 127.0.0.1
    smtp_connect_timeout 15
}
  
vrrp_script chk_haproxy { 
    script "/usr/bin/killall -0 haproxy"
    interval 2 
    weight 2 
}
  
vrrp_instance LB_VIP {
    interface enp0s8
    state BACKUP
    priority 100
    virtual_router_id 51
  
    smtp_alert
  
    authentication {
        auth_type AH
        auth_pass [email protected]
    }
    unicast_src_ip 192.168.58.4 # Private IP address of the backup haproxy
    unicast_peer {
        192.168.57.7	# Private IP address of the master haproxy
   }
  
    virtual_ipaddress {
        192.168.100.200
    }
     
    track_script {
        chk_haproxy
    }
}

Note that the notification section is optional. You can however follow the link below to install and configure Postfix to use Gmail SMTP as relay;

Configure Postfix to Use Gmail SMTP on Ubuntu 20.04

Read more about the configuration parameters used above on Keepalived man pages and Keepalived Configuration synopsis.

Running Keepalived on Ubuntu 20.04

You can now start and enable Keepalived to run on system boot on all nodes;

systemctl enable --now keepalived

Check the status on Master Node;

systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
     Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-06-06 18:24:20 UTC; 6s ago
   Main PID: 7097 (keepalived)
      Tasks: 2 (limit: 2282)
     Memory: 2.1M
     CGroup: /system.slice/keepalived.service
             ├─7097 /usr/sbin/keepalived --dont-fork
             └─7107 /usr/sbin/keepalived --dont-fork

Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Initial state master is incompatible with AH authentication - clearing
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Registering gratuitous ARP shared channel
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: VRRP_Script(chk_haproxy) succeeded
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Changing effective priority from 100 to 102
Jun 06 18:24:20 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: (LB_VIP) Entering MASTER STATE
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 18:24:23 lb01.kifarunix-demo.com Keepalived_vrrp[7107]: SMTP alert successfully sent

You can as well check the status on the slave node.

Check the IP address assigment;

On the master node;

ip --brief add
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp0s3           UP             10.0.2.15/24 fe80::a00:27ff:fe9d:888e/64 
enp0s8           UP             192.168.100.81/24 192.168.100.200/32 fe80::a00:27ff:feba:9e8c/64 
enp0s9           UP             192.168.57.7/24 fe80::a00:27ff:fe31:b7db/64

See the highlighted line, it is assigned a floating IP, 192.168.100.200/32.

On the slave;

lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp0s3           UP             10.0.2.15/24 fe80::a00:27ff:fefe:fc06/64 
enp0s8           UP             192.168.100.80/24 fe80::a00:27ff:fe75:9eae/64 
enp0s9           UP             192.168.58.4/24 fe80::a00:27ff:fef4:41/64

Now, let us take down interface enp0s8, on the master node. Be sure to login from the console before you can take the interface down.

ip link set enp0s8 down

Check Keepalived status on the backup node;

systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
     Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-06-06 19:24:27 UTC; 26min ago
   Main PID: 9948 (keepalived)
      Tasks: 2 (limit: 2282)
     Memory: 2.3M
     CGroup: /system.slice/keepalived.service
             ├─9948 /usr/sbin/keepalived --dont-fork
             └─9949 /usr/sbin/keepalived --dont-fork

Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Registering gratuitous ARP shared channel
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering BACKUP STATE (init)
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: VRRP_Script(chk_haproxy) succeeded
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Changing effective priority from 99 to 101
Jun 06 19:24:27 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: (LB_VIP) Entering MASTER STATE
Jun 06 19:50:39 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: Remote SMTP server [127.0.0.1]:25 connected.
Jun 06 19:50:40 lb02.kifarunix-demo.com Keepalived_vrrp[9949]: SMTP alert successfully sent.

Configure HAProxy with Keepalived Virtual IP

The only change that we are going to make on our HAProxy configuration file is to adjust the frontend bind IP and the stats IP address to the VIP or the floating IP, which in our case is 192.168.100.200

vim /etc/haproxy/haproxy.cfg
...
frontend kifarunixlb
        bind 192.168.100.200:443 ssl crt /etc/ssl/certs/haproxy.pem
        default_backend webapps
        option forwardfor
...
listen stats
        bind 192.168.100.200:8443 ssl crt /etc/ssl/certs/haproxy.pem
        stats enable
...

Save and exit the config file.

Restart HAProxy.

systemctl restart haproxy

You should now be able to access your web apps even when one of the HAProxy servers goes down via the floating IP. Sample screenshot for our setup is in below;

That marks the end of our guide on how to configure highly available HAProxy with Keepalived on Ubuntu 20.04

Further Reading

Keepalived Documentation

Keepalived User Guide

Other tutorials

How to Install and Configure Pound as Apache HTTP Load balancer on Ubuntu 16.04

LEAVE A REPLY

Please enter your comment!
Please enter your name here