Setup Rsyslog Server on Ubuntu 20.04

In this tutorial, you will learn how to setup rsyslog server on Ubuntu 20.04. Rsyslog is a multi-threaded implementation of syslogd (a system utility providing support for message logging), with features that include:

• reliable syslog over TCP, SSL/TLS and RELP
• on-demand disk buffering
• email alerting
• writing to MySQL or PostgreSQL databases (via separate output plugins)
• permitted sender lists
• filtering on any part of the syslog message
• on-the-wire message compression
• fine-grained output format control
• failover to backup destinations
• enterprise-class encrypted syslog relaying
  .
  It is the default syslogd on Debian systems.

Rsyslog can be configured in a client/server model. When configured as a client, it sends logs to a remote server over the network via TCP/UDP protocols. As a server, it receives logs over the network from remote client on port 514 TCP/UDP or any custom port on which it is configured to listen on.

Rsyslog filters syslog messages based on selected filters. You may want to check out our previous article on basic introduction to rsyslog filters.

Configuring Rsyslog Server on Ubuntu 20.04

Are you using Debian 10? Check the link below;

Setup Rsyslog Server on Debian 10

Install Rsyslog on Ubuntu 20.04

Rsyslog is the default syslogd on Debian systems and is usually installed on Ubuntu 20.04 by default.

You can verify this by checking the version of installed rsyslog.

apt list -a rsyslog

Listing... Done
rsyslog/focal-updates,now 8.2001.0-1ubuntu1.1 amd64 [installed,automatic]
rsyslog/focal 8.2001.0-1ubuntu1 amd64

If for any reasons it is not installed, run the command below to install it.

apt update

apt install rsyslog -y

Once the installation is done, start and enable the rsyslog service.

systemctl enable --now rsyslog

Setting up Rsyslog Server

Now that rsyslog is installed and running, you need to configure it to run in server mode. As stated above, rsyslog can be configured as client to sent logs to a central logging server or a server to receive and store logs from other systems.

In this guide, we setup Rsyslog as a server on an Ubuntu 20.04 box.

Open the ryslog configuration file for editing;

vim /etc/rsyslog.conf

Define Rsyslog Server Protocol and Port

To begin with, define the protocol and port you want to receive logs on.

You can choose to use UDP or TCP and any port of your choice.

Note that TCP syslog reception is way more reliable than UDP syslog and still pretty fast. The main reason is, that UDP might suffer of message loss. This happens when the syslog server must receive large bursts of messages. If the system buffer for UDP is full, all other messages will be dropped. With TCP, this will not happen. But sometimes it might be good to have a UDP server configured as well. That is, because some devices (like routers) are not able to send TCP syslog by design. In that case, you would need both syslog server types to have everything covered.

In this setup, we will configure Rsyslog to use both UDP and TCP protocols for logs reception over the ports 514 and 50514 respectively.

By default UDP syslog is received on port 514.

Enable UDP syslog reception:

Within the /etc/rsyslog.conf configuration file, uncomment the lines for UDP syslog reception in the MODULES section as shown below;

...
#################
#### MODULES ####
#################
...

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Enable TCP syslog reception:

TCP syslog may need to use a different port because often the RPC service is using this port as well.

To set rsyslog to run on a different TCP port, say TCP port, 50514, uncomment the TCP reception lines and change the port as shown below;

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="50514")

Save and exit the file;

Restart rsyslog service;

systemctl restart rsyslog

Verify that rsyslog is now listening on two ports;

ss -4altunp | grep 514

udp    UNCONN  0       0                   0.0.0.0:514            0.0.0.0:*      users:(("rsyslogd",pid=52382,fd=5))                                            
tcp    LISTEN  0       25                  0.0.0.0:50514          0.0.0.0:*      users:(("rsyslogd",pid=52382,fd=7))

You may notice that UDP port has no LISTEN state because it is connectionless and has no concept of “listening”, “established”, “closed”, or anything like that.

Allow Rsyslog through Firewall

If firewall is running, open rsyslog through it.

ufw allow 514/udp

ufw allow 50514/tcp

Define Allowed Senders

You may also want to explicitly set the remote clients that are allowed to to send syslog messages to rsyslogd. To achieve this, you can set a global directive using the $AllowedSender directive.

Allowed sender lists can be defined for UDP and TCP senders separately. The syntax to specify them is:

$AllowedSender [UDP/TCP], ip[/bits], ip[/bits]

• ip[/bits] is a machine or network ip address as in “192.0.2.0/24” or “192.0.2.10”. If the /bits part is omitted, a single host is assumed. “/0” is not allowed, because that would match any sending system.
• Hostnames, with and without wildcards, may also be provided. If so, the result of revers DNS resolution is used for filtering. Multiple allowed senders can be specified in a comma-delimited list.

It is good to specify senders with high traffic volume before those with lower volume.

To allow specific hosts for either UDP or TCP logging, enter the following lines;

vim /etc/rsyslog.conf

...
###########################
#### GLOBAL DIRECTIVES ####
###########################
# $AllowedSender - specifies which remote systems are allowed to send syslog messages to rsyslogd
$AllowedSender UDP, 192.168.57.0/24, [::1]/128, *.example.net, servera.example.com
$AllowedSender TCP, 192.168.58.0/24, [::1]/128, *.example.net, servera.example.com

The hostnames must be resolvable since before ACL is updated, they will be resolved into their individual IPs.

Also note that the above directives only allow UDP reception from 192.168.57.0/24 and TCP reception from 192.168.58.0/24.

As much as allowing specific hosts via this directive, a good idea to impose allowed sender limitations via firewalling.

For example, to allow hosts from the 192.168.57.0/24 and 192.168.58.0/24 networks;

ufw allow from 192.168.57.0/24 to any port 514 proto udp
ufw allow from 192.168.57.0/24 to any port 50514 proto tcp

ufw allow from 192.168.58.0/24 to any port 514 proto udp
ufw allow from 192.168.58.0/24 to any port 50514 proto tcp

Configure Rsyslog Template

Templates are a key feature of rsyslog. Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates.

To create a template use the following syntax in /etc/rsyslog.conf:

$template TEMPLATE_NAME,"text %PROPERTY% more text", [OPTION]

Thus, we can create our template like;

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="50514")

#Custom template to generate the log filename dynamically based on the client's IP address.
$template RemInputLogs, "/var/log/remotelogs/%FROMHOST-IP%/%PROGRAMNAME%.log"
*.* ?RemInputLogs

This will categorize logs received from remote host into log files for specific programs responsible for the generation of that log.

Once you are done with configuration, save and exit the file;

You can now restart the rsyslog service by running the command below. Before you can restart rsyslogd, run a configuration check.

rsyslogd -f /etc/rsyslog.conf -N1

rsyslogd: version 8.2001.0, config validation run (level 1), master config /etc/rsyslog.conf
 rsyslogd: End of config validation run. Bye.

If all is well, proceed to restart rsyslog.

systemctl restart rsyslog

Rsyslogd is now ready to receive logs from remote hosts.

Configure Remote Rsyslog Client to Forward logs Rsyslog Server

Now it is time to configure the remote client to send syslog messages to the remote syslog server. Login and proceed as follows.

Verify Remote Rsyslog Server Ports Connection

To verify connectivity to remote rsyslog server TCP port 50514, run the command below;

telnet 192.168.57.3 50514

Trying 192.168.57.3...
Connected to 192.168.57.3.
Escape character is '^]'.
^]

telnet>

Verify connectivity to UDP port 514. Since you cannot telnet to UDP port 514, use netcat command. On the server, run the command below;

nc -ul 514

On the client, run the command below, press ENTER and type anything. You should be able to see what you type on the server.

nc -u 192.168.57.3 514

If all is good, edit the client system rsyslog configuration file as shown below;

vim /etc/rsyslog.conf

To send authentication logs over port 514/UDP, add the following line at the end of the file. Note, this should be done from hosts under the 192.168.57.0/24 as per the AllowedSender directive.

# Send logs to remote syslog server over UDP
auth,authpriv.* @192.168.57.3:514

To send all logs over port 50514/TCP, add the following line at the end of the file. Note, this should be done from hosts under the 192.168.58.0/24 as per the AllowedSender directive.

# Send logs to remote syslog server over TCP 50514
*.* @@192.168.57.3:50514

As a cushion just in case the remote rsyslog server goes down and your logs are so important you don’t want to loose, set the rsyslog disk queue for buffering in the rsyslog configuration file as shown below;

# Define Disk Queue Buffer in case the server goes down
$ActionQueueFileName queue # define a file name for disk assistance.
$ActionQueueMaxDiskSpace 1g  # The maximum size that all queue files together will use on disk.
$ActionQueueSaveOnShutdown on  # specifies that data should be saved at shutdown
$ActionQueueType LinkedList  # holds enqueued messages in memory which makes the process very fast. 
$ActionResumeRetryCount -1  # prevents rsyslog from dropping messages when retrying to connect if server is not responding,

Restart the rsyslog service on the client.

systemctl restart rsyslog

You can now log out of the client and login again. The authentication logs should be available on rsyslog server.

Login to the Rsyslog server and verify the same.

ls /var/log/remotelogs/

127.0.0.1/     192.168.57.35/

In our case, we send only authentication logs to remote rsyslog server.

ls /var/log/remotelogs/192.168.57.35/

CRON.log  sshd.log  sudo.log  su.log

tail -5 /var/log/remotelogs/192.168.57.35/sshd.log 

2021-03-25T22:14:19+03:00 debian sshd[590]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.57.1  user=root
2021-03-25T22:14:22+03:00 debian sshd[590]: Failed password for root from 192.168.57.1 port 56714 ssh2

Other Tutorials

Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server

Configure Syslog on Solaris 11.4 for Remote Logging

Want to use NXLog to forward logs? Check out our article by following the link below;

Configure NXLog to Forward System Logs to Rsyslog Server on Ubuntu 20.04

How to Configure Remote Logging with Rsyslog on Ubuntu 18.04