Configure Remote Logging with Rsyslog on Ubuntu 18.04

In this tutorial, you will learn how to configure remote logging with Rsyslog on Ubuntu 18.04. Configuring remote logging with Rsyslog on Ubuntu allows you to centralize and store log data on a remote server. Remote logging has several important benefits:

• Makes it easier to monitor and analyze log data from multiple systems in one location.
• Simplifies the troubleshooting process, as you can review logs from various sources simultaneously.
• Storing logs on a separate server protects them from potential tampering or deletion on the local systems.
• In the event of a security incident, having centralized logs aids in forensic analysis, helping to identify the source and nature of the breach.
• Compliance to various standards by providing a unified and secure log storage solution.

Configuring Remote Logging with Rsyslog on Ubuntu 18.04

Rsyslog can be configured in a client/server model. When configured as a client, it sends logs to a remote server over the network via TCP/UDP protocols. As a server, it receives logs over the network from remote client on port 514 TCP/UDP.

Rsyslog filters syslog messages based on selected filters. You may want to check out our previous article on basic introduction to rsyslog filters.

Install Rsyslog on Ubuntu 18.04

Rsyslog is installed on Ubuntu 18.04 by default. You can verify this by checking the version of installed rsyslog.

rsyslogd -v

If it is not installed, run the command below to install it.

apt install rsyslog -y

Once the installation is done, start and enable the rsyslog service.

systemctl enable --now rsyslog

Allow Rsyslog through Firewall

If firewall is running, open rsyslog through it.

ufw allow 514/udp

Well, are you also interested in configuring syslog/rsyslog on Solaris 11.4? Check the links below;

Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server

Configure Syslog on Solaris 11.4 for Remote Logging

Want to use NXLog to forward logs? Check out our article by following the link below;

Configure NXLog to Forward System Logs to Rsyslog Server on Ubuntu 18.04

Configure Ubuntu 18.04 as a Log Server

Now that rsyslog is installed and running, you need to configure it to run in server mode. To do so, edit the /etc/rsyslog.conf configuration file and uncomment the lines for UDP syslog reception in the MODULES section as shown below;

vim /etc/rsyslog.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Note that TCP syslog reception is way more reliable than UDP syslog and still pretty fast. The main reason is, that UDP might suffer of message loss. This happens when the syslog server must receive large bursts of messages. If the system buffer for UDP is full, all other messages will be dropped. With TCP, this will not happen. But sometimes it might be good to have a UDP server configured as well. That is, because some devices (like routers) are not able to send TCP syslog by design. In that case, you would need both syslog server types to have everything covered.

By default UDP syslog is received on port 514. TCP syslog may need a different port because often the RPC service is using this port as well.

To set rsyslog to run on a different TCP port, say TCP port, 50514, uncomment the TCP reception lines and change the port as shown below;

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="50514")

Verify that rsyslog is now listening on two ports;

netstat -4altunp | grep 514

tcp        0      0 0.0.0.0:50514           0.0.0.0:*               LISTEN      10814/rsyslogd      
udp        0      0 0.0.0.0:514             0.0.0.0:*                           10814/rsyslogd

You may notice that UDP port has no LISTEN state because it is connectionless and has no concept of “listening”, “established”, “closed”, or anything like that.

Open the new port on UFW;

ufw allow 50514/tcp

Define Allowed Senders

You may also want to explicitly set the remote clients that are allowed to to send syslog messages to rsyslogd. To achieve this, you can set a global directive using the $AllowedSender directive.

Allowed sender lists can be defined for UDP and TCP senders separately. The syntax to specify them is:

$AllowedSender [UDP/TCP], ip[/bits], ip[/bits]

Where:

• ip[/bits] is a machine or network ip address as in “192.0.2.0/24” or “192.0.2.10”. If the /bits part is omitted, a single host is assumed. “/0” is not allowed, because that would match any sending system.

• Hostnames, with and without wildcards, may also be provided. If so, the result of revers DNS resolution is used for filtering. Multiple allowed senders can be specified in a comma-delimited list.

It is good to specify senders with high traffic volume before those with lower volume.

As much as allowing specific hosts via this directive, a good idea to impose allowed sender limitations via firewalling.

To allow specific hosts for either UDP or TCP logging, enter the following lines;

vim /etc/rsyslog.conf

...
###########################
#### GLOBAL DIRECTIVES ####
###########################
# $AllowedSender - specifies which remote systems are allowed to send syslog messages to rsyslogd
$AllowedSender UDP, 192.168.43.0/24, [::1]/128, *.example.net, servera.example.com
$AllowedSender TCP, 192.168.43.0/24, [::1]/128, *.example.net, servera.example.com

Configure Rsyslog Template

Templates are a key feature of rsyslog. Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates. To create a template use the following syntax in /etc/rsyslog.conf:

$template TEMPLATE_NAME,"text %PROPERTY% more text", [OPTION]

Thus, we can create our template like;

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="50514")

#Custom template to generate the log filename dynamically based on the client's IP address.
$template RemInputLogs, "/var/log/remotelogs/%FROMHOST-IP%/%PROGRAMNAME%.log"
*.* ?RemInputLogs

Once you are done with configuration, you can now restart the rsyslog service by running the command below. Before you can restart rsyslogd, run a configuration check.

rsyslogd -f /etc/rsyslog.conf -N1

rsyslogd: version 8.32.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

If all is well, proceed to restart rsyslog.

systemctl restart rsyslog

Rsyslogd is now ready to receive logs from remote hosts.

Configure Remote Client

Now it is time to configure the remote client to send syslog messages to the remote syslog server. Login and proceed as follows.

Verify Remote Ports Connection

To verify connectivity to remote rsyslog server TCP port 50514, run the command below;

telnet 192.168.43.154 50514

Trying 192.168.43.154...
Connected to 192.168.43.154.
Escape character is '^]'.
^]

telnet>

Verify connectivity to UDP port 514. Since you cannot telnet to UDP port 514, use netcat command. On the server, run the command below;

nc -ul 514

On the client, run the command below, press ENTER and type anything. You should be able to see what you type on the server.

nc -u 192.168.43.154 514

Configure Client Rsyslog to Sent Logs to Remote Log Server

If all is good, edit the rsyslog configuration file as shown below;

vim /etc/rsyslog.conf

To send authentication logs over port 514/UDP, add the following line at the end of the file.

# Send logs to remote syslog server over UDP
auth,authpriv.* @192.168.43.154:514

To send all logs over port 50514/TCP, add the following line at the end of the file.

# Send logs to remote syslog server over TCP 50514
*.* @@192.168.43.154:50514

As a cushion just in case the remote rsyslog server goes down and your logs are so important you don’t want to loose, set the rsyslog disk queue for buffering in the rsyslog configuration file as shown below;

# Send logs to remote syslog server over UDP
auth,authpriv.* @192.168.43.154:514

# Define Disk Queue Buffer in case the server goes down
$ActionQueueFileName queue # define a file name for disk assistance.
$ActionQueueMaxDiskSpace 1g  # The maximum size that all queue files together will use on disk.
$ActionQueueSaveOnShutdown on  # specifies that data should be saved at shutdown
$ActionQueueType LinkedList  # holds enqueued messages in memory which makes the process very fast. 
$ActionResumeRetryCount -1  # prevents rsyslog from dropping messages when retrying to connect if server is not responding,

Restart the rsyslog service on the client.

systemctl restart rsyslog

You can now log out of the client and login again. The authentication logs should be available on rsyslog server.

Login to the server and verify the same.

ls /var/log/remotelogs/

127.0.0.1  192.168.43.214

Verify Log Reception on Remote Log Server

In our case, we send only authentication logs to remote rsyslog server.

ls -1 /var/log/remotelogs/192.168.43.214/

sshd.log 
sudo.log
su.log
systemd-logind.log

tail -5 /var/log/remotelogs/192.168.43.214/sshd.log

2024-01-13T11:01:33+03:00 mydevapp sshd[13430]: Disconnected from user amos 192.168.43.149 port 60808
2024-01-13T11:01:33+03:00 mydevapp sshd[13363]: pam_unix(sshd:session): session closed for user amos
2024-01-13T11:01:37+03:00 mydevapp sshd[13569]: pam_ecryptfs: Passphrase file wrapped
2024-01-13T11:01:37+03:00 mydevapp sshd[13567]: Accepted password for amos from 192.168.43.149 port 60854 ssh2
2024-01-13T11:01:37+03:00 mydevapp sshd[13567]: pam_unix(sshd:session): session opened for user amos by (uid=0)

That is all on Rsyslog remote logging.