Configure NXLog to Forward System Logs to Rsyslog Server on Ubuntu

By
koromicha
-
0
6172

In this guide, we are going to learn how to configure NXLog to forward system logs to Rsyslog server on Ubuntu. There are various NXLog log collection solutions. In this guide, we are going to configure the opensource version of NXLog.

Configure NXLog to Forward System Logs to Rsyslog Server on Ubuntu

Download NXLog CE Installer

NXLog is not available on the default Ubuntu repositories.

Configure Rsyslog client to send lo...
Configure Rsyslog client to send local logs to remote Rsyslog Server

Hence, download the DEB package and install it with dpkg package manager.

To download NXLog install binary, navigate to NXLog community edition downloads page and grab one for Ubuntu. It can be Ubuntu 16.04, 18.04, 20.04, 22.04. Download current release version of your respective Ubuntu release installer.

wget https://nxlog.co/system/files/products/files/348/nxlog-ce_3.1.2319_ubuntu18_amd64.deb

You can also check our previuos articles on configuration of Rsyslog and Syslog by following the links below;

Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server

Configure Syslog on Solaris 11.4 for Remote Logging

How to Configure Remote Logging with Rsyslog on Ubuntu 18.04

Install NXLog CE on Ubuntu

Once you have downloaded the DEB binary, install it as shown below;

sudo apt install ./nxlog-ce_3.1.2319_ubuntu18_amd64.deb
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'nxlog-ce' instead of './nxlog-ce_3.1.2319_ubuntu18_amd64.deb'
The following package was automatically installed and is no longer required:
  libnetplan0
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  libapr1 libdbi1 libgdbm-compat4 libperl5.26 perl-modules-5.26
Suggested packages:
  libdbd-mysql libdbd-pgsql libdbd-sqlite3 libdbd-freetds
Recommended packages:
  perl
The following NEW packages will be installed:
  libapr1 libdbi1 libgdbm-compat4 libperl5.26 nxlog-ce perl-modules-5.26
0 upgraded, 6 newly installed, 0 to remove and 41 not upgraded.
Need to get 6,444 kB/7,824 kB of archives.
After this operation, 46.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 /root/nxlog-ce_3.1.2319_ubuntu18_amd64.deb nxlog-ce amd64 3.1.2319 [1,380 kB]
Get:2 https://mirror.hetzner.com/ubuntu/packages bionic-updates/main amd64 perl-modules-5.26 all 5.26.1-6ubuntu0.6 [2,764 kB]
Get:3 https://mirror.hetzner.com/ubuntu/packages bionic/main amd64 libapr1 amd64 1.6.3-2 [90.9 kB]
Get:4 https://mirror.hetzner.com/ubuntu/packages bionic/main amd64 libdbi1 amd64 0.9.0-5 [27.3 kB]
Get:5 https://mirror.hetzner.com/ubuntu/packages bionic/main amd64 libgdbm-compat4 amd64 1.14.1-6 [6,084 B]
Get:6 https://mirror.hetzner.com/ubuntu/packages bionic-updates/main amd64 libperl5.26 amd64 5.26.1-6ubuntu0.6 [3,556 kB]
Fetched 6,444 kB in 0s (13.5 MB/s) 
Selecting previously unselected package perl-modules-5.26.
(Reading database ... 26514 files and directories currently installed.)
Preparing to unpack .../0-perl-modules-5.26_5.26.1-6ubuntu0.6_all.deb ...
Unpacking perl-modules-5.26 (5.26.1-6ubuntu0.6) ...
Selecting previously unselected package libapr1:amd64.
Preparing to unpack .../1-libapr1_1.6.3-2_amd64.deb ...
Unpacking libapr1:amd64 (1.6.3-2) ...
Selecting previously unselected package libdbi1:amd64.
Preparing to unpack .../2-libdbi1_0.9.0-5_amd64.deb ...
Unpacking libdbi1:amd64 (0.9.0-5) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../3-libgdbm-compat4_1.14.1-6_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.14.1-6) ...
Selecting previously unselected package libperl5.26:amd64.
Preparing to unpack .../4-libperl5.26_5.26.1-6ubuntu0.6_amd64.deb ...
Unpacking libperl5.26:amd64 (5.26.1-6ubuntu0.6) ...
Selecting previously unselected package nxlog-ce.
Preparing to unpack .../5-nxlog-ce_3.1.2319_ubuntu18_amd64.deb ...
Unpacking nxlog-ce (3.1.2319) ...
Setting up libapr1:amd64 (1.6.3-2) ...
Setting up libdbi1:amd64 (0.9.0-5) ...
Setting up perl-modules-5.26 (5.26.1-6ubuntu0.6) ...
Setting up libgdbm-compat4:amd64 (1.14.1-6) ...
Setting up libperl5.26:amd64 (5.26.1-6ubuntu0.6) ...
Setting up nxlog-ce (3.1.2319) ...
Adding user nxlog to group nxlog
Created symlink /etc/systemd/system/multi-user.target.wants/nxlog.service → /lib/systemd/system/nxlog.service.
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...

Configure NXLog to Forward Logs on Ubuntu

Now that NXLog CE has been installed, you need to configure it to forward logs to the remote Rsyslog server.

The default configuration file for NXLog CE is /etc/nxlog/nxlog.conf.

This is how the default configuration file looks like;

cat /etc/nxlog/nxlog.conf
## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally under
## /usr/share/doc/nxlog-ce/ and is also available online at
## http://nxlog.org/docs

########################################
# Global directives                    #
########################################
User nxlog
Group nxlog

include /etc/nxlog/nxlog.d/*.conf
LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

########################################
# Modules                              #
########################################
<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in1>
    Module	im_udp
    Port	514
    Exec	parse_syslog_bsd();
</Input>

<Input in2>
    Module	im_tcp
    Port	514
</Input>

<Output fileout1>
    Module	om_file
    File	"/var/log/nxlog/logmsg.txt"
    Exec	if $Message =~ /error/ $SeverityValue = syslog_severity_value("error");
    Exec	to_syslog_bsd();
</Output>

<Output fileout2>
    Module	om_file
    File	"/var/log/nxlog/logmsg2.txt"
</Output>

########################################
# Routes                               #
########################################
<Route 1>
    Path	in1 => fileout1
</Route>

<Route tcproute>
    Path	in2 => fileout2
</Route>

NXLog can be configured to receive and read logs from different types of sources including;

  • log data received over the network
  • events stored in databases
  • messages read from files
  • data retrieved using executables

This guide focuses on configuring NXLog CE to receive, read and forward logs from system log files to a remote logging server.

Therefore, make a backup of the original configuration so that you can make any adjustments that suit your environment.

mv /etc/nxlog/nxlog.conf{,.original}

Create a new configuration file.

touch /etc/nxlog/nxlog.conf

The NXLog configuration file consists of;

  • global directives
  • module instances, and
  • routes

To begin with, set the ROOT to the main directory of NXLog configuration.

# Set the NXLog main directory
define ROOT /etc/nxlog

Define the Global directives

There are quite a number of global directives that can be set.

However, in its simplest, we will define;

  • the NXLog modules directory
  • directory to write the cached data
  • the logging level,
  • NXLog PID file,
  • NXLog working directory,
  • NXLog log file etc

This can be done by using the following directives respectively; ModuleDir, CacheDir, LogLevel, PidFile, SpoolDir respectively.

# Global Directives
Moduledir /usr/lib/nxlog/modules
CacheDir %ROOT%/data
SpoolDir %ROOT%/data
Pidfile /tmp/nxlog.pid
LogFile /var/log/nxlog/nxlog.log

Some of the other important global directives include User and Group.

NXLog runs as user nxlog by default with the limitation being this user cannot read the /var/log directory where most system logs are written to.

To circumvent this, NXlog can be set to run as root by omitting the User option. However, it is more secure if you can add NXlog user to a group with permissions to read the log file.

Define Input Module Directives

Basically, we are going to define basic input modules directives that read various log files for sending to a remote log server.

# Define Input Modules
<Input in1>
    Module      im_file
    File        "/var/log/auth.log"
    SavePos TRUE
    ReadFromLast TRUE
</Input>
<Input in2>
    Module      im_file
    File        "/var/log/syslog"
    SavePos TRUE
    ReadFromLast TRUE
</Input>

The SavePos directive ensures that log file state is cached when NXLog exits. The log file will be read from that position when NXLog starts. This can however be turned off by using NoCache directive.

The ReadFromLast directive ensures that NXLog reads the logs received after NXLog started.

Define the Processor Modules

Processor modules are used process logs between the Input and Output modules. This can be achieved by use of pm_buffer modules which supports both disk and memory log buffering. In this guide, we are going to set disk buffering. You also need to set the maximum size of the logs that can be buffered.

#Define Processor Modules
<Processor buffer>
   Module      pm_buffer
   MaxSize    512000   # Buffer logs upto 512MB
   Type         Disk   # Disk buffering
</Processor>

Define the Output Module

There are different types of output modules. We are going to set UDP as our output module. You can check about other modules here. You need to set the remote Host IP and Port.

# Define Output Modules
<Output udp>
    Module      om_udp
    Host        192.168.43.208
    Port        514
</Output>

Define the Route Directives

Define the data flow using the Path directive. More than one Input feeding logs into the route are comma separated. The list of Input modules is followed by an arrow (=>). Processor modules or Output modules follow after. Multiple Processors are separated by arrows. The syntax is;

Path INPUT1[, INPUT2...] => [PROCESSOR1 [=> PROCESSOR2...] =>] OUTPUT1[, OUTPUT2...]

Hence, we can define this in our configuration file as;

# Route definition
<Route 1>
    Path      in1,in2 => buffer => udp
</Route>

That is all about our configuration in its simplest form.

In general, it should look like;

# Set the NXLog main directory
define ROOT /etc/nxlog
# Global Directives
Moduledir /usr/lib/nxlog/modules
CacheDir %ROOT%/cache_dir
SpoolDir %ROOT%/spool_dir
Pidfile /tmp/nxlog.pid
LogFile /var/log/nxlog/nxlog.log
# Define Input Modules
<Input in1>
    Module      im_file
    File        "/var/log/auth.log"
    SavePos TRUE
    ReadFromLast TRUE
</Input>
<Input in2>
    Module      im_file
    File        "/var/log/syslog"
    SavePos TRUE
    ReadFromLast TRUE
</Input>
#Define Processor Modules
<Processor buffer>
   Module      pm_buffer
   MaxSize     512000
   Type        Disk
</Processor>
# Define Output Modules
<Output udp>
    Module      om_udp
    Host        0.0.0.0 # IP of Rsyslog Server
    Port        514
</Output>
# Route Definition
<Route 1>
    Path      in1,in2 => buffer => udp
</Route>

Create the Cache and Spool Directories.

mkdir -p /etc/nxlog/{cache_dir,spool_dir}

Check the configuration to verify the syntax;

nxlog -v

If you get INFO configuration OK, you are good to proceed.

Restart NXLog and set it to run on system boot.

systemctl restart nxlog
systemctl enable nxlog

Check the status.

systemctl status nxlog
● nxlog.service - NXLog daemon
   Loaded: loaded (/lib/systemd/system/nxlog.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-10-10 12:39:55 UTC; 56s ago
  Process: 1714 ExecStop=/usr/bin/nxlog -s (code=exited, status=0/SUCCESS)
  Process: 1715 ExecStartPre=/usr/bin/nxlog -v (code=exited, status=0/SUCCESS)
 Main PID: 1716 (nxlog)
    Tasks: 5 (limit: 2300)
   CGroup: /system.slice/nxlog.service
           └─1716 /usr/bin/nxlog -f

Oct 10 12:39:55 ubuntu-2gb-hel1-2 systemd[1]: Starting NXLog daemon...
Oct 10 12:39:55 ubuntu-2gb-hel1-2 nxlog[1715]: 2022-10-10 12:39:55 INFO configuration OK
Oct 10 12:39:55 ubuntu-2gb-hel1-2 systemd[1]: Started NXLog daemon.

Confirm that you can receive logs on the remote server.

As a POC, ssh into Ubuntu 18.04 server with nxlog running from a different server.

At the same time, tail the logs on remote Rsyslog server and there you go.

tail -f /var/log/remotelogs/192.168.43.203.log 
2022-10-16T19:32:40-04:00 u18svr sshd[21327]: Connection closed by 127.0.0.1 port 50630 [preauth]
2022-10-16T19:34:12-04:00 u18svr sshd[21335]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.43.149  user=kifarunix
2022-10-16T19:34:14-04:00 u18svr sshd[21335]: Failed password for kifarunix from 192.168.43.149 port 48224 ssh2
2022-10-16T19:34:18-04:00 u18svr sshd[21335]: Accepted password for kifarunix from 192.168.43.149 port 48224 ssh2
2022-10-16T19:34:18-04:00 u18svr sshd[21335]: pam_unix(sshd:session): session opened for user kifarunix by (uid=0)
2022-10-16T19:34:18-04:00 u18svr systemd-logind[581]: New session 26 of user kifarunix.

That is all about how to configure NXLog to forward system logs to Rsyslog server on Ubuntu. Feel free to read more about NXLog on their reference manual.

Related Tutorials

Install and Configure NXLog CE on Ubuntu 20.04

Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server

Configure Syslog on Solaris 11.4 for Remote Logging

How to Configure Remote Logging with Rsyslog on Ubuntu 18.04

RELATED ARTICLESMORE FROM AUTHOR

LEAVE A REPLY

Please enter your comment!
Please enter your name here