Forward Apache Logs to Central Log Server with Rsyslog

In this tutorial, you will learn how to forward Apache logs to central log server with rsyslog. Apache do not log to syslog by default. So, what if you want to forward them logs to central system where you can easily manage them? In that case, there are various tools which can be used to read and sent Apache logs to central logging systems.

Want to setup Central Log server with Rsyslog? Check the guides below;

Setup Rsyslog Server on Debian 10

Setup Rsyslog Server on Ubuntu 20.04

Forwarding Apache Logs to Central Log Server with Rsyslog

In this tutorial, we will explore two methods of forwarding Apache logs to central log server with rsyslog:

• Monitoring Apache Log File with Rsyslog Text File Input Module
• Configure Apache to Log to Syslog

Monitoring Apache Log File with Rsyslog Text File Input Module

The rsyslog text file input module (imfile), provides the ability to convert any standard text file into a syslog message. This module can read a log file line by line while passing each read line to rsyslog engine rules, which then applies filter conditions and selects which actions needs to be carried out. Empty lines are not processed, as they would result in empty syslog records. They are simply ignored.

In order to use Rsyslog text file input module to read Apache log file and forward it to remove Rsyslog server, you would create a configuration file like /etc/rsyslog.d/02-apache2.conf, with the content below;

vim /etc/rsyslog.d/02-apache2.conf

module(load="imfile" PollingInterval="10" statefile.directory="/var/spool/rsyslog")
input(type="imfile"
      File="/var/log/apache2/error.log"
      Tag="http_error"
      Severity="error"
      Facility="local6")
local6.error        @192.168.59.12:514

Save and exit the file.

• The first line specifices the Rsyslog module to load, which in this case is the imfile module.
• PollingInterval: Specifies how often the file is read for new data. You should never set the value of this parameter to 0 or you risk hogging your system CPU.
• statefile.directory: specify a dedicated directory for the storage of imfile state files. The file should exist and be writable by rsyslog before restarting rsyslog.
  ls -ald /var/spool/rsyslog/
drwx------ 2 syslog adm 4096 Feb 11 2020 /var/spool/rsyslog/
• type: Specifies the type of the module.
• File: Specifies the file to be polled.
• Tag: a tag to be assigned to messages read from the file above.
• Severity: syslog severity to be assigned to lines read from the file.
• Facility: syslog facility to be assigned to messages read from the file specified.
• The last line specifies that the these log lines be forwarded to a remote server, 192.168.59.12, on UDP port 514.

Verify the correctness of Rsyslog configuration file;

rsyslogd -N1 -f /etc/rsyslog.d/02-apache2.conf

rsyslogd: version 8.2006.0, config validation run (level 1), master config /etc/rsyslog.d/02-apache2.conf
rsyslogd: End of config validation run. Bye.

If no issues, restart Rsyslog;

systemctl restart rsyslog

Verify Reception of Apache Logs on Remote Log Server

You can use tcpdump to check if for any communication between the Rsyslog server and the client on UDP port 514; Replace the interface and the source host IP accordingly.

tcpdump -i enp0s8 src host 192.168.59.13 and udp port 514 -nn -vv

tcpdump: listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes
14:44:27.229660 IP (tos 0x0, ttl 64, id 35360, offset 0, flags [DF], proto UDP (17), length 1000)
    192.168.59.13.35486 > 192.168.59.12.514: [udp sum ok] SYSLOG, length: 972
	Facility local6 (22), Severity error (3)
	Msg: Mar 31 21:44:27 ubuntu20 http_error [Wed Mar 31 21:44:27.206708 2021] [:error] [pid 19242:tid 140596081583680] [client 127.0.0.1:51706] ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/ls' ) [file "/etc/apache2/modsecurity.d/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "496"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/ls found within ARGS:exec: /bin/ls"] [severity "2"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "127.0.1.1"] [uri "/"] [unique_id "161721626740.953787"] [ref "o1,6v11,7t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
...

You can also check your remote log file on the server. In our setup, we configured our Rsyslog log server to log remote logs per the program sending the logs on per host directory.

So in such a case, we have such logs from our Apache server;

ls /var/log/remotelogs/192.168.59.13/

http_error.log

Tailing this log file;

tail -f /var/log/remotelogs/192.168.59.13/http_error.log

2021-03-31T21:50:57-04:00 ubuntu20 http_error [Wed Mar 31 21:50:57.617055 2021] [:error] [pid 19243:tid 140595980871232] [client 192.168.59.1:43602] ModSecurity: Warning. Matched "Operator `Rx' with parameter `^[\\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `192.168.59.13' ) [file "/etc/apache2/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "722"] [id "920350"] [rev ""] [msg "Host header is a numeric IP address"] [data "192.168.59.13"] [severity "4"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "127.0.1.1"] [uri "/"] [unique_id "161721665745.833955"] [ref "o0,13v33,13"]
2021-03-31T21:50:57-04:00 ubuntu20 http_error [Wed Mar 31 21:50:57.620882 2021] [:error] [pid 19243:tid 140595980871232] [client 192.168.59.1:43602] ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:doc' (Value: `/bin/ls' ) [file "/etc/apache2/modsecurity.d/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "496"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/ls found within ARGS:doc: /bin/ls"] [severity "2"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "127.0.1.1"] [uri "/"] [unique_id "161721665745.833955"] [ref "o1,6v10,7t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
...

And that is how Rsyslog Text File Input Module can be used to read and forward application logs to a remote server.

Configure Apache to Log to Syslog

In order to forward logs via Rsyslog, the application generating logs should be able to write to syslog. Apache doesnt write to syslog by default.

In order to configure Apache to log to syslog, you can use a program like logger. Logger can be used to write application logs to syslog.

By default, the ErrorLog and CustomLog directives to set the name of the file to which the server will log any errors it encounters and any request made to the server respectively.

By default, these files are set like;

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

So error logs are logged to ${APACHE_LOG_DIR}/error.log while access logs are logged to ${APACHE_LOG_DIR}/access.log.

Now that we want to forward Apache logs to remote syslog server you can set the logging directive to pipe the logs to logger which can then write to a specific syslog facility, specifying the appropriate severity.

Let us take for example we want to log Apache error logs to syslog facility, local6 and severity we set to error i.e local6.err, edit the site configuration and set the value for the syslog directive as shown below;

vim /etc/apache2/sites-available/000-default.conf

...
ErrorLog "| /usr/bin/logger -thttp_error: -plocal6.err"
CustomLog ${APACHE_LOG_DIR}/access.log combined

Options passed to logger command; -t specifies the tag and -p specifies the syslog priority, facility.level.

Save and exit the file.

Next, configure Rsyslog to forward logs written to syslog priority, local6.err to a remote Rsyslog server.

echo "local6.err @192.168.59.12:514" >> /etc/rsyslog.conf

If you want to log directly to remote server without specifying the server on the rsyslog.conf file, use the line;

ErrorLog "| /usr/bin/logger -thttp_error: -plocal6.err -n 192.168.59.12 -P 514"

Check Apache for any configuration errors;

apachectl -t

If you get the output, Syntax OK, then you good to restart Apache;

systemctl restart apache2

Similarly, you can check Rsyslog configurations for any errors;

rsyslogd -N1

Restart Rsyslog;

systemctl restart rsyslog

You should now be able to receive the logs at the remote central log server.

One thing to note with this setting, ErrorLog "| /usr/bin/logger -thttp_error: -plocal6.err", it doesnt to logs the local system log file and hence, there is a high possibility that some logs may be lost.

To overcome this and log the Apache logs to both a local file and a remote server, use the tee command to pipe logs to logger and to the local log file as shown below;

ErrorLog "|/bin/sh -c '/usr/bin/tee -a /var/log/apache2/error.log | /usr/bin/logger -thttp_error: -plocal6.err'"
CustomLog ${APACHE_LOG_DIR}/access.log combined

Configure Rsyslog to sent logs on priority local6.err to remote log server.

Restart both Rsyslog and Apache;

systemctl restart rsyslog apache2

You should now be able log to the local file and to remote log server.

Other Tutorials

Configure Rsyslog on Solaris 11.4 to Send logs to Remote Log Server

Configure Syslog on Solaris 11.4 for Remote Logging

Install and Configure NXLog CE on Ubuntu 20.04