How to Debug Logstash Grok Filters

Welcome to our guide on how to debug Logstash Grok filters. Grok filter uses regular expressions to parse unstructured event data into fields. It is perfect for syslog logs, Apache and other web server logs, MySQL logs or any human readable log format.

This comes so handy if you want to extract different fields of an event data.

How to Debug Logstash Grok Filters

In this guide, we are going to use sample SSH authentication logs to debug Logstash Grok Patterns.

Configuring Logstash Plugins

Assuming that you have already installed Logstash, proceed to configure Logstash as follows. If you have not installed Logstash, see below links on how to;

Install and Configure Logstash 7 on Ubuntu 18/Debian 9.8

Install Logstash 7 on Fedora 30/Fedora 29/CentOS 7

Configure Logstash Input

To run a successful debugging, we are going to configure Logstash read events from standard input. The plugin responsible for this is usually installed by default. To verify installed plugins;

/usr/share/logstash/bin/logstash-plugin list | grep -i stdin
logstash-input-stdin

Create a Logstash plugin definition configuration file under /etc/logstash/conf.d and define the Input plugin to begin with.

vim /etc/logstash/conf.d/ssh-authentication.conf

input {
    stdin { }
}
...

Configure Grok Filter Plugin

Next, configure Logstash Filter plugin for whatever logs you need to parse or extract the fields. In this guide, we are using sample SSH authentication logs.

Grok filter is also installed by default.

/usr/share/logstash/bin/logstash-plugin list | grep -i filter-grok
logstash-filter-grok

Jul  1 05:49:25 fedora29 sshd[16748]: Accepted password for root from 192.168.0.103 port 45382 ssh2
Jul  1 05:23:45 fedora29 sshd[3603]: Failed password for root from 192.168.0.103 port 44074 ssh2

Our filter for the above sample log is define below


input {
    stdin { }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:src_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:source}.*" }
  }
}
...

To create Grok filter, you can use the Kibana Grok debugger or use the Heroku App Grok Debugger.

Configure Output Plugin

Configure Logstash to print the parsed event data to standard output for a convenient debugging. You need to configure the output plugin to outputs event data using the ruby “awesome_print” library, stdout { codec => rubydebug }.


input {
    stdin { }
}

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:dst_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:src_host}.*" }
  }
}
output {
  elasticsearch { hosts => ["192.168.0.106:9200"] }
  stdout { codec => rubydebug }
}

Once you have you Logstash configured, run the command below to verify if there are any configuration errors.

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

If you see, Configuration OK, then all fine. Otherwise, fix any would be errors before you can continue.

How to Debug Logstash Grok Filters

Now that you configuration is done, stop Logstash and run Grok filter debugging by running the command below;

systemctl stop logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ssh-authentication.conf --path.settings /etc/logstash/

Once you see the line, Successfully started Logstash API endpoint {:port=>9600}, paste you sample log line and press ENTER.


...
The stdin plugin is now waiting for input:
[2019-07-01T07:16:24,243][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-01T07:16:24,904][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
Jul  1 05:49:25 fedora29 sshd[16748]: Accepted password for root from 192.168.0.103 port 45382 ssh2 ENTER

You log line will be filtered if at all your Grok filter is fine.


...
Jul  1 05:49:25 fedora29 sshd[16748]: Accepted password for root from 192.168.0.103 port 45382 ssh2
{
         "auth_user" => "root",
          "@version" => "1",
          "src_host" => "192.168.0.103",
        "@timestamp" => 2019-07-01T04:37:29.244Z,
              "host" => "elastic.example.com",
           "message" => "Jul  1 05:49:25 fedora29 sshd[16748]: Accepted password for root from 192.168.0.103 port 45382 ssh2",
         "timestamp" => "Jul  1 05:49:25",
          "dst_host" => "fedora29",
    "syslog_program" => "sshd",
            "status" => "Accepted password"
}


Jul  1 05:23:45 fedora29 sshd[3603]: Failed password for root from 192.168.0.103 port 44074 ssh2
{
         "auth_user" => "root",
          "@version" => "1",
          "src_host" => "192.168.0.103",
        "@timestamp" => 2019-07-01T04:36:36.910Z,
              "host" => "elastic.example.com",
           "message" => "Jul  1 05:23:45 fedora29 sshd[3603]: Failed password for root from 192.168.0.103 port 44074 ssh2",
         "timestamp" => "Jul  1 05:23:45",
          "dst_host" => "fedora29",
    "syslog_program" => "sshd",
            "status" => "Failed password"
}

Well, from the output above, you can see that Logstash has parsed SSH authentication logs for failed and accepted password.

That is all on how to debug Logstash Grok filters. You can try other logs the same way.

How to Debug Logstash Grok Filters