Create Squid Logs Extractors on Graylog Server


Last updated on May 29th, 2019 at 02:17 pm

In this guide, we are going to go over how to create squid logs extractors on Graylog server. If you have pushed your Squid access logs to Graylog server via syslog, chances are they have not been parsed correctly to your liking. Use of Graylog makes it easy to to extract data from any text in the received message to message fields.

See our previous guides on Graylog Server by following the links below;

Install Graylog 3.0 on CentOS 7

Monitor Squid Access Logs with Graylog Server

Create Squid Logs Extractors on Graylog Server

Graylog Extractors can extract data using regular expressions, Grok patterns, substrings, or even by splitting the message into tokens by separator characters.

To create an extractors for a specific input, Navigate to System > Inputs and click on Manage Extractors.

Create Squid Logs Extractors on Graylog Server

You can also create an extractor for a specific message from Graylog search dashboard by clicking on the message as shown in the screenshot below;

Create Graylog Message Extractor

If you launched the extractor from Inputs section, click get started and load the message from the selected input.

Load message into graylog extractor

In this guide, we are going to use Grok pattern to extract fields on Squid access logs, as highlighted above and proceed to create the grok pattern to extract various fields in your message.

Graylog Extractor Grok pattern

For example, this is the grok pattern for the message used in this example. You can check sample Grok patterns under System > Grok Patterns.

%{NUMBER:req_time} %{INT:duration;int} %{IPV4:req_client_address} %{NOTSPACE:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:transfer_size} %{NOTSPACE:http_method} (%{URIPROTO:url_scheme}://)?(?\S+?)(:%{INT:url_port})?(/%{NOTSPACE:url_path})?\s+%{NOTSPACE:client_identity}\s+%{NOTSPACE:peer_code}/%{NOTSPACE:peerhost}\s+%{NOTSPACE:content_type}

A sample squid access log message is;

1556260467.596 8 TCP_MISS/200 2037 CONNECT - HIER_DIRECT/ -

You can test your grok pattern by clicking try against your message. If all is well, then you should see your fields extracted.

Graylog extractor preview

Next, set the title of the extractor and save it.

Save the graylog extractor

Click Create extractor to create and save your extractor grok pattern. After that, navigate to the Graylog search dashboard and your suid log messages should now have the correct fields as defined by the extractor.

Graylog extracted squid log fields

You have successfully created grok patterns to extract squid logs fields on Graylog server. Next, we are going to cover the creation of dashboards to ensure that you get good analytics visualization for your squid logs. See the link below;

Monitor Squid logs with Grafana and Graylog


Graylog Extractors


Please enter your comment!
Please enter your name here