Create Squid Logs Extractors on Graylog Server

|
Last Updated:
|
|

In this guide, we are going to go over how to create squid logs extractors on Graylog server. If you have pushed your Squid access logs to Graylog server via syslog, chances are they have not been parsed correctly to your liking. Use of Graylog makes it easy to to extract data from any text in the received message to message fields.

See our previous guides on Graylog Server by following the links below;

Install Graylog 3.0 on CentOS 7

Monitor Squid Access Logs with Graylog Server

Create Squid Logs Extractors on Graylog Server

Graylog Extractors can extract data using regular expressions, Grok patterns, substrings, or even by splitting the message into tokens by separator characters.

To create an extractors for a specific input, Navigate to System > Inputs and click on Manage Extractors.

Create Squid Logs Extractors on Graylog Server

You can also create an extractor for a specific message from Graylog search dashboard by clicking on the message as shown in the screenshot below;

Create Graylog Message Extractor

If you launched the extractor from Inputs section, click get started and load the message from the selected input.

Load message into graylog extractor

In this guide, we are going to use Grok pattern to extract fields on Squid access logs, as highlighted above and proceed to create the grok pattern to extract various fields in your message.

Graylog Extractor Grok pattern

For example, this is the grok pattern for the message used in this example. You can check sample Grok patterns under System > Grok Patterns.

%{NUMBER:req_time} %{INT:duration;int} %{IPV4:req_client_address} %{NOTSPACE:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:transfer_size} %{NOTSPACE:http_method} (%{URIPROTO:url_scheme}://)?(?\S+?)(:%{INT:url_port})?(/%{NOTSPACE:url_path})?\s+%{NOTSPACE:client_identity}\s+%{NOTSPACE:peer_code}/%{NOTSPACE:peerhost}\s+%{NOTSPACE:content_type}

A sample squid access log message is;

1556260467.596 8 192.168.45.27 TCP_MISS/200 2037 CONNECT 192.168.70.4:443 - HIER_DIRECT/192.168.70.4 -

You can test your grok pattern by clicking try against your message. If all is well, then you should see your fields extracted.

Graylog extractor preview

Next, set the title of the extractor and save it.

Save the graylog extractor

Click Create extractor to create and save your extractor grok pattern. After that, navigate to the Graylog search dashboard and your suid log messages should now have the correct fields as defined by the extractor.

Graylog extracted squid log fields

You have successfully created grok patterns to extract squid logs fields on Graylog server. Next, we are going to cover the creation of dashboards to ensure that you get good analytics visualization for your squid logs. See the link below;

Monitor Squid logs with Grafana and Graylog

Reference:

Graylog Extractors

SUPPORT US VIA A VIRTUAL CUP OF COFFEE

We're passionate about sharing our knowledge and experiences with you through our blog. If you appreciate our efforts, consider buying us a virtual coffee. Your support keeps us motivated and enables us to continually improve, ensuring that we can provide you with the best content possible. Thank you for being a coffee-fueled champion of our work!

Photo of author
koromicha
I am the Co-founder of Kifarunix.com, Linux and the whole FOSS enthusiast, Linux System Admin and a Blue Teamer who loves to share technological tips and hacks with others as a way of sharing knowledge as: "In vain have you acquired knowledge if you have not imparted it to others".

Leave a Comment