In this guide, we are going to go over how to create squid logs extractors on Graylog server. If you have pushed your Squid access logs to Graylog server via syslog, chances are they have not been parsed correctly to your liking. Use of Graylog makes it easy to to extract data from any text in the received message to message fields.
See our previous guides on Graylog Server by following the links below;
Install Graylog 3.0 on CentOS 7
Monitor Squid Access Logs with Graylog Server
Create Squid Logs Extractors on Graylog Server
Graylog Extractors can extract data using regular expressions, Grok patterns, substrings, or even by splitting the message into tokens by separator characters.
To create an extractors for a specific input, Navigate to System > Inputs and click on Manage Extractors.
You can also create an extractor for a specific message from Graylog search dashboard by clicking on the message as shown in the screenshot below;
If you launched the extractor from Inputs section, click get started and load the message from the selected input.
In this guide, we are going to use Grok pattern to extract fields on Squid access logs, as highlighted above and proceed to create the grok pattern to extract various fields in your message.
For example, this is the grok pattern for the message used in this example. You can check sample Grok patterns under System > Grok Patterns.
%{NUMBER:req_time} %{INT:duration;int} %{IPV4:req_client_address} %{NOTSPACE:squid_request_status}/%{NUMBER:http_status_code} %{NUMBER:transfer_size} %{NOTSPACE:http_method} (%{URIPROTO:url_scheme}://)?(?\S+?)(:%{INT:url_port})?(/%{NOTSPACE:url_path})?\s+%{NOTSPACE:client_identity}\s+%{NOTSPACE:peer_code}/%{NOTSPACE:peerhost}\s+%{NOTSPACE:content_type}A sample squid access log message is;
1556260467.596 8 192.168.45.27 TCP_MISS/200 2037 CONNECT 192.168.70.4:443 - HIER_DIRECT/192.168.70.4 -You can test your grok pattern by clicking try against your message. If all is well, then you should see your fields extracted.
Next, set the title of the extractor and save it.
Click Create extractor to create and save your extractor grok pattern. After that, navigate to the Graylog search dashboard and your suid log messages should now have the correct fields as defined by the extractor.
You have successfully created grok patterns to extract squid logs fields on Graylog server. Next, we are going to cover the creation of dashboards to ensure that you get good analytics visualization for your squid logs. See the link below;
Monitor Squid logs with Grafana and Graylog
Reference:
