Monitor Squid logs with Grafana and Graylog

|
Published:
|
|

In this guide, we are going to learn how to monitor squid logs with Grafana and Graylog. You can check our other guides on installing Graylog, forwarding squid logs to Graylog and creating Graylog squid log field extractors by following the links below;

Install Graylog 3.0 on CentOS 7

Monitor Squid Access Logs with Graylog Server

Create Squid Logs Extractors on Graylog Server

Monitor Squid logs with Grafana and Graylog

Grafana is an opensource tool for visualizing data collected from different types of data stores such as Prometheus, InfluxDB, Elasticsearch, Graphite, MySQL and several other databases. In this case of integrating it with Graylog, we will use Elasticsearch as our Grafana datasource.

To learn how to install Grafana on Ubuntu, Debian or Fedora, see the links below;

Configure Elasticsearch Remote Connection

Elasticsearch is listening on localhost by default in Graylog server. To configure it to allow remote connection, you need define an interface IP for the network.host parameter.

vim /etc/elasticsearch/elasticsearch.yml
...
network.host: GRAYLOG_SERVER_IP
...

If firewall is running, open the Elasticsearch ports

firewall-cmd --add-port=9200/tcp --permanent
firewall-cmd --reload
ufw allow from Graylog_IP to any port 9200 proto tcp

Restart Elasticsearch

systemctl restart elasticsearch

Verify Elasticsearch Connection

Login to Grafana and run the command below to verify connection to Elasticsearch by running the command below;

curl http://Graylog_IP_)R_HOSTNAME:9200
{
  "name" : "x55YNL_",
  "cluster_name" : "graylog",
  "cluster_uuid" : "CQBqPDoCRKW7tt955kq5Uw",
  "version" : {
    "number" : "6.8.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "65b6179",
    "build_date" : "2019-05-15T20:06:13.172855Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Create Graylog Squid Logs Elasticsearch Index Set

Graylog uses one or more sets of Elasticsearch indices to optimize search and analysis operations for speed and low resource consumption.

To create an index, navigate to System > Indices. Hit Create index set. On the index configuration page, set the name of the index, description, a unique prefix for use in Elasticsearch, number of Elasticsearch shards, index rotation strategy.

Graylog squid logs index
Graylog squid logs ES index

Once you are done, click save to save the index.

To verify the index name for your Elasticsearch datasource;

curl -XGET graylog.example.com:9200/_cat/indices?v
health status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   squidaccess_0 EiMgXL2UQqWym-5VZ-atDg   1   0       8859            0      1.9mb          1.9mb

Our index in this case is squidaccess_0. Note that using the Graylog Elasticsearch indices may bring issues due to constant rotation. We will look at the possible work around in our next guide.

Add Grafana Datasource

If connection to Elasticsearch from Grafana server is okay, proceed to create Grafana Elasticsearch datasource. To add Grafana datasource, navigate to Configuration > Data Sources.

Add Grafana Datasource

Click Add data source and choose Elasticsearch. Under the Elasticsearch datasource settings, set the name of the datasource, the URL of the Graylog Elasticsearch, Elasticsearch index prefix as defined in Graylog index above, time field name (timestamp).

Monitor Squid logs with Grafana and Graylog: Creating Grafana Elasticsearch datasource

Next, click Save & Test to test the connection to Elasticsearch datasource. If everything is fine, then you should get Index Ok.

Grafana Elasticsearch index Ok

Create Grafana Dashboard for Squid Logs

Once you have you Graylog Elasticsearch datasource added to Grafana, you need to create the dashboards for visualizing the data. This involves creating various queries for different dashboards you may want to have. You can also import a ready made dashboard.

To create a new or import Grafana dashboard, click on the HOME dropdown on the top left corner and choose whether to import dashboard json file or create a new one.

New grafana dashboard

For example, based on my Graylog squid log extractor, this is a simple dashboard that we have created.

Grafana-Graylog Squid access logs dashboard

Below are the panels that makes up this dashboard.

Total Traffic:

Grafana squid total traffic

Top Sites:

top sites

Top 10 Denied Sites:

top 10 denied

Top user:

top user

That is just the basics on how to monitor squid logs with Grafana and Graylog. We hope this was informative enough. Feel free to drop your comments. Enjoy.

SUPPORT US VIA A VIRTUAL CUP OF COFFEE

We're passionate about sharing our knowledge and experiences with you through our blog. If you appreciate our efforts, consider buying us a virtual coffee. Your support keeps us motivated and enables us to continually improve, ensuring that we can provide you with the best content possible. Thank you for being a coffee-fueled champion of our work!

Photo of author
koromicha
I am the Co-founder of Kifarunix.com, Linux and the whole FOSS enthusiast, Linux System Admin and a Blue Teamer who loves to share technological tips and hacks with others as a way of sharing knowledge as: "In vain have you acquired knowledge if you have not imparted it to others".

2 thoughts on “Monitor Squid logs with Grafana and Graylog”

  1. Hello,

    I appreciate your handwork in providing knowledge..
    Please also make tutorials to create dashboard that you have.

    Thanks

    Reply
  2. Hi!
    Thanks for articles! I have question:
    how i can show in Grafana transfer size(in bytes) to megabytes?
    I tryed like this but it not works:

    client_ip: * AND HOSTNAME: * AND http_status_code: 200 AND transfer_size: (*/1024/1024)

    Thanks!

    Reply

Leave a Comment