Monitor Docker Swarm Service Metrics using Grafana

|
Last Updated:
|
|

How can I monitor Docker swarm service metrics? In this tutorial, you will learn how to monitor Docker swarm service metrics using Grafana. Coupled with other tools such as Prometheus, cAdvisor, Grafana can let you create visualization for all your Docker swarm services metrics for monitoring.

Monitoring Docker Swarm Service Metrics using Grafana

In this guide, we will use Grafana, Prometheus and cAdvisor (Container Advisor) to monitor Docker swarm service metrics.

cAdvisor can be used as a data source to provide real-time docker swarm service metrics to Prometheus. The metrics on Prometheus can be used to visualize and analyze container metrics in Grafana.

In short, these are the tools we will use in this guide to show you how to monitor Docker swarm service metrics;

So, how can we be able to monitor Docker swarm metrics using Grafana, coupled with Prometheus, cAdvisor?

We will run Grafana and Prometheus as global swarm services, while cAdvisor as docker containers each on a swarm node.

Install Docker Engine

If you have not already installed Docker Engine, please refer to appropriate guides in the link below;

Install Docker in Linux

Setup Docker Swarm Cluster

Since we are dealing swarm services, you need to have setup swarm cluster that whose metrics need to be monitored.

If you haven’t, please refer to this guide to learn how to setup swarm cluster.

How to Setup Docker Swarm Cluster on Ubuntu

Create Docker Swarm Network to Interconnect Monitoring Tools

In order for the monitoring tools to communicate with each other, they should be in the same network. In a Docker Swarm, overlay networks can be used to provide communication between services, hence, create the network as follows.

The command below creates a Docker swarm network called monitoring_stack. You can use any name of your preference.

Since we also need to attach Docker containers to this network, make it attachable.

docker network create --driver overlay --attachable monitoring_stack

You check available Docker networks using the command below;

docker network ls

NETWORK ID     NAME               DRIVER    SCOPE
816c7f48bbb3   bridge             bridge    local
841640b7368f   docker_gwbridge    bridge    local
45c3672051a1   host               host      local
sxnbs83rwn5f   ingress            overlay   swarm
e5mklugaqapm   monitoring_stack   overlay   swarm
05990914584e   none               null      local

Deploy cAdvisor Docker Container

To begin with, let’s deploy cAdvisor as a Docker container on each node in the cluster so that it can collect swarm services runtime metrics from each node.

We have three nodes in our swarm cluster;

docker node ls
ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
him49eblt74amba1nghv2f8z1 *   swarm01    Ready     Active         Leader           20.10.22
4lxyd0f6d9h039y5itjiffqax     swarm02    Ready     Active         Reachable        20.10.12
dfmdl4wy4e7ouklu8mu7nqqh9     swarm03    Ready     Active         Reachable        20.10.22

You can simply run the command below to deploy cAdvisor and attach it to our custom Docker network above;


docker run -d \
  --name cadvisor-swarm01 \
  --restart unless-stopped \
  --network monitoring_stack \
  --volume /:/rootfs:ro \
  --volume /var/run:/var/run:rw \
  --volume /sys:/sys:ro \
  --volume /var/lib/docker/:/var/lib/docker:ro \
  gcr.io/cadvisor/cadvisor:v0.47.1

By default, cAdvisor exposes metrics on port 8080. This port will be reachable within container networks.

On the rest of the nodes, run the command, replacing the name of the cadvisor on each node.

cAdvisor container should now be up and running. For example on swarm node 2;

docker ps 

CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS                    PORTS              NAMES
6ed4926e9acf   gcr.io/cadvisor/cadvisor:v0.47.1   "/usr/bin/cadvisor -…"   46 seconds ago   Up 44 seconds (healthy)   8080/tcp           cadvisor-swarm02
41612d395de3   wordpress:latest                   "docker-entrypoint.s…"   24 minutes ago   Up 24 minutes             80/tcp             wordpress.3.8kma6ye90bw6e9j2uaj6er25v
5466529b2b4f   jasonrivers/nagios:latest          "/usr/local/bin/star…"   25 minutes ago   Up 25 minutes             80/tcp, 5667/tcp   nagios-server.2.3ueqfxnm2m8t1tclfgy93ebgt

Deploy Prometheus Docker Swarm Service

Next, let’s deploy Prometheus as a Docker swarm service to scrape collected swarm service metrics from cAdvisor and store them in its internal time series database which can later be read and visualized via Grafana.

First of all, before you can deploy Prometheus swarm service, create a Prometheus configuration file.

In our setup, we placed the Prometheus configuration file under /opt/prometheus directory. While creating Prometheus swarm service, we will mount this configuration file to the default Prometheus configuration file, /etc/prometheus/prometheus.yml.

vim /opt/prometheus/prometheus.yml

global:
  scrape_interval: 5s
  evaluation_interval: 10s

scrape_configs:
  - job_name: 'cadvisor'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['cadvisor-swarm01:8080','cadvisor-swarm02:8080','cadvisor-swarm03:8080']

In the configuration above, we use default settings for Prometheus and configured it to scrape the Docker swarm service metrics collected by cAdvisor container, running on each swarm node.

Prometheus will publish the metrics it scrapes on port 9090.

Thus, let’s create Prometheus Docker container using the official Prometheus Docker image and configure it to use our custom network created above, monitoring_stack.


docker service create \
-p 9090:9090 \
--mode global \
--network monitoring_stack \
--mount type=bind,src=/opt/prometheus/prometheus.yml,dst=/etc/prometheus/prometheus.yml \
--name prometheus \
prom/prometheus

Your Prometheus container should now be running and exposed via port 9090/tcp.

docker service ls --format '{{.Name}}\t{{.Ports}}'

nagios-server	*:8081->80/tcp
prometheus	*:9090->9090/tcp
wordpress	*:8880->80/tcp

Similarly, ensure you allow port 9090/tcp on firewall to allow you access the metrics externally!

Verify Prometheus Targets/Metrics from Prometheus Dashboard

You can now navigate to http://docker-host-IP:9090/targets to see Prometheus targets,

docker-host-IP can be IP of any of the swarm node.

Targets;

Monitor Docker Swarm Service Metrics using Grafana

Check Docker Container Metrics on Prometheus

Let’s check which container metrics are available.

Thus, on the dashboard, click Prometheus, and open metrics explorer.

Just type container and you should see quite a number of metrics;

metric

You can filter running Swarm services using the query;

container_tasks_state{container_label_com_docker_swarm_service_name=~".+.",state="running"}
get running swarm services

Deploy Grafana Docker Swarm Service

To get a better visualization of the Docker container metrics, you can deploy Grafana Docker swarm service across all the nodes in the cluster.

Create Grafana Data, Configuration and home directories volumes. Using a volume for Grafana data ensures that the dashboards, users, and other settings you create are persisted across container restarts. This is important if you want to keep your dashboards and other Grafana settings even if the container is updated or recreated.

for i in conf data home; do docker volume create grafana-$i; done

To deploy Grafana Docker container using the official image, grafana/grafana-oss and attach it to the custom network created above.


docker service create \
  --name grafana \
  --network monitoring_stack \
  --mount type=volume,source=grafana-data,target=/var/lib/grafana \
  --mount type=volume,source=grafana-home,target=/usr/share/grafana \
  --mount type=volume,source=grafana-config,target=/etc/grafana \
  --publish published=3000,target=3000 \
  grafana/grafana

Grafana is now running as a single replica.

If you run it as global or multi-replicated, you may not be able to login, getting Unauthorized, user token not found issue on the logs!

Open port 3000/tcp on firewall to allow external access to Grafana;

ufw allow 3000/tcp

Accessing Grafana Web Interface

You can now access Grafana web interface http://docker-host-IP:3000.

Default credentials are admin/admin. Reset the password and proceed to the Dashboard;

Monitor Docker swarm service Metrics using Grafana

Integrate Prometheus with Grafana For Monitoring

You can now integrate Prometheus with Grafana by adding Prometheus data source to Grafana. Check the link below;

Integrate Prometheus with Grafana For Monitoring

Data sources, once you have added the Prometheus data source.

As you can see, we specified the name of the Prometheus container on the data source URL because both Grafana and Prometheus are on same network.

Monitor Docker swarm service Metrics using Grafana

Create Docker Swarm Metrics Dashboards on Grafana

You can now create your own Grafana visualization dashboards for your Docker swarm cluster.

As a simple example, let’s create a simple visualization to display running, swarm cluster service tasks. Note that a task is a running instance of a service.

For example, using docker command, you can get information about the tasks running on your Swarm cluster using the docker service ps command, which shows the current status, node, and container ID of each task, …

e.g

docker service ps grafana

To create a Grafana dashboard, navigate to dashboards menu > New dashboard.

grafana dashboard menu

Add new panel;

You have three sections on the dashboard panel;

new dashboard panel

So, to create a visualization of specific metric, you need a query to fetch those metrics from the datasource, which in this case is our Preom

  • So from the Query panel, select a datasource (Prometheus in this example).
  • You can build your query using Builder which lets you select metrics and enter values (Builder) or simply switch to code which allows you to type the query manually.
    • with Builder, you can simply select your metric query from Metric drop-down and define the filters under Label filters.
      query builder for running swarm services
    • with Code, you can just type your query under metrics browser;
      Monitor Docker Swarm Service Metrics using Grafana

The query will count total number of running swarm service in last 5 minute, count(rate(container_last_seen{image=~".+",container_label_com_docker_swarm_service_id=~".+"}[5m])).

Under the Panel options;

  • change visualization type. We are using Stat in this example
  • Set the panel name e.g Running Containers. You can leave other options with default values
  • Value options: we leave defaults
  • Stat Styles, we only changed Color mode to Value and Graph Mode to none.
  • Standard Options unit Number.
  • everything else, defaults!
Monitor Docker Swarm Service Metrics using Grafana

When done, click Save/Apply to save the dashboard/visualization.

Docker container total CPU usage, RAM usage, Network I/O, Block I/O

Sample dashboard;

Monitor Docker Swarm Service Metrics using Grafana
Monitor Docker Swarm Service Metrics using Grafana
Monitor Docker Swarm Service Metrics using Grafana

Here is a sample dashboard JSON file;


{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "description": "Sample Docker Swarm Cluster",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "gnetId": 7007,
  "graphTooltip": 1,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "left",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [
            {
              "options": {
                "pattern": "\\w+-(\\S+):\\d+",
                "result": {
                  "index": 0,
                  "text": "$1"
                }
              },
              "type": "regex"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "container_label_com_docker_swarm_service_name"
            },
            "properties": [
              {
                "id": "displayName",
                "value": "Swarm Service"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "instance"
            },
            "properties": [
              {
                "id": "displayName",
                "value": "Swarm Node"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "Value"
            },
            "properties": [
              {
                "id": "displayName",
                "value": "No. Services"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 14,
        "x": 0,
        "y": 0
      },
      "id": 13,
      "options": {
        "footer": {
          "countRows": false,
          "fields": [
            "Value"
          ],
          "reducer": [],
          "show": true
        },
        "showHeader": true,
        "sortBy": [
          {
            "desc": true,
            "displayName": "No. Services"
          }
        ]
      },
      "pluginVersion": "9.4.7",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "count(container_last_seen{image=~\".+\",container_label_com_docker_swarm_service_id=~\".+\"}) by (instance,container_label_com_docker_swarm_service_name)",
          "format": "table",
          "instant": true,
          "legendFormat": "{{instance}}",
          "range": false,
          "refId": "A"
        }
      ],
      "title": "Swarm Services Per Node",
      "transformations": [
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true
            },
            "indexByName": {
              "Time": 0,
              "Value": 3,
              "container_label_com_docker_swarm_service_name": 2,
              "instance": 1
            },
            "renameByName": {}
          }
        }
      ],
      "type": "table"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "description": "These include both docker containers and swarm services tasks",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 6,
        "x": 14,
        "y": 0
      },
      "id": 3,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.4.7",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "count(rate(container_last_seen{image=~\".+\"}[$interval]))",
          "format": "time_series",
          "intervalFactor": 2,
          "range": true,
          "refId": "B",
          "step": 4
        }
      ],
      "title": "Total Swarm Cluster Containers",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "description": "These are swarm service containers/tasks",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 6,
        "x": 14,
        "y": 4
      },
      "id": 11,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.4.7",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "count(rate(container_last_seen{image=~\".+\",container_label_com_docker_swarm_service_id=~\".+\"}[$__rate_interval]))",
          "format": "time_series",
          "intervalFactor": 2,
          "range": true,
          "refId": "B",
          "step": 4
        }
      ],
      "title": "Total Swarm Tasks/Services",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "series",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 5,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineStyle": {
              "fill": "solid"
            },
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 8
      },
      "id": 19,
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "sum(irate(container_cpu_usage_seconds_total{container_label_com_docker_swarm_service_name=~\".+\"}[$__rate_interval])) by (container_label_com_docker_swarm_service_name)",
          "legendFormat": "{{container_label_com_docker_swarm_service_name}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Swarm Service CPU Usage",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "series",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "smooth",
            "lineStyle": {
              "fill": "solid"
            },
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 17
      },
      "id": 20,
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "sum(container_memory_rss{container_label_com_docker_swarm_service_name=~\".+\"}) by (container_label_com_docker_swarm_service_name)",
          "legendFormat": "{{container_label_com_docker_swarm_service_name}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Swarm Service Memory Usage",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "series",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "Bps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 21,
      "links": [],
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "9.4.7",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_swarm_service_name=~\".+\"}[$__rate_interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
          "range": true,
          "refId": "A",
          "step": 2
        }
      ],
      "title": "Incoming Network Traffic per Swarm Service",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "GbRl4cL4k"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "Bps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 33
      },
      "id": 8,
      "links": [],
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "9.4.7",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "GbRl4cL4k"
          },
          "editorMode": "code",
          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_swarm_service_name=~\".+\"}[$__rate_interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
          "range": true,
          "refId": "A",
          "step": 2
        }
      ],
      "title": "Outgoing Network Traffic per Swarm Service",
      "type": "timeseries"
    }
  ],
  "refresh": false,
  "revision": 1,
  "schemaVersion": 38,
  "style": "dark",
  "tags": [
    "prometheus",
    "cAdvisor",
    "node-exporter",
    "alertmanager"
  ],
  "templating": {
    "list": [
      {
        "current": {
          "isNone": true,
          "selected": false,
          "text": "None",
          "value": ""
        },
        "datasource": {
          "type": "prometheus",
          "uid": "GbRl4cL4k"
        },
        "definition": "query_result(count(container_last_seen{container_label_com_docker_swarm_service_name=~\".+\"}) by (instance))",
        "hide": 2,
        "includeAll": false,
        "label": "Swarm Node",
        "multi": false,
        "name": "node",
        "options": [],
        "query": {
          "query": "query_result(count(container_last_seen{container_label_com_docker_swarm_service_name=~\".+\"}) by (instance))",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "/instance=\"\\w+-(\\S+):\\d+\"/",
        "skipUrlSync": false,
        "sort": 2,
        "tagValuesQuery": "",
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "auto": true,
        "auto_count": 50,
        "auto_min": "50s",
        "current": {
          "selected": false,
          "text": "auto",
          "value": "$__auto_interval_interval"
        },
        "hide": 0,
        "label": "Interval",
        "name": "interval",
        "options": [
          {
            "selected": true,
            "text": "auto",
            "value": "$__auto_interval_interval"
          },
          {
            "selected": false,
            "text": "30s",
            "value": "30s"
          },
          {
            "selected": false,
            "text": "1m",
            "value": "1m"
          },
          {
            "selected": false,
            "text": "2m",
            "value": "2m"
          },
          {
            "selected": false,
            "text": "3m",
            "value": "3m"
          },
          {
            "selected": false,
            "text": "5m",
            "value": "5m"
          },
          {
            "selected": false,
            "text": "7m",
            "value": "7m"
          },
          {
            "selected": false,
            "text": "10m",
            "value": "10m"
          },
          {
            "selected": false,
            "text": "30m",
            "value": "30m"
          },
          {
            "selected": false,
            "text": "1h",
            "value": "1h"
          },
          {
            "selected": false,
            "text": "6h",
            "value": "6h"
          },
          {
            "selected": false,
            "text": "12h",
            "value": "12h"
          },
          {
            "selected": false,
            "text": "1d",
            "value": "1d"
          },
          {
            "selected": false,
            "text": "7d",
            "value": "7d"
          },
          {
            "selected": false,
            "text": "14d",
            "value": "14d"
          },
          {
            "selected": false,
            "text": "30d",
            "value": "30d"
          }
        ],
        "query": "30s,1m,2m,3m,5m,7m,10m,30m,1h,6h,12h,1d,7d,14d,30d",
        "queryValue": "",
        "refresh": 2,
        "skipUrlSync": false,
        "type": "interval"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "Swarm Cluster Monitoring",
  "uid": "364ipPOizz",
  "version": 8,
  "weekStart": ""
}

You can also check community created dashboards an import any that might impress you.

That marks the end of our guide on how to monitoring Docker swarm service metrics using Grafana/Prometheus and cAdvisor.

Other Tutorials

Monitor Docker Containers Metrics using Grafana

Connect to Remote Docker Environment on Docker Desktop

SUPPORT US VIA A VIRTUAL CUP OF COFFEE

We're passionate about sharing our knowledge and experiences with you through our blog. If you appreciate our efforts, consider buying us a virtual coffee. Your support keeps us motivated and enables us to continually improve, ensuring that we can provide you with the best content possible. Thank you for being a coffee-fueled champion of our work!

Photo of author
Kifarunix
Linux Certified Engineer, with a passion for open-source technology and a strong understanding of Linux systems. With experience in system administration, troubleshooting, and automation, I am skilled in maintaining and optimizing Linux infrastructure.

Leave a Comment