This tutorial serves as a guide to demystify different type of Kubernetes pods, in essence, Static Pods vs Mirror Pods vs DaemonSets. In Kubernetes, a pod is the fundamental unit of deployment, representing a logical collection of one or more containers that share resources resources such as storage volumes, network namespace, and IP addresses within the Kubernetes ecosystem. Pods serve as the basic building blocks for applications and services, encapsulating containers with shared storage/networking and a specification for how to run them.
Table of Contents
Static Pods vs Mirror Pods vs DaemonSets
In a Kubernetes ecosystem, what exactly is the difference between static pods, mirror pods and daemonsets?
Kubernetes Static Pods
What is a static pod in Kubernetes?
A static pod is a type of pod in Kubernetes that is managed directly by the kubelet on a specific node, without the involvement of the Kubernetes control plane (kube-apiserver). They are neither managed by controllers like Deployment or ReplicaSet.
Static Pods’ configuration manifest files are typically located in a directory watched by the kubelet (e.g., /etc/kubernetes/manifests). The kubelet monitors this directory and starts or stops these pods as necessary. Depending on your container runtime interface, you can watch these pods status by listing the running containers and checking their respective containers.
e.g
sudo crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
b44d8e7641002 3861cfcd7c04c 23 hours ago Running etcd 0 ed7b93674545c etcd-master-02
...
0c0e38d096416 53c535741fb44 25 hours ago Running kube-proxy 2 607aed66a0eef kube-proxy-46hnc
0535cad22c9a1 56ce0fd9fb532 27 hours ago Running kube-apiserver 1 e4bdd152fae58 kube-apiserver-master-02
20e4134130f4d e874818b3caac 27 hours ago Running kube-controller-manager 1 74e6d07084302 kube-controller-manager-master-02
ad611bac53b83 7820c83aa1394 27 hours ago Running kube-scheduler 1 52f0cab4cfeb3 kube-scheduler-master-02
The main Kubernetes control plane components (apiserver, scheduler, etcd, controller-manager) usually run as static pods. In a Kubeadm cluster, their manifests YAML configuration files reside under /etc/kubernetes/manifests.
ls /etc/kubernetes/manifests -1
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
Sample etcd.yaml contents.
sudo cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.122.59:2379
creationTimestamp: null
labels:
component: etcd
tier: control-plane
name: etcd
namespace: kube-system
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://192.168.122.59:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --experimental-initial-corrupt-check=true
- --experimental-watch-progress-notify-interval=5s
- --initial-advertise-peer-urls=https://192.168.122.59:2380
- --initial-cluster=master-02=https://192.168.122.59:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://192.168.122.59:2379
- --listen-metrics-urls=http://127.0.0.1:2381
- --listen-peer-urls=https://192.168.122.59:2380
- --name=master-02
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
image: registry.k8s.io/etcd:3.5.12-0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /health?exclude=NOSPACE&serializable=true
port: 2381
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: etcd
resources:
requests:
cpu: 100m
memory: 100Mi
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /health?serializable=false
port: 2381
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /var/lib/etcd
name: etcd-data
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
priority: 2000001000
priorityClassName: system-node-critical
securityContext:
seccompProfile:
type: RuntimeDefault
volumes:
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
- hostPath:
path: /var/lib/etcd
type: DirectoryOrCreate
name: etcd-data
status: {}
Kubelet service maybe configured to use a different static pod path. Therefore, to find out the current static pod path, you can check the value of the staticPodPath parameter in the Kubelet configuration file, /var/lib/kubelet/config.yaml.
grep staticPodPath /var/lib/kubelet/config.yaml
staticPodPath: /etc/kubernetes/manifests
Static Pods usually have the hostname of the node under which they are running suffixed on their names.
See example below;
kubectl get nodes -n kube-system
NAME READY STATUS RESTARTS AGE
...
etcd-master-01 1/1 Running 1 (27h ago) 3d2h
etcd-master-02 1/1 Running 0 23h
etcd-master-03 1/1 Running 0 23h
kube-apiserver-master-01 1/1 Running 1 (27h ago) 3d2h
kube-apiserver-master-02 1/1 Running 1 (27h ago) 3d2h
kube-apiserver-master-03 1/1 Running 1 (27h ago) 3d2h
kube-controller-manager-master-01 1/1 Running 1 (27h ago) 3d2h
kube-controller-manager-master-02 1/1 Running 1 (27h ago) 3d2h
kube-controller-manager-master-03 1/1 Running 1 (27h ago) 3d2h
...
...
kube-scheduler-master-01 1/1 Running 1 (27h ago) 3d2h
kube-scheduler-master-02 1/1 Running 1 (27h ago) 3d2h
kube-scheduler-master-03 1/1 Running 1 (27h ago) 3d2h
For examples on how to create Static Pods, refer to the documentation.
Kubernetes Mirror Pods
Mirror Pods are virtual representations of Static Pods. They provide a mechanism for local visibility of Static Pods within the Kubernetes API server. Mirror pods make it easy to discover and retrieve information about Static Pods using kubectl commands on the API server. See example of getting static pods above.
In essence, Mirror Pods are a byproduct of Static Pods. The kubelet creates a Mirror Pod for every detected Static Pod manifest file. However, any changes made to a Static Pod manifest only affect the actual Static Pod on the node it is running.
Kubernetes DaemonSets
A DaemonSet on the other hand is not a pod itself, but rather a resource that manages pods. So what is it exactly? A DaemonSet is a Kubernetes resource that ensure that all (or some) nodes run a copy of a specific pod. They automatically add or remove pods as nodes are added or removed from the cluster.
Common use cases include;
- Logging and Monitoring: DaemonSets are perfect for deploying logging agents like Fluentd or monitoring tools like Prometheus Node Exporter on every node in a cluster.
- Node-Specific Utilities: You can use DaemonSets to deploy utilities specific to each node, such as local volume storage provisioners.
- …
So, what mechanism does the DaemonSets use to ensure that all eligible nodes run a copy of a specific Pod?
- Node Selector: DaemonSets use a node selector to determine which nodes are eligible to run the DaemonSet pod. A node selector is a field in the DaemonSet specification (spec.template.spec.nodeSelector) that specifies a set of key-value pairs. These pairs match labels assigned to nodes. Only nodes that match these labels are considered eligible to run the DaemonSet pod.
- Node Affinity: In addition to node selectors, DaemonSets can also use node affinity settings (spec.template.spec.affinity.nodeAffinity) to further refine node selection. Node affinity allows DaemonSets to specify more complex rules based on node labels, such as required or preferred nodes for running the DaemonSet pod.
- Taints and Tolerations: DaemonSets can also utilize Kubernetes’ taints and tolerations mechanism (spec.template.spec.tolerations) to ensure that DaemonSet pods can tolerate specific node conditions (taints). This allows DaemonSets to schedule pods on nodes that have specific taints applied, ensuring flexibility in node selection based on cluster requirements.
To check available DaemonSets in a Kubernetes cluster, simply run the command;
kubectl get daemonsets --all-namespaces
Sample output;
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
calico-system calico-node 6 6 6 6 6 kubernetes.io/os=linux 3d2h
calico-system csi-node-driver 6 6 6 6 6 kubernetes.io/os=linux 3d2h
kube-system kube-proxy 6 6 6 6 6 kubernetes.io/os=linux 3d2h
To get more details about a deamonset;
kubectl describe daemonset kube-proxy -n kube-system
Name: kube-proxy
Selector: k8s-app=kube-proxy
Node-Selector: kubernetes.io/os=linux
Labels: k8s-app=kube-proxy
Annotations: deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 6
Current Number of Nodes Scheduled: 6
Number of Nodes Scheduled with Up-to-date Pods: 6
Number of Nodes Scheduled with Available Pods: 6
Number of Nodes Misscheduled: 0
Pods Status: 6 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: k8s-app=kube-proxy
Service Account: kube-proxy
Containers:
kube-proxy:
Image: registry.k8s.io/kube-proxy:v1.30.2
Port: <none>
Host Port: <none>
Command:
/usr/local/bin/kube-proxy
--config=/var/lib/kube-proxy/config.conf
--hostname-override=$(NODE_NAME)
Environment:
NODE_NAME: (v1:spec.nodeName)
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/kube-proxy from kube-proxy (rw)
Volumes:
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
Priority Class Name: system-node-critical
Node-Selectors: kubernetes.io/os=linux
Tolerations: op=Exists
Events: <none>
Read more about DaemonSets on the documentation page.
Conclustion
In summary;
- DaemonSet: Ensures a specific Pod runs on every node (or a subset) in your cluster. Ideal for deploying ubiquitous services like logging agents or monitoring tools across all nodes.
- Static Pod: A dedicated pod deployed directly on a specific node, managed by the
kubelet
service. - Mirror Pod: A virtual representation of a Static Pod residing within the Kubernetes API server. Provides limited visibility and management capabilities for the actual Static Pod running on specific nodes.