In this guide, we’ll walk you through the step-by-step process of how to safely upgrade kubeadm Kubernetes cluster. Upgrading a Kubernetes cluster that was originally deployed using kubeadm requires careful planning and execution to ensure minimal downtime and smooth operation.
Table of Contents
Upgrading Kubeadm Kubernetes Cluster
Important Cluster Upgrade Considerations
Before you even begin to think about the kubeadm Kubernetes cluster upgrade, there are important considerations you need to look into. Some of these are:
- Cluster Backup: Backup is Crucial. Murphy is always around the corner and you never know what might go wrong with the upgrade. You need to be able to recover your Kubernetes cluster data in case of any unforeseen circumstances. This is where backups become crucial. You can check how to backup and restore etcd in a Kubernetes cluster.
- Review Kubernetes Release Notes: Always review the official Kubernetes release notes for the version you’re targeting. This can help identify potential upgrade issues or breaking changes specific to that release.
- Static Control Plane Required: kubeadm upgrades only work with clusters using a static control plane and etcd pods or external etcd. If your setup differs, you might need to explore alternative upgrade methods.
- Disable Swap: Ensure swap is disabled on all cluster nodes. Check how to disable swap on Kubernetes nodes.
- Upgrade path: It is recommended to upgrade from a minor version to a next higher one (e.g., from 1.29.0 to 1.30.0), or from a patch release version to a higher one (e.g., from 1.30.1 to 1.30.2).
- It is recommended to match kubelet and kubeadm versions. However, in some cases, it is okay to use a kubelet that is three versions older than kubeadm.
Safely Upgrade Kubeadm Kubernetes Cluster
Kubernetes releases regular updates with new features, bug fixes, and security patches. Upgrading your cluster ensures you can leverage these improvements and maintain a secure environment for your applications.
What is the Order of Kubernetes Nodes Upgrade?
You need to start upgrading the control plane node before upgrading the worker nodes. If you have multiple control plane nodes, you need to start with the primary one, basically the one that is running as etcd cluster leader, first, then proceed with the rest of the control planes and the worker nodes in any order. This is summarized as:
- Upgrade primary control plane node
- Upgrade other control plane nodes
- Upgrade worker nodes
Determine the version of Kubernetes to upgrade to
To begin with, you need to check your current version of the cluster packages and of course the version you are upgrading to.
Current installed versions of kubeadm, kubectl, kubelet.
kubeadm version -o json{
"clientVersion": {
"major": "1",
"minor": "30",
"gitVersion": "v1.30.1",
"gitCommit": "6911225c3f747e1cd9d109c305436d08b668f086",
"gitTreeState": "clean",
"buildDate": "2024-05-14T10:49:05Z",
"goVersion": "go1.22.2",
"compiler": "gc",
"platform": "linux/amd64"
}
}
kubectl versionClient Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1
kubelet --versionKubernetes v1.30.1Cluster nodes;
kubectl get nodesNAME STATUS ROLES AGE VERSION
master-01 Ready control-plane 2d1h v1.30.1
master-02 Ready control-plane 2d1h v1.30.1
master-03 Ready control-plane 2d1h v1.30.1
worker-01 Ready <none> 2d1h v1.30.1
worker-02 Ready <none> 2d1h v1.30.1
worker-03 Ready <none> 2d1h v1.30.1
As you can see, we are on the release version v1.30.1 of Kubernetes. As per the release notes, the current latest release as of this writing is v1.30.2 which is one patch release version higher than our current running version.
As such, we will be upgrading our cluster to a patch release version.
Change the Kubernetes Package Repository
Chances are you are using the correct Kubernetes package repository and not the legacy ones (apt.kubernetes.io and yum.kubernetes.io) which have since deprecated. Instead, ensure you are using either of:
- pkgs.k8s.io
- pkgs.kubernetes.io
- packages.kubernetes.io
You can check which repository you are using;
grep -irlE "kubernetes|k8s" /etc/apt/This will print a file containing any of the kubernetes or k8s keywords.
Sample output;
/etc/apt/sources.list.d/kurbenetes.list
/etc/apt/trusted.gpg.d/k8s.gpgTherefore, our K8S package repository file is /etc/apt/sources.list.d/kurbenetes.list.
Get the contents of the file to confirm the repository address;
cat /etc/apt/sources.list.d/kurbenetes.listdeb https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /Based on the release version we are intending to upgrade to as seen in determining the version section above, the repository we have is enough to upgrade to the patch release v1.30.2.
Also, if you are using legacy repositories, be sure to update them before you can proceed.
Drain the Nodes Before Upgrade
Draining the nodes is not really necessary. However, If you are performing a minor version upgrade for any kubelet, you must first drain the node(s) that you are upgrading.
Upgrade Kubernetes Control Plane Nodes
As stated, you need to start the upgrade with control plane nodes. If you have multiple control plane nodes, start with the primary node. I would check which one is currently the etcd cluster leader to know the primary control plane.
sudo etcdctl endpoint status --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cluster \
-w table
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.60:2379 | 71adc069c6babb75 | 3.5.12 | 13 MB | false | false | 5 | 69810 | 69810 | |
| https://192.168.122.58:2379 | 7f56434149e2cc7f | 3.5.12 | 13 MB | false | false | 5 | 69810 | 69810 | |
| https://192.168.122.59:2379 | db4091f30b21595e | 3.5.12 | 13 MB | true | false | 5 | 69810 | 69810 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
As you can see, the node with the IP 192.168.122.59, is the current etcd leader and hence, we will treat this as the primary control plane.
Hence, login to your primary control plane. For example;
ssh [email protected]Now that we have confirmed that the package repository is all good, let’s check the current versions of kubeadm, kubelet and kubectl available.
apt-cache madison kubeadm kubelet kubectlSample output;
kubeadm | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubeadm | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubelet | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubelet | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubectl | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubectl | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
As you can see, the latest versions provided by the Kubernetes repositories for each of the package is v1.30.1-1.1
To get the latest release versions, run system package cache update.
sudo apt updateHit:1 http://de.archive.ubuntu.com/ubuntu noble InRelease
Get:2 https://download.docker.com/linux/ubuntu noble InRelease [48.8 kB]
Get:3 http://de.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Hit:4 http://de.archive.ubuntu.com/ubuntu noble-backports InRelease
Get:5 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:6 http://de.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [177 kB]
Get:7 http://de.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [49.1 kB]
Get:8 http://de.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [70.1 kB]
Get:9 http://de.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [14.3 kB]
Get:10 http://de.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [66.7 kB]
Get:11 http://de.archive.ubuntu.com/ubuntu noble-updates/universe Translation-en [25.1 kB]
Get:12 http://de.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [2,968 B]
Get:13 http://de.archive.ubuntu.com/ubuntu noble-updates/multiverse Translation-en [968 B]
Get:15 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [158 kB]
Get:16 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [41.5 kB]
Get:17 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [70.1 kB]
Get:18 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [14.3 kB]
Get:19 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [44.4 kB]
Get:20 http://security.ubuntu.com/ubuntu noble-security/universe Translation-en [17.0 kB]
Get:14 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb InRelease [1,186 B]
Get:21 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb Packages [5,226 B]
Fetched 1,059 kB in 1s (1,025 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
30 packages can be upgraded. Run 'apt list --upgradable' to see them.
Next, confirm the available versions;
apt-cache madison kubeadm kubelet kubectl kubeadm | 1.30.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubeadm | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubeadm | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubelet | 1.30.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubelet | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubelet | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubectl | 1.30.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubectl | 1.30.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
kubectl | 1.30.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
So, we now have the correct patch release version available and we ready to run the upgrades.
It is usually recommended to mark hold the Kubernetes packages to avoid automatic updates and keep a controlled version of the cluster. Check if these packages are hold;
sudo apt-mark showholdkubeadm
kubectl
kubelet
Thus, to upgrade any hold package, unhold it;
sudo apt-mark unhold kubeadm kubelet kubectlYou are now ready to upgrade.
Upgrade kubeadm on primary control plane node;
sudo apt updatesudo apt install kubeadmThe confirm;
kubeadm version -o yamlclientVersion:
buildDate: "2024-06-11T20:27:59Z"
compiler: gc
gitCommit: 39683505b630ff2121012f3c5b16215a1449d5ed
gitTreeState: clean
gitVersion: v1.30.2
goVersion: go1.22.4
major: "1"
minor: "30"
platform: linux/amd64
Upgrade the package on the other control plane nodes.
Once kubeadm package is upgraded across all control plane nodes, verify that the cluster can be upgraded and fetch the version to upgrade to on the primary control plane node;
sudo kubeadm upgrade plan[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.1
[upgrade/versions] kubeadm version: v1.30.2
...
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.30.2
...
If the cluster is upgrade-able, then you will get the command to use to apply the upgrade.
For example;
sudo kubeadm upgrade apply v1.30.2When prompted to confirm, do confirm and proceed. It may take a while to complete the upgrade.
After upgrade;
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.2". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
While it is not really required, but the for the sake of consistency, run the command on other control plane node just to ensure everything is upgraded.
sudo kubeadm upgrade nodeNext, upgrade the kubelet and the kubectl tool to the same version on all nodes (you can start with the leader):
As mentioned before, for minor release upgrade, you would have to drain other control plane nodes before proceeding with kubelet/kubectl upgrade.
Since we are doing a patch release, just proceed.
sudo apt install kubelet kubectlThe services, kubelet, will be restarted during upgrade.
After you are done with upgrade, hold the packages on all control plane nodes.
sudo apt-mark hold kubeadm kubelet kubectl(Again, if you had drained the other nodes, uncordon them)
Confirm the nodes versions;
kubectl get nodesNAME STATUS ROLES AGE VERSION
master-01 Ready control-plane 2d3h v1.30.2
master-02 Ready control-plane 2d2h v1.30.2
master-03 Ready control-plane 2d2h v1.30.2
worker-01 Ready <none> 2d2h v1.30.1
worker-02 Ready <none> 2d2h v1.30.1
worker-03 Ready <none> 2d2h v1.30.1
Upgrade Worker Nodes
Next, upgrade the worker nodes one by one.
Unhold the packages on all worker nodes;
sudo apt-mark unhold kubeadm kubelet kubectlUpgrade kubeadm;
sudo apt update;sudo apt install kubeadmUpgrade the Kubelet configuration;
sudo kubeadm upgrade nodeSample output;
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2070136212/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
Again the node drain condition applies here! So proceed accordingly.
Upgrade kubelet and kubectl;
sudo apt update;sudo apt install kubectl kubeletKubelet service will be restarted during upgrade.
If you had drained the node, it is now time to uncordon it.
Upgrade the next worker nodes using the same approach.
Mark and hold the packages;
sudo apt-mark hold kubeadm kubelet kubectlVerify Cluster Upgrade
You can now check the status of the cluster upgrade by listing the nodes;
kubectl get nodesNAME STATUS ROLES AGE VERSION
master-01 Ready control-plane 2d3h v1.30.2
master-02 Ready control-plane 2d3h v1.30.2
master-03 Ready control-plane 2d3h v1.30.2
worker-01 Ready <none> 2d3h v1.30.2
worker-02 Ready <none> 2d3h v1.30.2
worker-03 Ready <none> 2d3h v1.30.2
And all nodes are of the same version. Ensure that all nodes are in a ready state.
After a successful upgrade, I would recommended that you review cluster logs for any errors or warnings. Additionally, consider testing your applications to ensure they function as expected on the upgraded cluster.
Conclusion
Upgrading a Kubernetes cluster created with kubeadm involves several critical steps to maintain the reliability and security of your infrastructure. By following these steps, you can successfully upgrade your cluster while minimizing downtime and ensuring a smooth transition. Remember, planning, attention to detail, and following best practices are key to a seamless upgrade experience.
Read more on upgrading the cluster documentation.
