企业官网建设流程全解析

前言

本文演示kubernetes集群从v1.24.1升级到v1.29.15。

一、集群升级过程辅助命令

（1）查看节点上运行的pod。

kubectl get pod -o wide |grep <nodename>

（2）查看集群配置文件。

kubectl -n kube-system get cm kubeadm-config -o yaml

（3）查看当前集群节点。

kubectl get node

二、升级master节点

2.1、升级kubeadm。

# 更新包管理器
yum update
# 查看可用版本
apt-cache madison kubeadm
yum list | grep kubeadm
# 更新
yum update -y kubeadm
​
# 验证版本
kubeadm version

2.2、验证升级计划

（1）检查可升级到哪些版本，并验证你当前的集群是否可升级。

kubeadm upgrade plan

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.15
[upgrade/versions] kubeadm version: v1.29.15
I0327 11:28:43.151508 1125701 version.go:256] remote version is much newer: v1.32.3; falling back to: stable-1.29
[upgrade/versions] Target version: v1.29.15
[upgrade/versions] Latest version in the v1.28 series: v1.28.15
​
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.28.15 v1.29.15
​
Upgrade to the latest stable version:
​
COMPONENT CURRENT TARGET
kube-apiserver v1.28.15 v1.29.15
kube-controller-manager v1.28.15 v1.29.15
kube-scheduler v1.28.15 v1.29.15
kube-proxy v1.28.15 v1.29.15
CoreDNS v1.10.1 v1.11.1
etcd 3.5.15-0 3.5.16-0
​
You can now apply the upgrade by executing the following command:
​
kubeadm upgrade apply v1.29.15
​
_____________________________________________________________________
​
​
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
​
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________

注意下面的MANUAL字段：

The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
​
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________

指示哪些主键需要手动升级，如果是yes就要手动升级。

（2）显示哪些差异将被应用于现有的静态 pod 资源清单。

kubeadm upgrade diff 1.29.15

[upgrade/diff] Reading configuration from the cluster...
[upgrade/diff] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
--- /etc/kubernetes/manifests/kube-apiserver.yaml
+++ new manifest
@@ -40,7 +40,7 @@
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.15
+ image: registry.aliyuncs.com/google_containers/kube-apiserver:1.29.15
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
--- /etc/kubernetes/manifests/kube-controller-manager.yaml
+++ new manifest
@@ -28,7 +28,7 @@
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --use-service-account-credentials=true
- image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15
+ image: registry.aliyuncs.com/google_containers/kube-controller-manager:1.29.15
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
--- /etc/kubernetes/manifests/kube-scheduler.yaml
+++ new manifest
@@ -16,7 +16,7 @@
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
- image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.15
+ image: registry.aliyuncs.com/google_containers/kube-scheduler:1.29.15
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8

2.3、master节点升级

（1）升级到 1.29.15版本，此命令仅升级master节点（control plane）。

kubeadm upgrade apply v1.29.15

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.29.15"
[upgrade/versions] Cluster version: v1.28.15
[upgrade/versions] kubeadm version: v1.29.15
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.29.15" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-03-27-11-32-38/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2230279311"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-03-27-11-32-38/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-03-27-11-32-38/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2025-03-27-11-32-38/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config3777955110/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
​
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.29.15". Enjoy!
​
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

（2） 腾空节点，即将节点上除守护进程之外的其他进程调度到其他节点，同时将开启调度保护。

kubectl drain <nodename> --ignore-daemonsets

$ kubectl drain k8s-master1 --ignore-daemonsets
node/k8s-master1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-nxz4d, kube-system/kube-proxy-pbnk4
evicting pod kube-system/coredns-c676cc86f-twm96
evicting pod kube-system/coredns-c676cc86f-mdgbn
pod/coredns-c676cc86f-mdgbn evicted
pod/coredns-c676cc86f-twm96 evicted
node/k8s-master1 drained
​
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-nxz4d 1/1 Running 0 136m
kube-system coredns-c676cc86f-7stvs 0/1 Pending 0 60s
kube-system coredns-c676cc86f-vmkgv 0/1 Pending 0 60s
kube-system etcd-k8s-master1 1/1 Running 0 11m
kube-system kube-apiserver-k8s-master1 1/1 Running 0 10m
kube-system kube-controller-manager-k8s-master1 1/1 Running 0 10m
kube-system kube-proxy-pbnk4 1/1 Running 0 9m44s
kube-system kube-scheduler-k8s-master1 1/1 Running 0 9m58s
​
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready,SchedulingDisabled control-plane 162m v1.24.1

（3）升级kubelet与kubectl组件。

yum update -y kubelet

（4）重启 kubelet。

systemctl daemon-reload
systemctl restart kubelet

（5）解除调度保护。

kubectl uncordon <nodename>

三、升级node节点

（1）升级节点kubelet 配置。

kubeadm upgrade node

（2）腾空节点，同时开启调度保护，此命令请在master节点操作

kubectl drain <nodename> --ignore-daemonsets

（3）升级kubelet与kubectl组件。

yum update -y kubelet

（4）重启 kubelet。

systemctl daemon-reload
systemctl restart kubelet

（5）解除调度保护，master节点上执行该命令。

kubectl uncordon <nodename>

总结

每个版本的升级都不一样，所以要根据版本进行适当调整，不作为万能指导。 升级过程：

1. 升级master组件。

2. 升级worker节点组件，调度保护、排空节点、worker节点组件升级、解除保护。

Kubernetes集群的升级可以分为以下几个步骤：

1. 备份数据。在升级之前，需要备份Kubernetes集群的数据，包括访问控制、配置文件、数据卷等。

2. 选择升级方式。Kubernetes集群的升级方式可以分为两种：滚动升级和强制替换。滚动升级是指逐个升级每个节点，直到所有节点都升级完成。强制替换是指一次性替换所有节点，将旧节点直接替换为新节点。

3. 准备新版本。Kubernetes升级需要准备新版本的二进制文件和镜像文件。可以从Kubernetes官方网站下载最新版本的二进制文件和镜像文件，并上传到集群中的节点上。

4. 升级Master节点。首先需要升级Master节点，使用新版本的二进制文件替换旧版本的二进制文件，并启动新版本的Kubernetes API Server、ControllerKubernetes是一个快速发展的开源项目，为了保持其功能和安全性，集群的升级是必须的。

   • 查看升级文档：首先需要查看官方的升级文档，了解升级过程中需要注意的事项。

   • 备份数据：在升级前需要备份当前的数据，以防升级过程中的意外情况导致数据丢失。

   • 准备好备份：在升级前需要确保备份的可用性，以便在需要时能够 Manager和Scheduler。

5. 升级Node节点。接下来需要升级Node节点。首先需要将节点上的Kubelet和kube-proxy服务停止，使用新版本的二进制文件替换旧版本的二进制文件，然后启动新版本的Kubelet和kube-proxy服务。

6. 验证升级结果。升级完成后，需要验证恢复数据。

   • 升级前的测试：可以在测试环境中进行升级测试，以确保升级过程和升级后的集群正常运行。

   • 升级Node：首先需要升级每个Node节点中的Kubernetes组件，包括kubelet和kube-proxy等。

   • 升级Control Plane：然后需要升级Control Plane中的Kubernetes组件，包括kube-apiserver、kube-controller-manager和kube-scheduler等。

   • 升级Kubernetes对象：升级完Control Plane后，需要升级Kubernetes对象，如Deployment集群是否正常运行。可以使用kubectl命令查看集群的状态和资源对象的状态，确保所有的服务都能够正常访问。

7. 回滚升级。如果升级失败或出现问题，可以回滚到之前的版本。回滚的过程与升级的过程相同，只需要使用旧版本的二进制文件和镜像文件即可。

Kubernetes集群的升级需要仔细规划和准备，并按照一定的步骤进行操作。只有在备份数据、选择适当的升级方式、准备新版本、升级Master节点、升级Node节点、验证升级结果等步骤都完成后，才能确保集群的升级成功。、StatefulSet等。

升级后的检查：

• 验证集群状态：升级后需要验证集群的状态，包括Node节点的状态、Pod的状态、Service的状态等。

• 验证应用程序：升级后需要验证应用程序的运行状态，确保应用程序正常运行。

• 观察日志：如果发现问题，可以通过查看日志来排查问题原因。

Kubernetes集群升级是一个需要谨慎处理的过程，需要充分准备和测试，以确保升级过程的顺利和集群的稳定。在升级过程中，需要注意备份数据和备份的可用性，升级顺序和升级后的检查等问题，以确保集群的正常运行和应用程序的稳定性。