upgrade k8s (by quqi99)

upgrade k8s (by quqi99)

作者：张华 发表于：2023-11-17
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明()

本文只是从网上搜索一些升级k8s的理论学习，下面的步骤未实际测试。

理论学习 - upgrade k8s from 1.20.6 to 1.20.15 by kubeadm

refer: 云原生Kubernetes：K8S集群版本升级(v1.20.6 - v1.20.15) -

1, check verion
kubectl get nodes
kubectl version
kubeadm version
kubectl get componentstatuses
kubectl get deployments --all-namespaces2, upgrade kubeadm
apt install kubeadm=1.20.15*
kubeadm version3, upgrade master1
kubeadm upgrade plan
kubeadm upgrade apply v1.20.15
#in case it's offline mode, need to load the image first
docker image load -i kube-apiserver:v1.15.1.tar 
docker image load -i kube-scheduler:v1.15.1.tar 
docker image load -i kube-controller-manager:v1.15.1.tar 
docker image load -i kube-proxy:v1.15.1.tar
docker image list4, upgrade master2, but pls use 'kubeadm upgrade node' instead of 'kubeadm upgrade apply'
apt install kubeadm=1.20.15*
kubeadm version
kubeadm upgrade node5, upgrade kubelet and kubectl on master1
kubectl drain master1 --ignore-daemonsets
apt install kubelet=1.20.15*
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon master1
kubectl get nodes6, upgrade kubelet and k on master2
kubectl drain master2 --ignore-daemonsets
apt install kubelet=1.20.15*
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon master2
kubectl get nodes7, upgrade worker
apt install kubeadm=1.20.15*
kubeadm version
kubeadm upgrade node
kubectl drain worker1 --ignore-daemonsets --delete-emptydir-data
apt install kubelet=1.20.15*
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon worker1
kubectl get nodes8, verify the cluster
kubectl get nodes
kubeadm alpa certs check-expiration
kubectl get pods -n kube-system

实践 - Upgrade k8s from 1.21 to 1.26 by charm

下面upgrade采用n-1模式（Upgrade path: 1.21 --> 1.22 --> 1.23 --> 1.24 --> 1.25 --> 1.26），已在实际环境验证，it works.

0, deply k8s 1.21 test env via the bundle - .21/bundle.yamljuju scp kubernetes-master/0:config ~/.kube/config1, backup db
juju run-action etcd/leader snapshot --wait
juju scp etcd/1:/home/ubuntu/etcd-snapshots/etcd-snapshot-2023-11-21-02.25. .2, upgrade containerd to the latest stable charm revision (it's 1.23 without --revision, from revision 607 to 200) - .22/upgradingpls wait for the units to turn back to "active" state - /
juju upgrade-charm containerd
watch juju status containerd
# fix 'blocked' and 'idle' state
juju run -u <unit_in_blocked_state> 'hooks/update-status'3, upgrade etcd to 1.23 directly without --revision (from revision 607 to 768), and wait for the status to turn back to "active" 
juju upgrade-charm etcd
watch juju status etcd4, upgrade the additional charms to 1.23 one by one (after waiting for the status to turn back to "active" then start the next one)
#juju upgrade-charm easyrsa --revision 420  #we don't use --revision to use 420(1.22), we will upgrade it to 1.23 directly
juju upgrade-charm easyrsa                  #from 395(1.21) to 441(1.23)
juju upgrade-charm flannel                  #from 571 to 619
juju upgrade-charm kubeapi-load-balancer    #from 814 to 866
#juju upgrade-charm calico
#juju upgrade-charm hacluster-kubernetes-master
#juju upgrade-charm hacluster-dashboard
#juju upgrade-charm hacluster-keystone
#juju upgrade-charm hacluster-vault
#juju upgrade-charm telegraf
#juju upgrade-charm public-policy-routing
#juju upgrade-charm landscape-client
#juju upgrade-charm filebeat
#juju upgrade-charm ntp
#juju upgrade-charm nfs-client
#juju upgrade-charm nrpe-container
#juju upgrade-charm nrpe-host
#juju upgrade-charm prometheus-ceph-exporter5, upgrade k8s master from 1.21 to 1.22 (from revision 1034 to 1078)1.22 is in old charmstore, not in charmhub - /+bug/2043783/comments/2
#the revision for 1.22 is 1078 according to 1078 - .22/bundle.yaml
#should use ch: instead of cs: for the old storestore - 
#juju refresh kubernetes-master --switch ch:containers-kubernetes-master --revision 1078
#ERROR --switch and --revision are mutually exclusive
juju upgrade-charm kubernetes-master --revision 1078
watch juju status kubernetes-master          #wait for the status to turn back to "active"
juju config kubernetes-master channel=1.22/stable
juju run-action kubernetes-master/0 upgrade  #one by one as well, mainly install kube-proxy snap - /
juju run-action kubernetes-master/1 upgrade6, upgrade k8s worker from 1.21 to 1.22 (from revision 788 to 816)
# .22/bundle.yaml
juju upgrade-charm kubernetes-worker --revision 816
juju config kubernetes-worker channel=1.22/stable
juju run-action kubernetes-worker/0 upgrade         #one by one as well, mainly install kube-proxy snap
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade7, upgrade k8s master and worker from 1.22 to 1.23, only some charms (not all) are in charmhub for 1.23 - .23/bundle.yaml
juju upgrade-charm kubernetes-master --revision 1106
watch juju status kubernetes-master
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju upgrade-charm kubernetes-worker --revision 838
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade8, when upgrading charmed k8s to 1.24, which has relocated from juju store to charmhub, this means that upgrading each charm will require the use of --switch during the upgrade.# .24/bundle.yaml
juju upgrade-charm containerd --switch ch:containerd --channel 1.24/stable   #channel=1.24/stable, rev=27
juju upgrade-charm etcd --switch ch:etcd --channel 1.24/stable               #channel=1.24/stable, rev=701
juju upgrade-charm easyrsa --switch ch:easyrsa --channel 1.24/stable         #channel=1.24/stable, rev=15
juju upgrade-charm flannel --switch ch:flannel --channel 1.24/stable         #channel=1.24/stable, rev=28
#juju upgrade-charm calico --switch ch:calico --channel 1.24/stable
#juju upgrade-charm hacluster-kubernetes-master --switch ch:hacluster --channel latest/stable
#juju upgrade-charm hacluster-dashboard --switch ch:hacluster --channel latest/stable
#juju upgrade-charm hacluster-kesystone --switch ch:hacluster --channel latest/stable
#juju upgrade-charm hacluster-vault --switch ch:hacluster --channel latest/stable
#juju upgrade-charm telegraf --switch ch:telegraf --channel latest/stable
#juju upgrade-charm public-policy-routing
#juju upgrade-charm landscape-client
#juju upgrade-charm filebeat --switch ch:filebeat --channel latest/stable
#juju upgrade-charm ntp --switch ch:ntp --channel latest/stable
#juju upgrade-charm nfs-client
#juju upgrade-charm nrpe-container --switch ch:nrpe --channel latest/stable
#juju upgrade-charm nrpe-host --switch ch:nrpe --channel latest/stable
#juju upgrade-charm prometheus-ceph-exporter --switch ch:prometheus-ceph-exporter --channel latest/stable#for the charm 1.24, kubernetes-master is renamed to kubernetes=control-plane as well
juju upgrade-charm kubernetes-master --switch ch:kubernetes-control-plane --channel 1.24/stable  #channel=1.24/stable, rev=171
juju config kubernetes-master channel=1.24/stable  #for the following 'run-action ... upgrade'
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
# for the message "ceph-storage relation deprecated, use ceph-client instead" 
juju remove-relation kubernetes-master:ceph-storage ceph-mon
juju upgrade-charm kubernetes-worker --switch ch:kubernetes-worker --channel 1.24/stable
juju config kubernetes-worker channel=1.24/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade
kubectl get nodes -A9, upgrade k8s from 1.24 to 1.25
juju upgrade-charm containerd --channel 1.25/stable         #channel=1.25/stable, rev=41
juju upgrade-charm etcd --channel 1.25/stable               #channel=1.25/stable, rev=718
juju upgrade-charm easyrsa --channel 1.25/stable            #channel=1.25/stable, rev=26
juju upgrade-charm flannel --channel 1.25/stable            #channel=1.25/stable, rev=49
juju upgrade-charm kubernetes-master --channel 1.25/stable  #channel=1.25/stable, rev=219
juju config kubernetes-master channel=1.25/stable
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju config kubernetes-worker channel=1.25/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade
kubectl get nodes -A10, upgrade k8s from 1.25 to 1.26, need a 'juju refresh' after 'juju run-action'
juju upgrade-charm containerd --channel 1.26/stable         #channel=1.26/stable, rev=54
juju upgrade-charm etcd --channel 1.26/stable               #channel=1.26/stable, rev=728
juju upgrade-charm easyrsa --channel 1.26/stable            #channel=1.26/stable, rev=33
juju upgrade-charm flannel --channel 1.26/stable            #channel=1.26/stable, rev=63
juju upgrade-charm kubeapi-load-balancer --channel 1.26/stable
juju upgrade-charm kubernetes-master --channel 1.26/stable  #channel=1.26/stable, rev=247juju config kubernetes-master channel=1.26/stable
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju refresh kubernetes-master --channel 1.26/stablejuju config kubernetes-worker channel=1.26/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade
juju refresh kubernetes-worker --channel 1.26/stable
kubectl get nodes -A

客户问题

客户只是从1.21往1.22升级时运行‘juju upgrade-charm kubernetes-master’时发现charmhub里没有1.22 (1.22太老已经被删除了，它只有1.23到1.29, 可用‘juju info --series focal kubernetes-worker’查看), k8s升级没法从1.21直接跳到1.23啊。
那没有1.22怎么办？找产品team的把1.22加上？另外，可以使用local charm过渡吗 ( .22%2Bck2 ). 下面方法可以build 1.22 charm, 但可以从1.21升级到local 1.22再升级到1.23 on charmhub? 这个需要测试

#'python_version < "3.8"' don't match your environmentnIgnoring Jinja2: markers 'python_version >= "3.0" and python_version <= "3.4"'
# ppa:deadsnakes/ppa only has >=python3.5 as well, so have to use xenial instead
#juju add-machine --series jammy --constraints "mem=16G cores=8 root-disk=100G" -n 2
juju add-machine --series xenial -n 1
juju ssh 0
# but xenial is also using python3.5, and it said:
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.那在这个xenial的基础上继续通过源码来编译python3.2之后再build charm成功了
sudo apt-get install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev wget -y
wget .2.6/Python-3.
tar -xf Python-3.
cd Python-3.2.6/
./configure --enable-optimizations
make -j$(nproc)
sudo make altinstall
python3.2 --version
alias python=python3.2
alias python3=python3.2
sudo apt install build-essential -y
sudo apt install python3-pip python3-dev python3-nose python3-mock -y
cd $CHARM_LAYERS_DIR/..
charm build --debug ./layers/kubernetes-worker/
cd /home/ubuntu/charms/layers/builds/kubernetes-worker
zip -rq ../kubernetes-worker.charm .sudo snap install charm --classic
mkdir -p /home/ubuntu/charms
mkdir -p ~/charms/{layers,interfaces}
export JUJU_REPOSITORY=/home/ubuntu/charms
export CHARM_INTERFACES_DIR=$JUJU_REPOSITORY/interfaces
export CHARM_LAYERS_DIR=$JUJU_REPOSITORY/layers
export CHARM_BUILD_DIR=$JUJU_REPOSITORY/layers/builds
cd $CHARM_LAYERS_DIR
git clone .git kubernetes-worker
cd kubernetes-worker && git checkout -b 1.22+ck2 1.22+ck2
sudo apt install python3-virtualenv tox -y
cd .. && charm build --debug layers/kubernetes-worker/
#cd ${JUJU_REPOSITORY}/layers/builds/kubernetes-worker && tox -e func
cd /home/ubuntu/charms/layers/builds/kubernetes-worker
zip -rq ../kubernetes-worker.charm .

还有一个问题是，在1.24时charm kubernetes-master被更名为kubernetes-control-plane, charmhub中的版本目前是从1.23开始的已经是kubernetes-control-plane了，所以‘juju info --series focal kubernetes-control-plane’能看到信息，而‘juju info --series focal kubernetes-master’是看不到的。

测试环境搭建

20231120更新 - 注：下面的方法只改部分的容易出现问题，最终报了这个错“2023-11-20 04:35:53 INFO unit.kubernetes-control-plane/0.:316 status-set: waiting: Failed to setup auth-webhook tokens; will retry”，其实搭1.21的环境很简单，直接用1.21的release bundle即可。见：.21/bundle.yaml

需要搭建一个和客户环境尽可能一样的环境， 客户环境还在使用下列768的charmstore里的老charm.

charmstore - 
charmhub - 
charmstore - 
charmhub - 

  kubernetes-worker:charm: cs:~containers/kubernetes-worker-768channel: stable

目前测试工具默认生成的是用的charmhub中的latest/stable.

    charm: ch:kubernetes-control-planechannel: latest/stable

我的初步想法是使用"–use-stable-charms --charmstore --k8s-channel stable --revision-info ./juju_"在产生bundle时将ch变回cs：

./generate-bundle.sh -s focal --name k8s --num-control-planes 2 --num-workers 2 --calico --use-stable-charms --charmstore --k8s-channel stable --revision-info ./juju_

但是因为客户给的是juju_export_bundle的输出，而不是juju status的输出，所以上面的–revision-info不work, 这样上面命令产生的输出是：

cs:~containers/kubernetes-worker
channel: latest/stable

这样我的设想是：

• 手动编辑b/k8s/kubernetes.yaml只是将k8s master与k8s worker改成和客户一样的版本。 另外，charmstore里的stable revision 768已经不存在了，目前charmstore website已经停服了，所以没法查比768还新一点的版本是哪个，可以通过’juju deploy --series focal cs:containers-kubernetes-worker-770 test’增加数字一个个试，最终试出来是770. 所以最后改成：cs:~containers/kubernetes-worker-770
• 其他charm还是用charmstore里最新的stable版本, 这样做upgrade测试的时候这些upgrade都省了，将精力只集中在客户有问题的k8s worker 上, 所以最后改成：cs:~containers/kubernetes-master-1008

修改完b/k8s/kubernetes.yaml之后运行‘./generate-bundle.sh --name k8s --replay --run’完成测试环境搭建。

./generate-bundle.sh --name k8s --replay --run
watch -c juju status --color                                                    
sudo snap install kubectl --classic                                             
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config         
source <(kubectl completion bash)                                               
kubectl completion bash |sudo tee /etc/bash_completion.d/kubectl                
kubectl cluster-info

上游求助

似乎1.21可以直接升级到1.23 - /+bug/2043783/comments/1
1.23是一个临时的charmstore版本，1.24才是第一个正式charmstore版本(ch-only)，这是为什么在charmstore里看不到1.22的原因。
从这里（.22/bundle.yaml）可以找到 1.22发布时的revision号（如，对于workers是： cs:~containers/kubernetes-worker-816)

1, 当切换到charmstore的老版本charm时(如816)时用–switch, 但注意此时需要用ch来代替cs

# switch back to old charm 816 in charmstore (NOTE: should use ch instead of cs here) - 
juju refresh kubernetes-worker --switch ch:containers-kubernetes-worker --revision 816

2, 再切换回charmhub (1.24+)时，需要再到–switch

juju refresh kubernetes-worker --switch ch:kubernetes-worker --channel 1.2x/stable

3, 1.24版本有个从kubernetes-master到kubernetes-control-plane的更名.24/upgrading

juju refresh kubernetes-master --switch ch:kubernetes-control-plane --channel 1.24/stable

定稿 - test n-2 upgrade

下面n-2模式会work吗？

• 从1.21直接升级到1.23，跳过1.22 (1.21与1.23都在charmstore里）
• 从1.23升级到1.24, (1.24是charmhub里的第一个版本)
• 从1.24直接升级到1.26, n-2也跳过1.25 ?
• 下面运行upgrade-charm升级charm的命令可以同时运行，之后得等juju status能看到所有为active状态( 中间有时可运行 'juju run -u <unit_in_blocked_state> ‘hooks/update-status’ 命令解决), 之后再运行 'juju run-action xxx upgrade’命令来升级k8s-master与k8s-worker上的kube-proxy snap

1, upgrade k8s from 1.21 to 1.23 directly (1.23 is the last charm in the charmstore) - .23/bundle.yaml
juju upgrade-charm containerd --revision 200
juju upgrade-charm etcd --revision 655
juju upgrade-charm easyrsa --revision 441
juju upgrade-charm flannel --revision 619
juju upgrade-charm kubeapi-load-balancer --revision 866
juju upgrade-charm kubernetes-master --revision 1106
juju upgrade-charm kubernetes-worker --revision 838# 上面的命令可以同时运行，但不能同时和下面的命令同时运行
watch juju status  #wait for the status to 'active' againjuju config kubernetes-master channel=1.23/stable
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju config kubernetes-worker channel=1.23/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade2, upgrade k8s from 1.23 to 1.24, 1.24 is the first version in the charmhub, and kubernetes-master is renamed to kubernetes=control-plane as well
juju upgrade-charm containerd --switch ch:containerd --channel 1.24/stable
juju upgrade-charm etcd --switch ch:etcd --channel 1.24/stable
juju upgrade-charm easyrsa --switch ch:easyrsa --channel 1.24/stable
juju upgrade-charm flannel --switch ch:flannel --channel 1.24/stable
juju upgrade-charm kubernetes-master --switch ch:kubernetes-control-plane --channel 1.24/stable
juju upgrade-charm kubernetes-worker --switch ch:kubernetes-worker --channel 1.24/stablewatch juju status  #wait for the status to 'active' againjuju config kubernetes-master channel=1.24/stable
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju config kubernetes-worker channel=1.24/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade3, upgrade k8s from 1.24 to 1.26 (n+2 upgrade)
juju upgrade-charm containerd --channel 1.26/stable
juju upgrade-charm etcd --channel 1.26/stable
juju upgrade-charm easyrsa --channel 1.26/stable
juju upgrade-charm flannel --channel 1.26/stable
juju upgrade-charm kubernetes-master --channel 1.26/stable
juju config kubernetes-worker channel=1.26/stablewatch juju status  #wait for the status to 'active' againjuju config kubernetes-master channel=1.26/stable
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
juju config kubernetes-worker channel=1.26/stable
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
juju run-action kubernetes-worker/2 upgrade# 1.26引入了'juju refresh', 如果不运行下列refresh命令，那么用'kubectl get nodes'会看到worker的版本始终是1.24，而master是1.26，这样master端会一直看到： Waiting for kubelet,kube-proxy to start
#juju refresh kubernetes-master --channel 1.26/stable
juju refresh kubernetes-worker --channel 1.26/stable
kubectl get nodes

其它问题：

• upgrade juju 2.9.11 to 2.9.29+ to avoid: /+bug/1968931

1. backup the controller
juju model-config -m controller backup-dir="/var/snap/juju-db/common"
juju create-backup -m controller
2, Upgrade your juju client
snap refresh juju --channel=2.9/stable:
3, Upgrade the controller
juju upgrade-controller
4, Once done, upgrade your model
juju upgrade-model --dry-run
juju upgrade-model

• 当中间出现’juju status’不能回到active时，多半运行’juju run -u <unit_in_blocked_state> ‘hooks/update-status’'即可解决（或者重启unit也行). 对于ntp遇到的这种问题可能还要额外之前重启chronyd, 对于kubernetes-api遇到的这种问题可能还要额外之前重启crm api
• 如果涉及到ceph, 可能在升级到1.24时，需要处理 # for the message “ceph-storage relation deprecated, use ceph-client instead” , juju remove-relation kubernetes-master:ceph-storage ceph-mon
• 如果涉及到calico, 如果遇到‘stat /opt/calicoctl/kubeconfig: no such file or directory’， 可以用其他worker的/opt/calicoctl/kubeconfig作为workaround
• 注意：虽然自己测的可以skip level upgrade, 但这并不是官方推荐，向客户推荐时应该与官方保持一致 (), 就像o7k也一般不帮助客户升级的，要升级可以用bs服务，升级挂了是要背锅的