本次主要是为了模拟公司业务从非k8s如何迁移到k8s，及迁移前的准备工作。
（1）了解当前的业务环境，详细如下：
a. 当前项目中有哪些服务，因为后面要针对这些服务来制作容器镜像（Dockerfile）。
b. 完整的业务调用链路，前端、后端、存储、其他中间件等。
b. 部署到k8s后，进行运行测试、功能测试、压力测试等（由测试部门负责，运维不管）。（2）后期业务保障
a. 监控
b. 告警
c. 日志收集
d. cicd
e. 链路追踪等（3）理清上述问题后，列出迁移计划，什么时候做什么事，什么时候完成等。
a. 如购买服务器资源，列出采购清单。
b. 如前期的网络地址规划，创建VPC，划分子网。
（1）提高资源利用率、节约部署IT成本。
a. 自动化部署：使用Kubernetes可以实现自动化部署，可以大大减少手动部署的时间和人力成本。
b. 弹性伸缩：Kubernetes可以根据业务负载自动调整应用程序的副本数量，确保资源利用率高效，避免资源浪费。
c. 负载均衡：Kubernetes可以自动进行负载均衡，将请求分配到不同的应用程序副本中，确保每个应用程序都能够平均分配负载，提高资源利用率。
d. 资源隔离：Kubernetes可以实现资源隔离，确保每个应用程序都有足够的资源可用，避免资源争用和应用程序之间的干扰。
e. 可靠性和可用性：Kubernetes可以实现应用程序的高可靠性和可用性，确保业务能够持续运行，减少因为故障导致的停机时间和业务损失。（2）提高部署效率，基于kubernetes实现微服务的快速部署与交付、容器的批量调度与秒级启动。（3）实现横向扩容、灰度部署、回滚、链路追踪、服务治理等。（4）可根据业务负载进行自动弹性伸缩。（5）容器将环境和代码打包在镜像内，保证了测试与生产运行环境的一致性。（6）紧跟云原生社区技术发展的步伐，不给公司遗留技术债，为后期技术升级夯实了基础。（7）为个人储备前沿技术，提高个人level。
如上图：
（1）首先我们准备好基础镜像，比如centos、Ubuntu、alpine，在里面安装好我们所需的基础命令，这样第一个基础镜像就打好了。（2）基于自定义基础镜像，根据开发的需求来进行构建业务镜像，如上面有python、java、go、nginx。（3）再根据基础业务镜像，来区分不同的业务场景，比如A项目使用tomcat基础镜像，B项目使用nginx基础镜像。（4）最后就可以按照业务场景来选择使用什么镜像了。
[root@containerd-build-image ~]# mkdir -p /opt/k8s-data/dockerfile/system/centos
[root@containerd-build-image ~]# cd /opt/k8s-data/dockerfile/system/centos[root@containerd-build-image centos]# cat Dockerfile
#自定义Centos 基础镜像
FROM centos:7.9.2009 # 替换yum源
po /pos.d# 添加所需软件包并安装
ADD filebeat-7.12.1-x86_64.rpm /tmp
RUN yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop &&
rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm &&
ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime #&& useradd nginx -u 2088# 上传所需文件
[root@containerd-build-image centos]# mv /root/filebeat-7.12.1-x86_64.rpm ./
[root@containerd-build-image centos]# cp /pos.po ./
[root@containerd-build-image centos]# cp /pos.po ./
[root@containerd-build-image centos]# ll
-rw-r--r-- 1 root root 1923 6月 26 14:po
-rw-r--r-- 1 root root 561 6月 26 14:57 Dockerfile
-rw-r--r-- 1 root root 237 6月 26 14:po
-rw-r--r-- 1 root root 32600353 6月 26 14:47 filebeat-7.12.1-x86_64.rpm[root@containerd-build-image centos]# cat build-image.sh
#!/bin/bash
/usr/local/bin/nerdctl build -p/baseimages/magedu-centos-base:7.9.2009 . &&
/usr/local/bin/nerdctl p/baseimages/magedu-centos-base:7.9.2009
[root@containerd-build-image centos]# sh build-image.sh
[+] Building 231.4s (9/9)
[+] Building 232.1s (9/9) FINISHED
……省略部分内容
Loaded image: p/baseimages/magedu-centos-base:7.9.2009
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:4f8ce249d0db760721782e6331ae3145c3f4557b4645ad1a785da9b3b87481b4)
manifest-sha256:4f8ce249d0db760721782e6331ae3145c3f4557b4645ad1a785da9b3b87481b4: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:34073b4210e319aaedb179d2cb8c17157058943b1d7f9218d821f9195f7b3009: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 13.1s total: 3.8 Ki (295.0 B/s)
[root@containerd-build-image ~]# cd /opt/k8s-data/dockerfile/
[root@containerd-build-image dockerfile]# ls
system
[root@containerd-build-image dockerfile]# mkdir web
[root@containerd-build-image dockerfile]# cd web
[root@containerd-build-image web]# mkdir pub-images
[root@containerd-build-image web]# cd pub-images
[root@containerd-build-image pub-images]# mkdir jdk-1.8.212
[root@containerd-build-image pub-images]# cd jdk-1.8.212[root@containerd-build-image jdk-1.8.212]# cat Dockerfile
#JDK Base Image
p/baseimages/magedu-centos-base:7.9.2009ADD /usr/local/src/
RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk# 这个profile是等下给普通用户使用的
ADD profile /etc/profile# 这里声明的变量，是给root用户使用的，因为容器运行时，profile不会被加载
ENV JAVA_HOME /usr/local/jdk
ENV JRE_HOME $JAVA_HOME/jre
ENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin[root@containerd-build-image jdk-1.8.212]# ll -rt
总用量 190452
-rw-r--r-- 1 root root 332 6月 26 16:47 Dockerfile
-rw-r--r-- 1 root root 195013152 6月 26 16:59
-rw-r--r-- 1 root root 2105 6月 26 16:59 profile[root@containerd-build-image jdk-1.8.212]# cat profile
…省略部分内容
export LANG=en_US.UTF-8
export HISTTIMEFORMAT="%F %T `whoami` "export JAVA_HOME=/usr/local/jdk
export TOMCAT_HOME=/apps/tomcat
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$TOMCAT_HOME/bin:$PATH
export CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
[root@containerd-build-image jdk-1.8.212]# cat build-image.sh
#!/bin/bash
nerdctl build -t p/pub-images/jdk-base:v8.212 . &&
nerdctl push p/pub-images/jdk-base:v8.212
[root@containerd-build-image jdk-1.8.212]# sh build-image.sh
[root@containerd-build-image jdk-1.8.212]# nerdctl run -d --name p/pub-images/jdk-base:v8.212 tail -f /etc/hosts
0921ccf1a4b41e15038940fc92c9874469a6c7ea31eccf28870ae791c189b218
[root@containerd-build-image jdk-1.8.212]# nerdctl ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0921ccf1a4b4 p/pub-images/jdk-base:v8.212 "tail -f /etc/hosts" 5 seconds ago Up test
[root@containerd-build-image jdk-1.8.212]# nerdctl exec -it 0921ccf1a4b4 /bin/bash
[root@0921ccf1a4b4 /]# java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
[root@0921ccf1a4b4 /]# useradd test
[root@0921ccf1a4b4 /]# su - test
[test@0921ccf1a4b4 ~]$ java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
[test@0921ccf1a4b4 ~]$ exit
logout
[root@0921ccf1a4b4 /]# exit
exit
[root@containerd-build-image pub-images]# mkdir tomcat-base-8.5.43
[root@containerd-build-image pub-images]# cd tomcat-base-8.5.43
[root@containerd-build-image tomcat-base-8.5.43]# vim Dockerfile
[root@containerd-build-image tomcat-base-8.5.43]# cat Dockerfile
#Tomcat 8.5.43基础镜像
p/pub-images/jdk-base:v8.212RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pvADD apache-tomcat-8.5. /apps# 普通用户的创建根据需求来
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat
[root@ai-k8s-jump tmp]# ll apache-tomcat-8.5.
-rw-rw-r-- 1 ops ops 9717059 6月 27 10:02 apache-tomcat-8.5.
[root@containerd-build-image tomcat-base-8.5.43]# cat build-image.sh
#!/bin/bash
nerdctl build -p/pub-images/tomcat-base:v8.5.43 . &&
nerdctl p/pub-images/tomcat-base:v8.5.43
[root@containerd-build-image tomcat-base-8.5.43]# sh build-image.sh
# 创建临时容器
[root@containerd-build-image tomcat-base-8.5.43]# nerdctl run -it --rm -p 8080:p/pub-images/tomcat-base:v8.5.43 bash# 启动tomcat
[root@6434c11984b4 ~]# cd /apps/tomcat/bin/
[root@6434c11984b4 bin]# ./catalina.sh run
……省略部分内容
27-Jun-2023 10:19:57.997 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1237 ms
[root@containerd-build-image tomcat-base-8.5.43]# cd ..
[root@containerd-build-image pub-images]# ls
jdk-1.8.212 tomcat-base-8.5.43
[root@containerd-build-image pub-images]# mkdir tsk8s
[root@containerd-build-image pub-images]# cd tsk8s
[root@containerd-build-image tsk8s]# mkdir tomcat-app1
[root@containerd-build-image tsk8s]# cd tomcat-app1[root@containerd-build-image tomcat-app1]# cat Dockerfile
#tomcat web1
p/pub-images/tomcat-base:v8.5.43# 拷贝脚本，工作中该文件中的配置需要根据实际情况调整
ADD catalina.sh /apps/tomcat/bin/catalina.sh# 拷贝tomcat配置文件,工作中该文件中的配置需要根据实际情况调整
l /apps/tomcat/l# 拷贝业务代码
ADD /data/tomcat/webapps/app1/# 拷贝容器启动脚本，一般由开发提供
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh# 授权先关目录给之前创建的普通用户，后续tomcat服务由普通用户运行，保证容器安全
RUN chown -at /data/ /apps/ && chmod u+x /apps/tomcat/bin/catalina.sh /apps/tomcat/bin/run_tomcat.shEXPOSE 8080 8443CMD ["/apps/tomcat/bin/run_tomcat.sh"]
[root@containerd-build-image tomcat-app1]# cat run_tomcat.sh
#!/bin/bash# 切换到普通用户运行服务
su - tomcat -c "/apps/tomcat/bin/catalina.sh start"# 添加守护进程
tail -f /etc/hosts
[root@containerd-build-image tomcat-app1]# cat build-image.sh
#!/bin/bash# 该脚本执行时需要传递构建tag
TAG=$1nerdctl build -t p/tsk8s/tomcat-app1:${TAG} .
nerdctl push p/tsk8s/tomcat-app1:${TAG}
[root@containerd-build-image tomcat-app1]# ll catalina.l run_tomcat.sh
-rw-rw-r-- 1 root root 144 6月 27 10:37
-rw-rw-r-- 1 root root 23611 6月 27 10:37 catalina.sh
-rw-rw-r-- 1 root root 142 6月 27 10:55 run_tomcat.sh
-rw-rw-r-- 1 root root 6462 6月 27 10:l
[root@containerd-build-image tomcat-app1]# sh build-image.sh v1
[root@containerd-build-image tomcat-app1]# nerdctl run -it --rm -p 8080:p/tsk8s/tomcat-app1:v1
[root@k8s-harbor01 deployment]# cat tomcat-app1.yaml
kind: Deployment
apiVersion: apps/v1
metadata:labels:app: tomcat-app1name: tomcat-app1namespace: myserver
spec:replicas: 1selector:matchLabels:app: tomcat-app1template:metadata:labels:app: tomcat-app1spec:containers:- name: tomcat-app1image: p/tsk8s/tomcat-app1:v1imagePullPolicy: IfNotPresentports:- containerPort: 8080protocol: TCPname: httpenv:- name: "password"value: "123456"- name: "age"value: "18"volumeMounts:- name: tsk8s-imagesmountPath: /usr/local/nginx/html/webapp/imagesreadOnly: false- name: tsk8s-staticmountPath: /usr/local/nginx/html/webapp/staticreadOnly: falsevolumes: # 注意这里，需要创建持久化数据的目录- name: tsk8s-imagesnfs: server: 10.31.200.104path: /data/k8s-data/tsk8s/images- name: tsk8s-staticnfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/staticimagePullSecrets: # 注意这里，因为我之前访问harbor用的ip，后面改成域名了，又不想再到所有节点login一次，就用了一个Secrets，这个也需要创建- name: dockerhub-image-pull-key--- # 这里搞两个svc，一个nodeport是为了部署后验证，一个cluster IP是为了让等下部署的nginx pod调用
kind: Service
apiVersion: v1
metadata:labels:app: tomcat-app1name: tomcat-app1-nodeportnamespace: myserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 8080selector:app: tomcat-app1---
kind: Service
apiVersion: v1
metadata:labels:app: tomcat-app1name: tomcat-app1namespace: myserver
spec:ports:- name: httpport: 80protocol: TCPtargetPort: 8080selector:app: tomcat-app1
# 新建tsk8s、以及子目录images、static
[root@k8s-harbor01 tsk8s]# ls
images static
[root@k8s-harbor01 tsk8s]# pwd
/data/k8s-data/tsk8s
[root@k8s-harbor01 k8s-data]# docker p # 重新登录一次
[root@k8s-harbor01 k8s-data]# cat /root/.docker/config.json
{"auths": {&#p": {"auth": "YWRtaW46MTIzNDU2"}}
}[root@k8s-harbor01 k8s-data]# kubectl create secret generic dockerhub-image-pull-key --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson -n myserver
[root@k8s-harbor01 deployment]# kubectl get po,svc -n myserver
NAME READY STATUS RESTARTS AGE
pod/tomcat-app1-6844b69cb5-scfb4 1/1 Running 0 16hNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/tomcat-app1 ClusterIP 10.100.107.208 <none> 80/TCP 16h
service/tomcat-app1-nodeport NodePort 10.100.215.180 <none> 80:30929/TCP 14h
[root@containerd-build-image ~]# cd /opt/k8s-data/dockerfile/web/
[root@containerd-build-image web]# ls
pub-images
[root@containerd-build-image web]# mkdir nginx-base
[root@containerd-build-image web]# cd nginx-base
[root@containerd-build-image nginx-base]# cat Dockerfile
#Nginx Base Image
p/baseimages/magedu-centos-base:7.9.2009ADD nginx-1.22. /usr/local/src/RUN cd /usr/local/src/nginx-1.22.0 && ./configure && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx && rm -rf /usr/local/src/nginx-1.22. && useradd nginx -M -s /bin/nologin
[root@containerd-build-image nginx-base]# wget .22.
[root@containerd-build-image nginx-base]# ll -rt
总用量 1056
-rw-r--r-- 1 root root 1073322 6月 16 2022 nginx-1.22.
-rw-r--r-- 1 root root 306 6月 27 18:11 Dockerfile
[root@containerd-build-image nginx-base]# cat build-image.sh
#!/bin/bash
nerdctl build -t p/pub-images/nginx-base:v1.22.0 . &&
nerdctl p/pub-images/nginx-base:v1.22.0
[root@containerd-build-image nginx-base]# sh build-image.sh
[root@containerd-build-image tsk8s]# mkdir nginx
[root@containerd-build-image tsk8s]# cd nginx
[root@containerd-build-image nginx]# ls
[root@containerd-build-image nginx]# cat Dockerfile
#Nginx 1.22.0
p/pub-images/nginx-base:v1.22.f /usr/local/nginx/f
ADD /usr/local/nginx/html/webapp/
ADD index.html /usr/local/nginx/html/index.html#静态资源挂载路径
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/imagesEXPOSE 80 443CMD ["nginx"]
[root@containerd-build-image nginx]# ll -rt
总用量 20
-rw-r--r-- 1 root root 349 6月 27 18:37 Dockerfile
drwxrwxr-x 2 root root 24 6月 27 18:39 webapp
-rw-rw-r-- 1 root root 3127 6月 27 18:f # nginx配置文件根据实际需求调整
-rw-rw-r-- 1 root root 25 6月 27 18:39 index.html
-rw-rw-r-- 1 root root 381 6月 27 18:39 build-command.sh
-rw-rw-r-- 1 root root 234 6月 27 18:39 # f有几个改的地方
[root@containerd-build-image nginx]# f
user nginx nginx;
worker_processes auto;daemon off;events {worker_connections 1024;
}http {include pes;default_type application/octet-stream;sendfile on;keepalive_timeout 65;upstream tomcat_webserver {server server:80; # 同一个k8s集群中，可以直接通过调用k8s内部域名的方式来访问服务
}server {listen 80;server_name localhost;location / {root html;index index.html index.htm;}location /webapp {root html;index index.html index.htm;}location /app1 {proxy_pass tomcat_webserver;proxy_set_header Host $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Real-IP $remote_addr;}error_page 500 502 503 504 /50x.html;location = /50x.html {root html;}}
}[root@containerd-build-image nginx]# cat build-image.sh
#!/bin/bash
TAG=$1
nerdctl build -p/tsk8s/nginx-web1:${TAG} . &&
nerdctl p/tsk8s/nginx-web1:${TAG}
[root@containerd-build-image nginx]# sh build-image.sh v1
[root@k8s-harbor01 deployment]# cat nginx.yaml
kind: Deployment
apiVersion: apps/v1
metadata:labels:app: nginx-deploymentname: nginx-deploymentnamespace: myserver
spec:replicas: 1selector:matchLabels:app: nginx-deploymenttemplate:metadata:labels:app: nginx-deploymentspec:containers:- name: nginx-deploymentimage: p/tsk8s/nginx-web1:v1#command: ["/apps/tomcat/bin/run_tomcat.sh"]imagePullPolicy: IfNotPresentports:- containerPort: 80protocol: TCPname: http- containerPort: 443protocol: TCPname: httpsenv:- name: "password"value: "123456"- name: "age"value: "20"resources:limits:cpu: 500mmemory: 512Mirequests:cpu: 500mmemory: 256MivolumeMounts:- name: nginx-imagesmountPath: /usr/local/nginx/html/webapp/imagesreadOnly: false- name: nginx-staticmountPath: /usr/local/nginx/html/webapp/staticreadOnly: falsevolumes:- name: nginx-imagesnfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/images- name: nginx-staticnfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/staticimagePullSecrets: # 注意这里，因为我之前访问harbor用的ip，后面改成域名了，又不想再到所有节点login一次，就用了一个Secrets，这个也需要创建- name: dockerhub-image-pull-key
---
kind: Service
apiVersion: v1
metadata:name: nginx-deployment-nodeportnamespace: myserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 80- name: httpsport: 443protocol: TCPtargetPort: 443selector:app: nginx-deployment
[root@k8s-harbor01 deployment]# kubectl apply -f nginx.yaml[root@k8s-harbor01 deployment]# kubectl get po,svc -n myserver|grep nginx
pod/nginx-deployment-5b8fd687b9-jvk6f 1/1 Running 0 63s
service/nginx-deployment-nodeport NodePort 10.100.59.116 <none> 80:32035/TCP,443:32524/TCP 3m38s
这只是一个大概的链路图，具体的还有一些常用中间件：如注册中心、数据库、消息队列等。
主要是讲解类似zookeeper这种有状态服务的迁移，有以下特点：
（1）服务是集群模式。
（2）每一个pod都有一个独立的存储，意味着每个pod的数据都是独立的（不像nginx数据是共享的）。
可以自建，也可以使用dokerhub已有的，只要是jdk 8就行
[root@containerd-build-image ~]# nerdctl pull elevy/slim_java:8
[root@containerd-build-image ~]# nerdctl tag elevy/slim_java:p/baseimages/slim_java:8
[root@containerd-build-image ~]# nerdctl p/baseimages/slim_java:8
[root@containerd-build-image ~]# nerdctl run -it --p/baseimages/slim_java:8 sh
/ # java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
/ # exit
官方dockerfile：.7.1/Dockerfile
[root@containerd-build-image tsk8s]# mkdir zookeeper
[root@containerd-build-image tsk8s]# cd zookeeper
[root@containerd-build-image zookeeper]# vim Dockerfile
[root@containerd-build-image zookeeper]# cat Dockerfile
p/baseimages/slim_java:8# 设置环境变量，方便确认版本
ENV ZK_VERSION 3.4.14# 修改软件包安装源
ADD repositories /etc/apk/repositories# Download Zookeeper。这里直接添加二进制安装包到容器中，速度更快
COPY zookeeper-3.4. / # 安装包
COPY zookeeper-3.4.asc /asc # 校验文件
COPY KEYS /tmp/KEYS # 证书文件# 安装常用软件并设置必要环境
RUN apk add --no-cache --virtual .build-deps ca-certificates gnupg tar wget && ## Install dependenciesapk add --no-cache bash && ### Verify the signatureexport GNUPGHOME="$(mktemp -d)" && gpg -q --batch --import /tmp/KEYS && gpg -q --batch --no-auto-key-retrieve --verify /asc / && ## Set up directories#mkdir -p /zookeeper/data /zookeeper/wal /zookeeper/log && ## Installtar -x -C /zookeeper --strip-components=1 --no-same-owner -f / && ## Slim downcd /zookeeper && cp dist-maven/zookeeper-${ZK_VERSION}.jar . && rm -rf *.txt *.xml bin/*.cmd conf/* contrib dist-maven docs lib/*.txt lib/cobertura lib/jdiff recipes src zookeeper-*.asc zookeeper-*.md5 zookeeper-*.sha1 && ## Clean upapk del .build-deps && rm -rf /tmp/* "$GNUPGHOME"# 复制相关文件
COPY conf /zookeeper/conf/ # 配置文件
COPY bin/zkReady.sh /zookeeper/bin/ # 状态检测脚本
COPY entrypoint.sh / # 启动时加载的脚本# 声明环境变量
ENV PATH=/zookeeper/bin:${PATH} ZOO_LOG_DIR=/zookeeper/log ZOO_LOG4J_PROP="INFO, CONSOLE, ROLLINGFILE" JMXPORT=9010# 启动命令
ENTRYPOINT [ "/entrypoint.sh" ]# 启动命令的参数
CMD [ "zkServer.sh", "start-foreground" ]# 声明需要暴露的端口
EXPOSE 2181 2888 3888 9010
该文件中有很多系统级别的环境变量，内容需要结合zookeeper.yaml查看
#!/bin/bash#将MYID的值希尔MYID文件,如果变量为空就默认为1,MYID为pod中的系统级别环境变量
## 在一个zookeeper集群中，每个zookeeper节点都需要一个唯一的ID，此脚本中的MYID，是从zookeeper.yaml中获取的
# myid文件就只是为了保存id的
echo ${MYID:-1} > /zookeeper/data/myid #如果$SERVERS不为空则向下执行,SERVERS为pod中的系统级别环境变量、
## 在zookeeper集群中，会涉及到选主投票，数据同步等，所以要配置SERVERS来让集群中的节点互相能够通信，该SERVERS值也是从zookeeper.yaml中获取的
if [ -n "$SERVERS" ]; thenIFS=, read -a servers <<<"$SERVERS" #IFS为bash内置变量用于分割字符并将结果形成一个数组for i in "${!servers[@]}"; do #${!servers[@]}表示获取servers中每个元素的索引值,此索引值会用做当前ZK的IDprintf "nserver.%i=%s:2888:3888" "$((1 + $i))" "${servers[$i]}" >> /zookeeper/conf/zoo.cfg #打印结果并输出重定向到文件/zookeeper/conf/zoo.cfg，其中%i和%s的值来分别自于后面变量"$((1 + $i))" "${servers[$i]}"done
ficd /zookeeper
exec "$@" #$@变量用于引用给脚本传递的所有参数,传递的所有参数会被作为一个数组列表,exec为终止当前进程、保留当前进程id、新建一个进程执行新的任务,即CMD [ "zkServer.sh", "start-foreground" ]
[root@k8s-harbor01 deployment]# cat zookeerper.yaml
apiVersion: v1
kind: Service
metadata:name: zookeepernamespace: myserver
spec:ports:- name: clientport: 2181selector:app: zookeeper
---
apiVersion: v1
kind: Service
metadata:name: zookeeper1namespace: myserver
spec:type: NodePortports:- name: clientport: 2181nodePort: 32181- name: followersport: 2888- name: electionport: 3888selector:app: zookeeperserver-id: "1"
---
apiVersion: v1
kind: Service
metadata:name: zookeeper2namespace: myserver
spec:type: NodePortports:- name: clientport: 2181nodePort: 32182- name: followersport: 2888- name: electionport: 3888selector:app: zookeeperserver-id: "2"
---
apiVersion: v1
kind: Service
metadata:name: zookeeper3namespace: myserver
spec:type: NodePortports:- name: client # 客户端端口，给k8s集群外需要连接zookeeper集群的服务使用的port: 2181nodePort: 32183- name: followersport: 2888- name: electionport: 3888selector: # 这里选择了2个标签，只有同时拥有这2个标签的pod才会收到请求app: zookeeperserver-id: "3"
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper1namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "1"spec:volumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: harbor.linuxarchitect.io/magedu/zookeeper:v3.4.14imagePullPolicy: Alwaysenv:- name: MYIDvalue: "1"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-1volumes:- name: zookeeper-datadir-pvc-1persistentVolumeClaim:claimName: zookeeper-datadir-pvc-1
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper2namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "2"spec:volumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: harbor.linuxarchitect.io/magedu/zookeeper:v3.4.14imagePullPolicy: Alwaysenv:- name: MYIDvalue: "2"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-2volumes:- name: zookeeper-datadir-pvc-2persistentVolumeClaim:claimName: zookeeper-datadir-pvc-2
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper3namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "3"spec:volumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: harbor.linuxarchitect.io/magedu/zookeeper:v3.4.14imagePullPolicy: Alwaysenv:- name: MYIDvalue: "3"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-3volumes:- name: zookeeper-datadir-pvc-3persistentVolumeClaim:claimName: zookeeper-datadir-pvc-3
[root@containerd-build-image zookeeper]# cat Dockerfile
p/baseimages/slim_java:8# 设置环境变量，方便确认版本
ENV ZK_VERSION 3.4.14# 修改软件包安装源
ADD repositories /etc/apk/repositories# Download Zookeeper。这里直接添加二进制安装包到容器中，速度更快
COPY zookeeper-3.4. /
COPY zookeeper-3.4.asc /asc
COPY KEYS /tmp/KEYS# 安装常用软件并设置必要环境
RUN apk add --no-cache --virtual .build-deps ca-certificates gnupg tar wget && ## Install dependenciesapk add --no-cache bash && ### Verify the signatureexport GNUPGHOME="$(mktemp -d)" && gpg -q --batch --import /tmp/KEYS && gpg -q --batch --no-auto-key-retrieve --verify /asc / && ## Set up directories#mkdir -p /zookeeper/data /zookeeper/wal /zookeeper/log && ## Installtar -x -C /zookeeper --strip-components=1 --no-same-owner -f / && ## Slim downcd /zookeeper && cp dist-maven/zookeeper-${ZK_VERSION}.jar . && rm -rf *.txt *.xml bin/*.cmd conf/* contrib dist-maven docs lib/*.txt lib/cobertura lib/jdiff recipes src zookeeper-*.asc zookeeper-*.md5 zookeeper-*.sha1 && ## Clean upapk del .build-deps && rm -rf /tmp/* "$GNUPGHOME"# 复制相关文件
COPY conf /zookeeper/conf/
COPY bin/zkReady.sh /zookeeper/bin/
COPY entrypoint.sh /RUN chmod u+x /entrypoint.sh# 声明环境变量
ENV PATH=/zookeeper/bin:${PATH} ZOO_LOG_DIR=/zookeeper/log ZOO_LOG4J_PROP="INFO, CONSOLE, ROLLINGFILE" JMXPORT=9010# 启动命令
ENTRYPOINT ["/entrypoint.sh"]# 启动命令的参数
CMD [ "zkServer.sh", "start-foreground" ]# 声明需要暴露的端口
EXPOSE 2181 2888 3888 9010[root@containerd-build-image zookeeper]# ll -rt
总用量 36888
-rw-r--r-- 1 root root 2086 6月 28 14:54 Dockerfile
-rw-r--r-- 1 root root 125 6月 28 17:00 build-image.sh
-rw-rw-r-- 1 root root 2270 6月 28 17:01 zookeeper-3.
-rw-rw-r-- 1 root root 91 6月 28 17:01 repositories
-rw-rw-r-- 1 root root 63587 6月 28 17:01 KEYS
-rw-rw-r-- 1 root root 1015 6月 28 17:01 entrypoint.sh
-rw-rw-r-- 1 root root 836 6月 28 17:02 zookeeper-3.4.asc
-rw-rw-r-- 1 root root 37676320 6月 28 17:02 zookeeper-3.4.
drwxrwxr-x 2 root root 45 6月 28 17:02 conf
drwxrwxr-x 2 root root 24 6月 28 17:02 bin
[root@containerd-build-image zookeeper]# cat build-image.sh
#!/bin/bash
TAG=$1
nerdctl build -p/tsk8s/zookeeper:${TAG} . &&
nerdctl p/tsk8s/zookeeper:${TAG}
[root@containerd-build-image zookeeper]# sh build-image.sh v1
[root@containerd-build-image zookeeper]# nerdctl run -it --p/tsk8s/zookeeper:v1
……省略部分输出
2023-06-28 10:17:18,708 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 # 启动成功
[root@k8s-harbor01 volume]# cat zookeeper-persistentvolume.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:name: zookeeper-datadir-pv-1
spec:capacity:storage: 20GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/zookeeper-datadir-1---
apiVersion: v1
kind: PersistentVolume
metadata:name: zookeeper-datadir-pv-2
spec:capacity:storage: 20GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/zookeeper-datadir-2---
apiVersion: v1
kind: PersistentVolume
metadata:name: zookeeper-datadir-pv-3
spec:capacity:storage: 20GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/zookeeper-datadir-3
[root@k8s-harbor01 volume]# cat zookeeper-persistentvolumeclaim.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: zookeeper-datadir-pvc-1namespace: myserver
spec:accessModes:- ReadWriteOncevolumeName: zookeeper-datadir-pv-1 # 和pv强制绑定resources:requests:storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: zookeeper-datadir-pvc-2namespace: myserver
spec:accessModes:- ReadWriteOncevolumeName: zookeeper-datadir-pv-2 # 和pv强制绑定resources:requests:storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: zookeeper-datadir-pvc-3namespace: myserver
spec:accessModes:- ReadWriteOncevolumeName: zookeeper-datadir-pv-3 # 和pv强制绑定resources:requests:storage: 10Gi
[root@k8s-harbor01 tsk8s]# mkdir zookeeper-datadir-{1..3}
[root@k8s-harbor01 tsk8s]# ll -rt
总用量 0
drwxr-xr-x 2 root root 6 6月 27 15:38 images
drwxr-xr-x 2 root root 6 6月 27 15:38 static
drwxr-xr-x 2 root root 6 6月 28 18:35 zookeeper-datadir-3
drwxr-xr-x 2 root root 6 6月 28 18:35 zookeeper-datadir-2
drwxr-xr-x 2 root root 6 6月 28 18:35 zookeeper-datadir-1
[root@k8s-harbor01 volume]# kubectl apply -f zookeeper-persistentvolume.yaml
persistentvolume/zookeeper-datadir-pv-1 created
persistentvolume/zookeeper-datadir-pv-2 created
persistentvolume/zookeeper-datadir-pv-3 created[root@k8s-harbor01 volume]# kubectl get pv |grep zookeeper
zookeeper-datadir-pv-1 20Gi RWO Retain Available 27s
zookeeper-datadir-pv-2 20Gi RWO Retain Available 27s
zookeeper-datadir-pv-3 20Gi RWO Retain Available [root@k8s-harbor01 volume]# kubectl apply -f zookeeper-persistentvolumeclaim.yaml
persistentvolumeclaim/zookeeper-datadir-pvc-1 created
persistentvolumeclaim/zookeeper-datadir-pvc-2 created
persistentvolumeclaim/zookeeper-datadir-pvc-3 created[root@k8s-harbor01 volume]# kubectl get pvc -n myserver
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
zookeeper-datadir-pvc-1 Bound zookeeper-datadir-pv-1 20Gi RWO 118s
zookeeper-datadir-pvc-2 Bound zookeeper-datadir-pv-2 20Gi RWO 118s
zookeeper-datadir-pvc-3 Bound zookeeper-datadir-pv-3 20Gi RWO 118s
[root@k8s-harbor01 deployment]# cat zookeerper.yaml
apiVersion: v1
kind: Service
metadata:name: zookeepernamespace: myserver
spec:ports:- name: clientport: 2181selector:app: zookeeper
---
apiVersion: v1
kind: Service
metadata:name: zookeeper1namespace: myserver
spec:type: NodePortports:- name: clientport: 2181nodePort: 32181- name: followersport: 2888- name: electionport: 3888selector:app: zookeeperserver-id: "1"
---
apiVersion: v1
kind: Service
metadata:name: zookeeper2namespace: myserver
spec:type: NodePortports:- name: clientport: 2181nodePort: 32182- name: followersport: 2888- name: electionport: 3888selector:app: zookeeperserver-id: "2"
---
apiVersion: v1
kind: Service
metadata:name: zookeeper3namespace: myserver
spec:type: NodePortports:- name: client # 客户端端口，给k8s集群外需要连接zookeeper集群的服务使用的port: 2181nodePort: 32183- name: followersport: 2888- name: electionport: 3888selector: # 这里选择了2个标签，只有同时拥有这2个标签的pod才会收到请求app: zookeeperserver-id: "3"
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper1namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "1"spec:imagePullSecrets:- name: dockerhub-image-pull-keyvolumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: p/tsk8s/zookeeper:v1imagePullPolicy: Alwaysenv:- name: MYIDvalue: "1"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-1volumes:- name: zookeeper-datadir-pvc-1persistentVolumeClaim:claimName: zookeeper-datadir-pvc-1
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper2namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "2"spec:imagePullSecrets:- name: dockerhub-image-pull-keyvolumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: p/tsk8s/zookeeper:v1imagePullPolicy: Alwaysenv:- name: MYIDvalue: "2"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-2volumes:- name: zookeeper-datadir-pvc-2persistentVolumeClaim:claimName: zookeeper-datadir-pvc-2
---
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:name: zookeeper3namespace: myserver
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperserver-id: "3"spec:imagePullSecrets:- name: dockerhub-image-pull-keyvolumes:- name: dataemptyDir: {}- name: walemptyDir:medium: Memorycontainers:- name: serverimage: p/tsk8s/zookeeper:v1imagePullPolicy: Alwaysenv:- name: MYIDvalue: "3"- name: SERVERSvalue: "zookeeper1,zookeeper2,zookeeper3"- name: JVMFLAGSvalue: "-Xmx2G"ports:- containerPort: 2181- containerPort: 2888- containerPort: 3888volumeMounts:- mountPath: "/zookeeper/data"name: zookeeper-datadir-pvc-3volumes:- name: zookeeper-datadir-pvc-3persistentVolumeClaim:claimName: zookeeper-datadir-pvc-3[root@k8s-harbor01 deployment]# kubectl apply -f zookeerper.yaml
service/zookeeper created
service/zookeeper1 created
service/zookeeper2 created
service/zookeeper3 created
deployment.apps/zookeeper1 created
deployment.apps/zookeeper2 created
deployment.apps/zookeeper3 created[root@k8s-harbor01 deployment]# kubectl get po -n myserver |grep zook
zookeeper1-54fcfb896c-nr2d7 1/1 Running 1 (47s ago) 63s
zookeeper2-65789d49d8-z5n6r 1/1 Running 1 (49s ago) 63s
zookeeper3-5b94f474c7-7vbmb 1/1 Running 0 62s
[root@k8s-harbor01 deployment]# kubectl exec -it -n myserver zookeeper1-54fcfb896c-nr2d7 -- /bin/sh
/ # /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: follower # 集群模式
/ # cat /zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888/ #
/ # exit
[root@k8s-harbor01 deployment]# kubectl exec -it -n myserver zookeeper3-5b94f474c7-7vbmb -- /bin/sh
/ # /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: leader # 当前角色为主
/ # exit[root@k8s-harbor01 deployment]# cd /root/app/harbor/harbor/
[root@k8s-harbor01 harbor]# docker-compose down
Stopping nginx ... done
Stopping harbor-jobservice ... done
Stopping harbor-core ... done
Stopping registry ... done
Stopping redis ... done
Stopping harbor-db ... done
Stopping registryctl ... done
Stopping harbor-portal ... done
Stopping harbor-log ... done
Removing nginx ... done
Removing harbor-jobservice ... done
Removing harbor-core ... done
Removing registry ... done
Removing redis ... done
Removing harbor-db ... done
Removing registryctl ... done
Removing harbor-portal ... done
Removing harbor-log ... done
Removing network harbor_harbor
[root@k8s-harbor01 harbor]# kubectl delete po -n myserver zookeeper3-5b94f474c7-7vbmb
pod "zookeeper3-5b94f474c7-7vbmb" deleted[root@k8s-harbor01 harbor]# kubectl get po -n myserver |grep zoo
zookeeper1-54fcfb896c-nr2d7 1/1 Running 1 (16m ago) 16m
zookeeper2-65789d49d8-z5n6r 1/1 Running 1 (16m ago) 16m
zookeeper3-5b94f474c7-56mfb 0/1 ImagePullBackOff 0 19s[root@k8s-harbor01 harbor]# kubectl exec -it -n myserver zookeeper1-54fcfb896c-nr2d7 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: follower
/ # exit
[root@k8s-harbor01 harbor]# kubectl exec -it -n myserver zookeeper2-65789d49d8-z5n6r sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # /zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
ZooKeeper remote JMX Port set to 9010
ZooKeeper remote JMX authenticate set to false
ZooKeeper remote JMX ssl set to false
ZooKeeper remote JMX log4j set to true
Using config: /zookeeper/bin/../conf/zoo.cfg
Mode: leader # 这里可以看到新的主已经产生了
/ # exit# 重新启动节点
[root@k8s-harbor01 harbor]# docker-compose up -d
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating registry ... done
Creating harbor-portal ... done
Creating registryctl ... done
Creating harbor-db ... done
Creating redis ... done
Creating harbor-core ... done
Creating nginx ... done
Creating harbor-jobservice ... done[root@k8s-harbor01 harbor]# kubectl get po -n myserver |grep zoo
zookeeper1-54fcfb896c-nr2d7 1/1 Running 1 (18m ago) 18m
zookeeper2-65789d49d8-z5n6r 1/1 Running 1 (18m ago) 18m
zookeeper3-5b94f474c7-56mfb 0/1 ImagePullBackOff 0 2m29s
[root@k8s-harbor01 harbor]# kubectl delete po -n myserver zookeeper3-5b94f474c7-56mfb
pod "zookeeper3-5b94f474c7-56mfb" deleted[root@k8s-harbor01 harbor]# kubectl get po -n myserver |grep zoo
zookeeper1-54fcfb896c-nr2d7 1/1 Running 1 (18m ago) 19m
zookeeper2-65789d49d8-z5n6r 1/1 Running 1 (18m ago) 19m
zookeeper3-5b94f474c7-l5ftp 1/1 Running 0 9s
[root@containerd-build-image ~]# cd /opt/k8s-data/dockerfile/web/
[root@containerd-build-image web]# mkdir redis
[root@containerd-build-image web]# cd redis
[root@containerd-build-image redis]#
[root@containerd-build-image redis]# cat Dockerfile
#Redis Image
p/baseimages/magedu-centos-base:7.9.2009ADD redis-4.0. /usr/local/srcRUN ln -sv /usr/local/src/redis-4.0.14 /usr/local/redis && cd /usr/local/redis && make && cp src/redis-cli /usr/sbin/ && cp src/redis-server /usr/sbin/ && mkdir -pv /data/f /usr/local/fEXPOSE 6379#ADD run_redis.sh /usr/local/redis/run_redis.sh
#CMD ["/usr/local/redis/run_redis.sh"]ADD run_redis.sh /usr/local/redis/entrypoint.shENTRYPOINT ["/bin/sh", "/usr/local/redis/entrypoint.sh"]
[root@containerd-build-image redis]# cat run_redis.sh
#!/bin/bash/usr/sbin/redis-server /usr/local/ftail -f /etc/hosts
[root@containerd-build-image redis]# cat build-image.sh
#!/bin/bash
TAG=$1
nerdctl build -t p/tsk8s/redis:${TAG} . &&
nerdctl p/tsk8s/redis:${TAG}
[root@containerd-build-image redis]# ll
总用量 1776
-rw-rw-r-- 1 root root 115 6月 29 09:57 build-image.sh
-rw-r--r-- 1 root root 550 6月 29 09:53 Dockerfile
-rw-rw-r-- 1 root root 1740967 6月 29 09:55 redis-4.0.
-rw-rw-r-- 1 root root 58783 6月 29 09:f
-rw-rw-r-- 1 root root 85 6月 29 09:55 run_redis.sh
[root@containerd-build-image redis]# sh build-image.sh v1
[root@k8s-harbor01 deployment]# cat redis.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: devops-redisname: deploy-devops-redisnamespace: myserver
spec:replicas: 1selector:matchLabels:app: devops-redistemplate:metadata:labels:app: devops-redisspec:containers:- name: redis-containerimage: p/tsk8s/redis:v1imagePullPolicy: AlwaysvolumeMounts:- mountPath: "/data/redis-data/"name: redis-datadirvolumes:- name: redis-datadirpersistentVolumeClaim:claimName: redis-datadir-pvc-1imagePullSecrets:- name: dockerhub-image-pull-key
---
kind: Service
apiVersion: v1
metadata:labels:app: devops-redisname: srv-devops-redisnamespace: myserver
spec:type: NodePortports:- name: httpport: 6379targetPort: 6379selector:app: devops-redissessionAffinity: ClientIPsessionAffinityConfig:clientIP:timeoutSeconds: 10800
[root@k8s-harbor01 volume]# cat redis-persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-datadir-pv-1
spec:capacity:storage: 10GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/tsk8s/redis-datadir-1server: 10.31.200.104[root@k8s-harbor01 volume]# mkdir /data/k8s-data/tsk8s/redis-datadir-1[root@k8s-harbor01 volume]# kubectl apply -f redis-persistentvolume.yaml
persistentvolume/redis-datadir-pv-1 created
[root@k8s-harbor01 volume]# kubectl get pv |grep redis
redis-datadir-pv-1 10Gi RWO Retain Available 4s
[root@k8s-harbor01 volume]# cat redis-persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: redis-datadir-pvc-1namespace: myserver
spec:volumeName: redis-datadir-pv-1accessModes:- ReadWriteOnceresources:requests:storage: 10Gi[root@k8s-harbor01 volume]# kubectl apply -f redis-persistentvolumeclaim.yaml
persistentvolumeclaim/redis-datadir-pvc-1 created
[root@k8s-harbor01 volume]# kubectl get pvc -n myserver|grep redis
redis-datadir-pvc-1 Bound redis-datadir-pv-1 10Gi RWO 13s
[root@k8s-harbor01 ~]# kubectl get po,svc -n myserver |grep redis
pod/deploy-devops-redis-78cdf68d4d-kd7kk 1/1 Running 0 63m
service/srv-devops-redis NodePort 10.100.64.188 <none> 6379:32159/TCP 66m
[root@k8s-harbor01 ~]# ll /data/k8s-data/tsk8s/redis-datadir-1/
总用量 4
-rw-r--r-- 1 root root 111 6月 29 13:57 dump.rdb
[root@k8s-harbor01 volume]# cat redis-cluster-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv0
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis0---
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv1
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis1---
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv2
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis2---
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv3
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis3---
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv4
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis4---
apiVersion: v1
kind: PersistentVolume
metadata:name: redis-cluster-pv5
spec:capacity:storage: 5GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/redis5
[root@k8s-harbor01 ~]# mkdir /data/k8s-data/tsk8s/redis{0..5}
[root@k8s-harbor01 ~]# ll -rt /data/k8s-data/tsk8s/|grep redis
drwxr-xr-x 2 root root 22 6月 29 13:57 redis-datadir-1
drwxr-xr-x 2 root root 6 6月 29 14:36 redis5
drwxr-xr-x 2 root root 6 6月 29 14:36 redis4
drwxr-xr-x 2 root root 6 6月 29 14:36 redis3
drwxr-xr-x 2 root root 6 6月 29 14:36 redis2
drwxr-xr-x 2 root root 6 6月 29 14:36 redis1
drwxr-xr-x 2 root root 6 6月 29 14:36 redis0# k8s中创建pv
[root@k8s-harbor01 volume]# kubectl apply -f redis-cluster-pv.yaml
persistentvolume/redis-cluster-pv0 created
persistentvolume/redis-cluster-pv1 created
persistentvolume/redis-cluster-pv2 created
persistentvolume/redis-cluster-pv3 created
persistentvolume/redis-cluster-pv4 created
persistentvolume/redis-cluster-pv5 created[root@k8s-harbor01 volume]# kubectl get pv|grep redis-cluster
redis-cluster-pv0 5Gi RWO Retain Available 23s
redis-cluster-pv1 5Gi RWO Retain Available 23s
redis-cluster-pv2 5Gi RWO Retain Available 23s
redis-cluster-pv3 5Gi RWO Retain Available 23s
redis-cluster-pv4 5Gi RWO Retain Available 23s
redis-cluster-pv5 5Gi RWO Retain Available 23s
[root@k8s-harbor01 configmap]# f
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/f
cluster-node-timeout 5000
dir /var/lib/redis
port 6379[root@k8s-harbor01 configmap]# kubectl create cm redis-conf --from-file&#f -n myserver
configmap/redis-conf created
[root@k8s-harbor01 configmap]# kubectl get cm -n myserver|grep redis
redis-conf 1 8s[root@k8s-harbor01 configmap]# kubectl get cm -o yaml -n myserver redis-conf
apiVersion: v1
f: |+appendonly yescluster-enabled yescluster-config-file /var/lib/fcluster-node-timeout 5000dir /var/lib/redisport 6379kind: ConfigMap
metadata:creationTimestamp: "2023-06-29T07:11:43Z"name: redis-confnamespace: myserverresourceVersion: "13219950"uid: a41e1d2c-8505-4f61-a621-b76e91c42e10
[root@k8s-harbor01 sts]# cat redis-cluster.yaml
apiVersion: v1
kind: Service
metadata:name: redisnamespace: myserverlabels:app: redis
spec:selector:app: redisappCluster: redis-clusterports:- name: redisport: 6379clusterIP: None---
apiVersion: v1
kind: Service
metadata:name: redis-accessnamespace: myserverlabels:app: redis
spec:type: NodePortselector:app: redisappCluster: redis-clusterports:- name: redis-accessprotocol: TCPport: 6379targetPort: 6379---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: redisnamespace: myserver
spec:serviceName: redisreplicas: 6selector:matchLabels:app: redisappCluster: redis-clustertemplate:metadata:labels:app: redisappCluster: redis-clusterspec:terminationGracePeriodSeconds: 20 # pod优雅终止宽限期affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: appoperator: Invalues:- redistopologyKey: kubernetes.io/hostnamecontainers:- name: redisimage: p/tsk8s/redis:v1command:- "redis-server"args:- "/etc/f"- "--protected-mode"- "no"resources:requests:cpu: "500m"memory: "500Mi"ports:- containerPort: 6379name: redisprotocol: TCP- containerPort: 16379name: clusterprotocol: TCPvolumeMounts:- name: confmountPath: /etc/redis- name: datamountPath: /var/lib/redisvolumes:- name: confconfigMap:name: redis-confitems:- key: fpath: fimagePullSecrets:- name: dockerhub-image-pull-keyvolumeClaimTemplates: # 在sts中，可以指定一个pvc模板，它在pod启动时候会自动寻找可用的pv进行关联，就不用手动创建pvc了- metadata:name: datanamespace: myserverspec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 5Gi
[root@k8s-harbor01 sts]# kubectl apply -f redis-cluster.yaml
service/redis created
service/redis-access created
statefulset.apps/redis created[root@k8s-harbor01 sts]# kubectl get po -n myserver|grep redis
deploy-devops-redis-78cdf68d4d-kd7kk 1/1 Running 0 6h25m
redis-0 1/1 Running 0 88s
redis-1 1/1 Running 0 30s
redis-2 1/1 Running 0 26s
redis-3 1/1 Running 0 21s
redis-4 1/1 Running 0 17s
redis-5 1/1 Running 0 14s[root@k8s-harbor01 sts]# kubectl get pvc -n myserver |grep redis-cluster
data-redis-0 Bound redis-cluster-pv1 5Gi RWO 141m
data-redis-1 Bound redis-cluster-pv3 5Gi RWO 25m
data-redis-2 Bound redis-cluster-pv4 5Gi RWO 25m
data-redis-3 Bound redis-cluster-pv0 5Gi RWO 25m
data-redis-4 Bound redis-cluster-pv2 5Gi RWO 24m
data-redis-5 Bound redis-cluster-pv5 5Gi RWO 24m# 上面的data-redis-x和redis-cluster-pvx对应不上也没关系，数据能正常存储就行
[root@k8s-harbor01 yaml]# cat temp-unubtu.yaml
apiVersion: v1
kind: Pod
metadata:name: temp-ubuntunamespace: myserver
spec:containers:- name: temp-ubuntuimage: p/myserver/ubuntu:22.04command: ['tail', '-f', '/etc/hosts']imagePullSecrets:- name: dockerhub-image-pull-key[root@k8s-harbor01 yaml]# kubectl apply -f temp-unubtu.yaml[root@k8s-harbor01 yaml]# kubectl get po -n myserver|grep temp
temp-ubuntu 1/1 Running 0 37s[root@k8s-harbor01 yaml]# kubectl exec -it -n myserver temp-ubuntu -- /bin/bash
root@temp-ubuntu:/# apt update
root@temp-ubuntu:/# apt install python2.7 python-pip redis-tools dnsutils iputils-ping net-tools -y
root@temp-ubuntu:/# pip2 install --upgrade pip
root@temp-ubuntu:/# pip2 install redis-trib==0.5.1
root@temp-ubuntu:/# redis-trib.py create
`dig +server.svc.cluster.local`:6379
`dig +server.svc.cluster.local`:6379
`dig +server.svc.cluster.local`:6379
root@temp-ubuntu:/# redis-trib.py replicate
--master-addr `dig +server.svc.cluster.local`:6379
--slave-addr `dig +server.svc.cluster.local`:6379
root@temp-ubuntu:/# redis-trib.py replicate --master-addr `dig +server.svc.cluster.local`:6379 --slave-addr `dig +server.svc.cluster.local`:6379
root@temp-ubuntu:/# redis-trib.py replicate
--master-addr `dig +server.svc.cluster.local`:6379
--slave-addr `dig +server.svc.cluster.local`:6379
[root@k8s-harbor01 sts]# kubectl exec -it -n myserver redis-0 -- /bin/bash # 随便进一个pod就行
[root@redis-0 /]# redis-cli
127.0.0.1:6379> cluster info
cluster_state:ok # 集群状态
cluster_slots_assigned:16384 # 槽位分配
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:3
cluster_stats_messages_ping_sent:1804
cluster_stats_messages_pong_sent:1764
cluster_stats_messages_meet_sent:4
cluster_stats_messages_sent:3572
cluster_stats_messages_ping_received:1762
cluster_stats_messages_pong_received:1808
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:3572127.0.0.1:6379> cluster nodes
3541ad028af35947ff89ae3c56612d89b02824fa 10.200.135.143:6379@16379 slave 8cc64a2ef65477f94d5082c1b97f28db0b3b4954 0 1688365731693 2 connected
acf789c7ae98c98acd907f6602df23911a23b4cd 10.200.85.211:6379@16379 slave 9db5798d4d601982128eb4cfd86628ccc808f2e5 0 1688365733195 3 connected
9db5798d4d601982128eb4cfd86628ccc808f2e5 10.200.85.210:6379@16379 myself,master - 0 1688365731000 3 connected 10923-16383
8cc64a2ef65477f94d5082c1b97f28db0b3b4954 10.200.135.142:6379@16379 master - 0 1688365732000 2 connected 0-5461
f8fa6fe15857f994aae91ba11e096141605baa64 10.200.58.199:6379@16379 slave 60d5dbcea2b38b6a0606d6a03ef02f42e22560a1
# 写入测试数据
127.0.0.1:6379> set k1 v1
OK127.0.0.1:6379> get k1 # 由于集群模式下的数据是按照槽位存储的，所以在A节点写入的数据不一定会存储在A节点，我这里就报错了，提示我去10.200.85.210上查
(error) MOVED 12706 10.200.85.210:6379# 查询测试数据（10.200.85.210上就查出来了）
127.0.0.1:6379> KEYS k1
1) "k1"
127.0.0.1:6379> get k1
"v1"
[root@k8s-harbor01 volume]# cat mysql-persistentvolume.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-1namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-1server: 10.31.200.104
---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-2namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-2server: 10.31.200.104
---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-3namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-3server: 10.31.200.104
---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-4namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-4server: 10.31.200.104
---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-5namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-5server: 10.31.200.104---
apiVersion: v1
kind: PersistentVolume
metadata:name: mysql-datadir-6namespace: myserver
spec:capacity:storage: 50GiaccessModes:- ReadWriteOncenfs:path: /data/k8s-data/myserver/mysql-datadir-6server: 10.31.200.104[root@k8s-harbor01 volume]# mkdir /data/k8s-data/myserver/mysql-datadir-{1..6}
[root@k8s-harbor01 volume]# ls /data/k8s-data/myserver/mysql-datadir-*
/data/k8s-data/myserver/mysql-datadir-1:/data/k8s-data/myserver/mysql-datadir-2:/data/k8s-data/myserver/mysql-datadir-3:/data/k8s-data/myserver/mysql-datadir-4:/data/k8s-data/myserver/mysql-datadir-5:/data/k8s-data/myserver/mysql-datadir-6:
[root@k8s-harbor01 volume]# kubectl apply -f mysql-persistentvolume.yaml
persistentvolume/mysql-datadir-1 created
persistentvolume/mysql-datadir-2 created
persistentvolume/mysql-datadir-3 created
persistentvolume/mysql-datadir-4 created
persistentvolume/mysql-datadir-5 created
persistentvolume/mysql-datadir-6 created[root@k8s-harbor01 volume]# kubectl get pv |grep mysql
mysql-datadir-1 50Gi RWO Retain Available 3m45s
mysql-datadir-2 50Gi RWO Retain Available 3m45s
mysql-datadir-3 50Gi RWO Retain Available 3m45s
mysql-datadir-4 50Gi RWO Retain Available 3m45s
mysql-datadir-5 50Gi RWO Retain Available 3m45s
mysql-datadir-6 50Gi RWO Retain Available 3m45s
[root@k8s-harbor01 configmap]# cat mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:name: mysqlnamespace: myserverlabels:app: mysql
data:masterf: | # 如果pod是master，就挂载这个key# Apply this config only on the master.[mysqld]log-binlog_bin_trust_function_creators=1lower_case_table_names=1slavef: | # 如果pod是slave，就挂载这个key# Apply this config only on slaves.[mysqld]super-read-onlylog_bin_trust_function_creators=1[root@k8s-harbor01 configmap]# kubectl apply -f mysql-configmap.yaml
configmap/mysql created
[root@k8s-harbor01 configmap]# kubectl get cm -n myserver |grep mysql
mysql 2 10s
[root@k8s-harbor01 yaml]# mkdir svc
[root@k8s-harbor01 yaml]# cd svc
[root@k8s-harbor01 svc]# vi mysql-services.yaml
[root@k8s-harbor01 svc]# cat mysql-services.yaml
apiVersion: v1
kind: Service
metadata:namespace: myservername: mysqllabels:app: mysql
spec:ports:- name: mysqlport: 3306clusterIP: None # 无头类型的svcselector:app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: sql.
apiVersion: v1
kind: Service
metadata:name: mysql-read # 这个svc主要是给应用读的namespace: myserverlabels:app: mysql
spec:ports:- name: mysqlport: 3306selector:app: mysq
[root@k8s-harbor01 svc]# kubectl apply -f mysql-services.yaml
service/mysql created
service/mysql-read created[root@k8s-harbor01 svc]# kubectl get svc -nmyserver |grep mysql
mysql ClusterIP None <none> 3306/TCP 8s
mysql-read ClusterIP 10.100.236.134 <none> 3306/TCP 8s
[root@k8s-harbor01 sts]# cat mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:name: mysqlnamespace: myserver
spec:selector:matchLabels:app: mysqlserviceName: mysqlreplicas: 2template:metadata:labels:app: mysqlspec:initContainers:- name: init-mysql #初始化容器1、基于当前pod name匹配角色是master还是slave，并动态生成相对应的配置文件image: p/tsk8s/mysql:5.7.36command:- bash- "-c"- |set -ex# Generate mysql server-id from pod ordinal index.[[ `hostname` =~ -([0-9]+)$ ]] || exit 1 #匹配hostname的最后一位、最后是一个顺序叠加的整数ordinal=${BASH_REMATCH[1]}echo [mysqld] > /mnt/conf.d/server-idf# Add an offset to avoid reserved server-id=ho server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-idf# Copy appropriate conf.d files from config-map to emptyDir.if [[ $ordinal -eq 0 ]]; then #如果是master、则cpmaster配置文件cp /mnt/config-map/masterf /mnt/conf.d/else #否则cp slave配置文件cp /mnt/config-map/slavef /mnt/conf.d/fivolumeMounts:- name: conf #临时卷、emptyDirmountPath: /mnt/conf.d- name: config-mapmountPath: /mnt/config-map- name: clone-mysql #初始化容器2、用于生成mysql配置文件、并从上一个pod完成首次的全量数据clone(slave 3从slave2 clone，而不是每个slave都从master clone实现首次全量同步,但是后期都是与master实现增量同步)image: p/tsk8s/xtrabackup:1.0command:- bash- "-c"- |set -ex# Skip the clone if data already exists.[[ -d /var/lib/mysql/mysql ]] && exit 0# Skip the clone on master (ordinal index 0).[[ `hostname` =~ -([0-9]+)$ ]] || exit 1ordinal=${BASH_REMATCH[1]}[[ $ordinal -eq 0 ]] && exit 0 #如果最后一位是0(master)则退出clone过程# Clone data from at --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql #从上一个pod执行clone(binlog),xbstream为解压缩命令# Prepare the backup.xuextrabackup --prepare --target-dir=/var/lib/mysql #通过xtrabackup恢复binlogvolumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dcontainers:- name: mysql #业务容器1(mysql主容器)image: p/tsk8s/mysql:5.7.36env:- name: MYSQL_ALLOW_EMPTY_PASSWORDvalue: "1"ports:- name: mysqlcontainerPort: 3306volumeMounts:- name: data #挂载数据目录至/var/lib/mysqlmountPath: /var/lib/mysqlsubPath: mysql- name: conf #配置文件/etc/mysql/conf.dmountPath: /etc/mysql/conf.dresources: #资源限制requests:cpu: 500mmemory: 1GilivenessProbe: #存活探针exec:command: ["mysqladmin", "ping"]initialDelaySeconds: 30periodSeconds: 10timeoutSeconds: 5readinessProbe: #就绪探针exec:# Check we can execute queries over TCP (skip-networking is off)mand: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]initialDelaySeconds: 5periodSeconds: 2timeoutSeconds: 1- name: xtrabackup #业务容器2(xtrabackup),用于后期同步master 的binglog并恢复数据image: p/tsk8s/xtrabackup:1.0ports:- name: xtrabackupcontainerPort: 3307command:- bash- "-c"- |set -excd /var/lib/mysql# Determine binlog position of cloned data, if any.if [[ -f xtrabackup_slave_info ]]; then# XtraBackup already generated a partial "CHANGE MASTER TO" query# because we're cloning from an existing slave.mv xtrabackup_slave_info change_master_to.sql.in# Ignore xtrabackup_binlog_info in this case (it's useless).rm -f xtrabackup_binlog_infoelif [[ -f xtrabackup_binlog_info ]]; then# We're cloning directly from master. Parse binlog position.[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1rm xtrabackup_binlog_infoecho "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in #生成CHANGE MASTER命令fi# Check if we need to complete a clone by starting replication.if [[ -f change_master_to.sql.in ]]; thenecho "Waiting for mysqld to be ready (accepting connections)"until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; doneecho "Initializing replication from clone position"# In case of container restart, attempt this at-most-once.mv change_master_to.sql.in change_master_ig#执行CHANGE MASTER操作并启动SLAVEmysql -h 127.0.0.1 <<EOF$(<change_master_ig),MASTER_HOST=&#sql',MASTER_USER='root',MASTER_PASSWORD='',MASTER_CONNECT_RETRY=10;START SLAVE;EOFfi# Start a server to send backups when requested by peers. #监听在3307端口，用于为下一个pod同步全量数据exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"volumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dresources:requests:cpu: 100mmemory: 100Mivolumes:- name: confemptyDir: {}- name: config-mapconfigMap:name: mysqlimagePullSecrets:- name: dockerhub-image-pull-keyvolumeClaimTemplates:- metadata:name: dataspec:accessModes: ["ReadWriteOnce"]resources:requests:storage: 10Gi
[root@k8s-harbor01 sts]# docker pull mysql:5.7.36
[root@k8s-harbor01 sts]# docker tag mysql:5.7.p/tsk8s/mysql:5.7.36
[root@k8s-harbor01 sts]# docker p/tsk8s/mysql:5.7.36[root@k8s-harbor01 sts]# docker pull yizhiyong/xtrabackup
[root@k8s-harbor01 sts]# docker tag yizhiyong/xtrabackup:p/tsk8s/xtrabackup:1.0
[root@k8s-harbor01 sts]# docker p/tsk8s/xtrabackup:1.0
[root@k8s-harbor01 sts]# kubectl apply -f mysql-statefulset.yaml
[root@k8s-harbor01 sts]# kubectl get po -n myserver |grep mysql
mysql-0 2/2 Running 0 8m49s
mysql-1 2/2 Running 0 7m35s
[root@k8s-harbor01 ~]# kubectl scale sts -n myserver mysql --replicas=3
statefulset.apps/mysql scaled[root@k8s-harbor01 ~]# kubectl get po -A|grep mysql
myserver mysql-0 2/2 Running 0 4h55m
myserver mysql-1 2/2 Running 0 4h53m
myserver mysql-2 2/2 Running 0 77s # 差不多60s能起来
[root@k8s-harbor01 ~]# kubectl exec -it -n myserver mysql-2 -- /bin/bash
root@mysql-2:/# mysql
mysql> show slave statusG
*************************** 1. row ***************************Slave_IO_State: Waiting for master to send eventMaster_Host: sqlMaster_User: rootMaster_Port: 3306Connect_Retry: 10Master_Log_File: mysql-0-bin.000003Read_Master_Log_Pos: 154Relay_Log_File: mysql-2-relay-bin.000002Relay_Log_Pos: 322Relay_Master_Log_File: mysql-0-bin.000003Slave_IO_Running: Yes # io线程和sql线程为yes就可以了Slave_SQL_Running: Yes
……省略部分内容
[root@k8s-harbor01 ~]# kubectl exec -it -n myserver mysql-0 -- /bin/bash
root@mysql-0:/# mysql
mysql> show master statusG
*************************** 1. row ***************************File: mysql-0-bin.000003Position: 154Binlog_Do_DB: Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 secmysql> create database tsk8s;
Query OK, 1 row affected (0.05 sec)mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| tsk8s |
| xtrabackup_backupfiles |
+------------------------+
6 rows in set (0.07 sec)
[root@k8s-harbor01 ~]# kubectl exec -it -n myserver mysql-1 -- /bin/bash
root@mysql-1:/# mysql
mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| tsk8s |
| xtrabackup_backupfiles |
+------------------------+
6 rows in set (0.01 sec)
[root@containerd-build-image jenkins]# cat Dockerfile
#Jenkins Version 2.190.1
p/pub-images/jdk-base:v8.212ADD jenkins-2.319.2.war /apps/jenkins/jenkins.war
ADD run_jenkins.sh /usr/bin/
RUN chmod u+x /usr/bin/run_jenkins.shEXPOSE 8080 CMD ["/usr/bin/run_jenkins.sh"]
[root@containerd-build-image jenkins]# cat run_jenkins.sh
#!/bin/bash
cd /apps/jenkins && java -server -Xms1024m -Xmx1024m -Xss512k -jar jenkins.war --webroot=/apps/jenkins/jenkins-data --httpPort=8080
[root@containerd-build-image jenkins]# ll jenkins-2.319.2.war
-rw-r--r-- 1 root root 72248203 7月 6 13:51 jenkins-2.319.2.wa
# 编辑构建脚本
[root@containerd-build-image jenkins]# cat build-image.sh
#!/bin/bash
echo "即将开始就像构建,请稍等!" && echo 3 && sleep 1 && echo 2 && sleep 1 && echo 1
nerdctl build -t p/tsk8s/jenkins:v2.319.2 .
if [ $? -eq 0 ];thenecho "即将开始镜像上传,请稍等!" && echo 3 && sleep 1 && echo 2 && sleep 1 && echo 1nerdctl p/tsk8s/jenkins:v2.319.2 if [ $? -eq 0 ];thenecho "镜像上传成功!"elseecho "镜像上传失败"fi
elseecho "镜像构建失败,请检查构建输出信息!"
fi# 构建镜像
[root@containerd-build-image jenkins]# sh build-image.sh
[root@k8s-harbor01 deployment]# cat jenkins.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: jenkinsname: jenkinsnamespace: myserver
spec:replicas: 1selector:matchLabels:app: jenkinstemplate:metadata:labels:app: jenkinsspec:containers:- name: jenkinsimage: p/tsk8s/jenkins:v2.319.2 imagePullPolicy: IfNotPresent#imagePullPolicy: Alwaysports:- containerPort: 8080protocol: TCPname: httpvolumeMounts:- mountPath: "/apps/jenkins/jenkins-data/"name: jenkins-datadir- mountPath: "/root/.jenkins"name: jenkins-root-datadirvolumes:- name: jenkins-datadirpersistentVolumeClaim:claimName: jenkins-datadir-pvc- name: jenkins-root-datadirpersistentVolumeClaim:claimName: jenkins-root-data-pvcimagePullSecrets:- name: dockerhub-image-pull-key---
kind: Service
apiVersion: v1
metadata:labels:app: jenkinsname: jenkins-servicenamespace: mmyserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 8080selector:app: jenkins
# 编辑pv[root@k8s-harbor01 volume]# cat jenkins-pv.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:name: jenkins-datadir-pv
spec:capacity:storage: 100GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/jenkins-data ---
apiVersion: v1
kind: PersistentVolume
metadata:name: jenkins-root-datadir-pv
spec:capacity:storage: 100GiaccessModes:- ReadWriteOncenfs:server: 10.31.200.104path: /data/k8s-data/tsk8s/jenkins-root-data# 编辑pvc
[root@k8s-harbor01 volume]# cat jenkins-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: jenkins-datadir-pvcnamespace: myserver
spec:volumeName: jenkins-datadir-pvaccessModes:- ReadWriteOnceresources:requests:storage: 80Gi---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: jenkins-root-data-pvcnamespace: myserver
spec:volumeName: jenkins-root-datadir-pv accessModes:- ReadWriteOnceresources:requests:storage: 80Gi
[root@k8s-harbor01 volume]# mkdir /data/k8s-data/tsk8s/jenkins-data
[root@k8s-harbor01 volume]# mkdir /data/k8s-data/tsk8s/jenkins-root-data
# 创建pv
[root@k8s-harbor01 volume]# kubectl apply -f jenkins-pv.yaml
persistentvolume/jenkins-datadir-pv created
persistentvolume/jenkins-root-datadir-pv created
[root@k8s-harbor01 volume]# kubectl get pv |grep jenkins
jenkins-datadir-pv 100Gi RWO Retain Available 6s
jenkins-root-datadir-pv 100Gi RWO Retain Available 6s# 创建pvc
[root@k8s-harbor01 volume]# kubectl get pvc -A |grep jenkins
myserver jenkins-datadir-pvc Bound jenkins-datadir-pv 100Gi RWO 6s
myserver jenkins-root-data-pvc Bound jenkins-root-datadir-pv 100Gi RWO # 创建pod
[root@k8s-harbor01 volume]# cd ../deployment/
[root@k8s-harbor01 deployment]# kubectl apply -f jenkins.yaml
deployment.apps/jenkins unchanged
service/jenkins-service created
[root@k8s-harbor01 deployment]# kubectl get po,svc -n myserver|grep jenkins
pod/jenkins-7bc957c788-n8k7n 1/1 Running 0 12s
service/jenkins-service NodePort 10.100.246.230 <none> 80:31797/TCP 12s
[root@containerd-build-image provider]# cat Dockerfile
#Dubbo provider
p/pub-images/jdk-base:v8.212 ADD dubbo-demo-provider-2.1.5/ /apps/dubbo/provider
ADD run_java.sh /apps/dubbo/provider/binRUN yum install file nc -y && useradd nginx && inx /apps -R && chmod a+x /apps/dubbo/provider/bin/*.shCMD ["/apps/dubbo/provider/bin/run_java.sh"]
# 先准备好相关的配置
[root@containerd-build-image provider]# ll -rt
总用量 4
-rw-r--r-- 1 root root 353 7月 6 15:09 Dockerfile
drwxrwxr-x 5 root root 40 7月 6 16:21 dubbo-demo-provider-2.1.5# 编辑配置文件
[root@containerd-build-image provider]# vim dubbo-demo-provider-2.1.5/conf/dubbo.properties
ainer=log4j,spring
dubbo.application.name=demo-provider
dubbo.application.owner=
#istry.address=multicast://224.5.6.7:1234
istry.address=zookeeper://server.svc.cluster.local:2181 | zookeeper://server.svc.cluster.local:2181 | zookeeper://server.svc.cluster.local:2181 # 注意调整svc的地址itor.protocol=registry
dubbo.protocol.name=dubbo
dubbo.protocol.port=20880
dubbo.log4j.file=logs/dubbo-demo-provider.log
dubbo.log4j.level=WARN
[root@containerd-build-image provider]# cat run_java.sh
#!/bin/bash
su - nginx -c "/apps/dubbo/provider/bin/start.sh"
tail -f /etc/hosts
[root@containerd-build-image provider]# cat build-image.sh
#!/bin/bash
nerdctl build -p/tsk8s/dubbo-demo-provider:v1 . &&
nerdctl p/tsk8s/dubbo-demo-provider:v1
[root@containerd-build-image provider]# sh build-image.sh
[root@k8s-harbor01 deployment]# cat dubbo-provider.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: dubbo-providername: dubbo-providernamespace: myserver
spec:replicas: 1selector:matchLabels:app: dubbo-providertemplate:metadata:labels:app: dubbo-providerspec:containers:- name: dubbo-providerimage: p/tsk8s/dubbo-demo-provider:v1#command: ["/apps/tomcat/bin/run_tomcat.sh"]#imagePullPolicy: IfNotPresentimagePullPolicy: Alwaysports:- containerPort: 20880protocol: TCPname: httpimagePullSecrets:- name: dockerhub-image-pull-key
---
kind: Service
apiVersion: v1
metadata:labels:app: dubbo-providername: dubbo-provider-nodeportnamespace: myserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPselector:app: dubbo-provider
[root@k8s-harbor01 deployment]# kubectl apply -f dubbo-provider.yaml
deployment.apps/dubbo-provider created
service/dubbo-provider-nodeport created[root@k8s-harbor01 deployment]# kubectl get po,svc -n myserver |grep dubbo
pod/dubbo-provider-76d5f5cc9c-crs2p 1/1 Running 0 29s
service/dubbo-provider-nodeport NodePort 10.100.159.26 <none> 80:32589/TCP 29s
[root@containerd-build-image provider]# cd ..
[root@containerd-build-image dubbo]# mkdir consumer
[root@containerd-build-image dubbo]# cd consumer
[root@containerd-build-image consumer]# cat Dockerfile
#Dubbo consumer
p/pub-images/jdk-base:v8.212ADD dubbo-demo-consumer-2.1.5 /apps/dubbo/consumer
ADD run_java.sh /apps/dubbo/consumer/binRUN yum install file -y && useradd nginx && inx /apps -R && chmod a+x /apps/dubbo/consumer/bin/*.shCMD ["/apps/dubbo/consumer/bin/run_java.sh"]
# 先把所需文件上传到服务器
[root@containerd-build-image consumer]# ll -rt dubbo-demo-consumer-2.1.5
总用量 4
drwxrwxr-x 2 root root 30 7月 6 17:09 conf
drwxrwxr-x 2 root root 104 7月 6 17:09 bin
drwxrwxr-x 2 root root 4096 7月 6 17:09 lib# 修改配置
[root@containerd-build-image consumer]# cat dubbo-demo-consumer-2.1.5/conf/dubbo.properties
ainer=log4j,spring
dubbo.application.name=demo-consumer
dubbo.application.owner=
istry.address=zookeeper://server.svc.cluster.local:2181 | zookeeper://server.svc.cluster.local:2181 | zookeeper://server.svc.cluster.local:2181 # 主要还是修改svc地址
itor.protocol=registry
dubbo.log4j.file=logs/dubbo-demo-consumer.log
dubbo.log4j.level=WARN
[root@containerd-build-image consumer]# cat run_java.sh
#!/bin/bash
su - nginx -c "/apps/dubbo/consumer/bin/start.sh"
tail -f /etc/hosts
[root@containerd-build-image consumer]# cat build-image.sh
#!/bin/bash
nerdctl build -p/tsk8s/dubbo-demo-consumer:v1 . &&
nerdctl p/tsk8s/dubbo-demo-consumer:v1
[root@containerd-build-image consumer]# sh build-image.sh
[root@k8s-harbor01 deployment]# cat dubbo-consumer.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: dubbo-consumername: dubbo-consumernamespace: myserver
spec:replicas: 1selector:matchLabels:app: dubbo-consumertemplate:metadata:labels:app: dubbo-consumerspec:containers:- name: dubbo-consumerimage: p/tsk8s/dubbo-demo-consumer:v1 #command: ["/apps/tomcat/bin/run_tomcat.sh"]#imagePullPolicy: IfNotPresentimagePullPolicy: Alwaysports:- containerPort: 80protocol: TCPname: httpimagePullSecrets:- name: dockerhub-image-pull-key
---
kind: Service
apiVersion: v1
metadata:labels:app: dubbo-consumername: dubbo-consumer-nodeportnamespace: myserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 80selector:app: dubbo-consumer
[root@k8s-harbor01 deployment]# kubectl get po,svc -A |grep consumer
myserver pod/dubbo-consumer-6f65f7548b-rnvvj 1/1 Running 0 98m
myserver service/dubbo-consumer-nodeport NodePort 10.100.2.73 <none> 80:31200/TCP 98m# 检查生产消费
[root@k8s-harbor01 deployment]# kubectl exec -it -n myserver pod/dubbo-consumer-6f65f7548b-rnvvj -- /bin/bash
[root@dubbo-consumer-6f65f7548b-rnvvj /]# cd /apps/dubbo/consumer/logs/
[root@dubbo-consumer-6f65f7548b-rnvvj logs]# tail -f
dubbo-demo-consumer.log stdout.log
[root@dubbo-consumer-6f65f7548b-rnvvj logs]# tail stdout.log # 通过下面的日志可以看出，消费者消费的消息是从哪个pod来的
[19:17:58] Hello world3275, response form provider: 10.200.135.160:20880
[19:18:00] Hello world3276, response form provider: 10.200.135.160:20880
[19:18:02] Hello world3277, response form provider: 10.200.135.160:20880
[19:18:04] Hello world3278, response form provider: 10.200.135.160:20880
[19:18:06] Hello world3279, response form provider: 10.200.135.160:20880
[19:18:08] Hello world3280, response form provider: 10.200.135.160:20880
[19:18:10] Hello world3281, response form provider: 10.200.135.160:20880
[19:18:12] Hello world3282, response form provider: 10.200.135.160:20880
[19:18:14] Hello world3283, response form provider: 10.200.135.160:20880
[19:18:16] Hello world3284, response form provider: 10.200.135.160:20880
[root@dubbo-consumer-6f65f7548b-rnvvj logs]# exit
exit
[root@k8s-harbor01 deployment]# kubectl get po -o wide -A |grep 10.200.135.160
myserver dubbo-provider-76d5f5cc9c-crs2p 1/1 Running 0 135m 10.200.135.160 k8s-node03 <none> <none>
[root@containerd-build-image consumer]# cd ..
[root@containerd-build-image dubbo]# mkdir admin
[root@containerd-build-image dubbo]# cd admin
[root@containerd-build-image admin]# vi Dockerfile
#Dubbo dubboadmin
p/pub-images/tomcat-base:v8.5.l /apps/tomcat/l
ADD logging.properties /apps/tomcat/conf/logging.properties
ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
ADD dubboadmin.war /data/tomcat/webapps/dubboadmin.warRUN yum install unzip -y && cd /data/tomcat/webapps && unzip dubboadmin.war && rm -rf dubboadmin.war && chown -inx /data /apps && chmod u+x /apps/tomcat/bin/run_tomcat.shEXPOSE 8080 8443CMD ["/apps/tomcat/bin/run_tomcat.sh"]
[root@containerd-build-image admin]# cat run_tomcat.sh
#!/bin/bashsu - nginx -c "/apps/tomcat/bin/catalina.sh start"
su - nginx -c "tail -f /etc/hosts"
[root@containerd-build-image admin]# cat build-image.sh
#!/bin/bash
TAG=$1
nerdctl build -t p/tsk8s/dubboadmin:${TAG} . &&
nerdctl p/tsk8s/dubboadmin:${TAG}
[root@containerd-build-image admin]# ll -rt
总用量 54308
-rw-r--r-- 1 root root 22201 6月 22 2021 catalina.sh
-rw-r--r-- 1 root root 6427 6月 22 l
-rw-r--r-- 1 root root 3436 6月 22 2021 logging.properties
-rw-r--r-- 1 root root 27777984 6月 22 2021 dubboadmin.war.bak
-rw-r--r-- 1 root root 27777986 5月 11 18:14 dubboadmin.war
-rw-r--r-- 1 root root 605 7月 6 19:35 Dockerfile
-rw-r--r-- 1 root root 99 7月 6 19:36 run_tomcat.sh
-rw-r--r-- 1 root root 125 7月 6 19:39 build-image.sh
[root@containerd-build-image admin]# sh build-image.sh v1
[root@k8s-harbor01 deployment]# cat dubboadmin.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: dubboadminname: dubboadminnamespace: myserver
spec:replicas: 1selector:matchLabels:app: dubboadmintemplate:metadata:labels:app: dubboadminspec:containers:- name: dubboadminimage: p/tsk8s/dubboadmin:v1 #imagePullPolicy: IfNotPresentimagePullPolicy: Alwaysports:- containerPort: 8080protocol: TCPname: httpimagePullSecrets:- name: dockerhub-image-pull-key
---
kind: Service
apiVersion: v1
metadata:labels:app: dubboadminname: dubboadmin-nodeportnamespace: myserver
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 8080selector:app: dubboadmin
[root@k8s-harbor01 deployment]# kubectl get po,svc -A |grep admin
myserver pod/dubboadmin-86f886b5f6-zsfzn 1/1 Running 0 56s
myserver service/dubboadmin-nodeport NodePort 10.100.155.134 <none> 80:30730/TCP 56s
本文发布于:2024-01-28 05:33:14，感谢您对本站的认可！
本文链接：https://www.4u4v.net/it/17063911975154.html
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系，我们将在24小时内删除。
| 留言与评论（共有 0 条评论） |
