k8s高可用集群集群环境初始化实验环境主机名IP角色reg.westos.org192.168.36.151harbor私有镜像仓库k8s-master01192.168.36.155control-planehaproxykeepalivedk8s-master02192.168.36.156control-planehaproxykeepalivedk8s-master03192.168.36.157control-planehaproxykeepalivedk8s-worker01192.168.36.158workerk8s-worker02192.168.36.159worker使用现有的集群k8s123做master配置解析禁用swap配置yum源升级内核HAProxy 负载均衡1. 安装 HAProxy# yum install -y haproxy2. 配置 HAProxy# cat /etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option dontlognull option http-server-close option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s frontend monitor bind *:8000 mode http option httplog stats uri /status frontend k8s-apiserver bind *:8443 mode tcp option tcplog default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin server master1 192.168.36.152:6443 check fall 3 rise 2 server master2 192.168.36.153:6443 check fall 3 rise 2 server master3 192.168.36.154:6443 check fall 3 rise 2配置说明8000 ↓ HAProxy 状态监控页面 /status8443 ↓ Kubernetes API Server ↓ master1 / master2 / master3balance roundrobin表示采用轮询方式将请求分配给多个 Master。客户端 ↓ HAProxy :8443 ↓ ┌───────┬───────┬───────┐ ↓ ↓ ↓ master1 master2 master3 :6443 :6443 :6443check表示检查后端 Master 是否正常。3. 启动 HAProxy# systemctl enable --now haproxy.service检查# ss -antlp | grep haproxy应该看到LISTEN 0 2000 *:8000 LISTEN 0 2000 *:8443其它 Master 节点的 HAProxy 配置相同。Keepalived 高可用HAProxy 解决的是多个 Master 之间如何进行负载均衡。Keepalived 解决的是如果当前 HAProxy 所在节点发生故障谁接管 VIP。1. 安装 Keepalived# yum install -y keepalived2. 配置 Keepalived# cat /etc/keepalived/keepalived.conf! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script check_haproxy { script /etc/keepalived/check_haproxy.sh interval 5 weight -20 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.36.200 } track_script { check_haproxy } }关键参数virtual_ipaddress ↓ 192.168.36.200 ↓ VIP客户端只需要访问192.168.36.200:8443不需要关心具体是哪一台 Master 提供服务。优先级k8s-master01 100 k8s-master02 90 k8s-master03 80正常情况下MASTER → priority 100 BACKUP → priority 90 BACKUP → priority 80HAProxy 检测脚本# cat /etc/keepalived/check_haproxy.sh#!/bin/bash if ! ps aux | grep -q [h]aproxy; then systemctl start haproxy sleep 2 if ! ps aux | grep -q [h]aproxy; then systemctl stop keepalived fi fi添加执行权限# chmod x /etc/keepalived/check_haproxy.sh这个脚本的逻辑检查 HAProxy ↓ 正常 ↓ 继续运行 异常 ↓ 尝试启动 HAProxy ↓ 再次检查 ↓ 启动成功 → 继续运行 启动失败 → 停止 Keepalived ↓ VIP 发生切换启动 Keepalived# systemctl enable --now keepalived.service检查# systemctl status keepalived查看 VIP# ip addr应该可以看到192.168.36.200其他 Master 节点配置其他 Master 节点的keepalived.conf大部分配置保持一致。只需要修改k8s-master02state BACKUP priority 90k8s-master03state BACKUP priority 80kubeadm部署集群之前已经部署过的修改系统限制修改内核参数安装 IPVS 依赖部署containerd安装kubeadm、kubectl、kubelet集群初始化使用kubeadm初始化 Kubernetes 集群通过HAProxy Keepalived VIP提供统一的 API Server 入口使用Calico作为 Pod 网络插件使用Harbor作为私有镜像仓库。配置集群初始化文件# cat kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta4 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.36.152 #本机IP bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent imagePullSerial: true name: k8s-master01 #本机主机名 taints: null timeouts: controlPlaneComponentHealthCheck: 4m0s discovery: 5m0s etcdAPICall: 2m0s kubeletHealthCheck: 4m0s kubernetesAPICall: 1m0s tlsBootstrap: 5m0s upgradeManifests: 5m0s --- apiServer: certSANs: #添加VIP到证书 - 192.168.36.200 apiVersion: kubeadm.k8s.io/v1beta4 caCertificateValidityPeriod: 876000h0m0s certificateValidityPeriod: 87600h0m0s certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 192.168.36.200:8443 #负载均衡地址端口 controllerManager: {} dns: {} encryptionAlgorithm: RSA-2048 etcd: local: dataDir: /var/lib/etcd imageRepository: reg.westos.org/k8s #harbor私有仓库 kind: ClusterConfiguration kubernetesVersion: 1.34.1 #k8s部署版本 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 #service网段 podSubnet: 10.244.0.0/16 #Pod网段 proxy: {} scheduler: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs #kube-proxy使用IPVS模式 ipvs: scheduler: rr #调度算法 strictARP: true #高可用场景必须开启避免 ARP 冲突关键配置192.168.36.200:8443 ↓ HAProxy Keepalived ↓ 多个 Master:6443其中advertiseAddress当前 Master 节点 IP。controlPlaneEndpoint集群统一 API Server 地址也就是 HAProxy Keepalived 的 VIP。certSANs把 VIP 加入 API Server 证书避免通过 VIP 访问时证书校验失败。imageRepository使用 Harbor 私有仓库。serviceSubnetService 网段。podSubnetPod 网段。mode: ipvskube-proxy 使用 IPVS。strictARP: trueIPVS 高可用环境下开启。初始化集群要先清理之前集群残留的配置# 1. 先把重要数据备份etcd 快照、YAML 清单等 mkdir -p /backup/etcd ETCDCTL_API3 etcdctl \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/apiserver-etcd-client.crt \ --key/etc/kubernetes/pki/apiserver-etcd-client.key \ --endpoints https://127.0.0.1:2379 \ snapshot save /backup/etcd/etcd-final-$(date %F).bak cp -r /etc/kubernetes /backup/kubernetes-$(date %F) # 业务 YAML 也备份一下 # 2. reset 掉旧集群 kubeadm reset -f rm -rf /etc/kubernetes /var/lib/etcd /var/lib/kubelet/* \ /etc/cni/net.d $HOME/.kube # 3. 清理网络插件残留Calico / kube-proxy 的 iptables / ipvs iptables -F iptables -t nat -F iptables -t mangle -F iptables -X ipvsadm -C 2/dev/null # 4. 清理容器 crictl rm -f $(crictl ps -aq) 2/dev/null crictl rmi --prune 2/dev/null # 5. 重启 containerd 和 kubelet systemctl restart containerd kubelet在k8s-master01执行kubeadm init --configkubeadm-config.yaml --upload-certs初始化成功后会生成管理员 kubeconfigWorker 加入命令Master 加入命令certificate-key设置 KUBECONFIGexport KUBECONFIG/etc/kubernetes/admin.conf写入环境变量文件确保重启后依然生效echo export KUBECONFIG/etc/kubernetes/admin.conf ~/.bash_profile设置 kubectl 命令补齐安装yum install -y bash-completion开启补齐echo source (kubectl completion bash) ~/.bashrc source ~/.bashrc查看集群状态# kubectl get node刚初始化完成时NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane 9m42s v1.34.1此时NotReady通常是因为还没有安装 CNI 网络插件。初始化失败时重置节点如果初始化出现严重错误可以重置kubeadm reset -f --ignore-preflight-errorsall然后根据实际情况重新初始化。安装 Calico 网络插件1. 下载 Calicowget https://raw.githubusercontent.com/projectcalico/calico/v3.31.0/manifests/calico.yaml2. 修改镜像仓库将 Calico 镜像从官方仓库修改为 Harbor# sed -i s#quay.io/#reg.westos.org/#g calico.yaml3. 部署 Calico# kubectl apply -f calico.yaml4. 查看节点状态# kubectl get node正常NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 17m v1.34.1扩容 Master 节点在其他 Master 节点执行初始化时生成的kubeadm join命令。kubeadm join 192.168.36.200:8443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bd1808e77dd5669cfdd5c5b7859cf0ea9c8444b1c1bce990231eaefe80457b6e \ --control-plane --certificate-key 815568fa321d937c4a342dace14fbb657d77ee658431035b5fd218518691cf63注意实际实验中使用kubeadm init输出的 join 命令不要直接照抄示例中的 token 和 certificate-key。查看 Master 节点# kubectl get node例如NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 17m v1.34.1 k8s-master02 Ready control-plane 87s v1.34.1 k8s-master03 Ready control-plane 76s v1.34.1扩容 Worker 节点在 Worker 节点执行kubeadm join 192.168.36.200:8443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:bd1808e77dd5669cfdd5c5b7859cf0ea9c8444b1c1bce990231eaefe80457b6eWorker 加入后查看# kubectl get node例如NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 87m v1.34.1 k8s-master02 Ready control-plane 71m v1.34.1 k8s-master03 Ready control-plane 71m v1.34.1 k8s-worker01 Ready none 20s v1.34.1给 Worker 节点添加标签kubectl label nodes k8s-worker01 node-role.kubernetes.io/worker再次查看kubectl get node最终形成k8s-master01 control-plane k8s-master02 control-plane k8s-master03 control-plane k8s-worker01 workerkube-vip 云原生高可用官网Static Pods | kube-vip原理kube-vip 以静态 Pod形式部署在所有 master 节点通过ARP Leader 选举为一个 VIP 提供服务。任意 master 挂了VIP 自动漂移到其他节点 → 集群 apiserver 高可用。kubeadm init 前 ↓ kube-vip 先跑起来提供 VIP 192.168.36.200 ↓ kubeadm init 用 VIP 作为 controlPlaneEndpoint ↓ apiserver 高可用一、上传镜像到 Harbor# 从 ghcr.io 拉取或用镜像代理 docker pull ghcr.io/kube-vip/kube-vip:v1.2.4 # 打 tag 指向私有仓库 docker tag ghcr.io/kube-vip/kube-vip:v1.2.4 \ reg.westos.org/k8s/kube-vip:v1.2.4 # 推送 docker login reg.westos.org -u admin -p westos docker push reg.westos.org/k8s/kube-vip:v1.2.4二、从镜像提取二进制可选只是为了本机有kube-vip命令方便生成 YAML。直接用docker run也能生成可跳过这步。docker create --name demo ghcr.io/kube-vip/kube-vip:v1.2.4 docker cp demo:/kube-vip /usr/local/bin/kube-vip chmod x /usr/local/bin/kube-vip docker rm -f demo # 验证 kube-vip version # Version: v1.2.4三、在所有 master 节点放 kube-vip 静态 Pod1. 生成 YAML推荐用docker rundocker run --rm --nethost \ reg.westos.org/k8s/kube-vip:v1.2.4 manifest pod \ --interface eth0 \ --address 192.168.150.200 \ --controlplane \ --services \ --arp \ --leaderElection \ --image reg.westos.org/k8s/kube-vip:v1.2.4 \ | sudo tee /etc/kubernetes/manifests/kube-vip.yaml2. 或直接手写 YAMLapiVersion: v1 kind: Pod metadata: name: kube-vip namespace: kube-system spec: containers: - args: - manager env: - name: vip_arp value: true - name: port value: 6443 - name: vip_nodename valueFrom: fieldRef: fieldPath: spec.nodeName - name: vip_interface value: eth0 - name: vip_subnet value: 32 - name: dns_mode value: first - name: cp_enable value: true - name: cp_namespace value: kube-system - name: svc_enable value: true - name: svc_leasename value: plndr-svcs-lock - name: vip_leaderelection value: true - name: vip_leasename value: plndr-cp-lock - name: vip_leaseduration value: 5 - name: vip_renewdeadline value: 3 - name: vip_retryperiod value: 1 - name: address value: 192.168.150.200 - name: prometheus_server value: :2112 image: reg.westos.org/k8s/kube-vip:v1.2.4 imagePullPolicy: IfNotPresent name: kube-vip resources: {} securityContext: capabilities: add: - NET_ADMIN - NET_RAW drop: - ALL volumeMounts: - mountPath: /etc/kubernetes/admin.conf name: kubeconfig hostAliases: - hostnames: - kubernetes ip: 127.0.0.1 hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/super-admin.conf name: kubeconfig status: {}3. 检查三处必改字段说明image改成你的私有仓库地址vip_interface改成实际网卡名ip a查address改成你规划的 VIP4. 复制到所有 master 节点scp /etc/kubernetes/manifests/kube-vip.yaml rootk8s2:/etc/kubernetes/manifests/ scp /etc/kubernetes/manifests/kube-vip.yaml rootk8s3:/etc/kubernetes/manifests/四、初始化集群kubeadm-config.yml 里必须把controlPlaneEndpoint指向 VIPcontrolPlaneEndpoint: 192.168.150.200:6443 apiServer: certSANs: - 192.168.150.200kubeadm init --configkubeadm-config.yml --upload-certs别的节点正常加入集群五、验证1. Pod 是否 Runningcrictl ps | grep kube-vip # 输出示例 # xxx xxx 19 minutes ago Running kube-vip ... kube-vip-k8s1 kube-system2. VIP 是否绑到网卡ip a show eth0 | grep 192.168.150.200 # 输出示例 # inet 192.168.150.200/32 scope global deprecated eth0