构建高可用Kubernetes集群：架构设计与实践指南

1. 为什么需要高可用K8s集群在生产环境中单Master节点的Kubernetes集群存在明显的单点故障风险。当Master节点宕机时整个集群的控制平面将完全瘫痪导致无法创建新Pod、无法进行滚动更新等关键操作。根据CNCF 2023年度调查报告显示超过78%的企业在生产环境遭遇过因控制平面故障导致的业务中断。高可用(High Availability)架构通过多Master节点冗余部署配合负载均衡机制确保即使单个Master节点失效集群仍能维持正常运作。这种架构需要解决三个核心问题API Server的负载均衡与故障转移集群状态的一致性维护工作节点与控制平面的稳定通信2. 架构设计与核心组件选型2.1 三Master节点拓扑我们采用奇数节点(3个)的部署方式这是基于满足etcd的RAFT算法对多数派投票的要求3节点允许1节点故障5节点允许2节点故障在可靠性与资源消耗间取得平衡5节点会显著增加资源开销典型部署拓扑如下[Worker Nodes] ↑ [Haproxy Keepalived VIP] (TCP负载均衡层) ↑ [Master1] ↔ [Master2] ↔ [Master3] (运行kube-apiserver、etcd等组件)2.2 关键组件作用解析Keepalived提供虚拟IP(VIP)漂移能力通过VRRP协议实现主备选举健康检查确保只有健康节点持有VIPHaproxy四层(TCP)负载均衡器将API请求分发到后端多个Master节点会话保持与连接池管理IPVSLinux内核级负载均衡替代kube-proxy的iptables模式支持多种调度算法(rr/wrr/lc等)处理大规模服务时性能显著优于iptables实测数据在1000个Service的场景下IPVS的规则更新速度比iptables快3-5倍CPU消耗降低40%3. 环境准备与系统配置3.1 硬件资源规划节点类型数量vCPU内存磁盘网络Master34核8GB50GB千兆WorkerN4核8GB50GB千兆LB22核4GB20GB千兆3.2 Ubuntu系统基础配置在所有节点执行# 禁用swap sudo swapoff -a sudo sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab # 设置时区同步 sudo timedatectl set-timezone Asia/Shanghai sudo apt install -y chrony sudo systemctl enable --now chronyd # 加载内核模块 cat EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack EOF sudo modprobe -- br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack # 设置sysctl参数 cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 EOF sudo sysctl --system4. Keepalived与Haproxy部署4.1 Keepalived配置在两台LB节点上安装配置主备模式sudo apt install -y keepalived主节点配置(/etc/keepalived/keepalived.conf)global_defs { router_id LVS_MASTER } vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 101 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.100/24 } track_script { chk_haproxy } }备节点只需修改state BACKUP priority 1004.2 Haproxy配置两台LB节点相同配置(/etc/haproxy/haproxy.cfg)global log /dev/log local0 maxconn 20000 user haproxy group haproxy daemon defaults log global mode tcp timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend k8s-api bind *:6443 default_backend k8s-api backend k8s-api balance roundrobin option tcp-check server master1 192.168.1.101:6443 check fall 3 rise 2 server master2 192.168.1.102:6443 check fall 3 rise 2 server master3 192.168.1.103:6443 check fall 3 rise 2 listen stats bind *:1080 mode http stats enable stats hide-version stats uri /stats stats auth admin:password启动服务sudo systemctl enable --now haproxy keepalived5. K8s集群初始化与高可用配置5.1 安装容器运行时所有节点安装containerdsudo apt install -y containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml sudo sed -i s/SystemdCgroup false/SystemdCgroup true/ /etc/containerd/config.toml sudo systemctl restart containerd5.2 安装kubeadm/kubelet/kubectlsudo apt update sudo apt install -y apt-transport-https ca-certificates curl curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ / | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt update sudo apt install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl5.3 初始化第一个Master节点在master1执行sudo kubeadm init \ --control-plane-endpoint 192.168.1.100:6443 \ --upload-certs \ --pod-network-cidr10.244.0.0/16 \ --service-cidr10.96.0.0/12 \ --image-repository registry.aliyuncs.com/google_containers记录输出的join命令用于其他节点加入。5.4 加入其他Master节点使用带--control-plane参数的join命令sudo kubeadm join 192.168.1.100:6443 \ --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx \ --control-plane \ --certificate-key xxxx5.5 配置IPVS模式创建ConfigMapkubectl edit cm kube-proxy -n kube-system修改mode: ipvs ipvs: strictARP: true重启kube-proxykubectl rollout restart daemonset kube-proxy -n kube-system验证IPVS规则sudo ipvsadm -Ln6. 验证与故障排查6.1 高可用验证测试VIP漂移测试# 在主LB节点执行 sudo systemctl stop keepalived # 在备节点检查VIP获取情况 ip addr show ens33API Server故障转移# 随机停止一个Master节点的kube-apiserver sudo systemctl stop kube-apiserver # 观察集群状态应保持正常 kubectl get nodes6.2 常见问题解决问题1kubelet报错NodeNotReady检查网络插件是否安装确认容器运行时正常查看kubelet日志journalctl -u kubelet -f问题2Haproxy后端标记为DOWN检查Master节点6443端口是否监听验证网络连通性调整Haproxy的check间隔参数问题3etcd集群状态异常检查各节点etcd容器日志验证节点间时间同步使用etcdctl检查集群健康状态ETCDCTL_API3 etcdctl \ --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ endpoint health7. 生产环境优化建议网络性能调优为etcd流量配置独立网卡调整内核网络参数echo net.core.somaxconn 32768 /etc/sysctl.conf echo net.ipv4.tcp_tw_reuse 1 /etc/sysctl.conf监控与告警部署Prometheus监控集群状态对关键指标设置告警API Server延迟etcd写入延迟节点NotReady状态证书管理使用cert-manager自动续期证书定期备份/etc/kubernetes/pki目录升级策略采用滚动更新方式升级Master节点先升级kubeadm再升级控制平面组件最后升级kubelet和kubectl在长期运维过程中建议定期执行kubeadm alpha certs check-expiration检查证书过期时间kubectl get componentstatus查看组件健康状态kubectl top nodes监控资源使用情况