写在最前

在 Kubernetes 高可用部署中,通常涉及到 3 个 master 节点和多个 worker 节点。对于私有化部署,尤其在没有额外云服务资源(如阿里云 SLB)的情况下,我们不能依赖额外的机器,这样会增加不必要的开销。

为了实现 master 节点的高可用,通常使用 keepalivedHAProxy,但这种方式不适合将域名访问流量直接引入 master 节点,否则会影响其性能,甚至可能导致宕机。

虽然可以在 worker 节点上继续使用 keepalived 来创建第二个虚拟 IP(VIP)以供 Ingress 使用,但考虑到我们已经实现了容器化部署,继续依赖 keepalived 这样的传统工具显得不够优雅。此时,kube-vip 提供了一种更加简洁高效的解决方案。通过在指定节点中打上标签并部署 DaemonSet,使用容器化方式部署提供VIP是不是非常优雅呢。

1. kube-vip

https://github.com/kube-vip/kube-vip

1.1 rabc

kube-vip-rabc.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-vip
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: system:kube-vip-role
rules:
  - apiGroups: [""]
    resources: ["services/status"]
    verbs: ["update"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["list","get","watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list","get","watch", "update", "patch"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["list", "get", "watch", "update", "create"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list","get","watch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:kube-vip-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-vip-role
subjects:
- kind: ServiceAccount
  name: kube-vip
  namespace: kube-system

1.2 daemonset

kube-vip.yaml 中,虚拟 IP(VIP)需要自行配置,例如将 172.31.0.99 更改为所需的 VIP 地址,按你喜好自行决定。最重要的是节点亲和性配置,需要在目标机器上打上 kube-vip 标签,以确保 kube-vip 仅在这些节点上运行。可以使用以下命令为节点打标签:kubectl label nodes worker1 kube-vip=""

apiVersion: apps/v1
kind: DaemonSet
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: kube-vip
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: kube-vip
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kube-vip
                operator: Exists
      containers:
      - args:
        - manager
        env:
        - name: vip_arp
          value: "true"
        - name: vip_interface
          value: eth0
        - name: vip_cidr
          value: "32"
        - name: vipSubnet
          value: ""
        - name: cp_enable
          value: "true"
        - name: cp_namespace
          value: kube-system
        - name: vip_ddns
          value: "false"
        - name: svc_enable
          value: "true"
        - name: vip_leaderelection
          value: "true"
        - name: vip_leaseduration
          value: "5"
        - name: vip_renewdeadline
          value: "3"
        - name: vip_retryperiod
          value: "1"
        - name: address
          value: 172.31.0.99
        image: plndr/kube-vip:v1.0.2
        imagePullPolicy: Always
        name: kube-vip
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
            - SYS_TIME
      hostNetwork: true
      serviceAccountName: kube-vip
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
  updateStrategy: {}
status:
  currentNumberScheduled: 0
  desiredNumberScheduled: 0
  numberMisscheduled: 0
  numberReady: 0

2. kube-vip-cloud-provider

https://github.com/kube-vip/kube-vip-cloud-provider

kube-vip-cloud-provider 是一个 Kubernetes cloud provider 插件,它为 Kubernetes 集群提供了云环境中的 虚拟 IP (VIP) 功能。这使得 Kubernetes 集群可以使用 高可用性负载均衡动态 IP 管理,尤其是在没有云负载均衡器(如 AWS ELB 或 Google Cloud Load Balancer)的环境中,提供类似云负载均衡的功能。

2.1 configmap

配置 kube-vip 时,如果只有一个虚拟 IP 可用,可以将其范围设置为相同的 IP 地址,即将 range-global 配置为 172.31.0.99-172.31.0.99,这样确保使用的唯一 VIP 地址在配置中左右一致。

kind: ConfigMap
apiVersion: v1
metadata:
  name: kubevip
  namespace: kube-system
  annotations:
    kubesphere.io/creator: admin
data:
  range-global: 172.31.0.99-172.31.0.99

2.2 deployment

kube-vip-cloud-controller.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-vip-cloud-controller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: system:kube-vip-cloud-controller-role
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update", "list", "put"]
  - apiGroups: [""]
    resources: ["configmaps", "endpoints","events","services/status", "leases"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["nodes", "services"]
    verbs: ["list","get","watch","update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:kube-vip-cloud-controller-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-vip-cloud-controller-role
subjects:
- kind: ServiceAccount
  name: kube-vip-cloud-controller
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-vip-cloud-provider
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kube-vip
      component: kube-vip-cloud-provider
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kube-vip
        component: kube-vip-cloud-provider
    spec:
      containers:
      - command:
        - /kube-vip-cloud-provider
        - --leader-elect-resource-name=kube-vip-cloud-controller
        image: kubevip/kube-vip-cloud-provider:v0.0.12 
        name: kube-vip-cloud-provider
        imagePullPolicy: Always
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      serviceAccountName: kube-vip-cloud-controller
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 10
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
          - weight: 10
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists

写在最后