写在最前

在 Kubernetes 中部署 Redis 6 的 3 主 3 从的分片集群，真正的难点是 Redis Cluster 强依赖节点的固定 IP。Redis 在初始化时会把每个节点的 IP 写进集群元数据，一旦 IP 变化，主从关系就会失效，集群直接崩坏。而 Kubernetes 的 Pod IP 天然是不稳定的，即便使用 StatefulSet 与稳定名称，只要发生调度漂移，Redis 就会出现“IP 属于新节点，数据却来自旧节点 PVC”的矛盾，导致集群结构损坏。因此，StatefulSet + PVC 并不能解决问题，反而会制造混乱。

我们采用 Calico 的固定 IP 注解 cni.projectcalico.org/ipAddrs 为每个 Redis 节点分配稳定且可控的 Pod IP。
这种方式能够确保 Pod 在重启、漂移、节点重建等场景下仍能获得不变的 IP 地址，从而满足 Redis 集群对节点 IP 强一致性的要求。

在实现上，我们为 Redis 的每个节点分别创建 6 个独立的 Deployment，并为每个 Deployment 绑定一个对应的 PVC。随后，通过为每个 Deployment 添加cni.projectcalico.org/ipAddrs注解并设置唯一的固定 IP ，使其在 Calico 网络下始终分配相同的 IP。

由于需要为 Pod 分配固定 IP，因此 Deployment 必须使用 Recreate 更新策略。在更新过程中必须先删除旧 Pod，才能让 Calico 释放原有 IP 地址，从而确保新 Pod 可以顺利获取并占用指定的固定 IP。否则，若采用滚动更新方式，IP 地址将无法及时释放，导致新的 Pod 无法创建并出现卡住现象。

strategy:
   type: Recreate

1. docker 部署

2. kubernetes 部署

2.1 configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: redis-config
  namespace: default
  annotations:
    kubesphere.io/creator: admin
data:
  redis.conf: |-
    port 6379
    bind 0.0.0.0
    # 允许作为集群节点
    cluster-enabled yes
    # 节点的 cluster 配置文件（会自动创建）
    cluster-config-file nodes.conf
    # 超时时间
    cluster-node-timeout 5000
    # 开启 AOF
    appendonly yes
 
    # 自行变更，不可使用弱密码
    requirepass 123456
    masterauth 123456

2.2 deployment

我们需要创建6个redis-cluster的Deployment与6个redis-cluster的pvc分别与之对应绑定，其中cni.projectcalico.org/ipAddrs: '["172.244.1.11"]' 为固定的容器ip，每个Deployment分别+1 例如 172.244.1.12，172.244.1.13直到172.244.1.16

redis-cluster-1

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-1
  namespace: default
  labels:
    app: redis-cluster-1
  annotations:
    deployment.kubernetes.io/revision: '15'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-1
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-1
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.11"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-9xhdq0
          persistentVolumeClaim:
            claimName: redis-cluster-1
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-9xhdq0
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

redis-cluster-2

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-2
  namespace: default
  labels:
    app: redis-cluster-2
  annotations:
    deployment.kubernetes.io/revision: '3'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-2
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-2
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.12"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-on3pb0
          persistentVolumeClaim:
            claimName: redis-cluster-2
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-on3pb0
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

redis-cluster-3

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-3
  namespace: default
  labels:
    app: redis-cluster-3
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-3
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-3
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.13"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-9zcket
          persistentVolumeClaim:
            claimName: redis-cluster-3
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-9zcket
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

redis-cluster-4

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-4
  namespace: default
  labels:
    app: redis-cluster-4
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-4
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-4
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.14"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-6vcmgq
          persistentVolumeClaim:
            claimName: redis-cluster-4
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-6vcmgq
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

redis-cluster-5

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-5
  namespace: default
  labels:
    app: redis-cluster-5
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-5
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-5
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.15"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-1z78qc
          persistentVolumeClaim:
            claimName: redis-cluster-5
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-1z78qc
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

redis-cluster-6

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-cluster-6
  namespace: default
  labels:
    app: redis-cluster-6
  annotations:
    deployment.kubernetes.io/revision: '2'
    kubesphere.io/creator: admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-6
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-cluster-6
      annotations:
        cni.projectcalico.org/ipAddrs: '["172.244.1.16"]'
        kubesphere.io/creator: admin
        kubesphere.io/imagepullsecrets: '{}'
        kubesphere.io/restartedAt: '2025-11-28T10:17:31.386Z'
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: redis-config
          configMap:
            name: redis-config
            defaultMode: 420
        - name: volume-px25j8
          persistentVolumeClaim:
            claimName: redis-cluster-6
      containers:
        - name: redis
          image: 'redis:6.2.19'
          command:
            - redis-server
          args:
            - /etc/redis/redis.conf
          ports:
            - name: http-0
              containerPort: 6379
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: redis-config
              mountPath: /etc/redis/redis.conf
              subPath: redis.conf
            - name: volume-px25j8
              mountPath: /data
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      securityContext: {}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - redis-cluster
              topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

2.3 初始化集群

自行填写主机ip与redis密码

redis-cli -a xxxxxxxxxx \
--cluster create \
--cluster-replicas 1 \
172.244.1.11:6379 \
172.244.1.12:6379 \
172.244.1.13:6379  \
172.244.1.14:6379 \
172.244.1.15:6379 \
172.244.1.16:6379

2.4 service

创建6个service，其中我启用了nodeport自行决定自身是否开启。

app-redis1

kind: Service
apiVersion: v1
metadata:
  name: app-redis1
  namespace: default
  labels:
    app: app-redis1
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30031
  selector:
    app: redis-cluster-1
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

app-redis2

kind: Service
apiVersion: v1
metadata:
  name: app-redis2
  namespace: default
  labels:
    app: app-redis2
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30032
  selector:
    app: redis-cluster-2
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

app-redis3

kind: Service
apiVersion: v1
metadata:
  name: app-redis3
  namespace: default
  labels:
    app: app-redis3
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30033
  selector:
    app: redis-cluster-3
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

app-redis4

kind: Service
apiVersion: v1
metadata:
  name: app-redis4
  namespace: default
  labels:
    app: app-redis4
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30034
  selector:
    app: redis-cluster-4
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

app-redis5

kind: Service
apiVersion: v1
metadata:
  name: app-redis5
  namespace: default
  labels:
    app: app-redis5
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30035
  selector:
    app: redis-cluster-5
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

app-redis6

kind: Service
apiVersion: v1
metadata:
  name: app-redis6
  namespace: default
  labels:
    app: app-redis6
  annotations:
    kubesphere.io/creator: admin
spec:
  ports:
    - name: http-6379
      protocol: TCP
      port: 6379
      targetPort: 6379
      nodePort: 30036
  selector:
    app: redis-cluster-6
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  internalTrafficPolicy: Cluster

写在最后

如果觉得文章对你有用，请随意赞赏

redis

使用 K8S 或 Docker 快速部署 redis6, calico固定IP 集群

https://tanqidi.com/archives/18da8fed-bacd-44ee-91df-68014573c28b-7519e56b-e537020d-be96aad3

作者

世界哪有真情在

发布于

2025-11-28

更新于