etcd集群安装与配置

本文最后更新于 2025-01-14，距今已有 44 天，若文章内容或图片链接失效，请留言反馈。

写在最前

1. 前置条件

centos7.9 安装与配置

2. 部署流程

2.1 创建证书

在使用 cfssl gencert 命令生成证书时，-ca=ca.pem 和 -ca-key=ca-key.pem 是用于指定证书签名（CA）相关的证书和私钥。具体来说，它们的作用如下：

-ca=ca.pem：指定一个已经存在的 CA（Certificate Authority）证书，该证书用于签发新的证书。生成的新证书将由这个 CA 证书签名，确保其可信度。
-ca-key=ca-key.pem：指定对应于 ca.pem 的 CA 私钥，用于签名新的证书。在证书签发过程中，CA 会使用它的私钥对生成的证书进行签名。

总结：

这两个参数的作用是引用并使用已有的 CA 证书和私钥来签发新的证书（比如 etcd 的证书）。如果没有指定这两个参数，cfssl 会尝试创建自签名证书，而不是使用现有 CA 签发的证书。

例子：

假设你已有一个根 CA 证书 ca.pem 和它的私钥 ca-key.pem，并希望用它们为 etcd 生成一个证书，那么这些参数就用来引用这个根证书及其私钥进行签发。

mkdir -p /data/k8s-work && cd /data/k8s-work

cat > etcd-csr.json <<"EOF"
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.31.0.12",
    "172.31.0.13",
    "172.31.0.14"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "kubemsb",
    "OU": "CN"
  }]
}
EOF

# 使用cfssl生成证书，其中ca证书需要提前创建留给kubernetes使用
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

[root@k8s-master1 k8s-work]# ll
total 24
-rw-r--r-- 1 root root  356 Jan  2 23:29 ca-config.json
-rw-r--r-- 1 root root 1001 Jan  2 23:28 ca.csr
-rw-r--r-- 1 root root  256 Jan  2 23:28 ca-csr.json
-rw------- 1 root root 1679 Jan  2 23:28 ca-key.pem
-rw-r--r-- 1 root root 1359 Jan  2 23:28 ca.pem
-rw-r--r-- 1 root root  273 Jan  2 23:30 etcd-csr.json

2.2 下载安装

如果链接失效或者下载失败可以用我下载好的 etcd-v3.5.2-linux-amd64.tar.gz，想要更新的版本可以到github仓库下载即可。

https://github.com/etcd-io/etcd/releases

# 下载安装包
wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz

# 解压
tar -xvf etcd-v3.5.2-linux-amd64.tar.gz

# 安装
cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/

# 试验调用
[root@k8s-master1 k8s-work]# ll /usr/local/bin/ | grep etcd
-rwxr-xr-x 1 528287 89939 23588864 Feb  1  2022 etcd
-rwxr-xr-x 1 528287 89939 17993728 Feb  1  2022 etcdctl
-rwxr-xr-x 1 528287 89939 16068608 Feb  1  2022 etcdutl
[root@k8s-master1 k8s-work]# etcd --version
etcd Version: 3.5.2
Git SHA: 99018a77b
Go Version: go1.16.3
Go OS/Arch: linux/amd64

2.3 创建 etcd.conf 文件

其中要留意配置文件中的共性，每个节点上的配置文件对应的名称和IP都需要修改掉它。

2.3.1 master1

# 生成配置文件
cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.0.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.0.12:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.0.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.0.12:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.31.0.12:2380,etcd2=https://172.31.0.13:2380,etcd3=https://172.31.0.14:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

2.3.2 master2

# 生成配置文件
cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.0.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.0.13:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.0.13:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.0.13:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.31.0.12:2380,etcd2=https://172.31.0.13:2380,etcd3=https://172.31.0.14:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

2.3.3 master3

# 生成配置文件
cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.0.14:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.0.14:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.0.14:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.0.14:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.31.0.12:2380,etcd2=https://172.31.0.13:2380,etcd3=https://172.31.0.14:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

2.4 创建 etcd.service 文件

# 生成
cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2.5 文件分发

# 所有master节点都需要创建目录
mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

# 先复制到自己的目录
cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

# ssl 证书配置文件分发到k8s-master2, k8s-master3
for i in k8s-master2 k8s-master3; do scp /etc/etcd/ssl/* $i:/etc/etcd/ssl; done

# etcd.service 服务配置文件分发到k8s-master2, k8s-master3
for i in k8s-master2 k8s-master3; do scp etc/systemd/system/etcd.service $i:/etc/systemd/system; done

# 安装文件分发到其他master节点
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/

2.6 启动集群

如果启动失败需要检查一下 firewalld 和 iptables 是否关闭了。

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

2.7 验证集群状态

# 验证集群状态
[root@k8s-master1 k8s-work]# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.31.0.12:2379,https://172.31.0.13:2379,https://172.31.0.14:2379 endpoint health
+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://172.31.0.12:2379 |   true | 11.182209ms |       |
| https://172.31.0.13:2379 |   true | 12.274699ms |       |
| https://172.31.0.14:2379 |   true |   12.7094ms |       |
+--------------------------+--------+-------------+-------+

# 检查ETCD数据库性能
[root@k8s-master1 k8s-work]# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.31.0.12:2379,https://172.31.0.13:2379,https://172.31.0.14:2379 check perf
 59 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom    !  98.33%PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.091654s
PASS: Stddev is 0.002193s
PASS

# 列表成员
[root@k8s-master1 k8s-work]# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.31.0.12:2379,https://172.31.0.13:2379,https://172.31.0.14:2379 member list
+------------------+---------+-------+--------------------------+--------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+--------------------------+--------------------------+------------+
| 27de189f978d77f0 | started | etcd2 | https://172.31.0.13:2380 | https://172.31.0.13:2379 |      false |
| 7d78b925ef8c87aa | started | etcd1 | https://172.31.0.12:2380 | https://172.31.0.12:2379 |      false |
| dce414af7cc22cf7 | started | etcd3 | https://172.31.0.14:2380 | https://172.31.0.14:2379 |      false |
+------------------+---------+-------+--------------------------+--------------------------+------------+

# 节点集群状态
[root@k8s-master1 k8s-work]# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://172.31.0.12:2379,https://172.31.0.13:2379,https://172.31.0.14:2379 endpoint status
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://172.31.0.12:2379 | 7d78b925ef8c87aa |   3.5.2 |   22 MB |      true |      false |         3 |       8986 |               8986 |        |
| https://172.31.0.13:2379 | 27de189f978d77f0 |   3.5.2 |   22 MB |     false |      false |         3 |       8986 |               8986 |        |
| https://172.31.0.14:2379 | dce414af7cc22cf7 |   3.5.2 |   22 MB |     false |      false |         3 |       8986 |               8986 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

如果觉得文章对你有用，请随意赞赏

etcd

etcd集群安装与配置

https://tanqidi.com/archives/419a89e3-26e0-4371-80a4-c9b958739340

作者

世界哪有真情在

发布于

2025-01-14

更新于

2025-01-14

许可协议

CC BY 4.0

etcd集群安装与配置

写在最前

1. 前置条件

2. 部署流程

2.1 创建证书

2.2 下载安装

2.3 创建 etcd.conf 文件

2.3.1 master1

2.3.2 master2

2.3.3 master3

2.4 创建 etcd.service 文件

2.5 文件分发

2.6 启动集群

2.7 验证集群状态

作者

发布于

更新于

许可协议

评论