docker redis 集群搭建及主从容错切换演练
第一步，先拉取镜像docker pull redis
然后直接把容器都跑起来，这里用了redis-01 到redis-08
docker run -d --name redis-01 --net host --privileged=true -v /data/redis/redis_01:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6381
docker run -d --name redis-02 --net host --privileged=true -v /data/redis/redis_02:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6382
docker run -d --name redis-03 --net host --privileged=true -v /data/redis/redis_03:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6383
docker run -d --name redis-04 --net host --privileged=true -v /data/redis/redis_04:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6384
docker run -d --name redis-05 --net host --privileged=true -v /data/redis/redis_05:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6385
docker run -d --name redis-06 --net host --privileged=true -v /data/redis/redis_06:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6386
docker run -d --name redis-07 --net host --privileged=true -v /data/redis/redis_07:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6387
docker run -d --name redis-08 --net host --privileged=true -v /data/redis/redis_08:/data redis:latest --cluster-enabled
yes --appendonly yes --port 6388
docker run 创建并运行容器
–name 指定容器名字
–net 使用宿主机的IP和端口，默认
–privileged 获取宿主机root用户权限
-v 容器卷, 宿主机文件夹: 容器内部文件
redis:latest 镜像和版本号(latest是最新版本号)
–cluster-enabled 开启redis集群
–appendonly开启持久化
–port redis端口
运行docker容器的时候，他会自动把容器用到的宿主机文件夹建起来，如下面：
看一宿主机的IP地址，我们等会要用
[root@WIND scripts]# ifconfig | grep net
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
登录其中一个redis ,用redis-clis命令创建redis集群，命令格式：
redis-cli --cluster create IP:port --cluster-replicas 1
[root@WIND ~]# docker exec -it redis-01 /bin/bash
root@WIND:/data#redis-cli --cluster create 172.17.0.1:6381 172.17.0.1:6382 172.17.0.1:6383 172.17.0.1:6384 172.17.0.1:6385 172.17.0.1:6386 --cluster-replicas 1
这里的IP，就是我们刚才用ifconfig命令查看到的宿主机IP，注意不是容器内部的IP
root@WIND:/data# redis-cli --cluster create 172.17.0.1:6381 172.17.0.1:6382 172.17.0.1:6383 172.17.0.1:6384 172.17.0.1:6
385 172.17.0.1:6386 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.17.0.1:6385 to 172.17.0.1:6381
Adding replica 172.17.0.1:6386 to 172.17.0.1:6382
Adding replica 172.17.0.1:6384 to 172.17.0.1:6383
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381
slots:[0-5460] (5461 slots) master
M: 6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382
slots:[5461-10922] (5462 slots) master
M: f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383
slots:[10923-16383] (5461 slots) master
S: 964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384
replicates f0c8010c67b60bdbecfe8022c6670f2c75be213f
S: b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385
replicates 583c95d6be5823b381ae9a83d785b10a877c1fc4
S: 4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386
replicates 6d096c81307598ed804f61ed1a2b6d04ec226adb
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.17.0.1:6381)
M: 583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386
slots: (0 slots) slave
replicates 6d096c81307598ed804f61ed1a2b6d04ec226adb
S: b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385
slots: (0 slots) slave
replicates 583c95d6be5823b381ae9a83d785b10a877c1fc4
S: 964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384
slots: (0 slots) slave
replicates f0c8010c67b60bdbecfe8022c6670f2c75be213f
M: f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@WIND:/data#
上面展示的是创建过程，可以看到redis自动给我们建好主从了
Master[0] ， Master[1]， Master[2]是三个主节点
每个主节点的哈希槽的槽号也分配好了，分别是 0 - 5460， 5461 - 10922， 10923 - 16383
每个主节点对应的从节点，也给指定好了
主从对应关系是：
M(Master) S(Slave)
6383 6384
6381 6385
6382 6386
可以登录其中的某个节点，查看一下集群信息
root@WIND:/data# redis-cli -p 6381
127.0.0.1:6381> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:187
cluster_stats_messages_pong_sent:191
cluster_stats_messages_sent:378
cluster_stats_messages_ping_received:186
cluster_stats_messages_pong_received:187
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:378
127.0.0.1:6381>
查看节点信息：
127.0.0.1:6381> cluster nodes
6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382@16382 master - 0 1647265116369 2 connected 5461-10922
4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386@16386 slave 6d096c81307598ed804f61ed1a2b6d04ec226adb 0 1647265118
376 2 connected
b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385@16385 slave 583c95d6be5823b381ae9a83d785b10a877c1fc4 0 1647265118
000 1 connected
964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384@16384 slave f0c8010c67b60bdbecfe8022c6670f2c75be213f 0 1647265119
379 3 connected
f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383@16383 master - 0 1647265117000 3 connected 10923-16383
583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381@16381 myself,master - 0 1647265117000 1 connected 0-5460
127.0.0.1:6381>
mysefl，是当前登录节点，可以看到是6831端口对应的节点
583c95d6be5823b381ae9a83d785b10a877c1fc4 这种一长串的节点的id
跟进节点ID可以看到， b1dd918fa65ec11401900d1c471ff723d97b98bf挂载583c95d6be5823b381ae9a83d785b10a877c1fc4下面
前面可以看到98bf对应的端口是6385， 1fc4对应的端口是6381，也就是6381是6385的主节点，两者为主从关系
连接集群尝试一下读写
[root@WIND scripts]# docker exec -it redis-01 /bin/bash
root@WIND:/data# redis-cli -p 6381
127.0.0.1:6381>
127.0.0.1:6381> keys *
(empty array)
127.0.0.1:6381> set k1 v1
(error) MOVED 12706 172.17.0.1:6383
127.0.0.1:6381> set k2 v2
OK
127.0.0.1:6381> set k3 v3
OK
127.0.0.1:6381> set k4 v4
(error) MOVED 8455 172.17.0.1:6382
127.0.0.1:6381>
set k1 v1 报错了，说明数据没存储成功
上面redis-cli -p 6381单机版redis连接命令，用单机版连接方式连接，然后操作集群，肯定会报错啊
防止路由失效，我们连接集群的时候，需要加-c 参数， redis-cli -p 6381 -c 这样连接
[root@WIND scripts]# docker exec -it redis-01 /bin/bash
root@WIND:/data#
root@WIND:/data# redis-cli --cluster check 172.17.0.1:6381
172.17.0.1:6381 (583c95d6...) -> 2 keys | 5461 slots | 1 slaves.
172.17.0.1:6382 (6d096c81...) -> 1 keys | 5462 slots | 1 slaves.
172.17.0.1:6383 (f0c8010c...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.1:6381)
M: 583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386
slots: (0 slots) slave
replicates 6d096c81307598ed804f61ed1a2b6d04ec226adb
S: b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385
slots: (0 slots) slave
replicates 583c95d6be5823b381ae9a83d785b10a877c1fc4
S: 964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384
slots: (0 slots) slave
replicates f0c8010c67b60bdbecfe8022c6670f2c75be213f
M: f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
可以用–cluster check参数，查看已经建好的集群
我们停止其中一个主节点，看看会有什么问题
[root@WIND ~]# docker stop redis-01
redis-01
[root@WIND ~]#
停了其中一个Mater之后，再次登录redis查看节点信息
root@WIND:/data# redis-cli -p 6382 -c
127.0.0.1:6382> cluster nodes
6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382@16382 myself,master - 0 1647299515000 2 connected 5461-10922
4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386@16386 slave 6d096c81307598ed804f61ed1a2b6d04ec226adb 0 1647299516000 2 connected
583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381@16381 master,fail - 1647299430605 1647299423073 1 disconnected
b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385@16385 master - 0 1647299516506 7 connected 0-5460
f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383@16383 master - 0 1647299518516 3 connected 10923-16383
964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384@16384 slave f0c8010c67b60bdbecfe8022c6670f2c75be213f 0 1647299517511 3 connected
可以看到，其中有一行，包含6381 fail的字眼
再往redis里面写数据看看
127.0.0.1:6382> get k1
-> Redirected to slot [12706] located at 172.17.0.1:6383
"v1"
172.17.0.1:6383> get k2
-> Redirected to slot [449] located at 172.17.0.1:6385
"v2"
172.17.0.1:6385> get k3
"v3"
172.17.0.1:6385> get k4
-> Redirected to slot [8455] located at 172.17.0.1:6382
"v4"
写数据正常
马上查看集群信息
root@WIND:/data# redis-cli --cluster check 172.17.0.1:6382
Could not connect to Redis at 172.17.0.1:6381: Connection refused
*** WARNING: 172.17.0.1:6385 claims to be slave of unknown node ID 583c95d6be5823b381ae9a83d785b10a877c1fc4.
172.17.0.1:6382 (6d096c81...) -> 1 keys | 5462 slots | 1 slaves.
172.17.0.1:6383 (f0c8010c...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 2 keys in 2 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.1:6382)
M: 6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386
slots: (0 slots) slave
replicates 6d096c81307598ed804f61ed1a2b6d04ec226adb
S: b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385
slots: (0 slots) slave
replicates 583c95d6be5823b381ae9a83d785b10a877c1fc4
M: f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384
slots: (0 slots) slave
replicates f0c8010c67b60bdbecfe8022c6670f2c75be213f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
root@WIND:/data#
只有2个Maste了（还是3个Slave）
看到有6381 connect refused , 还有WARNING信息：
WARNING: 172.17.0.1:6385 claims to be slave of unknown node ID 583c95d6be5823b381ae9a83d785b10a877c1fc4
过一会再查看
root@WIND:/data# redis-cli --cluster check 172.17.0.1:6382
Could not connect to Redis at 172.17.0.1:6381: Connection refused
172.17.0.1:6382 (6d096c81...) -> 1 keys | 5462 slots | 1 slaves.
172.17.0.1:6385 (b1dd918f...) -> 2 keys | 5461 slots | 0 slaves.
172.17.0.1:6383 (f0c8010c...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.17.0.1:6382)
M: 6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386
slots: (0 slots) slave
replicates 6d096c81307598ed804f61ed1a2b6d04ec226adb
M: b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385
slots:[0-5460] (5461 slots) master
M: f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384
slots: (0 slots) slave
replicates f0c8010c67b60bdbecfe8022c6670f2c75be213f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
过一会再查看，Mater变成3个了，Slave变成2个了
原来是一个Salve顶上去做Mater了
可以看到新增的Master是98bf ， 也就是原来6381对应的那个Slave
[root@WIND ~]# docker start redis-01
redis-01
[root@WIND ~]#
重新启动启动我们的6381
root@WIND:/data# redis-cli -p 6382 -c
127.0.0.1:6382> cluster nodes
6d096c81307598ed804f61ed1a2b6d04ec226adb 172.17.0.1:6382@16382 myself,master - 0 1647299734000 2 connected 5461-10922
4f0d3d324ccffcfc4abe296322448ef223e2fd91 172.17.0.1:6386@16386 slave 6d096c81307598ed804f61ed1a2b6d04ec226adb 0 1647299734535 2 connected
583c95d6be5823b381ae9a83d785b10a877c1fc4 172.17.0.1:6381@16381 slave b1dd918fa65ec11401900d1c471ff723d97b98bf 0 1647299735538 7 connected
b1dd918fa65ec11401900d1c471ff723d97b98bf 172.17.0.1:6385@16385 master - 0 1647299732000 7 connected 0-5460
f0c8010c67b60bdbecfe8022c6670f2c75be213f 172.17.0.1:6383@16383 master - 0 1647299734000 3 connected 10923-16383
964f79107a5c7fb1d03df12d33592ccda5ae07ea 172.17.0.1:6384@16384 slave f0c8010c67b60bdbecfe8022c6670f2c75be213f 0 1647299733531 3 connected
发现6381变成slave了，也就是一个Mater挂掉之后，它原来对应的Slave会顶上去，变成Master
当这个Master再次启动在线的时候，他会变成新的Slave
如果要这个重新变回master，只需要原来的顶上去的Master停掉一会，6381上位，再启动即可
