0
点赞
收藏
分享

微信扫一扫

2021-07-05-redis-cluser集群最小分片数研究

律楷粑粑 2022-09-16 阅读 52
redis3df5虚拟化云计算


目录

​​摘要:​​

​​cluster集群-仅修改创建cluster时的master数量限制:​​

​​方案一:  redis-cluster集群-支持最小1个master的cluster集群-完整的修改代码​​

​​方案二: 不使用redis-cluser集群的faiover模块, 而是使用类似独立哨兵的模式，由独立的第三方服务来完成failover​​


摘要:

为了超过阿里云，成为中国最强的分布式redis云数据库，对redis集群的最小分片数做探讨。

分析缩小集群分片数后会引发的问题。

cluster集群-仅修改创建cluster时的master数量限制:

代码中写死了最小需要3个master节点, 直接修改源码以支持更小的分片

int node_len = cluster_manager.nodes->len; int replicas = config.cluster_manager_command.replicas; int masters_count = CLUSTER_MANAGER_MASTERS_COUNT(node_len, replicas); if (masters_count < 3) { clusterManagerLogErr( "*** ERROR: Invalid configuration for cluster creation.\n" "*** Redis Cluster requires at least 3 master nodes.\n" "*** This is not possible with %d nodes and %d replicas per node.", node_len, replicas); clusterManagerLogErr("\n*** At least %d nodes are required.\n", 3 * (replicas + 1)); return 0; }

cluster集群, 2分片

直接修改源码, 使最小的master节点数量为2.

int node_len = cluster_manager.nodes->len; int replicas = config.cluster_manager_command.replicas; int masters_count = CLUSTER_MANAGER_MASTERS_COUNT(node_len, replicas); if (masters_count < 2) { clusterManagerLogErr( "*** ERROR: Invalid configuration for cluster creation.\n" "*** Redis Cluster requires at least 2 master nodes.\n" "*** This is not possible with %d nodes and %d replicas per node.", node_len, replicas); clusterManagerLogErr("\n*** At least %d nodes are required.\n", 3 * (replicas + 1)); return 0; }

cluster集群创建成功

>>> Performing hash slots allocation on 4 nodes...Master[0] -> Slots 0 - 8191Master[1] -> Slots 8192 - 16383Adding replica 192.168.209.128:7003 to 192.168.209.128:7000Adding replica 192.168.209.128:7002 to 192.168.209.128:7001>>> Trying to optimize slaves allocation for anti-affinity[WARNING] Some slaves are in the same host as their masterM: 61b6f4d7d863b89033e3d1b2b58c791d45508da9 192.168.209.128:7000 slots:[0-8191] (8192 slots) masterM: 064ed2aadaab08337948dc5741044e3b6bdfae7e 192.168.209.128:7001 slots:[8192-16383] (8192 slots) masterS: 1e8c9092b4f5554c5f85d0c3e34693891eaf2287 192.168.209.128:7002 replicates 064ed2aadaab08337948dc5741044e3b6bdfae7eS: dd8423705b7c3a71f34b78ec0c4c625c5cf7876e 192.168.209.128:7003 replicates 61b6f4d7d863b89033e3d1b2b58c791d45508da9>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join>>> Performing Cluster Check (using node 192.168.209.128:7000)M: 61b6f4d7d863b89033e3d1b2b58c791d45508da9 192.168.209.128:7000 slots:[0-8191] (8192 slots) master 1 additional replica(s)M: 064ed2aadaab08337948dc5741044e3b6bdfae7e 192.168.209.128:7001 slots:[8192-16383] (8192 slots) master 1 additional replica(s)S: 1e8c9092b4f5554c5f85d0c3e34693891eaf2287 192.168.209.128:7002 slots: (0 slots) slave replicates 064ed2aadaab08337948dc5741044e3b6bdfae7eS: dd8423705b7c3a71f34b78ec0c4c625c5cf7876e 192.168.209.128:7003 slots: (0 slots) slave replicates 61b6f4d7d863b89033e3d1b2b58c791d45508da9[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes1e8c9092b4f5554c5f85d0c3e34693891eaf2287 192.168.209.128:7002@17002 slave 064ed2aadaab08337948dc5741044e3b6bdfae7e 0 1625474320603 2 connected064ed2aadaab08337948dc5741044e3b6bdfae7e 192.168.209.128:7001@17001 myself,master - 0 1625474320000 2 connected 8192-16383dd8423705b7c3a71f34b78ec0c4c625c5cf7876e 192.168.209.128:7003@17003 slave 61b6f4d7d863b89033e3d1b2b58c791d45508da9 0 1625474321616 4 connected61b6f4d7d863b89033e3d1b2b58c791d45508da9 192.168.209.128:7000@17000 master - 1625474312956 1625474312036 1 disconnected 0-8191

但是failover时候会出现投票的master节点没有过半,导致投票无法进行

1e8c9092b4f5554c5f85d0c3e34693891eaf2287 192.168.209.128:7002@17002 slave 064ed2aadaab08337948dc5741044e3b6bdfae7e 0 1625474654684 2 connected064ed2aadaab08337948dc5741044e3b6bdfae7e 192.168.209.128:7001@17001 myself,master - 0 1625474653000 2 connected 8192-16383dd8423705b7c3a71f34b78ec0c4c625c5cf7876e 192.168.209.128:7003@17003 slave 61b6f4d7d863b89033e3d1b2b58c791d45508da9 0 1625474654576 4 connected61b6f4d7d863b89033e3d1b2b58c791d45508da9 192.168.209.128:7000@17000 master,fail? - 1625474312956 1625474312036 1 disconnected 0-8191

cluster集群, 1分片

可以组件集群

>>> Performing hash slots allocation on 2 nodes...Master[0] -> Slots 0 - 16383Adding replica 192.168.209.128:7001 to 192.168.209.128:7000>>> Trying to optimize slaves allocation for anti-affinity[WARNING] Some slaves are in the same host as their masterM: 744e19b475557d1f17ca2fa65f68845a4fd55da9 192.168.209.128:7000 slots:[0-16383] (16384 slots) masterS: f0b6975f53c285174f71bf0ccfa72b8f48ceaba3 192.168.209.128:7001 replicates 744e19b475557d1f17ca2fa65f68845a4fd55da9>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join>>> Performing Cluster Check (using node 192.168.209.128:7000)M: 744e19b475557d1f17ca2fa65f68845a4fd55da9 192.168.209.128:7000 slots:[0-16383] (16384 slots) master 1 additional replica(s)S: f0b6975f53c285174f71bf0ccfa72b8f48ceaba3 192.168.209.128:7001 slots: (0 slots) slave replicates 744e19b475557d1f17ca2fa65f68845a4fd55da9[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes744e19b475557d1f17ca2fa65f68845a4fd55da9 192.168.209.128:7000@17000 master - 1625474846789 1625474846585 1 disconnected 0-16383f0b6975f53c285174f71bf0ccfa72b8f48ceaba3 192.168.209.128:7001@17001 myself,slave 744e19b475557d1f17ca2fa65f68845a4fd55da9 0 0 2 connected

但是无法完成failover

root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes744e19b475557d1f17ca2fa65f68845a4fd55da9 192.168.209.128:7000@17000 master,fail? - 1625474846789 1625474846585 1 disconnected 0-16383f0b6975f53c285174f71bf0ccfa72b8f48ceaba3 192.168.209.128:7001@17001 myself,slave 744e19b475557d1f17ca2fa65f68845a4fd55da9 0 0 2 connected

仅修改创建cluster集群时候限制总结:

通过修改创建cluster集群源码使可以减少分片，在不接管faiover模块的前提下，是无法正常提供功能的。

方案一:  redis-cluster集群-支持最小1个master的cluster集群-完整的修改代码

通过以上分析. 如果支持最小1个master的cluster集群，需要做以下修改:

  1. 创建cluster集群时候的master数量限制
  2. 判断节点fail时, 满足投票数量的限制
  1. 2个master节点的cluster集群, 则只能为1个
  1. 必须至少有1个master节点正常,集群才认为正常。
  2. 如果2个master节点都宕机，那么集群内所有的master都宕机，无法提供服务
  1. 1个master节点的cluster集群, 则只能为0个
  1. 一旦被判定为pfail, 则立即设置为fail
  2. 如果一主一从的master和slave都宕机，则集群都宕机，无法提供服务
  1. 从节点failover时, 满足投票数量限制
  1. 2个master节点的cluster集群, 则只能为1个
  1. 必须至少有1个master节点正常,集群才认为正常。
  2. 如果2个master节点都宕机，那么集群内所有的master都宕机，无法提供服务
  1. 1个master节点的cluster集群, 则只能为0个
  1. 一旦被判定为pfail, 则立即设置为fail
  2. 如果一主一从的master和slave都宕机，则集群都宕机，无法提供服务

代码调整:

clusterManagerCommandCreate 修改创建cluster的最小的master限制，将最小的3个master修改为1个

int node_len = cluster_manager.nodes->len; int replicas = config.cluster_manager_command.replicas; int masters_count = CLUSTER_MANAGER_MASTERS_COUNT(node_len, replicas); if (masters_count < 3) { clusterManagerLogErr( "*** ERROR: Invalid configuration for cluster creation.\n" "*** Redis Cluster requires at least 3 master nodes.\n" "*** This is not possible with %d nodes and %d replicas per node.", node_len, replicas); clusterManagerLogErr("\n*** At least %d nodes are required.\n", 3 * (replicas + 1)); return 0; }

2021-07-05-redis-cluser集群最小分片数研究_redis

clusterHandleSlaveFailover 从节点判断是否达到failover的投票数量以将自己提升为master

if (server.cluster->size < 3) { needed_quorum = server.cluster->size==2 ? 1 : 0; }

2021-07-05-redis-cluser集群最小分片数研究_f5_02

clusterCron 如果判断节点超时, 当master节点数量< 3时, 直接置为FAIL，而不是possiable fail

if (delay > server.cluster_node_timeout) {
            /* Timeout reached. Set the node as possibly failing if it is
             * not already in this state. */
            if (!(node->flags & (CLUSTER_NODE_PFAIL|CLUSTER_NODE_FAIL))) {
                serverLog(LL_DEBUG,"*** NODE %.40s possibly failing",
                    node->name);

                if (server.cluster->size < 3) {
                    node->flags |= CLUSTER_NODE_FAIL;
                } else {
                    node->flags |= CLUSTER_NODE_PFAIL;
                }

                update_state = 1;
            }
        }

​​​​​​

2021-07-05-redis-cluser集群最小分片数研究_f5_03

测试结果:

root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes936f19d1be23c11f3072c75a558a6cab674250d4 192.168.209.128:7000@17000 master - 0 1625477918714 1 connected 0-8191b67dbe1d90d7415b41d62052d1f439cb72ba62d4 192.168.209.128:7003@17003 slave 936f19d1be23c11f3072c75a558a6cab674250d4 0 1625477919237 4 connectedbbe94f4bfc4471afa39dfd392245a8483d5c6f1e 192.168.209.128:7002@17002 slave 62b189326aaebc2b8556febb9dbca4f90875def7 0 1625477918000 3 connected62b189326aaebc2b8556febb9dbca4f90875def7 192.168.209.128:7001@17001 myself,master - 0 1625477918000 2 connected 8192-16383root@bogon:~/work/redis/cluster# root@bogon:~/work/redis/cluster# root@bogon:~/work/redis/cluster# ps -ef |> ^Croot@bogon:~/work/redis/cluster# ps -ef | grep redis | grep 7000root 110971 1 0 05:38 pts/0 00:00:00 redis-server *:7000 [cluster]root@bogon:~/work/redis/cluster# kill 110971root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes936f19d1be23c11f3072c75a558a6cab674250d4 192.168.209.128:7000@17000 master - 1625477937263 1625477937058 1 disconnected 0-8191b67dbe1d90d7415b41d62052d1f439cb72ba62d4 192.168.209.128:7003@17003 slave 936f19d1be23c11f3072c75a558a6cab674250d4 0 1625477939090 4 connectedbbe94f4bfc4471afa39dfd392245a8483d5c6f1e 192.168.209.128:7002@17002 slave 62b189326aaebc2b8556febb9dbca4f90875def7 0 1625477938000 3 connected62b189326aaebc2b8556febb9dbca4f90875def7 192.168.209.128:7001@17001 myself,master - 0 1625477937000 2 connected 8192-16383root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes936f19d1be23c11f3072c75a558a6cab674250d4 192.168.209.128:7000@17000 master,fail - 1625477937263 1625477937058 1 disconnectedb67dbe1d90d7415b41d62052d1f439cb72ba62d4 192.168.209.128:7003@17003 master - 0 1625477941627 5 connected 0-8191bbe94f4bfc4471afa39dfd392245a8483d5c6f1e 192.168.209.128:7002@17002 slave 62b189326aaebc2b8556febb9dbca4f90875def7 0 1625477941119 3 connected62b189326aaebc2b8556febb9dbca4f90875def7 192.168.209.128:7001@17001 myself,master - 0 1625477940000 2 connected 8192-16383

root@bogon:~/work/redis/cluster# bash create.sh >>> Performing hash slots allocation on 2 nodes...Master[0] -> Slots 0 - 16383Adding replica 192.168.209.128:7001 to 192.168.209.128:7000>>> Trying to optimize slaves allocation for anti-affinity[WARNING] Some slaves are in the same host as their masterM: 55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000 slots:[0-16383] (16384 slots) masterS: 494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001 replicates 55c6c900e6b8cf18b017449284ce6f27f74f5883>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join>>> Performing Cluster Check (using node 192.168.209.128:7000)M: 55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000 slots:[0-16383] (16384 slots) master 1 additional replica(s)S: 494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001 slots: (0 slots) slave replicates 55c6c900e6b8cf18b017449284ce6f27f74f5883[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.0root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000@17000 master - 0 1625478602937 1 connected 0-16383494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001@17001 myself,slave 55c6c900e6b8cf18b017449284ce6f27f74f5883 0 0 2 connectedroot@bogon:~/work/redis/cluster# root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster failoverOKroot@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodes55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000@17000 slave 494cec9731c8c09d9821a4bc26b2498c0ff6a286 0 1625478611234 3 connected494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001@17001 myself,master - 0 0 3 connected 0-16383root@bogon:~/work/redis/cluster# ps -ef | grep redis | grep 7001root 111274 1 0 05:49 pts/0 00:00:00 redis-server *:7001 [cluster]root@bogon:~/work/redis/cluster# kill 111274root@bogon:~/work/redis/cluster# redis-cli -p 7001 cluster nodesCould not connect to Redis at 127.0.0.1:7001: Connection refusedroot@bogon:~/work/redis/cluster# redis-cli -p 7000 cluster nodes55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000@17000 myself,master - 0 0 4 connected 0-16383494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001@17001 master,fail - 1625478634680 1625478633659 3 disconnectedroot@bogon:~/work/redis/cluster# redis-cli -p 7000 cluster nodes55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000@17000 myself,master - 0 0 4 connected 0-16383494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001@17001 master,fail - 1625478634680 1625478633659 3 disconnectedroot@bogon:~/work/redis/cluster# cd 7001root@bogon:~/work/redis/cluster/7001# redis-server ./redis.conf &[1] 111347root@bogon:~/work/redis/cluster/7001# cd ..root@bogon:~/work/redis/cluster# redis-cli -p 7000 cluster nodes55c6c900e6b8cf18b017449284ce6f27f74f5883 192.168.209.128:7000@17000 myself,master - 0 0 4 connected 0-16383494cec9731c8c09d9821a4bc26b2498c0ff6a286 192.168.209.128:7001@17001 slave 55c6c900e6b8cf18b017449284ce6f27f74f5883 0 1625478663466 4 connected

经过以上修改后可以正常failover

方案二: 不使用redis-cluser集群的faiover模块, 而是使用类似独立哨兵的模式，由独立的第三方服务来完成failover

待考虑的方案是使用zookeeper， 并且使用zkfc的思想, 但是部署的可用性需要进一步讨论

举报

相关推荐

redis分片集群

qt

Redis分片集群

Redisredis集群Redis分片集群Redis高可用Java后端开发

【Redis】搭建分片集群

redis数据库缓存javaruby代码人生

redis之分片集群

redis数据库客户端数据JavaScript前端开发

【Redis】分片集群（四）

redis数据库java插槽数据代码人生

Redis搭建分片集群

c++开发语言

Redis集群系列六 —— 分片集群搭建

redis分片集群Redis配置文件Linux系统/运维

Redis集群搭建（主从、哨兵、分片）

redis数据库缓存

Redis分片集群搭建及其原理

xrvr交互

2021-05-07 奇迹晨起日记

思考生活，活在当下
0 条评论