文章目录
简介
一.条件及环境说明:
二.需求说明:
- 搭建redis哨兵集群。
- 高可用:有一个k8s节点死掉或重启pod之后也不影响使用。
- 高性能:读写本地磁盘实现高性能的io,共享存储如果资源及硬件性能不够的话,io将会是性能瓶颈。
- 数据安全:在发生切换,节点故障以后,数据要尽可能的保证完整
- 安装简单、管理维护容易
三.实现原理及说明
-
- 搭建redis哨兵集群,一主两从,三个哨兵进程。
-
- redis采用三个独立的statefulset,一个副本进行搭建,哨兵进程则采用一个deployment,三个副本搭建。
-
- redis进程采用节点亲和性及权重实现主备节点。
-
- redis采用 service headless服务,哨兵采用service 服务。
-
- 一组两从并根据权重主备,需要规划好六个节点的标签名称。
注:该方案搭建的redis,只有k8s集群内才能使用,不能通过LoadBlancer或NodePort的形式提供给集群外使用,因为哨兵内选举的主redis的IP都是pod的IP,当发生切换后,无法确定主redis是哪个。
四.详细步骤
4.1.规划节点标签
节点名称 | 角色 | 标签 |
---|---|---|
k8s-worker-120-81 | 主 | storage-selector=node-a-master |
k8s-worker-120-82 | 主 | storage-selector=node-b-master |
k8s-worker-120-83 | 主 | storage-selector=node-c-master |
k8s-worker-120-84 | 备 | storage-selector=node-a-slve |
k8s-worker-120-85 | 备 | storage-selector=node-a-slve |
k8s-worker-120-86 | 备 | storage-selector=node-a-slve |
注:这里的角色划分是指:redis由于是一主两从,每个redis的pod分别落在三个主的节点上,当主节点发生故障是,就切换到备节点,例如:名称为defaultapp-redis-a-0的pod,默认落在标签为storage-selector=node-a-master的节点上,当该节点发生故障是,就会根据权重匹配罗在storage-selector=node-a-slve的节点上。
4.2.创建configmap配置
apiVersion: v1
kind: ConfigMap
metadata:
name: defaultapp-redis-config
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-config
data:
sentinel.conf: |
sentinel monitor mymaster defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379 2
sentinel auth-pass mymaster redis#123
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
SENTINEL resolve-hostnames yes
SENTINEL announce-hostnames yes
redis-master.conf: |
maxmemory 256mb
requirepass redis#123
save 3600 1 300 10 60 100
dir /data/redis
masterauth redis#123
redis.conf: |
maxmemory 256mb
requirepass redis#123
save 3600 1 300 10 60 100
dir /data/redis
replicaof defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379
masterauth redis#123
run.sh: |
#!/bin/sh
role=$1
hname=$(hostname)
if [ $role == "master" ];then
echo "run redis master"
[ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
[ -d /data ] || mkdir /data
ln -s /home/redis/$hname /data/redis
redis-server /etc/redis/redis-master.conf
elif [ $role == "sentinel" ];then
echo "run redis sentinel"
mkdir -p /etc/redis
cp /home/redis/sentinel.conf /etc/redis/
redis-sentinel /etc/redis/sentinel.conf
else
echo "run redis slave"
[ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
[ -d /data ] || mkdir /data
ln -s /home/redis/$hname /data/redis
redis-server /etc/redis/redis.conf
fi
4.3.创建三个statefulset和service headless配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: defaultapp-redis-a
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-a
spec:
serviceName: "defaultapp-redis-a"
replicas: 1
selector:
matchLabels:
app: defaultapp-redis-a
template:
metadata:
labels:
app: defaultapp-redis-a
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-slave
containers:
- name: redis-server
image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
command: ["/bin/sh","/etc/redis/run.sh"]
args: ["master"]
volumeMounts:
- name: redis-config-volume
mountPath: /etc/redis/
- name: redis-data
mountPath: /home/redis
restartPolicy: Always
volumes:
- name: redis-data
hostPath:
path: /data/redis_data
type: ""
- name: redis-config-volume
configMap:
name: defaultapp-redis-config
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: defaultapp-redis-a
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-a
spec:
ports:
- port: 6379
clusterIP: None
selector:
app: defaultapp-redis-a
配置说明
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-slave
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: defaultapp-redis-b
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-b
spec:
serviceName: "defaultapp-redis-b"
replicas: 1
selector:
matchLabels:
app: defaultapp-redis-b
template:
metadata:
labels:
app: defaultapp-redis-b
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-b-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-b-slave
containers:
- name: redis-server
image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
command: ["/bin/sh","/etc/redis/run.sh"]
args: ["slave"]
volumeMounts:
- name: redis-config-volume
mountPath: /etc/redis/
- name: redis-data
mountPath: /home/redis
restartPolicy: Always
volumes:
- name: redis-data
hostPath:
path: /data/redis_data
type: ""
- name: redis-config-volume
configMap:
name: defaultapp-redis-config
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: defaultapp-redis-b
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-b
spec:
ports:
- port: 6379
clusterIP: None
selector:
app: defaultapp-redis-b
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: defaultapp-redis-c
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-c
spec:
serviceName: "defaultapp-redis-c"
replicas: 1
selector:
matchLabels:
app: defaultapp-redis-c
template:
metadata:
labels:
app: defaultapp-redis-c
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-c-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-c-slave
containers:
- name: redis-server
image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
command: ["/bin/sh","/etc/redis/run.sh"]
args: ["slave"]
volumeMounts:
- name: redis-config-volume
mountPath: /etc/redis/
- name: redis-data
mountPath: /home/redis
restartPolicy: Always
volumes:
- name: redis-data
hostPath:
path: /data/redis_data
type: ""
- name: redis-config-volume
configMap:
name: defaultapp-redis-config
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: defaultapp-redis-c
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-c
spec:
ports:
- port: 6379
clusterIP: None
selector:
app: defaultapp-redis-c
4.4.创建哨兵deployment配置和service配置
apiVersion: apps/v1
kind: Deployment
metadata:
name: defaultapp-redis-sentinel
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-sentinel
spec:
replicas: 3
selector:
matchLabels:
app: defaultapp-redis-sentinel
template:
metadata:
labels:
app: defaultapp-redis-sentinel
spec:
containers:
- name: redis-sentinel
image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
ports:
- containerPort: 26379
command: ["/bin/sh","/home/redis/run.sh"]
args: ["sentinel"]
volumeMounts:
- name: redis-config-volume
mountPath: /home/redis
restartPolicy: Always
volumes:
- name: redis-config-volume
configMap:
name: defaultapp-redis-config
---
apiVersion: v1
kind: Service
metadata:
name: defaultapp-redis-sentinel
namespace: default-ns
labels:
appname: default-app
app: defaultapp-redis-sentinel
spec:
ports:
- port: 26379
targetPort: 26379
type: LoadBalancer
selector:
app: defaultapp-redis-sentinel
五.安装说明
- 需要先在k8s的节点创建/data/redis_data的路径来存放数据。
- 使用配置前将配置中的defaultapp替换成自己需要的命名,配置的命名空间为:default-ns,修改成自己的命名空间,appname为default-app,修改成自己的应用名称,该标签也可以删除。
- 配置中的镜像地址采用了私有的镜像地址,镜像是dockerhub上下载的redis:7.2.5-alpine3.20镜像到私有镜像仓库的,如果能直通外网的k8s可以直接用redis:7.2.5-alpine3.20。否则配置一个可以通的镜像地址。
- redis.conf或redis-master.conf,sentinel.conf中的密码redis#123替换成自己的密码。
- 启动时是以第一个statefulset:defaultapp-redis-a 作为主节点的。
- 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。