前言
本文借鉴和参考:https://blog.51cto.com/u_15445837/6081230,如有侵权,请与我联系,感谢~
架构图
说明:
如上图,VIP地址是10.0.x.126
,如果主机10.0.x.125
宕机,会自动切换到备机10.0.x.23
,由于是双主,数据可以进行同步恢复。
部署流程
部署大致流程如下:
- 在
10.0.x.125
服务器上部署docker、master1 - 在
10.0.x.23
服务器上部署docker、master2 - 在两台服务器上安装
keeplived
形成高可用 10.0.x.126
是vip
配置第一台Master
准备Master配置文件my.cnf
## 创建相关目录和文件
mkdir -p /data/mysql
mkdir -p /data/mysql/master1
mkdir -p /data/mysql/master1/config
touch /data/mysql/master1/config/my.cnf
chmod +x /data/mysql/master1/config/my.cnf
cat > /data/mysql/master1/config/my.cnf <<EOF
[mysqld]
server_id=110
user=mysql
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
default-authentication-plugin=mysql_native_password
max_connections=1000
max_connect_errors=1000
log_bin=mysql_binlog_master1
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow.log
log_error=/var/lib/mysql/mysql.err
log_replica_updates=ON
enforce_gtid_consistency=ON
gtid_mode=ON
plugin-load="rpl_semi_sync_source=semisync_source.so;rpl_semi_sync_replica=semisync_replica.so"
#rpl_semi_sync_source_enabled=ON
#rpl_semi_sync_replica_enabled=ON
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
[client]
default-character-set=utf8mb4
EOF
运行第一个master
cat > /data/mysql/master1/start-master1.sh <<EOF
#!/bin/bash
docker run -d \
--name mysql_master1 \
--restart always \
-p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=test#123 \
-e TZ="Asia/Shanghai" \
-v /data/mysql/master1/config:/etc/mysql/conf.d \
-v /data/mysql/master1/data:/var/lib/mysql \
mysql:8.0.29 --lower-case-table-names=1 --authentication_policy=mysql_native_password
EOF
chmod +x /data/mysql/master1/start-master1.sh
sh /data/mysql/master1/start-master1.sh
启动半同步插件
此步主要是为了将rpl_semi_sync_source_enabled=ON
和rpl_semi_sync_replica_enabled=ON
的前#
去掉。
验证半同步插件是否安装:
mysql> INSTALL PLUGIN rpl_semi_sync_source SONAME 'semisync_source.so';
mysql> INSTALL PLUGIN rpl_semi_sync_replica SONAME 'semisync_replica.so';
说明:mysql会检测semisync_source.so
和semisync_replica.so
插件的情况,如果没有安装就不能设置rpl_semi_sync_source_enabled
和rpl_semi_sync_replica_enabled
。
cat > /data/mysql/master1/config/my.cnf <<EOF
[mysqld]
server_id=110
user=mysql
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
default-authentication-plugin=mysql_native_password
max_connections=1000
max_connect_errors=1000
log_bin=mysql_binlog_master1
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow.log
log_error=/var/lib/mysql/mysql.err
log_replica_updates=ON
enforce_gtid_consistency=ON
gtid_mode=ON
plugin-load="rpl_semi_sync_source=semisync_source.so;rpl_semi_sync_replica=semisync_replica.so"
rpl_semi_sync_source_enabled=ON
rpl_semi_sync_replica_enabled=ON
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
[client]
default-character-set=utf8mb4
EOF
重启mysql
docker restart mysql_master1
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "show variables like 'rpl_semi_sync_%';"
+---------------------------------------------+------------+
| Variable_name | Value |
+---------------------------------------------+------------+
| rpl_semi_sync_replica_enabled | ON |
| rpl_semi_sync_replica_trace_level | 32 |
| rpl_semi_sync_source_enabled | ON |
| rpl_semi_sync_source_timeout | 10000 |
| rpl_semi_sync_source_trace_level | 32 |
| rpl_semi_sync_source_wait_for_replica_count | 1 |
| rpl_semi_sync_source_wait_no_replica | ON |
| rpl_semi_sync_source_wait_point | AFTER_SYNC |
+---------------------------------------------+------------+
配置第二台Master
准备Master配置文件my.cnf
## 创建相关目录和文件
mkdir -p /data/mysql
mkdir -p /data/mysql/master2
mkdir -p /data/mysql/master2/config
touch /data/mysql/master2/config/my.cnf
chmod +x /data/mysql/master2/config/my.cnf
cat > /data/mysql/master2/config/my.cnf <<EOF
[mysqld]
server_id=111
user=mysql
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
default-authentication-plugin=mysql_native_password
max_connections=1000
max_connect_errors=1000
log_bin=mysql_binlog_master2
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow.log
log_error=/var/lib/mysql/mysql.err
log_replica_updates=ON
enforce_gtid_consistency=ON
gtid_mode=ON
plugin-load="rpl_semi_sync_source=semisync_source.so;rpl_semi_sync_replica=semisync_replica.so"
#rpl_semi_sync_source_enabled=ON
#rpl_semi_sync_replica_enabled=ON
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
[client]
default-character-set=utf8mb4
运行第二个master
cat > /data/mysql/master2/start-master2.sh <<EOF
#!/bin/bash
docker run -d \
--name mysql_master2 \
--restart always \
-p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=test#123 \
-e TZ="Asia/Shanghai" \
-v /data/mysql/master2/config:/etc/mysql/conf.d \
-v /data/mysql/master2/data:/var/lib/mysql \
mysql:8.0.29 --lower-case-table-names=1 --authentication_policy=mysql_native_password
EOF
chmod +x /data/mysql/master2/start-master2.sh
sh /data/mysql/master2/start-master2.sh
启动半同步插件
此步主要是为了将rpl_semi_sync_source_enabled=ON
和rpl_semi_sync_replica_enabled=ON
的前#
去掉。
验证半同步插件是否安装:
mysql> INSTALL PLUGIN rpl_semi_sync_source SONAME 'semisync_source.so';
mysql> INSTALL PLUGIN rpl_semi_sync_replica SONAME 'semisync_replica.so';
说明:mysql会检测semisync_source.so
和semisync_replica.so
插件的情况,如果没有安装就不能设置rpl_semi_sync_source_enabled
和rpl_semi_sync_replica_enabled
。
cat > /data/mysql/master2/config/my.cnf <<EOF
[mysqld]
server_id=111
user=mysql
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
default-authentication-plugin=mysql_native_password
max_connections=1000
max_connect_errors=1000
log_bin=mysql_binlog_master2
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow.log
log_error=/var/lib/mysql/mysql.err
log_replica_updates=ON
enforce_gtid_consistency=ON
gtid_mode=ON
plugin-load="rpl_semi_sync_source=semisync_source.so;rpl_semi_sync_replica=semisync_replica.so"
rpl_semi_sync_source_enabled=ON
rpl_semi_sync_replica_enabled=ON
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=sys
binlog-ignore-db=performance_schema
[client]
default-character-set=utf8mb4
EOF
重启mysql
docker restart mysql_master2
docker exec -it mysql_master2 mysql -uroot -ptest#123 -e "show variables like 'rpl_semi_sync_%';"
+---------------------------------------------+------------+
| Variable_name | Value |
+---------------------------------------------+------------+
| rpl_semi_sync_replica_enabled | ON |
| rpl_semi_sync_replica_trace_level | 32 |
| rpl_semi_sync_source_enabled | ON |
| rpl_semi_sync_source_timeout | 10000 |
| rpl_semi_sync_source_trace_level | 32 |
| rpl_semi_sync_source_wait_for_replica_count | 1 |
| rpl_semi_sync_source_wait_no_replica | ON |
| rpl_semi_sync_source_wait_point | AFTER_SYNC |
+---------------------------------------------+------------+
双主配置
在进行双主配置时,需要进行一些准备工作,如mysql客户端安装
、创建同步账号
等。
mysql客户端安装
具体步骤,可参考Centos7.9仅安装mysql8客户端工具。
两台机器上创建主从同步账号
登录其中一台master上操作即可。
mysql> create user 'rpl_admin'@'%' identified with mysql_native_password by 'rpl_admin123456';
mysql> grant replication slave on *.* to 'rpl_admin'@'%';
mysql> flush privileges;
Master1作为主,Master2作为备份
Master1的IP地址为:10.0.x.125
,Master2的IP地址为:10.0.x.23
。
编写配置文件:
touch /data/mysql/rplShell.sh
chmod +x /data/mysql/rplShell.sh
cat > /data/mysql/rplShell.sh <<EOF
#!/bin/bash
MASTER_RPL_USER=rpl_admin
MASTER_RPL_PASSWORD=rpl_admin123456
# master and slaves admin user
ADMIN_USER=root
ADMIN_PASSWORD=test#123
MASTER_HOST=10.0.x.125
MASTER_PORT=3306
BACKUP_HOST=10.0.x.23
BACKUP_PORT=3306
RESULT=`mysql -h$MASTER_HOST -P$MASTER_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e "SHOW MASTER STATUS;" | grep -v grep |tail -n +2| awk '{print $1,$2}'`
LOG_FILE_NAME=`echo $RESULT | grep -v grep | awk '{print $1}'`
LOG_FILE_POS=`echo $RESULT | grep -v grep | awk '{print $2}'`
SYNC_SQL="change master to master_host='$MASTER_HOST', master_port=$MASTER_PORT, master_user='$MASTER_RPL_USER',master_password='$MASTER_RPL_PASSWORD',master_log_file='$LOG_FILE_NAME',master_log_pos=$LOG_FILE_POS;"
START_SYNC_SQL="start REPLICA;"
STATUS_SQL="show REPLICA status\G;"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$SYNC_SQL"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$START_SYNC_SQL"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$STATUS_SQL"
EOF
执行脚本并检查:
sh /data/mysql/rplShell.sh
*************************** 1. row ***************************
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Master2作为主,Master1作为备份
Master1的IP地址为:10.0.x.125
,Master2的IP地址为:10.0.x.23
。
编写配置文件:
touch /data/mysql/rplShell.sh
chmod +x /data/mysql/rplShell.sh
cat > /data/mysql/rplShell.sh <<EOF
#!/bin/bash
MASTER_RPL_USER=rpl_admin
MASTER_RPL_PASSWORD=rpl_admin123456
# master and slaves admin user
ADMIN_USER=root
ADMIN_PASSWORD=test#123
MASTER_HOST=10.0.x.23
MASTER_PORT=3306
BACKUP_HOST=10.0.x.125
BACKUP_PORT=3306
RESULT=`mysql -h$MASTER_HOST -P$MASTER_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e "SHOW MASTER STATUS;" | grep -v grep |tail -n +2| awk '{print $1,$2}'`
LOG_FILE_NAME=`echo $RESULT | grep -v grep | awk '{print $1}'`
LOG_FILE_POS=`echo $RESULT | grep -v grep | awk '{print $2}'`
SYNC_SQL="change master to master_host='$MASTER_HOST', master_port=$MASTER_PORT, master_user='$MASTER_RPL_USER',master_password='$MASTER_RPL_PASSWORD',master_log_file='$LOG_FILE_NAME',master_log_pos=$LOG_FILE_POS;"
START_SYNC_SQL="start REPLICA;"
STATUS_SQL="show REPLICA status\G;"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$SYNC_SQL"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$START_SYNC_SQL"
mysql -h$BACKUP_HOST -P$BACKUP_PORT -u"$ADMIN_USER" -p"$ADMIN_PASSWORD" -e"$STATUS_SQL"
EOF
执行脚本并检查:
sh /data/mysql/rplShell.sh
*************************** 1. row ***************************
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
安装 keepsalived
在安装keepalived出现问题:需要:libmysqlclient.so.18
主要步骤:
wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-community-libs-compat-5.7.25-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.25-1.el7.x86_64.rpm
## 再次安装
yum install keepalived -y
Master1主机配置
编写keepalived配置文件:
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id HA-M1
script_user root
}
vrrp_script chk_mysql {
script /etc/keepalived/chk_mysql.sh
interval 3
fall 3
rise 2
}
vrrp_instance VI_MYSQL {
state MASTER
interface ens192
virtual_router_id 100
priority 100
advert_int 1
unicast_src_ip 10.0.x.125
unicast_peer {
10.0.x.23
}
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.x.126/24 dev ens192 label ens192:vip
}
track_script {
chk_mysql
}
}
EOF
说明:
-
router_id:标识,两台需不一样
-
state MASTER:表明这台是Master角色
* priority:优先级,MASTER角色比BACKUP高
* virtual_router_id:虚拟路由编号,两台需要一致,两个节点不能一样
* 10.0.x.126/24 dev ens192 label ens192:vip:VIP地址是10.0.x.126
* script /etc/keepalived/chk_mysql.sh:MySQL的检测脚本,定时去检测MySQL进程是否挂掉,会重启,interval是为间隔时间(2s检测一次)
配置Keepalived的MySQL状态监测脚本:
touch /etc/keepalived/chk_mysql.sh
chmod +x /etc/keepalived/chk_mysql.sh
cat > /etc/keepalived/chk_mysql.sh <<EOF
#!/bin/bash
MYSQL=/usr/bin/mysql
MYSQL_HOST=10.0.x.125
MYSQL_USER=root
MYSQL_PASSWORD=test#123
CHECK_TIME=3
#mysql is working MYSQL_OK is 1 , mysql down MYSQL_OK is 0
MYSQL_OK=1
function check_mysql_helth (){
$MYSQL -h $MYSQL_HOST -u$MYSQL_USER -p${MYSQL_PASSWORD} -e "show status;" >/dev/null 2>&1
if [ $? = 0 ] ;then
MYSQL_OK=1
else
MYSQL_OK=0
fi
return $MYSQL_OK
}
while [ $CHECK_TIME -ne 0 ]
do
let "CHECK_TIME -= 1"
check_mysql_helth
if [ $MYSQL_OK = 1 ] ; then
CHECK_TIME=0
exit 0
fi
if [ $MYSQL_OK -eq 0 ] && [ $CHECK_TIME -eq 0 ]
then
systemctl stop keepalived
exit 1
fi
sleep 1
done
EOF
Master2备机配置
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id HA-M2
script_user root
}
vrrp_script chk_mysql {
script /etc/keepalived/chk_mysql.sh
interval 3
fall 3
rise 2
}
vrrp_instance VI_MYSQL {
state BACKUP
interface ens192
virtual_router_id 100
nopreempt
priority 50
advert_int 1
unicast_src_ip 10.0.x.23
unicast_peer {
10.0.x.125
}
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.x.126/24 dev ens192 label ens192:vip
}
track_script {
chk_mysql
}
}
EOF
注意:注意router_id
需要不一样,state BACKUP
表明是备机,virtual_router_id
需要一致,priority
要比MASTER低!
配置Keepalived的MySQL状态监测脚本:
touch /etc/keepalived/chk_mysql.sh
chmod +x /etc/keepalived/chk_mysql.sh
cat > /etc/keepalived/chk_mysql.sh <<EOF
#!/bin/bash
MYSQL=/usr/bin/mysql
MYSQL_HOST=10.0.x.23
MYSQL_USER=root
MYSQL_PASSWORD=test#123
CHECK_TIME=3
#mysql is working MYSQL_OK is 1 , mysql down MYSQL_OK is 0
MYSQL_OK=1
function check_mysql_helth (){
$MYSQL -h $MYSQL_HOST -u$MYSQL_USER -p${MYSQL_PASSWORD} -e "show status;" >/dev/null 2>&1
if [ $? = 0 ] ;then
MYSQL_OK=1
else
MYSQL_OK=0
fi
return $MYSQL_OK
}
while [ $CHECK_TIME -ne 0 ]
do
let "CHECK_TIME -= 1"
check_mysql_helth
if [ $MYSQL_OK = 1 ] ; then
CHECK_TIME=0
exit 0
fi
if [ $MYSQL_OK -eq 0 ] && [ $CHECK_TIME -eq 0 ]
then
systemctl stop keepalived
exit 1
fi
sleep 1
done
EOF
启动Keepalived
systemctl start keepalived
systemctl enable keepalived
双主高可用测试
数据同步测试
进入Master1,创建数据库test_db,表test_user ,并插入一条数据:
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "CREATE DATABASE test_db;"
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "CREATE TABLE test_db.test_user (id INT(11),name VARCHAR(25),age int(11) );"
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "INSERT INTO test_db.test_user values (1,'zhangsan',18);"
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "SELECT * FROM test_db.test_user;"
查看Master1及Master2的数据同步情况:
docker exec -it mysql_master1 mysql -uroot -ptest#123 -e "SELECT * FROM test_db.test_user;"
+------+----------+------+
| id | name | age |
+------+----------+------+
| 1 | zhangsan | 18 |
+------+----------+------+
docker exec -it mysql_master2 mysql -uroot -ptest#123 -e "SELECT * FROM test_db.test_user;"
+------+----------+------+
| id | name | age |
+------+----------+------+
| 1 | zhangsan | 18 |
+------+----------+------+
高可用测试
停止master1
主机上的mysql:
docker stop mysql_master1
注意: Master备机上做写操作,第一次写会发生半同步等待(10s左右),第二次写会退化成异步复制。
master1主机恢复测试:
docker start mysql_master1
systemctl start keepalived
小结
如果杀掉master1
主机上的mysql:
-
VIP会漂移到
master2主机
上,ifconfig能看到VIP信息。 -
master2主机的Keepalived从BACKUP状态转换为了MASTER状态。
-
master2主机上做写操作,第一次写会发生半同步等待(10s左右),第二次写会退化成异步复制。
恢复master1
主机mysql:
-
ifconfig看下VIP会漂移回来,并且备机上的VIP会消失。
-
刚才在master2主机上写入的数据同步过来了。
-
Keepalived的状态是从FAULT转变为BACKUP,最后变成MASTER。
添加 mysql 监控
待监控mysql服务需先创建 mysql 监控专用账号,登录其中一台master主机即可:
mysql> CREATE USER 'monitoring'@'%' IDENTIFIED BY 'test#0131';
Query OK, 0 rows affected (0.09 sec)
mysql> GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO 'monitoring'@'%';
Query OK, 0 rows affected (0.01 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)
登入consulmanager点击“ 云资源管理 ”,“ MySQL管理 ”,“ 自建MySQL管理 ”,点击新增或导入来添加待监控mysql服务信息。
添加完成后,我们可以通过Mysqld Exporter Dashboard
数据平台看到新增的 mysql 资源已经被监控起来了。
参考文档
- 部署MySQL8+keepalived双主热备高可用.md