如下图所示，使用Keepalived实现简单的GrayLog高可用
(图片点击放大查看)
Node1:192.168.31.64
Node2:192.168.31.66
虚拟VIP——192.168.31.65
具体步骤如下
1、yum安装keepalived
yum -y install keepalived
(图片点击放大查看)
2、配置keepalived.conf
cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-default
echo -n | tee /etc/keepalived/keepalived.conf
vim /etc/keepalived/keepalived.conf
global_defs {
# Keepalived process identifier
router_id graylog
}
# Script to check whether graylog is running or not
vrrp_script check_graylog {
script "/etc/keepalived/check_graylog.sh"
interval 3
weight 50
fall 2
}
# Virtual interface - The priority specifies the order in which the assigned interface to take over in a failover
vrrp_instance VI_Graylog {
state MASTER
interface ens33
virtual_router_id 80
priority 110
# The virtual ip address shared between the two GrayLog Web Server which will float
virtual_ipaddress {
192.168.31.65/24 dev ens33
}
track_script {
check_graylog
}
authentication {
auth_type PASS
auth_pass Keepalived2023
}
notify_master "/etc/keepalived/master.sh"
notify_backup "/etc/keepalived/backup.sh"
notify_fault "/etc/keepalived/fault.sh"
}
(图片点击放大查看)
(图片点击放大查看)
备节点修改如下位置
scp /etc/keepalived/keepalived.conf root@192.168.31.66:/etc/keepalived/keepalived.confvim /etc/keepalived/keepalived.conf
state BACKUP
priority 80
(图片点击放大查看)
3、编写notify脚本
vim /etc/keepalived/master.sh
添加如下行
#!/bin/bash
LOGFILE=/var/log/keepalived-state.log
date >> $LOGFILE
echo "[MASTER]" >> $LOGFILE
webhook_url="https://oapi.dingtalk.com/robot/send?access_token=4f851XXXXXXXXcaf25e6eabad63f88aXX"
NodeIP=`(hostname -I)`
set_payload_file(){
cat > /opt/payload_result.json << \EOF
{
"msgtype": "actionCard",
"actionCard": {
"title":"当前节点已切换成Master节点",
"text":"
##### 当前节点已切换成Master节点 \n
> ##### <font color=#67C23A> 【Master节点IP】</font> :<font color=#FF0000> templateIP </font> \n
"
}
}
EOF
}
set_payload_file
sed -i "s^templateIP^$NodeIP^g" /opt/payload_result.json
respnotallow=$(curl -sS -H "Content-Type: application/json" -X POST -d @/opt/payload_result.json "${webhook_url}")
if [ $? -eq 0 ]; then
echo "Alert sent successfully"
else
echo "Failed to send alert: ${response}"
fi
(图片点击放大查看)
vim /etc/keepalived/backup.sh
添加如下行
#!/bin/bash
LOGFILE=/var/log/keepalived-state.log
date >> $LOGFILE
echo "[BACKUP]" >> $LOGFILE
vim /etc/keepalived/fault.sh
添加如下行
#!/bin/bash
LOGFILE=/var/log/keepalived-state.log
date >> $LOGFILE
echo "[FAULT]" >> $LOGFILE
(图片点击放大查看)
4、编写track_script
vim /etc/keepalived/check_graylog.sh
添加如下行
#!/bin/bash
Graylog_PID=`ps -ef | grep '/bin/graylog-server' | grep -v grep | awk '{print $2}'`
if [ -z $Graylog_PID ]; then
exit 1
else
exit 0
fi
(图片点击放大查看)
chmod a+x /etc/keepalived/*.sh
scp /etc/keepalived/*.sh root@192.168.31.66:/etc/keepalived/
(图片点击放大查看)
5、启动keepalived
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
systemctl enable keepalived
systemctl start keepalived
systemctl status keepalived
(图片点击放大查看)
(图片点击放大查看)
这时Node1为主节点
6、Keepalived高可用测试验证
手动关闭主节点的graylog-server服务
这时虚拟VIP会漂移到备节点
(图片点击放大查看)
再启动主节点的graylog-server服务
这时Node1为切回Master 这时虚拟VIP会漂移到主节点
(图片点击放大查看)
本文参考如下链接完成
https://linuxhandbook.com/load-balancing-setup/
https://www.linuxtechi.com/setup-highly-available-nginx-keepalived-linux/
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/keepalived_install_example1#doc-wrapper
https://www.redhat.com/sysadmin/advanced-keepalived
https://www.howtoforge.com/how-to-setup-highly-available-nginx-with-keepalived-on-centos-8/
https://www.atlantic.net/dedicated-server-hosting/how-to-vrrp-keepalived-configuration/
https://tobru.ch/keepalived-check-and-notify-scripts/
https://docs.oracle.com/en/operating-systems/oracle-linux/6/admin/section_hxz_zdw_pr.html一、Keepalived优化
1、Keepalived设置成不抢占模式
具体地方：keepalived.conf配置文件中
1）、vrrp_script check_graylog中注释掉#weight 50
2）、vrrp_instance VI_Graylog 添加 nopreempt true
(图片点击放大查看)
2、添加服务重启进程PID变化时检测机制
#!/bin/bash
# 检测的服务名称
SERVICE_NAME="graylog"
# 获取当前服务的 PID
CURRENT_PID=`ps -ef | grep '/bin/graylog-server' | grep -v grep | awk '{print $2}'`
# 如果获取到的 PID 为空，则服务可能已经停止，返回 1
if [ -z "${CURRENT_PID}" ]; then
exit 1
fi
# 如果上一次记录的 PID 和当前 PID 不同，则返回 1
if [ "${CURRENT_PID}" != "$(cat /var/run/${SERVICE_NAME}.pid)" ]; then
echo "${CURRENT_PID}" > /var/run/${SERVICE_NAME}.pid
exit 1
fi
# 记录当前 PID 到文件中
echo "${CURRENT_PID}" > /var/run/${SERVICE_NAME}.pid
# 如果上述条件都不满足，则返回 0，表示服务正常
exit 0
(图片点击放大查看)
3、状态变化时告警通知显示当前时间
master.sh backup.sh fault.sh 三个脚本均做了优化
(图片点击放大查看)
主备切换演示效果如下
1、节点1重启graylog-server服务，节点1先是Fault状态，这时节点2成为Master，待节点1重启完成后变为Backup状态
(图片点击放大查看)
(图片点击放大查看)
2、节点2停止graylog-server服务，这时节点2变为Fault状态，节点1成为Master
(图片点击放大查看)
(图片点击放大查看)
二、进程监测脚本
vim /opt/check_graylogserver_process.sh
#!/bin/bash
# 进程名称
PROCESS_NAME="graylog"
# 进程状态通过读写文件来更新计数
PROCESS_STATUS_FILE="/tmp/process_status.txt"
# 告警计数通过读写文件来更新计数
ALERT_COUNT_FILE="/tmp/alert_count.txt"
# 告警次数阈值
ALERT_THRESHOLD=3
# 钉钉机器人Webhook地址
WEBHOOK_URL_graylog="https://oapi.dingtalk.com/robot/send?access_token=87ab6ee5b3045a8e896222c7b8889d14dcb6ebf94ecacd759f1b737a7cef5408"
# 发送告警
send_alert() {
current_time=$(date "+%Y-%m-%d %H:%M:%S")
message="{\"msgtype\":\"text\",\"text\":{\"content\":\"Node1节点(IP:192.168.31.64)的 $PROCESS_NAME 进程未运行已超过【$ALERT_COUNT】分钟，请立即处理！\n当前时间：$current_time\"}}"
curl -s -H "Content-Type: application/json" -d "$message" "$WEBHOOK_URL_graylog" > /dev/null
}
send_alert_tips() {
current_time=$(date "+%Y-%m-%d %H:%M:%S")
message="{\"msgtype\":\"text\",\"text\":{\"content\":\"Node1节点(IP:192.168.242.64)的 $PROCESS_NAME 进程未运行告警已发送三次，后续不再发送提醒，请及时关注并处理！\n当前时间：$current_time\"}}"
curl -s -H "Content-Type: application/json" -d "$message" "$WEBHOOK_URL_graylog" > /dev/null
}
# 发送恢复告警
send_recovery_alert() {
current_time=$(date "+%Y-%m-%d %H:%M:%S")
message="{\"msgtype\":\"text\",\"text\":{\"content\":\"Node1节点(IP:192.168.242.64)的 $PROCESS_NAME 进程已恢复正常运行！\n恢复时间：$current_time\"}}"
curl -s -H "Content-Type: application/json" -d "$message" "$WEBHOOK_URL_graylog" > /dev/null
}
# 检测进程是否运行
check_process() {
# 使用ps命令结合grep命令检测进程是否存在
process_status=$(ps -ef | grep -v grep | grep "$PROCESS_NAME" | grep "server.conf")
# 如果进程不存在
if [ -z "$process_status" ]; then
echo "--------------------------------"
echo 当前时间:`date "+%Y-%m-%d %H:%M:%S"`
echo "进程 $PROCESS_NAME 未运行"
# 增加告警次数
ALERT_COUNT=$(cat $ALERT_COUNT_FILE)
ALERT_COUNT=$((ALERT_COUNT+1))
echo $ALERT_COUNT > $ALERT_COUNT_FILE
if [ $ALERT_COUNT -le $ALERT_THRESHOLD ]; then
# 发送告警
echo "--------------------------------"
echo 当前时间:`date "+%Y-%m-%d %H:%M:%S"`
echo AlertCount:$ALERT_COUNT
echo "告警次数+1,发送(进程不运行)告警"
send_alert
echo "--------------------------------"
fi
# 如果告警次数超过3次，则不再发送告警
if [ $ALERT_COUNT -eq $ALERT_THRESHOLD ]; then
echo "--------------------------------"
echo 当前时间:`date "+%Y-%m-%d %H:%M:%S"`
echo AlertCount:$ALERT_COUNT
echo "已达到告警次数上限次数3次，后续将不再发送告警"
send_alert_tips
echo "--------------------------------"
fi
if [ $ALERT_COUNT -gt $ALERT_THRESHOLD ]; then
echo "--------------------------------"
echo 当前时间:`date "+%Y-%m-%d %H:%M:%S"`
echo AlertCount:$ALERT_COUNT
echo "已超过告警次数上限次数3次，后续将不再发送告警"
echo "--------------------------------"
exit 0
fi
# 将进程状态标志设置为未运行
echo 0 > $PROCESS_STATUS_FILE
else
# 进程存在
echo "进程 $PROCESS_NAME 在运行"
# 获取告警故障计数
ALERT_COUNT=$(cat $ALERT_COUNT_FILE)
# 如果之前有告警且进程状态为未运行，发送恢复告警
PROCESS_STATUS=$(cat $PROCESS_STATUS_FILE)
if [ $ALERT_COUNT -gt 0 ] && [ $PROCESS_STATUS -eq 0 ]; then
echo "--------------------------------"
echo 当前时间:`date "+%Y-%m-%d %H:%M:%S"`
echo "进程已恢复运行，发送恢复告警"
send_recovery_alert
echo "--------------------------------"
fi
# 进程存在，重置告警次数
ALERT_COUNT=0
echo $ALERT_COUNT > /tmp/alert_count.txt
# 将进程状态标志设置为运行中
echo 1 > $PROCESS_STATUS_FILE
fi
}
# 执行检测进程的函数
check_processcrontab -e 添加如下行
* * * * * /opt/check_graylogserver_process.sh >> /var/log/check_graylogserver_process.log
touch /tmp/process_status.txt
touch /tmp/alert_count.txt
(图片点击放大查看)
效果如下
1、当graylog-server进程死掉后就超过1，2，3分钟没有起来，前三分钟就每次一条告警 ，超过3分钟发送后续不再发送提醒的1条告警，后续不再发出告警
(图片点击放大查看)
2、当graylog-server进程恢复后提示恢复，就1条恢复告警
(图片点击放大查看)
当然这个脚本可以单独使用
