prometheus alertmanager完整过程

下载：https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz

cat > /usr/lib/systemd/system/prometheus.service <<EOF
[Unit]
Description=prometheus
After=network-online.target
[Service]
User=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

或者启动方式：  nohup /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
               
systemctl daemon-reload
systemctl start prometheus
systemctl status prometheus
systemctl enable prometheus
访问：http://192.168.100.101:9090/


cat > /usr/lib/systemd/system/alertmanager.service <<EOF
[Unit]
Description=Alertmanager
After=network-online.target
[Service]
Type=simple
User=prometheus
Restart=on-failure
ExecStart=/usr/local/alertmanager/alertmanager --config.file=/usr/local/alertmanager/alertmanager.yml --storage.path=/opt/alertmanager/data
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start alertmanager &&
systemctl status alertmanager &&
systemctl enable alertmanager

# prometheus关联alertmanager
vi /etc/prometheus/prometheus.yaml
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 127.0.0.1:9093



cat > /usr/lib/systemd/system/blackbox_exporter.service<<EOF
[Unit]
Description=blackbox_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/opt/blackbox_exporter/blackbox_exporter --config.file=/opt/blackbox_exporter/blackbox.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload &&
systemctl start blackbox_exporter &&
systemctl status blackbox_exporter &&
systemctl enable blackbox_exporter

vim /etc/prometheus/prometheus.yml 添加内容：
- job_name: 'k8s'   ### 新增加job
    static_configs:
    - targets: ['192.168.100.201:9100','192.168.100.202:9100'] 



其他组件：
https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
https://github.com/prometheus/node_exporter/releases/download/v1.4.0-rc.0/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz
https://github.com/prometheus/blackbox_exporter/releases/download/v0.22.0/blackbox_exporter-0.22.0.linux-amd64.tar.gz

mysql监控：https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
/usr/local/mysqld_exporter --config.my-cnf="/usr/local/.my.cnf" &
vi .my.cnf
[client]
user=root
password=123456


监控节点：
下载：https://github.com/prometheus/node_exporter/releases/download/v1.4.0-rc.0/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz

[root@localhost opt]# vi /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=node_exporter
After=network-online.target
 
[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.targe


4.启动
nohup /usr/local/bin/node_exporter &
nohup /usr/local/bin/node_exporter >> /usr/local/bin/node_exporter.log 2>&1 &
或者
systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter

5. ss -ant|grep 9100



官网地址：https://grafana.com
wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.1.6-1.x86_64.rpm
sudo yum install grafana-enterprise-9.1.6-1.x86_64.rpm
systemctl start grafana-server.service  #启动
systemctl stop grafana-server.service  #停止
systemctl restart grafana-server.service  #重启
systemctl status grafana-server.service  #查看状态

http://192.168.100.101:3000/login   admin   admin
导入模板：https://grafana.com/grafana/dashboards/1860-node-exporter-full/



告警规则参考： https://blog.51cto.com/u_12227788/5522616