0
点赞
收藏
分享

微信扫一扫

Prometheus exporter 安装说明

静守幸福 2022-06-17 阅读 57

变更记录

  • 2022-06-17 编写node_exporter
  • 2022-06-15 编写mysqld_exporter

node_exporter

一、binary部署

1.1、安装node_exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar -zxf node_exporter-1.3.1.linux-amd64.tar.gz
mv node_exporter-1.3.1.linux-amd64 /data/monitor/node_exporter

1.2、配置systemd启动node_exporter

cat node_exporter-9400.service

[Unit]
Description=node_exporter service
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
LimitNOFILE=1000000
#LimitCORE=infinity
LimitSTACK=10485760

ExecStart=/data/monitor/node_exporter/scripts/run_node_exporter.sh
Restart=always

RestartSec=15s

[Install]
WantedBy=multi-user.target

启动脚本

cat run_node_exporter.sh

#!/bin/bash
set -e

# WARNING: This file was auto-generated. Do not edit!
# All your edit might be overwritten!
DEPLOY_DIR=/data/monitor/node_exporter
cd "${DEPLOY_DIR}" || exit 1

exec > >(tee -i -a "/data/monitor/node_exporter/log/node_exporter.log")
exec 2>&1
exec bin/node_exporter \
--web.listen-address=":9400" \
--collector.tcpstat \
--collector.systemd \
--collector.mountstats \
--collector.meminfo_numa \
--collector.interrupts \
--collector.vmstat.fields="^.*" \
--log.level="info"

二、k8s部署

2.1 编写node_exporter Daemonset yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter-9400
namespace: kube-system
spec:
selector:
matchLabels:
app: node-exporter-9400
template:
metadata:
labels:
app: node-exporter-9400
spec:
containers:
- args:
- --web.listen-address=0.0.0.0:9400
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
image: harbor.foxchan.com/prom/node-exporter:v1.3.1
imagePullPolicy: IfNotPresent
name: node-exporter-9400
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host/proc
name: proc
- mountPath: /host/sys
name: sys
- mountPath: /host/root
mountPropagation: HostToContainer
name: root
readOnly: true
dnsPolicy: ClusterFirst
hostNetwork: true
hostPID: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /proc
type: ""
name: proc
- hostPath:
path: /sys
type: ""
name: sys
- hostPath:
path: /
type: ""
name: root
tolerations:
# 为了 node exporter 能够被调度到 master 节点中运行,我们需要为 Pod 添加容忍度属性
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
updateStrategy:
type: OnDelete

2.2 编写 svc

由于 node exporter Pod 分散在各个节点,为了便于 Prometheus 收集这些 node exporter 的 Pod IP,需要创建 Endpoint 统一收集,这里通过创建 Service 自动生成 Endpoint 来达到目的。

apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9400"
prometheus.io/path: /metrics
name: node-exporter-9400
namespace: kube-system
labels:
app: node-exporter-9400
spec:
ports:
- name: http
port: 9400
protocol: TCP
targetPort: 9400
selector:
app: node-exporter-9400

2.3 加入prometheus 自动发现

在 Kubernetes 下,Promethues 通过与 Kubernetes API 集成,目前主要支持5中服务发现模式,分别是:Node、Service、Pod、Endpoints、Ingress

prometheus 添加如下

global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_timeout: 10s
rule_files:
- /prometheus-rules/*.rules.yml
scrape_configs:

- job_name: 'kubernetes-node-exporter'

kubernetes_sd_configs:
- role: endpoints
api_server: https://k8s.foxchan.com:8443
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

relabel_configs:
- source_labels: [__meta_kubernetes_endpoints_name]
regex: 'node-exporter-9400'
action: keep

mysqld_exporter

1、安装mysql_exporter

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
tar -zxf mysqld_exporter-0.14.0.linux-amd64.tar.gz
mv mysqld_exporter-0.14.0.linux-amd64 /data/monitor/mysqld_exporter

2、在MySQL中创建用户并授权

密码如果有特殊字符建议放到中间,避免exporter无法识别

CREATE USER  'mysqld_exporter'@'localhost'  IDENTIFIED BY  'exporter!12ASzx' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

3、写一个mysqld_exporter的配置文件

cat /data/monitor/mysqld_exporter/.my.cnf 
[client]
host=localhost
user=exporter
password=exporter!12ASzx
socket=/var/lib/mysql/mysql.sock

4、配置systemd启动mysqld_exporter

cat mysqld_exporter-9401.service

[Unit]
Description=mysqld_exporter
After=network.target

[Service]
ExecStart=/data/monitor/mysqld_exporter/scripts/run_mysqld_exporter.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动脚本

cat run_mysqld_exporter.sh

#!/bin/bash
set -e

# WARNING: This file was auto-generated. Do not edit!
# All your edit might be overwritten!
DEPLOY_DIR=/data/monitor/mysqld_exporter
cd "${DEPLOY_DIR}" || exit 1

exec > >(tee -i -a "/data/monitor/mysqld_exporter/log/mysqld_exporter.log")
exec 2>&1
exec bin/mysqld_exporter \
--web.listen-address=":9401" \
--config.my-cnf /data/monitor/mysqld_exporter/.my.cnf \
--log.level="info" \
--collect.info_schema.processlist \
--collect.info_schema.innodb_metrics \
--collect.info_schema.innodb_tablespaces \
--collect.info_schema.innodb_cmp \
--collect.info_schema.innodb_cmpmem

5、确认监控指标正常

curl http://localhost:9401/metrics|grep mysql_up
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 295k 0 295k 0 0 9.9M 0 --:--:-- --:--:-- --:--:-- 10.3M
# HELP mysql_up Whether the MySQL server is up.
# TYPE mysql_up gauge
mysql_up 1

6、在Prometheus的server端添加job任务

vim  prometheus.yml
scrape_configs:
- file_sd_configs:
- files:
- mysql.yml
job_name: MySQL
metrics_path: /metrics
relabel_configs:
- source_labels: [__address__]
regex: (.*)
target_label: __address__
replacement: $1

mysql.yml

- labels:
instance: mysql1:3306 # grafana显示的实例的别名
targets:
- 172.18.0.23:9401 # mysqld_exporter暴露的端口
- labels:
instance: mysql2:3306 # grafana显示的实例的别名
targets:
- 172.18.0.23:9402 # mysqld_exporter暴露的端口

7、在grafana中导入MySQL监控图表

点击import,在弹出界面中输入7362,数据源选择Prometheus

8、配置mysql_exporter告警规则

cat mysql_rules.yml

groups:
- name: mysql.rules
rules:
- alert: MysqlDown
expr: up == 0
for: 0m
labels:
severity: critical
annotations:
title: 'MySQL down'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL instance is down"

- alert: MysqlRestarted
expr: mysql_global_status_uptime < 60
for: 0m
labels:
severity: info
annotations:
title: 'MySQL Restarted'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL has just been restarted, less than one minute ago"

- alert: MysqlTooManyConnections(>80%)
expr: avg by (instance) (rate(mysql_global_status_threads_connected[1m])) / avg by (instance) (mysql_global_variables_max_connections) * 100 > 80
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL too many connections (> 80%)'
description: "Mysql实例: 【{{ $labels.instance }}】, More than 80% of MySQL connections are in use, Current Value: {{ $value }}%"

- alert: MysqlThreadsRunningHigh
expr: mysql_global_status_threads_running > 40
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL Threads_Running High'
description: "Mysql实例: 【{{ $labels.instance }}】, Threads_Running above the threshold(40), Current Value: {{ $value }}"

- alert: MysqlQpsHigh
expr: sum by (instance) (rate(mysql_global_status_queries[2m])) > 500
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL QPS High'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL QPS above 500"

- alert: MysqlSlowQueries
expr: increase(mysql_global_status_slow_queries[1m]) > 0
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL slow queries'
description: "Mysql实例: 【{{ $labels.instance }}】, has some new slow query."

- alert: MysqlTooManyAbortedConnections
expr: round(increase(mysql_global_status_aborted_connects[5m])) > 20
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL too many Aborted connections in 2 minutes'
description: "Mysql实例: 【{{ $labels.instance }}】, {{ $value }} Aborted connections within 2 minutes"

- alert: MysqlTooManyAbortedClients
expr: round(increase(mysql_global_status_aborted_clients[120m])) > 10
for: 2m
labels:
severity: warning
annotations:
title: 'MySQL too many Aborted connections in 2 hours'
description: "Mysql实例: 【{{ $labels.instance }}】, {{ $value }} Aborted Clients within 2 hours"

- alert: MysqlSlaveIoThreadNotRunning
expr: mysql_slave_status_master_server_id > 0 and ON (instance) mysql_slave_status_slave_io_running == 0
for: 0m
labels:
severity: critical
annotations:
title: 'MySQL Slave IO thread not running'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL Slave IO thread not running"

- alert: MysqlSlaveSqlThreadNotRunning
expr: mysql_slave_status_master_server_id > 0 and ON (instance) mysql_slave_status_slave_sql_running == 0
for: 0m
labels:
severity: critical
annotations:
title: 'MySQL Slave SQL thread not running'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL Slave SQL thread not running"

- alert: MysqlSlaveReplicationLag
expr: mysql_slave_status_master_server_id > 0 and ON (instance) (mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay) > 30
for: 1m
labels:
severity: critical
annotations:
title: 'MySQL Slave replication lag'
description: "Mysql实例: 【{{ $labels.instance }}】, MySQL replication lag"

- alert: MysqlInnodbLogWaits
expr: rate(mysql_global_status_innodb_log_waits[15m]) > 10
for: 0m
labels:
severity: warning
annotations:
title: 'MySQL InnoDB log waits'
description: "Mysql实例: 【{{ $labels.instance }}】, innodb log writes stalling"

9、将告警规则集成到Prometheus

prometheus.yml

rule_files:
- "/rules/*.yml"

10、mysqld_exporter监控多个数据库

可以根据配置文件分开监控,日志也分开

拷贝2个mysqld_exporter安装包
cp -r mysqld_exporter-0.14.1.linux-amd64 mysqld_exporter_3306
cp -r mysqld_exporter-0.14.1.linux-amd64 mysqld_exporter_3307

修改监控文件为对应的库的配置
cd/ data/mysqld_exporter_3307
cat .my.cnf
[client]
host = 127.0.0.1
user = exporter
password = expoter12ssdc3
socket = /tmp/mysql_3307.sock
修改启动文件也根据情况修改

11、常见报错

  • level=error msg=“Error pinging mysqld: Error 1045: Access denied for user ‘root’@‘localhost’ (using password: YES)” source=“exporter.go:146”

1、如上报错,是授权问题,数据库中记得授权,并且查看mysqld_exporter下的my.cnf中的配置

2、如果授权密码都没问题,就要注意密码最后一位不能是字符,特别是#,mysqld_exporter 不识别,导致密码不对无法登陆

  • level=error msg=“Error pinging mysqld: this user requires old password authentication. If you still want to use it, please add ‘allowOldPasswords=1’ to your DSN. See also https://github.com/go-sql-driver/mysql/wiki/old_passwords” source=“exporter.go:146

见于老数据库,是由于old_paswords参数导致,启动时候设置DSN的环境变量来启动,不用配置文件 格式

DATA_SOURCE_NAME=用户名:密码@unixl(Sock)/?参数=值&参数=值or
DATA_SOURCE_NAME=用户名:密码@(ip地址:端口)/?参数=值&参数=
DATA SOURCE NANE=exporter:expoter12Ssdc3unix\(/tmp/mysql 3306.sock\)/?allowOldPasswords=true
举报

相关推荐

0 条评论