Prometheus + Grafana

1.1 运行用户创建

groupadd prometheus
useradd -g prometheus -m -d /opt/prometheus/ -s /sbin/nologin prometheus

1.2 prometheus server安装

可以访问 github.com 进行下载tar包

wget https://github.com/prometheus/prometheus/releases/download/v2.32.1/prometheus-2.32.1.darwin-amd64.tar.gz
tar xzf prometheus-2.14.0.linux-amd64.tar.gz -C /opt/
cd /opt/prometheus-2.14.0.linux-amd64 && mv prometheus-2.14.0.linux-amd64 prometheus

1.3 prometheus配置语法校验

建议每次修改prometheus配置之后, 都进行语法校验, 以免导致 prometheus server无法启动. 
./promtool check config prometheus.yml

1.4 测试启动Prometheus

此时采用默认配置启动 prometheus server 看下界面, 稍后介绍如何监控Linux 服务器.

./prometheus --config.file=prometheus.yml
http://192.168.0.71:9090/graph

1.6 设置prometheus系统服务,并配置开机启动

vim  /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=prometheus
# --storage.tsdb.path是可选项，默认数据目录在运行目录的./dada目录中
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --web.enable-lifecycle --storage.tsdb.path=/opt/prometheus/data --storage.tsdb.retention=60d
Restart=on-failure

[Install]
WantedBy=multi-user.target

修改启动文件权限，知道其他启动方式可以通过 shell 脚本启动。

chown prometheus:prometheus /usr/lib/systemd/system/prometheus.service

设置开机启动

systemctl daemon-reload
systemctl enable prometheus.service
systemctl status prometheus.service
systemctl restart prometheus.service

2. Prometheus 配置监控其他Linux主机

2.1 node_exporter安装配置

运行用户添加

groupadd prometheus
useradd -g prometheus -m -d /usr/local/node_exporter/ -s /sbin/nologin prometheus

下载node_server 同样此安装包 github.com 也有

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.darwin-amd64.tar.gz
tar -zxf node_exporter-1.3.1.darwin-amd64.tar.gz -C /usr/local/
mv /usr/local/node_exporter-1.3.1 /usr/local/node_exporter

系统服务配置 node_exporter

vim /usr/lib/systemd/system/node_exporter.service 
[Unit]
Description=node_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target

chown prometheus:prometheus /usr/lib/systemd/system/node_exporter.service [Ubuntu系统路径不同：/lib/systemd/system/]
chown -R prometheus:prometheus /usr/local/node_exporter*

####启动 node_exporter 服务并设置开机启动

systemctl daemon-reload
systemctl enable node_exporter.service
systemctl start node_exporter.service
systemctl status node_exporter.service
systemctl restart node_exporter.service
systemctl start node_exporter.service
systemctl stop node_exporter.service

node_export 启动完毕后会开启一个 9100 tcp 端口：http://192.168.0.71:9100/metrics 可以手动访问测试

客户端机器配置完成，现在配置服务端将客户端添加至监控中。

[root@localhost ~]# cd /opt/prometheus/
[root@localhost prometheus]# ls
console_libraries  consoles  data  LICENSE  NOTICE  prometheus  prometheus.rules.yml  prometheus.yml  promtool  sd_cfg
[root@localhost prometheus]# vim prometheus.yml  # 这里展示的是不完全的配置文件
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'Linux'
    file_sd_configs:
    - files: ['/opt/prometheus/sd_cfg/Linux.yml']
      refresh_interval: 5s

配置一个 Linux 群组：并创建一个对于的文件夹和文件。

[root@localhost prometheus]# cat sd_cfg/Linux.yml 
- targets: ['localhost:9100']
  labels:
    name: Linux-node1 
#[这里建议给每个主机打个有意义的标签,方便识别.]
- targets: ['192.168.0.102:9100']
  labels:
    name: Linux-node2

检查配置文件：

[root@localhost prometheus]# ./opt/prometheus/promtool check config prometheus.yml 
Checking prometheus.yml
  SUCCESS: 1 rule files found

Checking prometheus.rules.yml
  SUCCESS: 1 rules found

3 数据展示Grafana安装配置可以直接下载 8.3版本企业版，开源版下载不了

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.2-1.x86_64.rpm
sudo yum install grafana-enterprise-8.3.2-1.x86_64.rpm
systemctl enable grafana-server.service
systemctl start grafana-server.service