4、Prometheus及相关组件容器部署及alertmanager高可用

Prometheus及相关组件容器部署教程

环境准备

10.0.0.201部署：
prometheus-server、grafana-server、consul与consul-exporter，节点级监控node_exporter

10.0.0.202部署：
tomcat、mysqld mysqld_exporter、nginx nginx_exporter

10.0.0.203部署：
blakebox_exporter、alertmanager

10.0.0.204部署：
victoriametrics-cluster实现prometheus远程持久存储

各主机本地解析:
cat /etc/hosts
10.0.0.201      prometheus.magedu.com grafana.magedu.com consul.magedu.com
10.0.0.202      server01.magedu.com mysql.magedu.com nginx.magedu.com tomcat.magedu.com
10.0.0.203      server02.magedu.com alert.magedu.com probe.magedu.com
10.0.0.204      server03.magedu.com vmselect.magedu.com vminsert.magedu.com

各主机安装docker和docker-compose
apt -y install docke
apt -y install docker-compose

各主机下载编写好的各组件的docker-compose.yml文件
[root@ubuntu2004 ~]#git clone https://github.com/iKubernetes/learning-prometheus.git
正克隆到 'learning-prometheus'...
remote: Enumerating objects: 576, done.
remote: Counting objects: 100% (76/76), done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 576 (delta 34), reused 56 (delta 20), pack-reused 500
接收对象中: 100% (576/576), 571.99 KiB | 253.00 KiB/s, 完成.
处理 delta 中: 100% (306/306), 完成.
[root@ubuntu2004 ~]#ls
learning-prometheus 

所有组件在learning-prometheus/08-prometheus-components-compose/目录下

部署Consul

10.0.0.201部署：
[root@ubuntu2004 08-prometheus-components-compose]#cd
[root@ubuntu2004 ~]#cd learning-prometheus/08-prometheus-components-compose/consul-and-exporter/
[root@ubuntu2004 consul-and-exporter]#ls
consul_configs  docker-compose.yml

查看注册到Consul之上的各服务的文件
[root@ubuntu2004 consul-and-exporter]#ls consul_configs/
alertmanager.json        grafana.json             nginx-exporter.json      prometheus-servers.json
consul-exporter.json     mysqld-exporter.json     nodes.json               tomcat.json

部署Consul（拉取镜像并启动）
[root@ubuntu2004 consul-and-exporter]#docker-compose up -d
Pulling consul-exporter (prom/consul-exporter:v0.8.0)...
v0.8.0: Pulling from prom/consul-exporter
3cb635b06aa2: Pull complete
34f699df6fe0: Pull complete
63dbd0b9144a: Pull complete
Digest: sha256:4139fe6f0c57b6df35dcd423e81f4025d3201038f6168ecc8ceec73ceeb8a14e
Status: Downloaded newer image for prom/consul-exporter:v0.8.0
Creating consul-and-exporter_consul_1 ... done
Creating consul-and-exporter_consul-exporter_1 ... done

查看是否UP和端口有无问题
[root@ubuntu2004 consul-and-exporter]#docker-compose ps

在物理机进行域名解析后访问http://consul.magedu.com:8500/ui/dc1/services

各服务已被注册到Consul之上

部署prometheus-server

10.0.0.201部署：
[root@ubuntu2004 consul-and-exporter]#cd ../prometheus-server/
[root@ubuntu2004 prometheus-server]#ls
docker-compose.yml  prometheus  README.md
[root@ubuntu2004 prometheus-server]#cd prometheus/
[root@ubuntu2004 prometheus]#ls
examples-no-alertmanager.yml  examples-with-remote-storage.yml  examples.yml  prometheus.yml  rules  targets

其中prometheus.yml文件只配置了各个节点的监控及prometheus自身的监控（默认配置），用于简单测试，看prometheus-server是否能够正常启动
prometheus-server正常启动之后，随后用examples-no-alertmanager.yml文件替代prometheus.yml，基于consul服务得发现，监控各个组件。监控各组件之后，每个组件都不是正常的，目前是把告警功能禁掉了

基于prometheus.yml部署prometheus-server
[root@ubuntu2004 prometheus]#cd ..
[root@ubuntu2004 prometheus-server]#pwd
/root/learning-prometheus/08-prometheus-components-compose/prometheus-server
[root@ubuntu2004 prometheus-server]#docker-compose up -d

访问：http://prometheus.magedu.com:9090/

prometheus-server启动无误，更换配置文件，监控其他组件

[root@ubuntu2004 prometheus-server]#cd prometheus/
[root@ubuntu2004 prometheus]#mv prometheus.yml prometheus.yml.bak
[root@ubuntu2004 prometheus]#cp examples-no-alertmanager.yml prometheus.yml

重载配置文件
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

访问：http://prometheus.magedu.com:9090/，服务都被发现

部署node_exporter

10.0.0.201:部署
[root@ubuntu2004 08-prometheus-components-compose]#pwd
/root/learning-prometheus/08-prometheus-components-compose
[root@ubuntu2004 08-prometheus-components-compose]#cd node-exporter/
[root@ubuntu2004 node-exporter]#ls
docker-compose.yml
[root@ubuntu2004 node-exporter]#docker-compose up -d

访问：http://prometheus.magedu.com:9090/，node_exporter已启动

部署grafana-server

10.0.0.201：部署
[root@ubuntu2004 node-exporter]#cd ../grafana/
[root@ubuntu2004 grafana]#ls
config.monitoring  docker-compose.yml  grafana
[root@ubuntu2004 grafana]#docker-compose up -d

访问http://grafana.magedu.com:3000/ 账号：admin，密码magedu.com

10.0.0.202部署node_expoeter（余下203、204部署流程一样）

[root@ubuntu2004 ~]#cd learning-prometheus/08-prometheus-components-compose/
[root@ubuntu2004 08-prometheus-components-compose]#cd node-exporter/
[root@ubuntu2004 node-exporter]#docker-compose up -d

部署Nginx、Nginx_exporter

10.0.0.202部署
[root@ubuntu2004 node-exporter]#cd ../nginx-and-exporter/
[root@ubuntu2004 nginx-and-exporter]#ls
docker-compose.yml  nginx
[root@ubuntu2004 nginx-and-exporter]#docker-compose up -d

访问http://nginx.magedu.com/

部署mysqld、mysqld_exporter

10.0.0.202部署
[root@ubuntu2004 nginx-and-exporter]#cd ../mysqld-and-exporter/
[root@ubuntu2004 mysqld-and-exporter]#ls
docker-compose.yml  mysql
[root@ubuntu2004 mysqld-and-exporter]#docker-compose up -d

exporter是无法正常认证到mysql之上的，需要进入到mysql之上，完成对应的授权用户权限的创建
[root@ubuntu2004 mysqld-and-exporter]#docker-compose exec mysqld /bin/sh
sh-4.2# mysql
mysql> CREATE USER 'exporter'@'172.31.%.%' IDENTIFIED BY 'exporter';
mysql> GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'exporter'@'172.31.%.%';
mysql> GRANT SELECT ON performance_schema.* TO 'exporter'@'172.31.%.%';

查看mysql被监控的指标数据
[root@ubuntu2004 mysqld-and-exporter]#curl localhost:9104/metrics

访问：http://prometheus.magedu.com:9090/，mysqld_exporter已启动

部署TOMCAT

10.0.0.202部署
[root@ubuntu2004 mysqld-and-exporter]#cd ../tomcat-and-metrics/
[root@ubuntu2004 tomcat-and-metrics]#ls
docker-compose.yml  tomcat

原生的tomcat镜像没有内建测量系统，基于tomcat原有镜像做dockerfile，按需现场打镜像
[root@ubuntu2004 tomcat-and-metrics]#cd tomcat/
[root@ubuntu2004 tomcat]#ls
context.xml  Dockerfile  sources.list  tomcat-users.xml

临时构建镜像，基于镜像的结果，完成容器的启动
[root@ubuntu2004 tomcat-and-metrics]#docker-compose up -d

访问http://tomcat.magedu.com:8080/metrics/，进入Manager App 账号：tomcat，密码：magedu.com

部署blakebox_exporter

10.0.0.203:部署
[root@ubuntu2004 ~]#cd learning-prometheus/08-prometheus-components-compose/blackbox-exporter/
[root@ubuntu2004 blackbox-exporter]#ls
configs  docker-compose.yml
[root@ubuntu2004 blackbox-exporter]#docker-compose up -d
[root@ubuntu2004 blackbox-exporter]#docker-compose ps

检测网站定义：
[root@ubuntu2004 blackbox-exporter]#cat ../prometheus-server/prometheus/targets/sites-all.yaml 
- targets:
  - http://www.magedu.com
  - http://www.google.com
  labels:
    app: blackbox

relabel_configs:给prometheus-server所在的node_exporter打标签

10.0.0.201：
[root@ubuntu2004 prometheus]#vim prometheus.yml
 35   # All nodes
 36   - job_name: 'nodes'
 37     consul_sd_configs:
 38     - server: "consul.magedu.com:8500"
 39       tags:
 40       - "nodes"
 41       refresh_interval: 1m
 42 
 43     relabel_configs:
 44     - source_labels:
 45         - __scheme__
 46         - __address__
 47         - __metrics_path__
 48       regex: "(http|https)(.*)"
 49       separator: ""
 50       target_label: "endpoint"
 51       replacement: "${1}://${2}"
 52       action: replace
 
 set nu显示行数
 ：43，52s@^@    @gi #整体移动几格，使用几个空格符

重载配置文件
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

relabel_configs:给blackbox打标签

10.0.0.201：
[root@ubuntu2004 prometheus]#vim prometheus.yml
 95   # Blackbox Exporter
 96   - job_name: 'blackbox'
 97     metrics_path: /probe
 98     params:
 99       module: [http_2xx]  # Look for a HTTP 200 response.
100     file_sd_configs:
101     - files:
102       - targets/sites-*.yaml
103       refresh_interval: 1m
104     relabel_configs:
105       - source_labels: [__address__]
106         target_label: __param_target
107       - source_labels: [__param_target]
108         target_label: instance
109       - target_label: __address__
110         replacement: "probe.magedu.com:9115"  # Blackbox exporter.
111       - target_label: region
112         replacement: "local"
113 
114       - regex: "(job|app)"
115         replacement: ${1}_name
116         action: labelmap
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload
可查看日志确认
[root@ubuntu2004 prometheus]#docker-compose logs prometheus prometheus

访问http://prometheus.magedu.com:9090/targets?search=，可发现标签已改

metric_relabel_configs指标重写：如删除mysqld_exporter以“go_info”为名称前缀的指标

10.0.0.201:
[root@ubuntu2004 prometheus]#vim prometheus.yml
70   # mysqld-exporter
 71   - job_name: 'mysqld-exporter'
 72     consul_sd_configs:
 73     - server: "consul.magedu.com:8500"
 74       tags:
 75       - "mysqld-exporter"
 76       refresh_interval: 1m
 77 
 78     metric_relabel_configs:
 79     - source_labels:
 80       - __name__
 81       regex: "go_info.*"
 82       action: drop
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

在下一个采样周期，其采集信息就会断开

部署alertmanager

10.0.0.203部署：
[root@ubuntu2004 ~]#cd learning-prometheus/08-prometheus-components-compose/
[root@ubuntu2004 08-prometheus-components-compose]#ls
alertmanager                 blackbox-exporter    images               node-exporter      tomcat-and-metrics
alertmanager-and-dingtalk    consul-and-exporter  mysqld-and-exporter  prometheus-server  victoriametrics-cluster
alertmanager-gossip-cluster  grafana              nginx-and-exporter   README.md

钉钉告警、企业微信告警、邮件告警定义在alertmanager-and-dingtalk目录中

alertmanager-gossip-cluster目录中定义了运行三个alertmanager实例构建为集群，基于流言协议传递各个实例批次之间对外告警的相关状态信息


[root@ubuntu2004 08-prometheus-components-compose]#cd alertmanager-and-dingtalk/
[root@ubuntu2004 alertmanager-and-dingtalk]#cd alertmanager/
[root@ubuntu2004 alertmanager]#ls
config-with-route-trees.yml  config.yml  dingtalk_template.tmpl  email_template.tmpl  wechat_template.tmpl

config.yml为默认的配置文件
config-with-route-trees.yml是包含复杂路由数的示例

其配置文件中 templates字段对应的模板路径
templates:
  - '/etc/alertmanager/email_template.tmpl'
  - '/etc/alertmanager/wechat_template.tmpl'
  - '/etc/alertmanager/dingtalk_template.tmpl'
对应路径
[root@ubuntu2004 alertmanager-and-dingtalk]#ls alertmanager/
config-with-route-trees.yml  config.yml  dingtalk_template.tmpl  email_template.tmpl  wechat_template.tmpl

运行alertmanager
[root@ubuntu2004 alertmanager-and-dingtalk]#docker-compose up

在prometheus-server上定义监控规则

10.0.0.201:
[root@ubuntu2004 rules]#pwd
/root/learning-prometheus/08-prometheus-components-compose/prometheus-server/prometheus/rules
[root@ubuntu2004 rules]#ls
alert-rules-blackbox-exporter.yml  alert-rules-example.yml  alert-rules-node-exporter.yml  kubernetes-rules.yml
alert-rules-consul-exporter.yaml   alert-rules-mysqld.yml   alert-rules-prometheus.yml     record-rules-mysqld.yml

以 record开头的为记录规则
以 alert开头的为告警规则

定义好监控规则之后在prometheus.yml配置文件中启用并定义alertmanager的服务地址

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alert.magedu.com:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - rules/alert-rules-*.yml
  - rules/record-rules-*.yml
  # - "first_rules.yml"
  # - "second_rules.yml"

重载配置文件，使其生效
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

停掉并拆除当前的alertmanager，更换到alertmanager-gossip-cluster环境中去

（alertmanager-gossip-cluster目录中定义了运行三个alertmanager实例构建为集群，基于流言协议传递各个实例批次之间对外告警的相关状态信息）

[root@ubuntu2004 alertmanager-and-dingtalk]#docker-compose down
Removing alertmanager-and-dingtalk_alertmanager_1                ... done
Removing alertmanager-and-dingtalk_prometheus-webhook-dingtalk_1 ... done
Removing network alertmanager-and-dingtalk_monitoring

[root@ubuntu2004 alertmanager-and-dingtalk]#cd ..
[root@ubuntu2004 08-prometheus-components-compose]#cd alertmanager-gossip-cluster/

一旦启用该实例，就有三个alertmanager服务可以调用，用于发送告警信息
[root@ubuntu2004 alertmanager-gossip-cluster]#docker-compose up

验证服务是否正常开启，可访问节点的三个端口

http://10.0.0.203:9093/#/status

http://10.0.0.203:19093/#/status

http://10.0.0.203:29093/#/status

此时，集群如果被真正用起来，需要配置prometheus-server监控这三个实例，并把这三个实例作为alertmanager-server使用

10.0.0.201：
[root@ubuntu2004 prometheus]#pwd
/root/learning-prometheus/08-prometheus-components-compose/prometheus-server/prometheus
[root@ubuntu2004 prometheus]#vim prometheus.yml
# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alert.magedu.com:9093
      - alert.magedu.com:19093
      - alert.magedu.com:29093

在consul上把剩余的两个alertmanager-服务注册
[root@ubuntu2004 consul_configs]#cat alertmanager-19093.json 
{
      "id": "alertmanager-02",
      "name": "alertmanager-02",
      "address": "alert.magedu.com",
      "port": 19093,
      "tags": ["alertmanager-02"],
      "checks": [{
        "http": "http://alert.magedu.com:19093/metrics",
        "interval": "5s"
      }]
}
[root@ubuntu2004 consul_configs]#curl -XPUT --data @alertmanager-19093.json http://localhost:8500/v1/agent/service/register

[root@ubuntu2004 consul_configs]#cat alertmanager-29093.json 
{
      "id": "alertmanager-03",
      "name": "alertmanager-03",
      "address": "alert.magedu.com",
      "port": 29093,
      "tags": ["alertmanager-03"],
      "checks": [{
        "http": "http://alert.magedu.com:29093/metrics",
        "interval": "5s"
      }]
}
[root@ubuntu2004 consul_configs]#curl -XPUT --data @alertmanager-29093.json http://localhost:8500/v1/agent/service/register

重载prometheus-server配置文件
[root@ubuntu2004 prometheus]#pwd
/root/learning-prometheus/08-prometheus-components-compose/prometheus-server/prometheus
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

部署victoriametrics-cluster实现prometheus远程持久存储

10.0.0.204:
[root@ubuntu2004 victoriametrics-cluster]#cat README.md 
# VictoriaMetrics 示例集群

#### 配置Prometheus以之为远程存储：
remote_write:    # 远程写入到远程 VM 存储  - url: http://vminsert.magedu.com:8480/insert/0/prometheusremote_read:  - url: http://vmselect.magedu.com:8481/select/0/prometheusremote_write:    # 远程写入到远程 VM 存储
  - url: http://vminsert.magedu.com:8480/insert/0/prometheus

remote_read:
  - url: http://vmselect.magedu.com:8481/select/0/prometheusremote_write:    # 远程写入到远程 VM 存储
  - url: http://vminsert.magedu.com:8480/insert/0/prometheus

remote_read:
  - url: http://vmselect.magedu.com:8481/select/0/prometheus


其中的0为多租户模型下的租户ID。


#### 配置Grafana以之为数据源：

添加新数据源，将数据加载路径定义为：http://vmselect.magedu.com:8481/select/0/prometheus

配置完成后，导入Dashboard，即可在Grafana中查看基于该数据源的Dashboard。

[root@ubuntu2004 victoriametrics-cluster]#cat docker-compose.yml 
version: '3.6'

networks:
  vm_net:
    driver: bridge

volumes:
  strgdata-1: {}
  strgdata-2: {}
  grafanadata: {}

services:
  vmstorage-1:
    container_name: vmstorage-1
    image: victoriametrics/vmstorage:v1.83.1-cluster
    ports:
      - 8482
      - 8400
      - 8401
    volumes:
      - strgdata-1:/storage
    networks:
      - vm_net
    command:
      - '--storageDataPath=/storage'
    restart: always

  vmstorage-2:
    container_name: vmstorage-2
    image: victoriametrics/vmstorage:v1.83.1-cluster
    networks:
      - vm_net
    ports:
      - 8482
      - 8400
      - 8401
    volumes:
      - strgdata-2:/storage
    command:
      - '--storageDataPath=/storage'
    restart: always

  vminsert:
    container_name: vminsert
    image: victoriametrics/vminsert:v1.83.1-cluster
    depends_on:
      - "vmstorage-1"
      - "vmstorage-2"
    command:
      - '--storageNode=vmstorage-1:8400'
      - '--storageNode=vmstorage-2:8400'
    ports:
      - 8480:8480
    networks:
      - vm_net
    restart: always

  vmselect:
    container_name: vmselect
    image: victoriametrics/vmselect:v1.83.1-cluster
    depends_on:
      - "vmstorage-1"
      - "vmstorage-2"
    command:
      - '--storageNode=vmstorage-1:8401'
      - '--storageNode=vmstorage-2:8401'
      #- '--vmalert.proxyURL=http://vmalert:8880'
    networks:
      - vm_net
    ports:
      - 8481:8481
    restart: always

[root@ubuntu2004 victoriametrics-cluster]#docker-compose up -d

定义prometheus-server

10.0.0.201：

[root@ubuntu2004 prometheus]# vim prometheus.yml
...
remote_write:
  - url: http://vminsert.magedu.com:8480/insert/0/prometheus

remote_read:
  - url: http://vmselect.magedu.com:8481/select/0/prometheus

重载配置文件
[root@ubuntu2004 prometheus]#curl -XPOST localhost:9090/-/reload

查看各端口指标暴露接口
[root@ubuntu2004 victoriametrics-cluster]#curl localhost:8480/metrics
[root@ubuntu2004 victoriametrics-cluster]#curl localhost:8481/metrics

远端存储接口是否生效http://prometheus.magedu.com:9090/config

grafana配置新数据源

监控vmselect和vminsert

[root@ubuntu2004 config]#cat vm*
{
     "id": "vminsert",
     "name": "vminsert",
     "address": "vminsert.magedu.com",
     "port": 8480,
     "tags": ["vminsert"],
     "checks": [{
       "http": "http://vminsert.magedu.com:8480/metrics",
       "interval": "5s"
     }]
}

{
     "id": "vmselect",
     "name": "vmselect",
     "address": "vmselect.magedu.com",
     "port": 8481,
     "tags": ["vmselect"],
     "checks": [{
       "http": "http://vmselect.magedu.com:8481/metrics",
       "interval": "5s"
     }]
}

[root@ubuntu2004 config]#curl -XPUT --data @vminsert.json http://localhost:8500/v1/agent/service/register
[root@ubuntu2004 config]#curl -XPUT --data @vmselect.json http://localhost:8500/v1/agent/service/register





[root@ubuntu2004 local]#curl -XPOST http://localhost:9090/-/reload
[root@ubuntu2004 local]#cat prometheus/prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alert.magedu.com:9093
          
  

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - rules/record-rules-*.yml
  - rules/alert-rules-*.yml

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label job=<job_name>job=<job_name> to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "nodes"
      refresh_interval: 1m
    metric_relabel_configs:
    - source_labels:
      - __name__
      regex: "go_info.*"
      action: drop
  
  - job_name: "consul_exporter"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "consul_exporter"
      refresh_interval: 1m
  
  - job_name: "mysqld_exporter"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "mysqld_exporter"
      refresh_interval: 1m

  - job_name: "nginx_exporter"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "nginx_exporter"
      refresh_interval: 1m

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
    - targets:
      - "www.baidu.com"
      - "www.google.com"
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: "vminsert:9115"  # 指向实际的Blackbox exporter.
      - target_label: region
        replacement: "remote"
    
  - job_name: "grafana"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "grafana"
      refresh_interval: 1m

  - job_name: "vminsert"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "vminsert"
      refresh_interval: 1m


  - job_name: "vmselect"
    metrics_path: '/metrics'
    scheme: 'http'
    consul_sd_configs:
    - server: "localhost:8500"
      tags:
      - "vmselect"
      refresh_interval: 1m


remote_write:
  - url: http://vminsert.magedu.com:8480/insert/0/prometheus

remote_read:
  - url: http://vmselect.magedu.com:8481/select/0/prometheus