0
点赞
收藏
分享

微信扫一扫

如何优雅的使用Prometheus来做进程监控

逸省 2022-10-18 阅读 91
docker配置文件字节数系统/运维

作者：崔亮的博客

博客地址：https://www.cuiliangblog.cn

Process Exporter简介

process exporter功能

如果想要对主机的进程进行监控，例如chronyd，sshd等服务进程以及自定义脚本程序运行状态监控。我们使用node exporter就不能实现需求了，此时就需要使用process exporter来做进程状态的监控。
项目地址：https://github.com/ncabatoff/process-exporter

process exporter配置

process exporter的配置文件为yaml格式，需要在配置文件中指明需要监控的进程，并将进程分组，一个进程只能属于一个组。

  • 基本格式

process_names:
  - name: "{{.Matches}}" # 匹配模板
    cmdline: 
    - 'redis-server' # 匹配名称

  • 匹配模板

{{.Comm}}

包含原始可执行文件的名称，即/proc//stat

{{.ExeBase}}

包含可执行文件的名称(默认)

{{.ExeFull}}

包含可执行文件的路径

{{.Username}}

包含的用户名

{{.Matches}}

包含所有正则表达式而产生的匹配项（建议使用）

{{.PID}}

包含进程的PID，一个PID仅包含一个进程（不建议使用）

{{.StartTime}}

包含进程的开始时间（不建议使用）

process exporter常用指标

namedprocess_namegroup_num_procs

运行的进程数

namedprocess_namegroup_states

Running/Sleeping/Other/Zombie状态的进程数

namedprocess_namegroup_cpu_seconds_total

获取/proc/[pid]/stat 进程CPU utime、stime状态时间

namedprocess_namegroup_read_bytes_total

获取/proc/[pid]/io 进程读取字节数

namedprocess_namegroup_write_bytes_total

获取/proc/[pid]/io 进程写入字节数

namedprocess_namegroup_memory_bytes

获取进程使用的内存字节数

namedprocess_namegroup_open_filedesc

获取进程使用的文件描述符数量

namedprocess_namegroup_thread_count

运行的线程数

namedprocess_namegroup_thread_cpu_seconds_total

获取线程CPU状态时间

namedprocess_namegroup_thread_io_bytes_total

获取线程IO字节数

Process Exporter部署

实验场景：需要监控k8s集群每台linux服务器的chronyd sshd docker kubelet进程运行状态，当有进程异常时，触发告警。

exporter配置文件

使用configmap的方式挂载配置文件内容，使用Matches模板正则匹配指定名称的进程。

apiVersion: v1
kind: ConfigMap
metadata:
  name: process-exporter-config
  namespace: monitoring
data:
  process-exporter-config.yaml: |-
    process_names:
    - name: "{{.Matches}}"
      cmdline:
      - 'docker'
    - name: "{{.Matches}}"
      cmdline:
      - 'kubelet'
    - name: "{{.Matches}}"
      cmdline:
      - 'chronyd'
    - name: "{{.Matches}}"
      cmdline:
      - 'sshd'

exporter控制器

由于集群内每台主机都需要进行进程监控，所以使用daemonset控制器。将配置文件和宿主机/proc目录挂载至pod中。

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: process-exporter
  namespace: monitoring
  labels:
    app: process-exporter
spec:
  selector:
    matchLabels:
      app: process-exporter
  template:
    metadata:
      labels:
        app: process-exporter
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: process-exporter
          image: ncabatoff/process-exporter
          args:
            - -config.path=/config/process-exporter-config.yaml
          ports:
            - containerPort: 9256
          resources:
            requests:
              cpu: 10m
              memory: 10Mi
            limits:
              cpu: 150m
              memory: 180Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          volumeMounts:
            - name: proc
              mountPath: /proc
            - name: config
              mountPath: /config
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: config
          configMap:
            name: process-exporter-config

  • exporter访问验证

访问任意一个process exporter的metrics，验证是否正常采集到指标

[root@k8s-master exporter]# kubectl get pod -n monitoring -o wide 
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
alertmanager-6f944cf9fb-x49bl        1/1     Running   3          122d    10.244.3.187    k8s-work1    <none>           <none>
blackbox-exporter-547df5d6bd-sppfh   1/1     Running   4          124d    10.244.2.108    k8s-work3    <none>           <none>
grafana-7b45bfcd8b-plcqq             1/1     Running   5          125d    10.244.3.184    k8s-work1    <none>           <none>
node-exporter-2rg5h                  1/1     Running   4          124d    10.244.0.50     k8s-master   <none>           <none>
node-exporter-rzl4v                  1/1     Running   4          124d    10.244.3.188    k8s-work1    <none>           <none>
node-exporter-sk8r8                  1/1     Running   5          124d    10.244.2.110    k8s-work3    <none>           <none>
node-exporter-xqwb5                  1/1     Running   11         124d    10.244.1.143    k8s-work2    <none>           <none>
process-exporter-46nqr               1/1     Running   0          5m10s   192.168.10.11   k8s-work1    <none>           <none>
process-exporter-9d5kd               1/1     Running   0          5m10s   192.168.10.10   k8s-master   <none>           <none>
process-exporter-kl8vc               1/1     Running   0          5m10s   192.168.10.13   k8s-work3    <none>           <none>
process-exporter-t8gd6               1/1     Running   0          5m10s   192.168.10.12   k8s-work2    <none>           <none>
prometheus-0                         1/1     Running   0          10m     10.244.3.200    k8s-work1    <none>           <none>
[root@k8s-master exporter]# curl 192.168.10.11:9256/metrics
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="alertmanager"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="calico-node"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="containerd-shim"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="coredns"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="dockerd"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="flanneld"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="grafana-server"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="kube-controller"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="kube-proxy"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="node_exporter"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="process-exporte"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:docker]",threadname="prometheus"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:kubelet]",threadname="kubelet"} 0
namedprocess_namegroup_thread_major_page_faults_total{groupname="map[:kubelet]",threadname="metrics-server"} 0
# HELP namedprocess_namegroup_thread_minor_page_faults_total Minor page faults for these threads
# TYPE namedprocess_namegroup_thread_minor_page_faults_total counter
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="alertmanager"} 136
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="calico-node"} 0
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="containerd-shim"} 144
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="coredns"} 0
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="dockerd"} 0
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="flanneld"} 26
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="grafana-server"} 5
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="kube-controller"} 2
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="kube-proxy"} 0
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="node_exporter"} 8028
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="process-exporte"} 2389
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:docker]",threadname="prometheus"} 953
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:kubelet]",threadname="kubelet"} 4
namedprocess_namegroup_thread_minor_page_faults_total{groupname="map[:kubelet]",threadname="metrics-server"} 3

namedprocess开头的指标均为process exporter采集的信息，指标采集正常，接下来配置prometheus的targets

prometheus targets配置

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yaml: |-
    global:
      scrape_interval: 30s
      eval（.*):10250'
        replacement: '${1}:9256'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        action: replace
        target_label: ip

  • prometheus targets验证

如何优雅的使用Prometheus来做进程监控_dockerimage.png

  • 输入指标名称验证

如何优雅的使用Prometheus来做进程监控_配置文件_02image.png

prometheus采集无误，接下来我们配置告警规则。由于namedprocess_namegroup_num_procs这个指标显示的是指定服务的进程数，我们可以设置条件，当这个指标值为0的时候触发告警。

prometheus rules配置

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rule
  labels:
    name: prometheus-rule
  namespace: monitoring
data:
  alert-rules.yaml: |-
    groups:
    - name: node-alert
      rules:
      - alert: service not running
        expr: namedprocess_namegroup_num_procs == 0
        for: 1m
        labels:
          severity: warning
          team: server
        annotations:
          summary: "{{$labels.ip}} service status not running"
          description: "{{$labels.ip}} {{$labels.groupname}} service status not running"
          value: "{{$labels.groupname}}"

  • 接下来。我们将node3的chronyd服务人为停止，模拟触发故障，查看alert信息

如何优雅的使用Prometheus来做进程监控_字节数_03image.png

grafana配置

process exporter已经为我们准备好了现成的grafana dashboard模板，我们只需要直接导入即可，模板ID为249

如何优雅的使用Prometheus来做进程监控_docker_04


举报

相关推荐

技术分享| 如何使用Prometheus实现系统进程监控

Prometheus系统进程监控数据监控服务器运维开发系统/运维

如何使用ConsulManager来优雅的管理云主机监控与站点监控

看板docker应用场景

如何使用Lombok来优雅的编码

程序员自我修养javaweb收藏SSM+shiro等学习

如何使用 Prometheus 监控SpringBoot系统

k8sspring bootprometheusgrafana云原生云计算yyds干货盘点

如何优雅的使用Consul管理站点监控

flasknginxdocker

prometheus监控如何工作

kubernetes容器云原生基础设施用例虚拟化云计算

如何使用padding空间来做css

paddingHtml/CSS前端开发

SpringBoot使用prometheus监控

SpringBoot程序猿阵线联盟...SpringB...Spring-...技术干货Spring ...

使用prometheus监控docker

docker文件系统安装部署运维

使用prometheus监控redis

prometheusPrometheus微服务架构和实践javaWeb学习
0 条评论