日志收集系统loki+promtail+Grafana 部署
转自 https://zhuanlan.zhihu.com/p/639304209
一、简 介
Loki是受Prometheus启发由Grafana Labs团队开源的水平可扩展,高度可用的多租户日志聚合系统。 开发语言: Google Go。它的设计具有很高的成本效益,并且易于操作。使用标签来作为索引,而不是对全文进行检索,也就是说,你通过这些标签既可以查询日志的内容也可以查询到监控的数据签,极大地降低了日志索引的存储。系统架构十分简单,由以下3个部分组成 :
Loki 是主服务器,负责存储日志和处理查询 。
promtail 是代理,负责收集日志并将其发送给 loki 。
Grafana 用于 UI 展示。
只要在应用程序服务器上安装promtail来收集日志然后发送给Loki存储,就可以在Grafana UI界面通过添加Loki为数据源进行日志查询(如果Loki服务器性能不够,可以部署多个Loki进行存储及查询)。作为一个日志系统不光只有查询分析日志的能力,还能对日志进行监控和报警。
Loki文档网址:https://grafana.com/docs/loki/latest/
下载网址:https://github.com/grafana/loki/releases
二、原理分析
promtail收集并将日志发送给loki的 Distributor 组件
Distributor会对接收到的日志流进行正确性校验,并将验证后的日志分批并行发送到Ingester
Ingester 接受日志流并构建数据块,压缩后存放到所连接的存储后端
Querier 收到HTTP查询请求,并将请求发送至Ingester 用以获取内存数据 ,Ingester 收到请求后返回符合条件的数据 ;
如果 Ingester 没有返回数据,Querier 会从后端存储加载数据并遍历去重执行查询 ,通过HTTP返回查询结果
三、与 ELK 比较优势
ELK虽然功能丰富,但规模复杂,资源占用高,操作苦难,很多功能往往用不上,有点杀鸡用牛刀的感觉。
loki 不对日志进行全文索引。通过存储压缩非结构化日志和索引元数据,Loki 操作起来会更简单,更省成本。
通过使用与 Prometheus 相同的标签记录流对日志进行索引和分组,这使得日志的扩展和操作效率更高。
安装部署简单快速,且受 Grafana 原生支持。
C/S架构
Server:192.168.17.10 loki grafana
Client:192.168.17.11 promtail
Server:192.168.17.10 loki grafana
[root@localhost ~]# mkdir -p /usr/local/loki && mkdir -p /data/loki/{index,chunks}
[root@localhost ~]# cd /usr/local/loki/
[root@localhost loki]# wget https://github.com/grafana/loki/releases/download/v2.3.0/loki-linux-amd64.zip
[root@localhost loki]# wget https://dl.grafana.com/oss/release/grafana-7.1.0-1.x86_64.rpm
##部署loki
[root@localhost loki]# vi /usr/local/loki/loki.yaml
---
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 192.168.17.10 #loki所在主机
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2022-08-01
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h #每张表的时间范围7天
storage_config:
boltdb:
directory: /data/loki/index #索引文件存储地址
filesystem:
directory: /data/loki/chunks #块存储地址
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
# 最大可查询历史日期 28天,这个时间必须是schema_config中的period的倍数,否则报错。
max_look_back_period: 672h
# 表的保留期28天
table_manager:
retention_deletes_enabled: true
retention_period: 672h
配置systemd系统启动:
[root@localhost loki]# vi /etc/systemd/system/loki.service
[Unit]
Description=loki
After=network.target
[Service]
ExecStart=/usr/local/loki/loki-linux-amd64 -config.file=/usr/local/loki/loki.yaml &>> /usr/local/loki/loki-3100.log
Restart=on-failure
[Install]
WantedBy=multi-user.target
[root@localhost loki]# systemctl daemon-reload
[root@localhost loki]# systemctl start loki && systemctl enable loki
[root@localhost loki]# netstat -ntlp |grep loki
tcp6 0 0 :::9095 :::* LISTEN 23440/loki-linux-am
tcp6 0 0 :::3100 :::* LISTEN 23440/loki-linux-am
部署grafana
[root@localhost loki]# yum install -y grafana-7.1.0-1.x86_64.rpm
[root@localhost loki]# systemctl daemon-reload
[root@localhost loki]# systemctl enable grafana-server.service
[root@localhost loki]# systemctl start grafana-server.service
[root@localhost loki]# netstat -ntlp |grep grafana
tcp6 0 0 :::3000 :::* LISTEN 23622/grafana-serve
Client:192.168.17.11 promtail
[root@localhost ~]# mkdir -p /usr/local/promtail/
[root@localhost ~]# wget https://github.com/grafana/loki/releases/download/v2.3.0/promtail-linux-amd64.zip
[root@localhost ~]# unzip promtail-linux-amd64.zip
[root@localhost ~]# vi /usr/local/promtail/promtail.yaml
# Promtail Server Config
server:
http_listen_port: 9080
grpc_listen_port: 0
# Positions
positions:
filename: ./positions.yaml
# Loki服务器的地址
clients:
- url: http://192.168.17.10:3100/loki/api/v1/push
scrape_configs:
- job_name: systemd #区分从其他日志组收集的日志
static_configs:
- targets:
- 192.168.17.11
labels:
job: systemd
host: 192.168.17.11
__path__: /var/log/messages #收集日志的路径
[root@localhost ~]# vi /etc/systemd/system/promtail.service
[Unit]
Description=promtail
After=network.target
[Service]
ExecStart=/usr/local/promtail/promtail-linux-amd64 \
-config.file=/usr/local/promtail/promtail.yaml &>> /usr/local/promtail/promtail-9080.log
Restart=on-failure
[Install]
WantedBy=multi-user.target
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl start promtail && systemctl enable promtail
[root@localhost ~]# netstat -nutlp |grep promtail
tcp6 0 0 :::9080 :::* LISTEN 34045/promtail-linu
tcp6 0 0 :::33470 :::* LISTEN 34045/promtail-linu
访问grafana http://192.168.17.10:3000/
用户 admin
密码 admin
查询方式/正则匹配示例
收集系统日志:
收集NGINX日志:
[root@localhost promtail]# vim promtail.yaml
# Promtail Server Config
server:
http_listen_port: 9080
grpc_listen_port: 0
# Positions
positions:
filename: ./positions.yaml
# Loki服务器的地址
clients:
- url: http://192.168.17.10:3100/loki/api/v1/push
scrape_configs:
- job_name: nginx #区分从其他日志组收集的日志
static_configs:
- targets:
- 192.168.17.11
labels:
job: nginx
host: 192.168.17.11
__path__: /var/log/nginx/access.log #收集nginx日志的路径
收集tomcat日志:
在tomcat端安装promtai,并配置 promtai.yaml
[root@localhost ~]# vi /usr/local/promtail/promtail.yaml
# Promtail Server Config
server:
http_listen_port: 9080
grpc_listen_port: 0
# Positions
positions:
filename: ./positions.yaml
# Loki服务器的地址
clients:
- url: http://192.168.17.10:3100/loki/api/v1/push
scrape_configs:
- job_name: tomcat
static_configs:
- targets:
- 192.168.17.12
labels:
job: tomcat
host: 192.168.17.12
__path__: /usr/local/tomcat/logs/catalina.out #收集tomcat日志的路径
收集docker nginx访问日志:
[root@localhost ~]# mkdir -p /opt/nginx
[root@localhost ~]# cd /opt/nginx/
[root@localhost nginx]# mkdir {conf,conf.d,html,logs}
###拷贝一份完整得nginx配置文件
[root@localhost ~]# docker run -d --name nginx1 -p 8899:80 nginx:latest
[root@localhost ~]# docker cp nginx1:/var/log/nginx /opt/nginx/logs
[root@localhost ~]# docker cp nginx1:/etc/nginx/nginx.conf /opt/nginx/conf/
[root@localhost ~]# docker cp nginx1:/etc/nginx/conf.d/ /opt/nginx/
###启动docker 挂载本地目录
[root@localhost ~]# docker run -d --privileged=true --name nginx -p 7000:80 -v /opt/nginx/conf/nginx.conf:/etc/nginx/nginx.conf -v /opt/nginx/conf.d:/etc/nginx/conf.d -v /opt/nginx/logs/:/var/log/nginx -v /opt/nginx/html:/usr/share/nginx/html nginx:latest
[root@localhost promtail]# vi promtail.yaml
# Promtail Server Config
server:
http_listen_port: 9080
grpc_listen_port: 0
# Positions
positions:
filename: ./positions.yaml
# Loki服务器的地址
clients:
- url: http://192.168.17.10:3100/loki/api/v1/push
scrape_configs:
- job_name: docker-nginx
static_configs:
- targets:
- 192.168.17.13
labels:
job: docker-nginx
host: 192.168.17.13
__path__: /opt/nginx/logs/access.log #收集nginx日志的路径