ansible-playbook之安装filebeat部署

一.准备环境：

1.centos7 环境

2.安装ansible环境

3.filebeat安装文件下载路径：

       https://artifacts.elastic.co/downloads/kibana/kibana-8.6.2-linux-x86_64.tar.gz

4.filebeat安装，作者用192.168.126.128作为filebeat安装环境

二.规划：

1.变量规划：

安装路径：   filebeat_dir: "/kingdee/yzj_monitor"

运行用户：   filbeat_user: "filebeat"

安装包版本： filebeat_tgz: "kibana-8.6.2-linux-x86_64.tar.gz"

es服务端：   elasticsearch_host："192.168.126.128:9200,192.168.126.129:9200,192.168.126.130:9200"

写入索引：   myindex："applog"

索引副本数： number_of_replicas： 0

索引分片数： number_of_shards： 1

需要读取的日志路径：

appnameinfo:

  - {servername: "systemlog",logpath: "/var/log/messge*"}

  - {servername: "mytestlog",logpath: "/kingdee/*.log"}

es用户名称：esusername: "logadmin"

es用户密码：espassword: "Mytest@@@@@1203"

2.filebeat目录规划：

安装目录：/kingdee/yzj_monitor

三.编写ansible-playbook：

1.创建ansible-playbook剧本目录

mkdir -p roles/filebeat/{files,tasks,templates,vars}

2.模板文件配置

cd roles/filebeat/templates

#filebeat配置文件

vim filebeat.yml.j2

###定义配置文件路径######
filebeat.config.inputs:
  path: {{filebeat_dir}}/filebeat/config/*.yml
  enabled: true


###定义filebeat日志文件路径####
logging.level: info
logging.to_files: true
logging.files:
  path: {{filebeat_dir}}/filebeat/logs
  keepfiles: 3
  name: filebeat.log


###写入elasticsearch库中######
output.elasticsearch:
  hosts: ["http://{{elasticsearch_host.split(',')[0]}}","http://{{elasticsearch_host.split(',')[1]}}","http://{{elasticsearch_host.split(',')[2]}}"]    ##es的ip及端口
  index: "{{myindex}}"                    ##创建的索引名
  username: "{{esusername}}"               ##es的账号
  password: "{{espassword}}"           ##es的密码
setup.template.name: "{{myindex}}"      ##保持和index一致即可
setup.template.pattern: "{{myindex}}-*" ##保持和index-*即可
setup.ilm.enabled: false

#日志文件配置

vim app_log.yml.j2

{%for appname in appnameinfo%}
- type: log
  paths:
    - {{appname.logpath}}
  #exclude_lines: [^DBG]
  #exclude_files: [.gz]
  fields:
     host: {{ansible_ssh_host}}
     type: {{appname.servername}}
  fields_under_root: true
  scan_frequency: 5s
  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 500
  multiline.timeout: 5s
{%endfor%}

#生成别名索引文件

vim myindex.json.j2

{
  "aliases": {
    "{{myindex}}": {
      "is_write_index": true
    }
  },
  "settings": {
    "number_of_replicas": {{number_of_replicas}},
    "number_of_shards": {{number_of_shards}}
  }
}

#生成生命周期文件【根据实际生产需求、设置host.warm.cold.delete阶段条件】

vim policy.json.j2

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "12h",
            "max_docs": "20000000",
            "max_size": "10gb"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "allocate": {
            "include": {
              "box_type": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "2d",
        "actions": {
          "allocate": {
            "include": {
              "box_type": "cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

#生成模板文件

vim template.json.j2

{
  "order": 1,
  "index_patterns": [
    "{{myindex}}-*"
  ],
  "settings": {
    "number_of_shards": {{number_of_shards}},
    "number_of_replicas": {{number_of_replicas}},
    "index.lifecycle.name": "{{myindex}}_policy",
    "index.lifecycle.rollover_alias": "{{myindex}}",
    "index.routing.allocation.include.boxtype": "all"
  }
}

#设置集群检测生命周期配置

vim cluster_settings.json.j2

{
  "persistent": { #永久设置
    "indices": {
      "lifecycle": {
        "poll_interval": "1h" 
      }
    }
  },
  "transient": {  #临时设置
    "indices": {
      "lifecycle": {
        "poll_interval": "1h"
      }
    }
  }
}

#手动绑定生命周期【注：一般不需要设置这项，作者只是提醒各位大佬，需要手动设置的索引，这样设置就OK】

vim settings.json.j2

{
  "settings": {
    "index.lifecycle.name": "{{myindex}}_policy",
    "index.lifecycle.rollover_alias": "{{myindex}}",
  }
}

2.tasks任务文件

cd roles/filebeat/tasks

#安装filebeat任务

vim install_filebeat.yml

---
- name: "useradd {{filebeat_user}}"
  user: name={{filebeat_user}} state=present
  become: yes

- name: "mkdir {{filebeat_dir}}"
  file: path={{filebeat_dir}} owner={{filebeat_user}} group={{filebeat_user}} mode=0755 state=directory
  become: yes

- name: "set applog config"
  template: src=app_log.yml.j2 dest=/tmp/app_log.yml
  become: yes

- name: "copy {{filebeat_tgz}} to fliebeate server"
  copy: src={{filebeat_tgz}} dest={{filebeat_dir}}/{{filebeat_tgz}}
  become: yes
  ignore_errors: yes

- name: "tar -xf {{filebeat_tgz}}"
  shell: |
    tar -xf {{filebeat_dir}}/{{filebeat_tgz}} -C {{filebeat_dir}}
    mv {{filebeat_dir}}/{{filebeat_tgz.split('.tar')[0]}} {{filebeat_dir}}/filebeat
  become: yes
  ignore_errors: yes

- name: "mkdir {{filebeat_dir}}/filebeat/{config,logs}"
  file: path={{filebeat_dir}}/filebeat/{{item}} state=directory
  with_items:
    - config
    - logs
  become: yes

- name: "copy filebeat.yml app_log.yml to {{filebeat_dir}}/filebeat/config"
  template: src={{item}} dest={{filebeat_dir}}/filebeat/config/{{item.split('.j2')[0]}}
  with_items:
    - filebeat.yml.j2
    - app_log.yml.j2
  become: yes

- name: "chown -R yzj:yzj {{filebeat_dir}}/filebeat"
  file: path={{filebeat_dir}}/filebeat owner={{filebeat_user}} group={{filebeat_user}} mode=0755 state=directory recurse=yes
  become: yes

- name: "reomve fields.yml"
  file: path={{filebeat_dir}}/filebeat/fields.yml state=absent
  become: yes

- name: "start filebeat"
  shell: su - {{filebeat_user}} -c "nohup {{filebeat_dir}}/filebeat/filebeat -c {{filebeat_dir}}/filebeat/config/filebeat.yml >> {{filebeat_dir}}/filebeat/logs/filebeat.log 2>&1 &" && sleep 10
  become: yes

- name: "check filebeat server"
  shell: ps -ef|grep  filebeat|grep -v grep|awk '{print $2}'
  register: filebeat_process
  become: yes

- name: "print filebeat_process"
  debug: "msg={{filebeat_process}}"

- name: "remove {{filebeat_tgz}}"
  file: path={{filebeat_dir}}/{{filebeat_tgz}} state=absent
  become: yes

#创建别名索引、生命周期、模板等

vim create_applog.yml

---
- name: "create index {{myindex}}"
  uri:
    url: "http://{{elasticsearch_host.split(',')[0]}}/%3C{{myindex}}-%7Bnow%2Fd%7D-000001%3E"
    method: "PUT"
    user: "{{esusername}}"
    password: "{{espassword}}"
    body_format: json
    force_basic_auth: yes
    status_code: 200
    body: "{{lookup('template','myindex.json.j2')}}"
  ignore_errors: yes

- name: "create  {{myindex}} policy"
  uri:
    url: "http://{{elasticsearch_host.split(',')[0]}}/_ilm/policy/{{myindex}}_policy"
    method: "PUT"
    user: "{{esusername}}"
    password: "{{espassword}}"
    body_format: json
    force_basic_auth: yes
    status_code: 200
    body: "{{lookup('template','policy.json.j2')}}"

- name: "create {{myindex}} template"
  uri:
    url: "http://{{elasticsearch_host.split(',')[0]}}/_template/{{myindex}}_template"
    method: "PUT"
    user: "{{esusername}}"
    password: "{{espassword}}"
    body_format: json
    force_basic_auth: yes
    status_code: 200
    body: "{{lookup('template','template.json.j2')}}"

##根据之前现有环境，该索引是否需要手动执行，需要则放开
#- name: "bind  {{myindex}} policy"
#  uri:
#    url: "http://{{elasticsearch_host.split(',')[0]}}/{{myindex}}/_settings"
#    method: "PUT"
#    user: "{{esusername}}"
#    password: "{{espassword}}"
#    body_format: json
#    force_basic_auth: yes
#    status_code: 200
#    body: "{{lookup('template','settings.json.j2')}}"


- name: "settings  {{myindex}} policy"
  uri:
    url: "http://{{elasticsearch_host.split(',')[0]}}/_cluster/settings"
    method: "PUT"
    user: "{{esusername}}"
    password: "{{espassword}}"
    body_format: json
    force_basic_auth: yes
    status_code: 200
    body: "{{lookup('template','cluster_settings.json.j2')}}"

- name: "print curl"
  debug:
    msg: "curl http://{{elasticsearch_host.split(',')[0]}}/_cat/nodes?pretty --user {{esusername}}:'{{espassword}}'"

#一些es操作基本操作【可以作为参考，非任务文件】

vim readme

#查索引情况，red、yellow、green、文档数量、大下、副本数、分片数
get _cat/indices

#查看现有模板
get _cat/templates

#查看已创建索引设置
get applog-2023.04.13-000001/_settings

#删除索引
delete  applog-2023.04.13-000001

#查看applog别名信息
get _alias/applog

#删除索引模板
delete _index_template/applog_template

#查看applog模板信息
GET _cat/templates/applog_template

#查看别名索引设置
get applog/_settings

#查看索引是否满足滚动策略
POST applog/_rollover?dry_run=true
{
  "conditions": {
    "max_age": "12h",
    "max_docs": 100,
    "max_size": "5gb"
  }
}

#满足滚动策略则执行滚动
POST applog/_rollover
{
  "conditions": {
    "max_age": "12h",
    "max_docs": 100,
    "max_size": "5gb"
  }
}


#创建applog别名索引：
PUT  %3Capplog-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "applog": {
      "is_write_index": true
    }
  },
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1
  }
}

#设置applog索引副本数
PUT applog/_settings
{
  "index":{
    "number_of_replicas": "0",
     "refresh_interval": "30s"
  }
}

#设置集群刷新生命周期频率，该参数会定时自动根据生命周期条件执行生命周期策略
PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1h"
  }
}

#查看集群设置
GET _cluster/settings

#生产生命周期applog_policy【根据生产实际需求修改条件参数】
PUT _ilm/policy/applog_policy
{
       "policy": {
         "phases": {
           "hot": {
             "actions": {
               "rollover": {
                 "max_age": "12h",
                 "max_docs": "20000000",
                 "max_size": "10gb"
               }
             }
           },
           "warm": {
             "min_age": "1d",
             "actions": {
               "allocate": {
                 "include": {
                   "box_type": "warm"
                 }
               }
             }
           },
           "cold": {
             "min_age": "2d",
             "actions": {
               "allocate": {
                 "include": {
                   "box_type": "cold"
                 }
               }
             }
           },
           "delete": {
             "min_age": "5d",
             "actions": {
               "delete": {}
             }
           }
         }
       }
}

#模板绑定索引生命周期
PUT _template/applog_template
{
  "order": 1,
  "index_patterns": [
    "applog-*"
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index.lifecycle.name": "applog_policy", #生命周期名称
    "index.lifecycle.rollover_alias": "applog", #执行滚动策略的别名索引
    "index.routing.allocation.include.boxtype": "all"
  }
}


#手动绑定索引绑定生命周期【一般不需要手动执行】
PUT applog/_settings
{
  "settings": {
     "number_of_replicas": 0,
    "number_of_shards": 1,
    "index.lifecycle.name": "applog_policy",
    "index.lifecycle.rollover_alias": "applog_policy",
    "index.number_of_replicas": 0
  }
}

#主任务文件

vim main.yml

---
- include_tasks: install_filebeat.yml
- include_tasks: create_applog.yml

三.拿取安装包文件

cd role/filebeat/files

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.6.1-linux-x86_64.tar.gz -O filebeat-8.6.1-linux-x86_64.tar.gz

整体ansible-playbook文件目录如下：

四.编写hosts文件，与roles文件在同级目录：

vim hosts

[filebeat]
filebeat1 ansible_ssh_host=192.168.126.128

五.编写运行yml文件,与roles文件在同级目录：

vim startinstall_filebeat.yml

---
- hosts: filebeat
  remote_user: yzj
  gather_facts: no
  become: no
  vars:
    - appnameinfo:
        - {servername: "systemlog",logpath: "/var/log/messge*"}
        - {servername: "mytestlog",logpath: "/kingdee/*.log"}
    - filebeat_dir: "/kingdee/yzj_monitor"
    - filebeat_tgz: "filebeat-8.6.1-linux-x86_64.tar.gz"
    - filebeat_user: "filebeat"
    - esusername: "logadmin"
    - espassword: "Kingdee@1203"
    - elasticsearch_host: "192.168.126.128:9200,192.168.126.129:9200,192.168.126.130:9200"
    - myindex: "huawei123"
    - number_of_replicas: 0
    - number_of_shards: 1

  roles:
    - role: filebeat

六.运行elasticsearch安装脚本：

ansible-playbook -i hosts startinstall_filebeat.yml

#查看状态：

curl http://192.168.126.128:9200/applog/_settings?pretty --user logadmin:Mytest@@@@@1203'

至此，ansible-playbook部署filebeat+elasticsearch+kibana已经完成，我们后面将进行下一章，挑战一下k8s使用ansible-playbook,进行部署；感谢各位读者大佬！！