​

背景

本来自建的单机ES提供搜索服务，随着数据的变多，性能已经成为严重的问题，现在准备迁移到腾讯云ES集群

实战示例

1. 购买腾讯云白金版ES集群7.5.1

集群自带 es+kibana 但是不带logstash，所以还要自己安装一个

2. 下载logstash7.5.1+JDK

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.tar.gz
wget https://mirrors.yangxingzhen.com/jdk/jdk-11.0.1_linux-x64_bin.tar.gz

3.配置JDK 设置环境变量

tar xvf jdk-11.0.1_linux-x64_bin.tar.gz
mv jdk-11.0.1 /usr/java
vim .bash_profile
export JAVA_HOME=/usr/java export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

source .bash_profile

4.验证jdk安装是否成功

java -version

5.安装logstash

tar -xvf logstash-7.5.1.tar.gz
mv logstash-7.5.1 /usr/local/logstash

6.配置mysql-connector-java

下载jar包，放入logstash安装目录的tools目录下

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.23/mysql-connector-java-8.0.23.jar

7.配置文件修改

input {
       jdbc {
               jdbc_connection_string => "jdbc:mysql://localhost:3306/dbname?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull"

               jdbc_user => "es_igper"
               jdbc_password => "123456"

               jdbc_driver_library => "/usr/local/logstash/tools/mysql-connector-java-8.0.23.jar"

               jdbc_driver_class => "com.mysql.jdbc.Driver"
               jdbc_paging_enabled => "true"
               jdbc_page_size => "50000"


               clean_run => false
               use_column_value => true
               tracking_column => id
               record_last_run => true
               last_run_metadata_path => "/usr/local/logstash/config/igp/run/live_v1_run"
               
               statement => "select * from live where live.id>(:sql_last_value-5000) limit 10000"

               schedule => "* * * * *"

               type => "live"
       }
}
filter {

}
output {
if [type]=="live" {
       elasticsearch {
               hosts => ["localhost:9200"]
               index => "live_v1"
               document_id => "%{id}"
               user => "elastic"
               password => "es1234"
       }
       stdout {
               codec => json_lines
       }
}
}

因为腾讯云的ES集群开始了认证模式，所以output要加入账号密码

user => "elastic"
password => "es1234"

8.启动测试

nohup ./bin/logstash &