clickhouse 集群搭建步骤

编号

需求项

操作环境

需求细节内容

分配主机

1

操作系统要求

ch01,ch02,ch03

linux版本：CentOS 7

如果使用VMWare，则关闭vmware的overcommit

2

文件系统要求

ch01,ch02,ch03

建议为EXT4

3

CPU要求

ch01,ch02,ch03

1. 使用支持SSE4.2指令集的x86_64架构的CPU

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

SSE 4.2 supported

2. 建议使用 睿频加速 和 超线程 技术

4

内存要求

ch01,ch02,ch03

建议32G以上

5

SWAP要求

ch01,ch02,ch03

生产环境禁用swap file

6

建卷要求

ch01,ch02,ch03

1. 单独挂载数据卷：/data

2. clickhouse软件要求至少2G可用空间，数据空间根据实际确定

3. 建议使用 SSD，如果没有，7200转SATA硬盘也可以

7

关闭文件系统/分区的atime选项

ch01,ch02,ch03

cat /etc/fstab，看到对应的分区是noatime ,nodiratime

8

关闭透明大页

ch01,ch02,ch03

cat /sys/kernel/mm/transparent_hugepage/enabled

应该看到[never]

如果这个文件不存在，则检查

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

如果2个文件都不存在，那么就是在系统内核中移除了THP。

9

禁用SELinux

ch01,ch02,ch03

当前生效： setenforce 0

重启后生效：

vi /etc/selinux/config

SELINUX=disabled

10

设置/etc/hosts

ch01,ch02,ch03

加入各clickhouse与zookeeper节点信息

vi /etc/hosts

192.168.13.81    ch01

192.168.13.82    ch02

192.168.13.83    ch03

192.168.13.100    zk01

192.168.13.119    zk02

192.168.13.120    zk03

11

NTP时间同步

ch01,ch02,ch03

crontab -e

* * * * * /usr/sbin/ntpdate 192.168.0.11

12

调整limit设置

ch01,ch02,ch03

vi /etc/security/limits.conf

clickhouse soft nproc 2047

clickhouse hard nproc 16384

clickhouse soft nofile 1024

clickhouse hard nofile 65536

防火墙实施

12

网络防火墙要求

ch01,ch02,ch03

clickhouse间双向开通8123,9000端口

zookeeper到clickhouse单向开通8123,9000端口

clickhouse到zookeeper单向开通2181端口

13

本地防火墙要求

ch01,ch02,ch03

关闭并禁用本地防火墙

systemctl stop firewalld

systemctl disable firewalld

IP要求

各zookeeper与clickhouse服务器尽量在相同网段

编号

需求项

操作环境

需求细节内容

1

下载rpm包

ch01,ch02,ch03

下载以下4个rpm包，并上传至服务器目录

clickhouse-client-21.8.14.5-2.noarch.rpm      

clickhouse-common-static-21.8.14.5-2.x86_64.rpm

clickhouse-server-21.8.14.5-2.noarch.rpm      

clickhouse-server-common-19.4.0-2.noarch.rpm  

2

安装rpm包

ch01,ch02,ch03

rpm -ivh clickhouse*

期间会要求输入default用户密码

Enter password for default user:

3

创建clickhouse目录

ch01,ch02,ch03

mkdir -p /data/prd/clickhouse/21.8.14/ch9000/log

mkdir -p /data/prd/clickhouse/21.8.14/ch9000/data

mkdir -p /data/prd/clickhouse/21.8.14/ch9000/tmp

chown -R clickhouse.clickhouse /data

4

修改配置文件

ch01,ch02,ch03

vi /etc/clickhouse-server/config.xml

# 修改以下参数

<level>warning</level>

<log>/data/prd/clickhouse/21.8.14/ch9000/log/clickhouse-server.log</log>

<errorlog>/data/prd/clickhouse/21.8.14/ch9000/log/clickhouse-server.err.log</errorlog>

<!-- Path to data directory, with trailing slash. -->

<path>/data/prd/clickhouse/21.8.14/ch9000/data/</path>

<!-- Path to temporary data for processing hard queries. -->

<tmp_path>/data/prd/clickhouse/21.8.14/ch9000/tmp/</tmp_path>

<listen_host>::</listen_host>

<interserver_http_host>IBG-ClickHouse-DB-CWP01</interserver_http_host>

5

启动并enable

clickhouse服务

ch01,ch02,ch03

systemctl start clickhouse-server

systemctl enable clickhouse-server

编号

需求项

操作环境

需求细节内容

安装java环境

1

下载java安装包

zk01,zk02,zk03

可以从Oracle官网或GitHub下载

https://www.oracle.com/java/technologies/downloads https://github.com/AdoptOpenJDK/openjdk8-binaries/releases

2

解压及软链接

zk01,zk02,zk03

tar -zxvf jdk-8u311-linux-x64.tar.gz -C /usr/local/

ln -s /usr/local/jdk1.8.0_311 /usr/local/java

3

设置java环境变量

zk01,zk02,zk03

vim /etc/profile

export JAVA_HOME=/usr/local/java

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH

source /etc/profile

4

查看版本

zk01,zk02,zk03

java -version

安装zookeeper

1

设置/etc/hosts

zk01,zk02,zk03

加入各zookeeper节点信息

vi /etc/hosts

192.168.13.100    zk01

192.168.13.119    zk02

192.168.13.120    zk03

2

下载zookeeper安装包

zk01,zk02,zk03

https://zookeeper.apache.org/releases.html

3

创建目录

zk01,zk02,zk03

mkdir -p  /data/app/deploy

mkdir -p  /data/app/data/zookeeper/data

mkdir -p /data/app/data/zookeeper/logs

4

解压及软链接

zk01,zk02,zk03

cd /data/app/deploy

tar -zxf apache-zookeeper-3.5.9-bin.tar.gz

ln -snf apache-zookeeper-3.5.9-bin zookeeper

5

配置环境变量

zk01,zk02,zk03

vim /etc/profile

export ZOOKEEPER_HOME=/data/app/deploy/zookeeper

export PATH=$PATH:$ZOOKEEPER_HOME/bin

source /etc/profile

1

添加配置文件

zk01,zk02,zk03

2

配置myid

zk01,zk02,zk03

echo 1 > /data/app/data/zookeeper/data/myid  # 节点1

echo 2 > /data/app/data/zookeeper/data/myid  # 节点2

echo 3 > /data/app/data/zookeeper/data/myid  # 节点3

3

配置zookeeper日志

zk01,zk02,zk03

vi zookeeper-env.sh

#!/usr/bin/env bash

#tip:custom configurationfile，do not amend the zkEnv.sh file

#chang the log dir and output of rolling file

ZOO_LOG_DIR=/data/app/logs/zookeeper

ZOO_LOG4J_PROP="INFO,ROLLINGFILE"

chmod 755 zookeeper-env.sh

4

修改日志输入形式

zk01,zk02,zk03

vi log4j.properties

zookeeper.root.logger=INFO, ROLLINGFILE

#zookeeper.root.logger=INFO, CONSOLE

# Max log file size of 10MB

log4j.appender.ROLLINGFILE.MaxFileSize=128MB

# uncomment the next line to limit number of backup files

log4j.appender.ROLLINGFILE.MaxBackupIndex=10

5

配置运行zookeeper的JVM

zk01,zk02,zk03

vi java.env

#!/usr/bin/env bash

#config the jvm parameter in a reasonable,note that shell be source so that do not need to use export

#set classpath,here do not set

#export CLASSPATH=""

#set jvm start parameter , also can set JVMFLAGS variable

SERVER_JVMFLAGS="-Xmx1G -Xms1G -Djute.maxbuffer=8388608 \

    -verbose:gc -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps \

    -Xloggc:/data/app/logs/zookeeper/zookeeper-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=16M

    -XX:MaxGCPauseMillis=50"

chmod 755 java.env

6

启动zookeeper服务

zk01,zk02,zk03

./bin/zkServer.sh start

7

配置为系统服务

zk01,zk02,zk03

vi /usr/lib/systemd/system/zookeeper.service

[Unit]

Description=Zookeeper Service unit Configuration

After=network.target

[Service]

Type=forking

Environment=JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk

ExecStart=/data/app/deploy/zookeeper/bin/zkServer.sh start

ExecStop=/data/app/deploy/zookeeper/bin/zkServer.sh stop

PIDFile=/data/app/deploy/zookeeper/zookeeper_server.pid

KillMode=none

User=dji

Group=dji

Restart=on-failure

[Install]

WantedBy=multi-user.target

需求项

操作环境

需求细节内容

配置集群信息

ch01,ch02,ch03

    <remote_servers>

        <!-- Test only shard config for testing distributed storage -->

        <ads_cluster>

            <shard>

                <!-- 数据自动同步 -->

                <internal_replication>true</internal_replication>

                <replica>

                    <host>ch01</host>

                    <port>9000</port>

                </replica>

                <replica>

                    <host>ch02</host>

                    <port>9000</port>

                </replica>

                <replica>

                    <host>ch03</host>

                    <port>9000</port>

                </replica>

            </shard>

        </ads_cluster>

    </remote_servers>

配置zookeeper信息

ch01,ch02,ch03

    <zookeeper>

        <node>

            <host>zk01</host>

            <port>2181</port>

        </node>

        <node>

            <host>zk02</host>

            <port>2181</port>

        </node>

        <node>

            <host>zk03</host>

            <port>2181</port>

        </node>

    </zookeeper>

复制表的参数替换

ch01,ch02,ch03

    <macros incl="macros" optional="true" />

    <macros>

        <shard>01</shard>

        <replica>replica01</replica>

    </macros>

建库测试

任意ch节点

create database hydb on cluster ads_cluster;

建表测试（不带参数）

ch01,ch02,ch03

CREATE TABLE hydb.hytest(EventDate DateTime,CounterID UInt32,UserID UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/01/hytest','replica01')PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID))SETTINGS index_granularity = 8192;

CREATE TABLE hydb.hytest(EventDate DateTime,CounterID UInt32,UserID UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/01/hytest','replica02')PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID))SETTINGS index_granularity = 8192;

CREATE TABLE hydb.hytest(EventDate DateTime,CounterID UInt32,UserID UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/01/hytest','replica03')PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID))SETTINGS index_granularity = 8192;

插入与查询

任意ch节点

insert into hydb.hytest VALUES (now(), 10000, 10000);

select * from hydb.hytest;

建表测试（带参数）

任意ch节点

CREATE TABLE hydb.events_local ON CLUSTER ads_cluster (ts_date Date,user_id Int64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/events_local','{replica}') PARTITION BY ts_date ORDER BY (ts_date);

需求项

命名规则

权限

操作目录

备注

DBA用户

default

所有权限

vi user.xml

# 修改

<access_management>1</access_management>

启用RBAC权限控制，可通过SQL执行创建/删除用户、角色及授权等操作

业务读写用户

业务名_rw

业务DB所有权限（待定）

CREATE USER ads_rw@'%' IDENTIFIED BY 'xxxxx' ON CLUSTER ads_cluster;

GRANT all ON ads_db.* TO ads_rw ON CLUSTER ads_cluster;

业务只读用户

业务名_r

业务DB只读权限

CREATE USER ads_r@'%' IDENTIFIED BY 'xxxxx' ON CLUSTER ads_cluster;

GRANT SELECT ON ads_db.* TO ads_r ON CLUSTER ads_cluster;

序号

参数

参数说明

默认值

建议值

备注

0

config.xml（前面已提及的参数不再重复解释）

1

max_connections

最大连接数

4096

4096

参考阿里云ClickHouse设置

2

max_server_memory_usage

ClickHouse 最大可用内存

0（Auto）

默认大小=内存大小 * max_server_memory_usage_to_ram_ratio

如果设置值大于计算值，会被减少至计算值

3

max_server_memory_usage_to_ram_ratio

ClickHouse可用内存占总内存的比例

0.9

设为0表示可用所有服务器内存

4

max_concurrent_queries

同时可处理的MergeTree相关表最大请求数

100

100

0表示不限制，参考阿里云ClickHouse设置

5

max_concurrent_insert_queries

同时处理的最大INSERT数

0

0

0表示不限制

6

max_concurrent_select_queries

同时处理的最大SELECT数

0

0

0表示不限制

7

max_concurrent_queries_for_user

每个用户同时可处理的MergeTree相关表最大请求数

0

0

0表示不限制

8

max_table_size_to_drop

如果MergeTree表的大小超过max_table_size_to_drop（以字节为单位），则无法使用 DROP 将其删除

50GB

0

0表示不限制

9

max_partition_size_to_drop

如果MergeTree分区的大小超过max_partition_size_to_drop（以字节为单位），则无法使用DROP将其删除

50GB

0

0表示不限制

10

query_log

结合log_queries=1，记录执行的语句

log_queries=0

审计需求，注意日志大小及性能影响。一般建成带TTL属性的表，自动清理数据

12

keep_alive_timeout

ClickHouse在关闭连接之前等待传入请求的秒数

3

300

参考阿里云ClickHouse设置

参考：

config.xml    https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings/

user.xml       https://clickhouse.com/docs/en/operations/settings/settings-profiles/