环境说明
名词解释
- 中控机器 存储 OceanBase 数据库安装包和集群配置信息的机器。
- 目标机器 安装 OceanBase 数据库的机器。
- OBD OceanBase Deployer，OceanBase 开源软件的安装部署工具，简称为 OBD。
- OBProxy OceanBase Database Proxy，OceanBase 高性能反向代理服务器，简称为 OBProxy
-
机器列表
| IP地址 | 主机名 | 机器配置 | 系统版本 | 内核版本 | 角色 |
|---|---|---|---|---|---|
| 192.168.31.150 | obcontrol | 1C2G | CentOS 7.5 | Linux3.10.0 | OBD，中控机 |
| 192.168.31.151 | z1 | 2C8G | CentOS 7.5 | Linux3.10.0 | OBserver |
| 192.168.31.152 | z2 | 2C8G | CentOS 7.5 | Linux3.10.0 | OBserver |
| 192.168.31.153 | z3 | 2C8G | CentOS 7.5 | Linux3.10.0 | OBserver |
部署准备
配置hosts
四台机器都要配置hosts
[root@obcontrol ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.31.150 obcontrol
192.168.31.151 z1
192.168.31.152 z2
192.168.31.153 z3
配置免密登录
以下操作在中控机进行
ssh-keygen -t rsa
scp -p ~/.ssh/id_rsa.pub root@z1:/root/.ssh/authorized_keys
scp -p ~/.ssh/id_rsa.pub root@z2:/root/.ssh/authorized_keys
scp -p ~/.ssh/id_rsa.pub root@z3:/root/.ssh/authorized_keys
配置NTP时钟同步
服务端（中控机）上配置
cat /etc/ntp.conf
server 127.127.1.0
service ntpd restart
chkconfig ntpd on
客户端（OBserver）上配置
cat /etc/ntp.conf
server 192.168.31.150 prefer
service ntpd restart
chkconfig ntpd on
查看NTP信息
[root@z2 .ssh]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
-time.cloudflare 10.211.8.4 3 u 34 64 3 139.541 -3.923 0.495
*tock.ntp.infoma .GPS. 1 u 35 64 3 223.843 2.985 0.957
+119.28.183.184 100.122.36.196 2 u 35 64 3 40.520 0.713 0.576
+119.28.206.193 100.122.36.196 2 u 31 64 3 45.111 1.564 0.914
obcontrol .INIT. 16 u - 64 0 0.000 0.000 0.000
配置 limits.conf
所有机器进行如下配置：
cat /etc/security/limits.conf
root soft nofile 655350
root hard nofile 655350
* soft nofile 655350
* hard nofile 655350
* soft stack 20480
* hard stack 20480
* soft nproc 655360
* hard nproc 655360
* soft core unlimited
* hard core unlimited
配置 sysctl.conf
由于只是测试用，我们只配置fs.aio-max-nr
cat /etc/sysctl.conf
# # for oceanbase
# ## 修改内核异步 I/O 限制
fs.aio-max-nr=1048576
sysctl -p
fs.aio-max-nr = 1048576
关闭防火墙
关闭防火墙
systemctl stop firewalld.service
永久关闭防火墙
systemctl disable firewalld.service
查看防火墙状态
systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
Nov 11 11:56:01 z3 systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 11 11:56:14 z3 systemd[1]: Started firewalld - dynamic firewall daemon.
Nov 11 15:55:00 z3 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Nov 11 15:55:01 z3 systemd[1]: Stopped firewalld - dynamic firewall daemon.
关闭SELinux
vim /etc/selinux/config
SELINUX=disabled
重启服务器，使更改生效
setenforce 0
查看更改是否生效
sestatus
创建用户
groupadd -g 300 ob
useradd -u 300 -g ob ob
passwd ob
在线部署
部署模式
本次测试采用三副本部署模式，共使用四台机器。
- 使用一台机器部署 OBProxy、Grafana、Prometheus。
- 使用三台机器部署 OceanBase 集群。
安装 all-in-one package
在官网下载all-in-one package https://open.oceanbase.com/softwareCenter/community 解压安装包
tar zxvf oceanbase-all-in-one.4.0.0.0-beta-100120221102135736.el7.x86_64.tar.gz
安装
cd oceanbase-all-in-one/bin/
./install.sh
name: grafana
version: 7.5.17
release:1
arch: x86_64
md5: 1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6
add /software/oceanbase-all-in-one/rpms/grafana-7.5.17-1.el7.x86_64.rpm to local mirror
name: obagent
version: 1.2.0
release:4.el7
arch: x86_64
md5: 0e8f5ee68c337ea28514c9f3f820ea546227fa7e
add /software/oceanbase-all-in-one/rpms/obagent-1.2.0-4.el7.x86_64.rpm to local mirror
name: obproxy-ce
version: 4.0.0
release:5.el7
arch: x86_64
md5: de53232a951184fad75b15884458d85e31d2f6c3
add /software/oceanbase-all-in-one/rpms/obproxy-ce-4.0.0-5.el7.x86_64.rpm to local mirror
name: oceanbase-ce
version: 4.0.0.0
release:100000272022110114.el7
arch: x86_64
md5: 42611dc51ca9bb28f36e60e4406ceea4a74914c7
add /software/oceanbase-all-in-one/rpms/oceanbase-ce-4.0.0.0-100000272022110114.el7.x86_64.rpm to local mirror
name: oceanbase-ce-libs
version: 4.0.0.0
release:100000272022110114.el7
arch: x86_64
md5: 188919f8128394bf9b62e3989220ded05f1d14da
add /software/oceanbase-all-in-one/rpms/oceanbase-ce-libs-4.0.0.0-100000272022110114.el7.x86_64.rpm to local mirror
name: prometheus
version: 2.37.1
release:10000102022110211.el7
arch: x86_64
md5: 58913c7606f05feb01bc1c6410346e5fc31cf263
add /software/oceanbase-all-in-one/rpms/prometheus-2.37.1-10000102022110211.el7.x86_64.rpm to local mirror
Disable remote ok
#####################################################################
Install Finished
=====================================================================
Setup Environment: source ~/.oceanbase-all-in-one/bin/env.sh
Quick Start: obd demo
More Details: obd -h
=====================================================================
检查是否安装成功
# which obd
/root/.oceanbase-all-in-one/obd/usr/bin/obd
# which obclient
/root/.oceanbase-all-in-one/obclient/u01/obclient/bin/obclient
部署 OceanBase 集群
编辑default-example.yaml 位置在/root/.oceanbase-all-in-one/conf/autodeploy
## Only need to configure when remote login is required
user:
username: ob
password: ob123
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
oceanbase-ce:
servers:
- name: z1
# Please don't use hostname, only IP can be supported
ip: 192.168.31.151
- name: z2
ip: 192.168.31.152
- name: z3
ip: 192.168.31.153
global:
# Please set devname as the network adaptor's name whose ip is in the setting of severs.
# if set severs as "127.0.0.1", please set devname as "lo"
# if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
devname: enp0s3
cluster_id: 1
redo_dir: /oceanbase/redo
data_dir: /oceanbase/data
# please set memory limit to a suitable value which is matching resource.
memory_limit: 6G
system_memory: 4G
stack_size: 512K
max_syslog_file_count: 4
# observer cluster name, consistent with obproxy's cluster_name
appname: ob_cluster
root_password: 123456
proxyro_password: 123456
z1:
mysql_port: 2881
rpc_port: 2882
home_path: /oceanbase/home/observer
zone: zone1
z2:
mysql_port: 2881
rpc_port: 2882
home_path: /oceanbase/home/observer
zone: zone2
z3:
mysql_port: 2881
rpc_port: 2882
home_path: /oceanbase/home/observer
zone: zone3
obproxy-ce:
servers:
- 192.168.31.150
global:
listen_port: 2883
home_path: /oceanbase/home/obproxy
# oceanbase root server list
# format: ip:mysql_port,ip:mysql_port
rs_list: 192.168.31.151:2881;192.168.31.152:2881;192.168.31.153:2881
enable_cluster_checkout: false
# observer cluster name, consistent with oceanbase-ce's appname
cluster_name: ob_cluster
obproxy_sys_password: 123456
observer_sys_password: 123456
在中控机部署OceanBase集群
obd cluster autodeploy ob_cluster -c default-example.yaml
install oceanbase-ce-4.0.0.0 for local ok
install obproxy-ce-4.0.0 for local ok
Cluster param config check ok
Open ssh connection ok
Generate observer configuration ok
Generate obproxy configuration ok
install oceanbase-ce-4.0.0.0 for local ok
install obproxy-ce-4.0.0 for local ok
+--------------------------------------------------------------------------------------------+
| Packages |
+--------------+---------+------------------------+------------------------------------------+
| Repository | Version | Release | Md5 |
+--------------+---------+------------------------+------------------------------------------+
| oceanbase-ce | 4.0.0.0 | 100000272022110114.el7 | 42611dc51ca9bb28f36e60e4406ceea4a74914c7 |
| obproxy-ce | 4.0.0 | 5.el7 | de53232a951184fad75b15884458d85e31d2f6c3 |
+--------------+---------+------------------------+------------------------------------------+
Repository integrity check ok
Parameter check ok
Open ssh connection ok
Cluster status check ok
Initializes observer work home ok
Initializes obproxy work home ok
Remote oceanbase-ce-4.0.0.0-100000272022110114.el7-42611dc51ca9bb28f36e60e4406ceea4a74914c7 repository install ok
Remote oceanbase-ce-4.0.0.0-100000272022110114.el7-42611dc51ca9bb28f36e60e4406ceea4a74914c7 repository lib check !!
Remote obproxy-ce-4.0.0-5.el7-de53232a951184fad75b15884458d85e31d2f6c3 repository install ok
Remote obproxy-ce-4.0.0-5.el7-de53232a951184fad75b15884458d85e31d2f6c3 repository lib check ok
Try to get lib-repository
install oceanbase-ce-libs-4.0.0.0 for local ok
Remote oceanbase-ce-libs-4.0.0.0-100000272022110114.el7-188919f8128394bf9b62e3989220ded05f1d14da repository install ok
Remote oceanbase-ce-4.0.0.0-100000272022110114.el7-42611dc51ca9bb28f36e60e4406ceea4a74914c7 repository lib check ok
ob_cluster deployed
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1007: (192.168.31.151) The recommended number of max user processes is 12288 (Current value: 4096)
[WARN] (192.168.31.151) clog and data use the same disk (/)
[WARN] OBD-1007: (192.168.31.152) The recommended number of max user processes is 12288 (Current value: 4096)
[WARN] (192.168.31.152) clog and data use the same disk (/)
[WARN] OBD-1007: (192.168.31.153) The recommended number of max user processes is 12288 (Current value: 4096)
[WARN] (192.168.31.153) clog and data use the same disk (/)
Check before start obproxy ok
Start observer ok
observer program health check ok
Connect to observer ok
Initialize cluster ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize cluster ok
Wait for observer init ok
+--------------------------------------------------+
| observer |
+----------------+---------+------+-------+--------+
| ip | version | port | zone | status |
+----------------+---------+------+-------+--------+
| 192.168.31.151 | 4.0.0.0 | 2881 | zone1 | ACTIVE |
| 192.168.31.152 | 4.0.0.0 | 2881 | zone2 | ACTIVE |
| 192.168.31.153 | 4.0.0.0 | 2881 | zone3 | ACTIVE |
+----------------+---------+------+-------+--------+
obclient -h192.168.31.151 -P2881 -uroot -p123456 -Doceanbase
+--------------------------------------------------+
| obproxy |
+----------------+------+-----------------+--------+
| ip | port | prometheus_port | status |
+----------------+------+-----------------+--------+
| 192.168.31.150 | 2883 | 2884 | active |
+----------------+------+-----------------+--------+
obclient -h192.168.31.150 -P2883 -uroot@proxysys -p123456 -Doceanbase
ob_cluster running
连接 OceanBase 数据库
obclient -h192.168.31.150 -P2883 -uroot@proxysys -p123456
ERROR 2027 (HY000): received malformed packet
[root@obcontrol autodeploy]# obclient -h192.168.31.150 -P2883 -uroot -p123456
Welcome to the OceanBase. Commands end with ; or \g.
Your OceanBase connection id is 12
Server version: OceanBase_CE 4.0.0.0 (r100000272022110114-6af7f9ae79cd0ecbafd4b1b88e2886ccdba0c3be) (Built Nov 1 2022 14:57:18)
Copyright (c) 2000, 2018, OB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient [(none)]>
obclient [(none)]>
obclient [(none)]>
obclient [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| LBACSYS |
| mysql |
| oceanbase |
| ORAAUDITOR |
| SYS |
| test |
+--------------------+
7 rows in set (0.029 sec)
obclient [(none)]> use oceanbase;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
obclient [oceanbase]> show tables;
+-------------------------------------------+
| Tables_in_oceanbase |
+-------------------------------------------+
| CDB_INDEXES |
| CDB_IND_COLUMNS |
| CDB_IND_PARTITIONS |
| CDB_IND_SUBPARTITIONS |
| CDB_OBJECTS |
| CDB_OB_ARCHIVELOG |
问题梳理
问题一
[ERROR] (192.168.31.151) /home not enough disk space. (Avail: 10.2G, Need: 50.0G).
[ERROR] (192.168.31.152) /home not enough disk space. (Avail: 10.2G, Need: 50.0G).
[ERROR] (192.168.31.153) /home not enough disk space. (Avail: 10.2G, Need: 50.0G).
原因是observer三个节点home_path设置在/home目录下，而由于安装系统时候/home目录只设置了10G空间。 解决方法： 修改default-example.yaml文件的home_path配置，寻找空间大于50G的目录。
问题二
[WARN] OBD-1007: (192.168.31.151) The recommended number of max user processes is 12288 (Current value: 4096)
[ERROR] OBD-2000: (192.168.31.151) not enough memory. (Free: 7.0G, Need: 8.0G)
[WARN] (192.168.31.151) clog and data use the same disk (/)
[WARN] OBD-1007: (192.168.31.152) The recommended number of max user processes is 12288 (Current value: 4096)
[ERROR] OBD-2000: (192.168.31.152) not enough memory. (Free: 7.0G, Need: 8.0G)
[WARN] (192.168.31.152) clog and data use the same disk (/)
[WARN] OBD-1007: (192.168.31.153) The recommended number of max user processes is 12288 (Current value: 4096)
[ERROR] OBD-2000: (192.168.31.153) not enough memory. (Free: 7.0G, Need: 8.0G)
[WARN] (192.168.31.153) clog and data use the same disk (/)
解决方法 每台observer最大物理内存8G，memory_limit设置为8G。安装提示内存不足，修改default-example.yaml memory_limit: 6G。
问题三
[ERROR] Deploy "ob_cluster" is deployed. You could not deploy an deployed cluster.
See https://www.oceanbase.com/product/ob-deployer/error-codes .
解决方法 重新执行部署之前，需要对部署一半的集群进行清理。
obd cluster destroy ob_cluster
参考文档： https://www.oceanbase.com/docs https://zhuanlan.zhihu.com/p/448637597
