PG数据库高可用方案改进_repmgr+postgresql+witness

最近买了本书，老外的Postgresql高可用。这本书方案比较全。对其中的repmgr章节进行了测试。测试的架构图为：

具体的就不多说了。

归档配置请参照

​​https://blog.51cto.com/lishiyan/3867038​​

一、安装包

pg_partman11-4.2.0-1.rhel7.x86_64.rpm
postgresql11-11.5-1PGDG.rhel7.x86_64.rpm
postgresql11-contrib-11.5-1PGDG.rhel7.x86_64.rpm
postgresql11-devel-11.5-1PGDG.rhel7.x86_64.rpm
postgresql11-libs-11.5-1PGDG.rhel7.x86_64.rpm
postgresql11-server-11.5-1PGDG.rhel7.x86_64.rpm
repmgr_11-5.2.1-1.rhel7.x86_64.rpm
repmgr_11-devel-5.2.1-1.rhel7.x86_64.rpm
timescaledb_11-1.4.0-1.rhel7.x86_64.rpm

二、主机hosts文件

# vi /etc/hosts
127.0.0.1   localhost 
10.10.10.111 pgha01
10.10.10.112 pgha02
10.10.10.113 pgwitness

配置10.10.10.1

三、操作系统安装包

# yum install libicu libicu-devel libxslt
# yum install perl-libs
# yum install python-psycopg2

四、创建repmgr目录

# mkdir /var/log/postgresql
# chown postgres.postgres /var/log/postgresql
# mkdir /etc/repmgr/11
# chown postgres.postgres /etc/repmgr

五、Postgresql环境变量

-bash-4.2$ cat .bash_profile 
[ -f /etc/profile ] && source /etc/profile
PGDATA=/var/lib/pgsql/11/data
export PGDATA
# If you want to customize your settings,
# Use the file below. This is not overridden
# by the RPMS.
[ -f /var/lib/pgsql/.pgsql_profile ] && source /var/lib/pgsql/.pgsql_profile

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

export PGPORT=5432
export PGHOME=/usr/pgsql-11
export PGDATA=/zdata/pg/pgmain
export PATH=$PGHOME/bin:$PATH
export MANPATH=$PGHOME/share/man:$MANPATH
export LANG=en_US.UTF-8
export DATE='date +"%Y%m%d%H%M"'
export LD_LIBRARY_PATH=$PGHOME/lib:$LD_LIBRARY_PATH
export PGHOST=$PGDATA
export PGUSER=postgres
export PGDATABASE=postgres

PATH=$PATH:$HOME/.local/bin:$HOME/bin:$PGHOME/bin
export PATH
-bash-4.2$

六、编辑repmgr配置文件

配置文件放置于/etc/repmgr/11目录

10.10.10.111
主机配置
-bash-4.2$ cat /etc/repmgr/11/repmgr.conf
node_id = 111
node_name = 'pgha01'
conninfo = 'host=pgha01 port=5432 dbname=repmgr user=repmgr'
data_directory = '/zdata/pg/pgmain'
pg_bindir = '/usr/pgsql-11/bin'
use_replication_slots = 'yes'
log_level = 'DEBUG'
log_file = '/var/log/postgresql/repmgr.log'
monitoring_history = 'true'
failover = 'automatic'
primary_visibility_consensus = 'true'
promote_command = '/usr/pgsql-11/bin/repmgr standby promote --siblings-follow --log-to-file'
follow_command = 'repmgr standby follow -f /etc/repmgr/11/repmgr.conf -W --upstream-node-id=%n '
ssh_options='-q -o StrictHostKeyChecking=no -o ConnectTimeout=10'
event_notification_command='bash /usr/local/bin/repmgr_promote_self.sh %n'
event_notifications='standby_promote'
service_start_command = 'sudo systemctl start postgresql-11'
service_stop_command = 'sudo systemctl stop postgresql-11'
service_restart_command = 'sudo systemctl restart postgresql-11'
service_reload_command = 'sudo systemctl reload postgresql-11'


10.10.10.112
主机配置
-bash-4.2$ more repmgr.conf
node_id = 112
node_name = 'pgha02'
conninfo = 'host=pgha02 port=5432 dbname=repmgr user=repmgr'
data_directory = '/zdata/pg/pgmain'
pg_bindir = '/usr/pgsql-11/bin'
use_replication_slots = 'yes'
log_level = 'DEBUG'
log_file = '/var/log/postgresql/repmgr.log'
monitoring_history = 'true'
failover = 'automatic'
primary_visibility_consensus = 'true'
promote_command = '/usr/pgsql-11/bin/repmgr standby promote --siblings-follow --log-to-file'
follow_command = 'repmgr standby follow -f /etc/repmgr/11/repmgr.conf -W --upstream-node-id=%n '
ssh_options='-q -o StrictHostKeyChecking=no -o ConnectTimeout=10'
event_notification_command='bash /usr/local/bin/repmgr_promote_self.sh %n'
event_notifications='standby_promote'
service_start_command = 'sudo systemctl start postgresql-11'
service_stop_command = 'sudo systemctl stop postgresql-11'
service_restart_command = 'sudo systemctl restart postgresql-11'
service_reload_command = 'sudo systemctl reload postgresql-11'


10.10.10.113
主机配置
-bash-4.2$ more /etc/repmgr/11/repmgr.conf
node_id = 113
node_name = 'pgwitness'
conninfo = 'host=pgwitness port=5432 dbname=repmgr user=repmgr'
data_directory = '/zdata/pg/pgmain'
pg_bindir = '/usr/pgsql-11/bin'
log_level = 'DEBUG'
log_file = '/var/log/postgresql/repmgr.log'
monitoring_history = 'true'
primary_visibility_consensus = 'true'



###### 设置三台机器的账号密码 ######
-bash-4.2$ cat /var/lib/pgsql/.pgpass 
10.10.10.111:5432:repmgr:repmgr:repmgr
10.10.10.112:5432:repmgr:repmgr:repmgr
10.10.10.113:5432:repmgr:repmgr:repmgr
10.10.10.111:5432:postgres:postgres:123456
10.10.10.112:5432:postgres:postgres:123456
10.10.10.113:5432:postgres:postgres:123456
127.0.0.1:5432:postgres:postgres:123456
localhost:5432:postgres:postgres:123456
127.0.0.1:5432:repmgr:repmgr:repmgr
localhost:5432:repmgr:repmgr:repmgr

七、postgresql用户的sudo权限

我的网卡是ens37

postgres用户sudo权限目的是，数据库切换时执行root用户权限命令
# vi /etc/sudoers
## Same thing without a password
# %wheel  ALL=(ALL) NOPASSWD: ALL
postgres  ALL=(ALL) NOPASSWD: ALL
postgres ALL = NOPASSWD: \
/usr/bin/systemctl stop postgresql-11, \
/usr/bin/systemctl start postgresql-11, \
/usr/bin/systemctl restart postgresql-11, \
/usr/bin/systemctl reload postgresql-11, \
/usr/sbin/ip addr add 10.10.10.100/32 dev ens37 label ens37\:pg, \
/usr/sbin/ip addr del 10.10.10.100/32 dev ens37 label ens37\:pg, \
/usr/sbin/arping -b -A -c 3 -I ens37 10.10.10.100

八、配置postgresql免密

postgres用户ssh免密
postgres用户ssh免密,实现数据库切换时，数据库集群内其它数据库节点执行远程调用命令。
数据库所有节点，执行以下操作：
生成ssh秘钥
所有节点
# su - postgres
$ ssh-keygen -t rsa -N ''
$ ssh-copy-id postgres@pgha01
$ ssh-copy-id postgres@pgha02
$ ssh-copy-id postgres@pgwitness
验证SSH免密
$ ssh pgha01 date;ssh pgha02 date;ssh pgwitness date

九、VIP切换脚本

VIP切换脚本
数据库节点pgha01,pgha02，执行以下操作：
[root@pgha01 11]# more /usr/local/bin/repmgr_promote_self.sh 
#!/usr/bin/env bash
# pgha01 :10.10.10.111
# pgha02 : 10.10.10.112
# VIP network ens37
VIP=10.10.10.100
IFACE=ens37
PATH=/usr/pgsql-11/bin:$PATH
VIP_CONF="${VIP}/32 dev ${IFACE} label ${IFACE}:pg"
##IPDOWN
ssh postgres@pgha02 sudo ip addr del ${VIP_CONF}
sleep 5;
##IPUP
sudo /usr/sbin/ip addr add ${VIP_CONF}
sudo /usr/sbin/arping -b -A -c 3 -I ${IFACE} ${VIP}
exit 0
执行权限
# chmod +x /usr/local/bin/repmgr_promote_self.sh
[root@pgha01 11]# 
执行权限
# chmod +x /usr/local/bin/repmgr_promote_self.sh

十、初始化

创建primary节点
数据库primary节点，执行以下操作：
primary节点初始化
 initdb -D $PGDATA -U postgres --locale=en_US.UTF8 -E UTF8 -k
 
配置postgresql.conf文件

6.3.postgresql.conf
postgresql.conf主要修改项：
postgresql.conf主要修改项：
listen_addresses = '*'
port = 15432   # 非默认端口
max_connections = 3000   # OLTP 3000，OLAP 300
tcp_keepalives_idle = 60 
tcp_keepalives_interval = 10
tcp_keepalives_count = 6
shared_buffers = 256MB   # 物理内存25%
huge_pages = on
work_mem = 16MB      # oltp,建议16MB，OLAP建议64MB
effective_cache_size = 1256MB     # 物理内存75% 
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 4     # HDD 4，SSD 1.1
effective_io_concurrency = 2   # HDD 2，SSD，200
min_wal_size = 2GB
max_wal_size = 8GB     # OLTP 8GB，OLAP，16GB
maintenance_work_mem = 2GB   # 生产环境大内存环境建议2GB
wal_log_hints = on
shared_preload_libraries = 'repmgr'
full_page_writes = on
archive_mode = on
archive_command = '/bin/true'
wal_keep_segments = 3000  # 根据磁盘空间调整
hot_standby_feedback = on   # 开启后建议监控表膨胀
log_destination = 'stderr'
logging_collector = on
log_directory = 'log'
log_filename = 'postgresql-%a.log'
log_file_mode = 0600
log_truncate_on_rotation = on
log_rotation_age = 1d
log_rotation_size = 0
log_min_duration_statement = 5000
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0
track_io_timing = on
autovacuum = on
log_autovacuum_min_duration = 0
shared_preload_libraries = 'pg_stat_statements,repmgr' # 根据情况添加

配置pg_hba.conf


# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:

host    all             all             127.0.0.1/32            trust
host    all             repmgr        10.10.10.0/24        trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
host    all             all             0.0.0.0/0               md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust
host    replication     repmgr       10.10.10.0/24           trust

十一、创建repmgr用户和数据库

启动数据库
[root@pgha01 ~]# systemctl start postgresql-11 
[root@pgha01 ~]# systemctl status postgresql-11 
创建用户
[root@pgha01 ~]# su - postgres
$ psql -h localhost
 
create user repmgr replication login encrypted WITH SUPERUSER REPLICATION password 'repmgr';
CREATE DATABASE repmgr OWNER repmgr;

十二、注册repmgr到主节点

注册primary节点
注册主节点
[root@pgha01 ~]# su - postgres
$ repmgr primary register
显示集群状态
$ repmgr cluster show

十三、启动repmgr守护进程

启动primary节点repmgr服务
[root@pgha01 ~]# systemctl start repmgr11
[root@pgha01 ~]# systemctl status repmgr11

十四、创建备库节点并启动，repmgr备库注册

创建standby节点
数据库standby节点，执行以下操作：
克隆standby节点
[root@pgha02 ~]# su - postgres
$ repmgr standby clone -h pgha01 -U repmgr -d repmgr
启动standby数据库服务
[root@pgha02 ~]# systemctl start postgresql-11
[root@pgha02 ~]# systemctl status postgresql-11
注册standby节点

注册standby节点
[root@pgha02 ~]# su - postgres
$ repmgr standby register
检查集群状态
$ repmgr cluster show
启动standby数据库repmgr进程
[root@pgha02 ~]# systemctl start repmgr11
[root@pgha02 ~]# systemctl status repmgr11

十五、创建witness数据库

创建witness节点
数据库witness节点，执行以下操作：
初始化witness节点数据库
[root@pgwitness usr]# su - postgres
-bash-4.2$ initdb -D $PGDATA -U postgres --locale=en_US.UTF8 -E UTF8 -k
witness节点postgresql.conf
shared_preload_libraries = 'repmgr'  # witness节点仅保留此扩展


witness节点pg_hba.conf
# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             repmgr        10.10.10.0/24        trust
host    all             all             0.0.0.0/0               md5
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

witness节点启动数据库服务
[root@pgwitness ~]# systemctl start postgresql-11
[root@pgwitness ~]# systemctl status postgresql-11

witness节点创建repmgr用户
[root@pgwitness ~]# su - postgres
$ psql
postgres=# CREATE USER repmgr WITH SUPERUSER REPLICATION;
CREATE ROLE
postgres=# CREATE DATABASE repmgr OWNER repmgr;
CREATE DATABASE
postgres=# \q
$
注册witness节点
注册witness节点
[root@pgwitness ~]# su - postgres
$ repmgr witness register -h pgha01 -d repmgr
检查集群状态
$ repmgr cluster show

启动witness节点repmgr服务
[root@pgwitness ~]# systemctl start repmgr11
[root@pgwitness ~]# systemctl status repmgr11

十六、主库上添加VIP

10.primary节点初始化VIP
数据库primary节点，执行以下操作
启动VIP
[root@pgha01 ~]# su - postgres
$  sudo ip addr add 10.10.10.100/32 dev ens37 label ens37:pg

检查VIP
$ ifconfig

十七、switchover与failover

$ repmgr standby switchover -f /etc/repmgr/11/repmgr.conf 
$ repmgr standby switchover -f /etc/repmgr/11/repmgr.conf --siblings-follow --repmgrd-force-unpause


主节点宕机后，启动一次后，再关闭一次。然后进行rejoin
repmgr -f /etc/repmgr/11/repmgr.conf node rejoin -d 'host=10.10.10.112 user=repmgr dbname=repmgr connect_timeout=2' --force-rewind --dry-run --verbose
repmgr -f /etc/repmgr/11/repmgr.conf node rejoin -d 'host=10.10.10.112 user=repmgr dbname=repmgr connect_timeout=2' --force-rewind  --verbose

检查集群状态
$ repmgr cluster show