Nginx 实践案例：反向代理单台web;反向代理多组web并实现负载均衡-CFANZ编程社区

1. 架构及主机

架构说明：在Nginx 后面我们建好两组 web，其中一组为PC-Servers，模拟某个网站给PC电脑访问的；另外一组为M-Servers，模拟网站给手机和移动终端访问。我们在设置一台Nginx服务器实现访问不通域名，在各自的组内实现负载均衡轮询方式响应用户的请求，同时Nginx自带对后端服务器的健康检测，后端服务器不可用的时候会自动停止调度到故障的服务器上，充分保障了用户的体验。

Nginx 实践案例：反向代理单台web;反向代理多组web并实现负载均衡_nginx

# 六台主机
1  4台web服务器 ：
主机名：PC-WebServer-IP18
CentOS 8.4
IP:192.168.250.18
httpd web服务   页面内容 PC-WebServer-IP18 192.168.250.18

主机名：PC-WebServer-IP28
CentOS 8.4
IP:192.168.250.28
httpd web服务   页面内容 PC-WebServer-IP28 192.168.250.28

主机名：PC-WebServer-IP58
CentOS 8.4
IP:192.168.250.58
httpd web服务   页面内容 M-WebServer-IP58 192.168.250.58

主机名：PC-WebServer-IP68
CentOS 8.4
IP:192.168.250.68
httpd web服务   页面内容 M-WebServer-IP68 192.168.250.68

2  1台 Nginx 服务器 ：
主机名： Nginx-IP08
CentOS 8.4
IP:192.168.250.8/24
nginx version: nginx/1.21.6

3  1台client主机 ：
主机名：Client-IP172-18
CentOS 8.4
IP:172.16.0.18/24

2. 后端web主机的准备

基本思路：在两组四台服务器上分别安装好Apache，并定义好首页页面，确保后面测试直观显示效果。

# 基础环境包括CentOS操作系统的优化、防火墙关闭、同步时间等都要做好，我们按照规划的架构图对四台服务器进行分组并重新命名
# 修改服务器名称
[root@CentOS84-IP18 ]#hostnamectl set-hostname PC-WebServer-IP18
[root@CentOS84-IP18 ]#exit
[root@PC-WebServer-IP18 ]#

# 安装Apache 准备httpd服务
[root@PC-WebServer-IP18 ]#yum -y install httpd

# 用命令自动生成Apache页面文件，主机名写入到 indexTmp.html临时文件；IP地址追加到indexTmp.html临时文件；最后将这个indexTmp.html临时文件的内容合并到一行，并写入到 /var/www/html/index.html  这个Apache首页文件内。
[root@PC-WebServer-IP18 ]#hostname > /var/www/html/indexTmp.html
[root@PC-WebServer-IP18 ]#hostname -I >> /var/www/html/indexTmp.html
[root@PC-WebServer-IP18 ]#cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html
# 验证首页内容
[root@PC-WebServer-IP18 ]#cat /var/www/html/index.html
PC-WebServer-IP18 192.168.250.18

# 启动并设定开启自启 httpd 服务
[root@PC-WebServer-IP18 ]#systemctl  enable --now httpd
[root@PC-WebServer-IP18 ]#


##  下面用将上面的命令合并到一条命令内完成整个httpd的部署工作
# PC-WebServer-IP28 主机部署 httpd
[root@CentOS84-IP28 ]#hostnamectl set-hostname PC-WebServer-IP28
[root@CentOS84-IP28 ]#exit
[root@PC-WebServer-IP28 ]#yum -y install httpd;hostname > /var/www/html/indexTmp.html;hostname -I >> /var/www/html/indexTmp.html;cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html;systemctl  enable --now httpd
[root@PC-WebServer-IP28 ]#cat  /var/www/html/index.html
PC-WebServer-IP28 192.168.250.28
[root@PC-WebServer-IP28 ]#

# M-WebServer-IP58 主机部署 httpd
[root@CentOS84-IP58 ]#hostnamectl set-hostname M-WebServer-IP58
[root@CentOS84-IP58 ]#exit
[root@M-WebServer-IP58 ]#yum -y install httpd;hostname > /var/www/html/indexTmp.html;hostname -I >> /var/www/html/indexTmp.html;cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html;systemctl  enable --now httpd
[root@M-WebServer-IP58 ]#cat /var/www/html/index.html
M-WebServer-IP58 192.168.250.58
[root@M-WebServer-IP58 ]#

# M-WebServer-IP68 主机部署 httpd
[root@CentOS84-IP68 ]#hostnamectl set-hostname M-WebServer-IP68
[root@CentOS84-IP68 ]#exit
[root@M-WebServer-IP68 ]#yum -y install httpd;hostname > /var/www/html/indexTmp.html;hostname -I >> /var/www/html/indexTmp.html;cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html;systemctl  enable --now httpd
[root@M-WebServer-IP68 ]#cat /var/www/html/index.html
M-WebServer-IP68 192.168.250.68
[root@M-WebServer-IP68 ]#

######################################################################################
# 在终端客户端上不通过Nginx 反向道理功能直接访问这四台web服务器，确定网页内容
[root@Client-IP172-18 ]#curl 192.168.250.18                     
PC-WebServer-IP18 192.168.250.18
[root@Client-IP172-18 ]#curl 192.168.250.28                     
PC-WebServer-IP28 192.168.250.28
[root@Client-IP172-18 ]#curl 192.168.250.58                     
M-WebServer-IP58 192.168.250.58
[root@Client-IP172-18 ]#curl 192.168.250.68                     
M-WebServer-IP68 192.168.250.68
[root@Client-IP172-18 ]#

3. Nginx 服务器基础环境准备

# 基础环境包括CentOS操作系统的优化、防火墙关闭、同步时间等都要做好，我们按照规划的架构图对四台服务器进行分组并重新命名
[root@CentOS84-IP08 ]#
[root@CentOS84-IP08 ]#hostnamectl set-hostname Nginx-IP08
[root@CentOS84-IP08 ]#exit

# 查看CentOS8默认的 nginx 版本1.14  比较低了，查询网站已经到1.21，稳定版本1.20.2了
[root@Nginx-IP08 ]#yum info nginx
Name         : nginx
Epoch        : 1
Version      : 1.14.1
Release      : 9.module_el8.0.0+184+e34fea82
Architecture : x86_64
Size         : 570 k
Source       : nginx-1.14.1-9.module_el8.0.0+184+e34fea82.src.rpm
Repository   : AppStream
Summary      : A high performance web server and reverse proxy server
URL          : http://nginx.org/
License      : BSD
Description  : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
             : IMAP protocols, with a strong focus on high concurrency, performance and low
             : memory usage.

# 因为CentOS8自带的版本太低，我们从nginx 官网上复制 repo 文件，并写入到本机的相应目录下，准备yum 安装最新版本的nginx
[root@Nginx-IP08 ]#vim /etc/yum.repos.d/nginx.repo 
[root@Nginx-IP08 ]#cat /etc/yum.repos.d/nginx.repo    
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[root@Nginx-IP08 ]#yum info nginx
nginx stable repo                              23 kB/s |  36 kB     00:01    
nginx mainline repo                            50 kB/s |  95 kB     00:01    
Available Packages
Name         : nginx
Epoch        : 1
Version      : 1.21.6
Release      : 1.el8.ngx
Architecture : x86_64
Size         : 827 k
Source       : nginx-1.21.6-1.el8.ngx.src.rpm
Repository   : nginx-mainline
Summary      : High performance web server
URL          : https://nginx.org/
License      : 2-clause BSD-like license
Description  : nginx [engine x] is an HTTP and reverse proxy server, as well as
             : a mail proxy server.
# yum安装最新的 nginx
[root@Nginx-IP08 ]#yum -y install nginx
[root@Nginx-IP08 ]#rpm -qi nginx
Name        : nginx
Epoch       : 1
Version     : 1.21.6
Release     : 1.el8.ngx
Architecture: x86_64
Install Date: Mon 28 Mar 2022 05:20:25 AM CST
Group       : System Environment/Daemons
Size        : 2949219
License     : 2-clause BSD-like license
Signature   : RSA/SHA1, Tue 25 Jan 2022 11:45:07 PM CST, Key ID abf5bd827bd9bf62
Source RPM  : nginx-1.21.6-1.el8.ngx.src.rpm
Build Date  : Tue 25 Jan 2022 11:25:23 PM CST
Build Host  : ip-10-1-17-168.eu-central-1.compute.internal
Relocations : (not relocatable)
Vendor      : NGINX Packaging <nginx-packaging@f5.com>
URL         : https://nginx.org/
Summary     : High performance web server
Description :
nginx [engine x] is an HTTP and reverse proxy server, as well as
a mail proxy server.

# 默认的安装包等信息
[root@Nginx-IP08 ]#rpm -ql nginx
/etc/logrotate.d/nginx
/etc/nginx
/etc/nginx/conf.d
/etc/nginx/conf.d/default.conf
/etc/nginx/fastcgi_params
/etc/nginx/mime.types
/etc/nginx/modules
/etc/nginx/nginx.conf
/etc/nginx/scgi_params
/etc/nginx/uwsgi_params
/usr/lib/.build-id
/usr/lib/.build-id/ae
/usr/lib/.build-id/ae/80f1c107606755e59070162a192d5c7b250e37
/usr/lib/.build-id/ef
/usr/lib/.build-id/ef/61e35830a566768e73e0c62909fafa180ee175
/usr/lib/systemd/system/nginx-debug.service
/usr/lib/systemd/system/nginx.service
/usr/lib64/nginx
/usr/lib64/nginx/modules
/usr/libexec/initscripts/legacy-actions/nginx
/usr/libexec/initscripts/legacy-actions/nginx/check-reload
/usr/libexec/initscripts/legacy-actions/nginx/upgrade
/usr/sbin/nginx
/usr/sbin/nginx-debug
/usr/share/doc/nginx-1.21.6
/usr/share/doc/nginx-1.21.6/COPYRIGHT
/usr/share/man/man8/nginx.8.gz
/usr/share/nginx
/usr/share/nginx/html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html
/var/cache/nginx
/var/log/nginx
[root@Nginx-IP08 ]#rpm -qc nginx
/etc/logrotate.d/nginx
/etc/nginx/conf.d/default.conf
/etc/nginx/fastcgi_params
/etc/nginx/mime.types
/etc/nginx/nginx.conf
/etc/nginx/scgi_params
/etc/nginx/uwsgi_params

# yum安装后自动生成的service 文件，这个文件可以被编译安装借鉴，按照编译安装定义的内容修改即可被用于编译安装的启动文件
[root@Nginx-IP08 ]#cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/sh -c "/bin/kill -s HUP $(/bin/cat /var/run/nginx.pid)"
ExecStop=/bin/sh -c "/bin/kill -s TERM $(/bin/cat /var/run/nginx.pid)"

[Install]
WantedBy=multi-user.target

# 启动并设定开机启动
[root@Nginx-IP08 ]#systemctl enable --now nginx 
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.
[root@Nginx-IP08 ]#
[root@Nginx-IP08 ]#
[root@Nginx-IP08 ]#ss -ltn
State          Recv-Q     Send-Q      Local Address:Port      Peer Address:Port    Process 
LISTEN         0          511         0.0.0.0:80              0.0.0.0:*                           

# 查看yum安装的官网的版本编译的参数信息，我们常用的反向代理、地址透传、https等都编译进去了
[root@Nginx-IP08 ]#nginx -V                    
nginx version: nginx/1.21.6
built by gcc 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC) 
built with OpenSSL 1.1.1k  FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
[root@Nginx-IP08 ]#nginx -v
nginx version: nginx/1.21.6
[root@Nginx-IP08 ]#

4. 反向代理单台web

基本内容：我们先完成反向代理至后端服务器组内的一台WEB主机。

# 查看yum安装nginx的配置目录结构
[root@Nginx-IP08 ]#tree /etc/nginx/
/etc/nginx/
├── conf.d
│   └── default.conf   # server 默认配置文件
├── fastcgi_params
├── mime.types
├── modules -> ../../usr/lib64/nginx/modules
├── nginx.conf         # 主配置文件
├── scgi_params
└── uwsgi_params

2 directories, 6 files

# 在/etc/nginx/conf.d/ 子配置目录下创建两个组的配置文件
[root@Nginx-IP08 ]#vim /etc/nginx/conf.d/pc.conf  
[root@Nginx-IP08 ]#cat /etc/nginx/conf.d/pc.conf       
server {
listen 80;
  server_name www.shone.cn;
  location / {
  proxy_pass http://192.168.250.18;
  }
}

[root@Nginx-IP08 ]#vim /etc/nginx/conf.d/mobile.conf   
[root@Nginx-IP08 ]#cat /etc/nginx/conf.d/mobile.conf    
server {
  listen 80;
  server_name m.shone.cn;
  location / {
    proxy_pass http://192.168.250.58;
  }
}

# 重新引导Nginx，让新配置文件生效
[root@Nginx-IP08 ]#nginx -s reload 


######################################################################################
# 在终端客户机上测试访问
[root@Client-IP172-18 ]#curl www.shone.cn   
PC-WebServer-IP18 192.168.250.18

[root@Client-IP172-18 ]#curl m.shone.cn
M-WebServer-IP58 192.168.250.58
[root@CentOS84-IP172-18 ]#

5. 反向代理多组web并实现各组内的web-rs的负载均衡

基本内容：在上面实现一台反向代理的基础上，完成后面每组多台的反向代理，并在同一个组内实现负载均衡和自动的故障停止调度等

# 首先要修改主配置文件
[root@Nginx-IP08 ]#cat /etc/nginx/nginx.conf    

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {

######################  下面这部分就是新增加的内容 ######################
# PC-Servers组的配置，注释掉一些高级用法，用基础的可以测试下
    upstream PC-Servers {
      #hash $request_uri consistent;
      #hash $cookie_sessionid
      #ip_hash;
      #least_conn;
      server 192.168.250.18:80 weight=1 fail_timeout=5s max_fails=3; 
      server 192.168.250.28:80 weight=1 fail_timeout=5s max_fails=3;
      #server 127.0.0.1:80 weight=1 fail_timeout=5s max_fails=3 backup;
    }
    
# M-Servers组的配置，注释掉一些高级用法，用基础的可以测试下 
    upstream M-Servers {
      #hash $request_uri consistent;
      #hash $cookie_sessionid
      #ip_hash;
      #least_conn;
      server 192.168.250.58:80 weight=1 fail_timeout=5s max_fails=3; 
      server 192.168.250.68:80 weight=1 fail_timeout=5s max_fails=3;
      #server 127.0.0.1:80 weight=1 fail_timeout=5s max_fails=3 backup;
    }
#######################################################################

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
[root@Nginx-IP08 ]#

########  和上面主配置文件对应的是server子配置文件都要修改  ########
# PC-Servers组的server子配置文件
[root@Nginx-IP08 ]#cat /etc/nginx/conf.d/pc.conf       
server {
listen 80;
  server_name www.shone.cn;
  location / {
  proxy_pass http://PC-Servers;
  }
}

# M-Servers组的server子配置文件
[root@Nginx-IP08 ]#cat /etc/nginx/conf.d/mobile.conf    

server {
  listen 80;
  server_name m.shone.cn;
  location / {
    proxy_pass http://M-Servers;
  }
}

[root@Nginx-IP08 ]#

# 语法检测没错误
[root@Nginx-IP08 ]#nginx -t
# 重新引导Nginx，让新配置文件生效
[root@Nginx-IP08 ]#nginx -s reload 

######################################################################################
# 在终端客户机上测试访问，看到符合设定的规则轮询，我们确定测试停掉任何一个后端的web都会被停止调度
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP28 192.168.250.28
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP18 192.168.250.18
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP28 192.168.250.28
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP18 192.168.250.18
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP28 192.168.250.28
[root@Client-IP172-18 ]#curl www.shone.cn
PC-WebServer-IP18 192.168.250.18



[root@Client-IP172-18 ]#curl m.shone.cn  
M-WebServer-IP58 192.168.250.58
[root@Client-IP172-18 ]#curl m.shone.cn
M-WebServer-IP68 192.168.250.68
[root@Client-IP172-18 ]#curl m.shone.cn  
M-WebServer-IP58 192.168.250.58
[root@Client-IP172-18 ]#curl m.shone.cn
M-WebServer-IP68 192.168.250.68
[root@Client-IP172-18 ]#curl m.shone.cn
M-WebServer-IP58 192.168.250.58
[root@Client-IP172-18 ]#curl m.shone.cn
M-WebServer-IP68 192.168.250.68
[root@Client-IP172-18 ]#