使用ingress暴露kubernetes集群内部的pod服务
Kubernetes暴露服务的方式目前只有三种:LoadBlancer Service、ExternalName、NodePort Service、Ingress,
四层负载均衡调度器Service可以以标签的形式选定一组带有指定标签的Pod,并监控和自动负载他们的Pod IP,通过NodePort暴露。当客户端访问kubernetes集群内部的应用时,数据包的走向如下面流程所示:client—>node ip:port—>service ip:port—>pod ip:port。
采用NodePort方式暴露服务面临的问题是,服务一旦多起来,NodePort在每个节点上开启的端口会及其庞大,而且难以维护;而使用 DaemonSet 在每个Node上监听 80的nginx反向代理又有域名分配及动态更新问题,此时Ingress出现了。
ngress 包含两大组件:Ingress Controller 和 Ingress
Ingress Controller:可以理解为控制器,它通过不断的跟 Kubernetes API 交互,实时获取后端 Service、Pod 的变化,比如新增、删除等,然后结合 Ingress 定义的规则生成配置,然后动态更新上边的七层负载均衡调度器 Nginx 或者trafik负载均衡器,并刷新使配置生效,来达到服务自动发现的作用。
nginx:需要手动加载配置文件
traefik:定期自动加载配置文件,不需要手动干预,在微服务中几乎都会使用这种调度器
Ingress:定义规则,通过它定义某个域名的请求过来之后转发到集群中指定的 Service。它可以通过 Yaml 文件定义,更新。可以给一个或多个 Service 定义一个或多个 Ingress 规则。简单的理解就是你原来需要改Nginx配置,然后配置各种域名对应哪个 Service
使用七层负载均衡调度器的步骤
(1)部署ingress controller,traefik或者nginx
(2)创建service,用来分组pod
(3)创建pod应用,可以通过控制器创建pod
(4)创建ingress http,测试通过http访问k8s内部pod
(5)创建ingress https,测试通过https访问k8s内部pod
使用七层负载均衡调度器ingress controller时,当客户端访问kubernetes集群内部的应用时,数据包走向如下面流程所示:
client—>Nodeip:port----->IngressController—>service—>pod
如何创建Ingress资源
Ingress资源是基于HTTP虚拟主机或URL的转发规则,它在资源配置清单中的spec字段中嵌套了rules、backend和tls等字段进行定义。如下定义一个Ingress资源转发规则:
将发往tomcat.lucky.com的请求,代理给一个名字为myapp的Service资源。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-myapp
namespace: default
spec:
rules:
- host: tomcat.lucky.com
http:
paths:
- path:
backend:
serviceName: myapp
servicePort: 80
Ingress中的spec字段是Ingress资源的核心组成部分,主要包含以下3个字段:
**rules:**用于定义当前Ingress资源的转发规则列表;由rules定义规则,或没有匹配到规则,所有的流量会转发到由backend定义的默认后端。
**backend:**默认的后端用于服务那些没有匹配到任何规则的请求;定义Ingress资源时,必须要定义backend或rules两者之一,该字段用于让负载均衡器指定一个全局默认的后端。
**tls:**TLS配置,目前仅支持通过默认端口443提供服务,如果要配置指定的列表成员指向不同的主机,则需要通过SNI TLS扩展机制来支持该功能。
backend对象的定义由2个必要的字段组成:serviceName和servicePort,分别用于指定流量转发的后端目标Service资源名称和端口。
rules对象由一系列的配置的Ingress资源的host规则组成,这些host规则用于将一个主机上的某个URL映射到相关后端Service对象,其定义格式如下:
spec:
rules:
- hosts: <string>
http:
paths:
- path:
backend:
serviceName: <string>
servicePort: <string>
spec.rules.host属性值,目前暂不支持使用IP地址定义,也不支持IP:Port的格式,该字段留空,代表着通配所有主机名。
tls对象由2个内嵌的字段组成,仅在定义TLS主机的转发规则上使用。
**hosts:**包含 于 使用 的 TLS 证书 之内 的 主机 名称 字符串 列表, 因此, 此处 使用 的 主机 名 必须 匹配 tlsSecret 中的 名称。
secretName: 用于 引用 SSL 会话 的 secret 对象 名称, 在 基于 SNI 实现 多 主机 路 由 的 场景 中, 此 字段 为 可选。
Ingress资源类型
1、单Service资源型Ingress
2、基于URL路径进行流量转发
3、基于主机名称的虚拟主机
4、TLS类型的Ingress资源
单Service资源型Ingress
暴露单个服务的方法有多种,如NodePort、LoadBanlancer等等,当然也可以使用Ingress来进行暴露单个服务,只需要为Ingress指定default backend即可,
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: my-svc
servicePort: 80
Ingress控制器会为其分配一个IP地址接入请求流量,并将其转发至后端my-svc
Ingress Nginx部署
1、安装部署ingress controller Pod
创建命名空间
[root@master ingress]# cat namespace.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
[root@master ingress]# kubectl apply -f namespace.yaml
namespace/ingress-nginx created
[root@master ingress]# kubectl get namespace
NAME STATUS AGE
ingress-nginx Active 11s
创建
[root@master test]# cat rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
[root@master test]# cat with-rbac.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
[root@master test]# cat configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app: ingress-nginx
[root@master test]# cat default-backend.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
labels:
app: default-http-backend
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: whychoice/defaultbackend:1.4
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: ingress-nginx
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend
[root@master test]# cat tcp-services-configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
[root@master test]# cat udp-services-configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
[root@master test]# kubectl apply -f .
[root@master test]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/default-http-backend-78d75577fd-g5lfs 1/1 Running 0 29s
pod/nginx-ingress-controller-7c7d57b55d-g85w6 1/1 Running 0 29s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/default-http-backend ClusterIP 10.98.163.106 <none> 80/TCP 29s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/default-http-backend 1/1 1 1 29s
deployment.apps/nginx-ingress-controller 1/1 1 1 29s
NAME DESIRED CURRENT READY AGE
replicaset.apps/default-http-backend-78d75577fd 1 1 1 29s
replicaset.apps/nginx-ingress-controller-7c7d57b55d 1 1 1 29s
2.部署后端服务
1)部署后端服务
cat deploy-demo.yaml
[root@master ingress]# cat deploy-demo.yaml
#创建service为myapp
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
ports:
- name: http
targetPort: 80
port: 80
---
#创建后端服务的pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-backend-pod
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v2
ports:
- name: http
containerPort: 80
[root@master ingress]# kubectl apply -f deploy-demo.yaml
[root@master ingress]# kubectl get pod
NAME READY STATUS RESTARTS AGE
myapp-backend-pod-559ff5c66-5s4mk 1/1 Running 0 51s
myapp-backend-pod-559ff5c66-g9zkc 1/1 Running 0 51s
myapp-backend-pod-559ff5c66-lxpjz 1/1 Running 0 51s
2)创建ingress-controller的service
[root@master ingress]# cat service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 30080
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 30443
selector:
app: ingress-nginx
[root@master ingress]# ss -antup |grep 30443
[root@master ingress]# ss -antup |grep 30443
[root@master ingress]# kubectl apply -f service-nodeport.yaml
service/ingress-nginx created
[root@master ingress]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.98.163.106 <none> 80/TCP 2h
ingress-nginx NodePort 10.101.32.19 <none> 80:30080/TCP,443:30443/TCP 3s
http://192.168.1.12:30080/ 显示 default backend - 404
[root@master ingress]# cat ingress-myapp.yaml
apiVersion: extensions/v1beta1 #api版本
kind: Ingress #清单类型
metadata: #元数据
name: ingress-myapp #ingress的名称
namespace: default #所属名称空间
annotations: #注解信息
kubernetes.io/ingress.class: "nginx"
spec: #规格
rules: #定义后端转发的规则
- host: tomcat.lucky.com #通过域名进行转发
http:
paths:
- path: #配置访问路径,如果通过url进行转发,需要修改;空默认为访问的路径为"/"
backend: #配置后端服务
serviceName: myapp
servicePort: 80
[root@master ingress]# kubectl apply -f ingress-myapp.yaml
C:\Windows\System32\drivers\etc\hosts写入
192.168.1.12 tomcat.lucky.com
http://tomcat.lucky.com:30080/ 显示如下
Hello MyApp | Version: v2 | Pod Name
进入nginx-ingress-control查看,可以找到自动加入的域名
server_name tomcat.lucky.com
[root@master ingress]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-78d75577fd-g5lfs 1/1 Running 0
nginx-ingress-controller-7c7d57b55d-g85w6 1/1 Running 0
[root@master ingress]# kubectl exec -it -n ingress-nginx nginx-ingress-controller-7c7d57b55d-g85w6 -- /bin/bash
www-data@nginx-ingress-controller-7c7d57b55d-g85w6:/etc/nginx$ ls
fastcgi.conf koi-utf modsecurity owasp-modsecurity-crs uwsgi_params.default modules
fastcgi.conf.default koi-win scgi_params win-utf
fastcgi_params lua nginx.conf scgi_params.default fastcgi_params.default mime.types nginx.conf.default template geoip mime.types.default opentracing.json uwsgi_params
www-data@nginx-ingress-controller-7c7d57b55d-g85w6:/etc/nginx$ cat nginx.conf server {
server_name tomcat.lucky.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
location / {
5.部署ingress-测试代理tomcat服务
1)部署tomcat服务
[root@master ingress]# cat tomcat-demo.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
spec:
selector:
app: tomcat
release: canary
ports:
- name: http
targetPort: 8080
port: 8080
- name: ajp
targetPort: 8009
port: 8009
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deploy
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: tomcat
release: canary
template:
metadata:
labels:
app: tomcat
release: canary
spec:
containers:
- name: tomcat
image: tomcat:8.5-jre8-alpine
#此镜像在dockerhub上进行下载,需要查看版本是否有变化,hub.docker.com
ports:
- name: http
containerPort: 8080
name: ajp
containerPort: 8009
[root@master ingress]# kubectl apply -f tomcat-demo.yaml
[root@master ingress]# cat ingress-tomcat.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tomcat
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: tomcat.lucky6.com #主机域名
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
[root@master ingress]# kubectl apply -f ingress-tomcat.yaml
C:\Windows\System32\drivers\etc\hosts写入
192.168.1.12 tomcat.lucky6.com
http://tomcat.lucky6.com:30080/ 显示
Home Documentation Configuration Examples Wiki Mailing Lists
Find HelpApache Tomcat/8.5.41的首页
总结
①下载Ingress-controller相关的YAML文件,给Ingress-controller创建独立的名称空间;
②部署后端的服务,如myapp,并通过service进行暴露;
③部署Ingress-controller的service,以实现接入集群外部流量;
④部署Ingress,进行定义规则,使Ingress-controller和后端服务的Pod组进行关联。
构建TLS站点
(1)准备证书,在k8s的master节点操作
[root@master ingress]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
.........................+++
.....+++
e is 65537 (0x10001)
[root@master ingress]# ls
tls.key
[root@master ingress]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.lucky.com
(2)生成secret,在k8s的master节点操作
[root@master ingress]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key
[root@master ingress]# kubectl get secret
NAME TYPE DATA AGE
default-token-86dcd kubernetes.io/service-account-token 3 3d
tomcat-ingress-secret kubernetes.io/tls 2 18s
(3)查看tomcat-ingress-secret详细信息
[root@master ingress]# kubectl describe secret tomcat-ingress-secret
Name: tomcat-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1294 bytes
tls.key: 1675 bytes
(5)创建ingress
[root@master ingress]# cat ingress-tomcat-tls.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-tomcat-tls
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- tomcat.lucky.com
secretName: tomcat-ingress-secret
rules:
- host: tomcat.lucky.com
http:
paths:
- path:
backend:
serviceName: tomcat
servicePort: 8080
[root@master ingress]# kubectl apply -f ingress-tomcat-tls.yaml
ingress.extensions/ingress-tomcat-tls created
http://tomcat.lucky.com:30443/ 400 Bad Request
https://tomcat.lucky.com:30443/ Hello MyApp | Version: v2