Envoy Mesh中TLS常用场景
- Front Proxy面向下游客户端提供https服务，但Front Proxy、Mesh内部的各服务间依然使用http协议；
- https (下游) → http (上游)
- Front Proxy面向下游客户端提供https服务，而且Front Proxy、Mesh内部的各服务间也使用https协议；
- https (下游) → https (上游)
- 但是内部各Service间的通信也有如下两种情形
- 仅客户端验证服务端证书
- 客户端与服务端之间互相验证彼此的证书（mTLS）
- 注意：对于容器化的动态环境来说，证书预配和管理将成为显著难题
- Front Proxy直接以TCP Proxy的代理模式，在下游客户端与上游服务端之间透传tls协议；
- https-passthrough
- 集群内部的东西向流量同样工作于https协议模型
TLS配置位置说明
Listener
Listener面向DownStream（下游客户端）通信，与下游建立tls通信时通常配置在Listener上；
与下游建立tls通信时Listener作为server端，需要提供证书和私钥；
Cluster
Cluster面向上游通信，与上游建立tls通信时通常配置在Cluster上；
与上游建立tls通信时Cluster作为客户端；
- 单向tls通信时；
- 需要ca证书，验证上游服务器证书；
- 双向tls通信时；
- 需要ca证书，验证上游服务器证书；
- 需要client端证书和私钥；
TLS环境配置实例
环境说明
- envoy：Front Proxy,地址为172.31.7.2，监听于443端口
- webserver_http-01：第一个后端服务 http
- webserver_http-01-sidecar：第一个后端服务的Sidecar Proxy,地址为172.31.7.11，监听于80端口
- webserver_http-02：第二个后端服务 http
- webserver_http-02-sidecar：第二个后端服务的Sidecar Proxy,地址为172.31.7.12, 监听于80端口
- webserver_tls-01：第三个后端服务 tls
- webserver_tls-01-sidecar：第三个后端服务的Sidecar Proxy,地址为172.31.7.13，监听于443端口
- webserver_tls-02：第四个后端服务 tls
- webserver_tls-02-sidecar：第四个后端服务的Sidecar Proxy,地址为172.31.7.14, 监听于443端口
- webserver_mtls-01：第五个后端服务 mtls
- webserver_mtls-01-sidecar：第五个后端服务的Sidecar Proxy,地址为172.31.7.15，监听于443端口
- webserver_mtls-02：第六个后端服务 mtls
- webserver_mtls-02-sidecar：第六个后端服务的Sidecar Proxy,地址为172.31.7.16, 监听于443端口
拓扑结构
创建证书目录
# mkdir -pv certs/{CA,front-envoy}
# touch certs/index.txt
# touch certs/serial
# echo "1000" > certs/serial
# echo "unique_subject = no" > certs/index.txt.attr # Sign multiple certs for the same CN
准备openssl.conf文件
查看代码
# environment variable values
CERT_DIR=certs
[ ca ]
# `man ca`
default_ca = CA_default
[ CA_default ]
# Directory and file locations.
dir = ${ENV::CERT_DIR}
certs = $dir
crl_dir = $dir/crl
new_certs_dir = $dir
database = $dir/index.txt
serial = $dir/serial
# certificate revocation lists.
crlnumber = $dir/crlnumber
crl = $dir/crl/intermediate-ca.crl
crl_extensions = crl_ext
default_crl_days = 30
default_md = sha256
name_opt = ca_default
cert_opt = ca_default
default_days = 375
preserve = no
policy = policy_loose
[ policy_loose ]
# Allow the CA to sign a range of certificates.
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
# `man req`
default_bits = 4096
distinguished_name = req_distinguished_name
string_mask = utf8only
default_md = sha256
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name
localityName = Locality Name
0.organizationName = Organization Name
organizationalUnitName = Organizational Unit Name
commonName = Common Name
# Certificate extensions (`man x509v3_config`)
[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
[ client_cert ]
basicConstraints = CA:FALSE
nsCertType = client
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
创建证书
创建ca证书
# openssl genrsa -out certs/CA/ca.key 4096
# openssl req -config openssl.conf -new -x509 -days 3650 -sha256 -key certs/CA/ca.key -extensions v3_ca -out certs/CA/ca.crt -subj /CN=envoy-ca
创建front-proxy server证书
# openssl genrsa -out certs/front-envoy/server.key 2048
# openssl req -config openssl.conf -new -sha256 -key certs/front-envoy/server.key -out certs/front-envoy/server.csr -subj /CN=front-proxy
# openssl ca -batch -config openssl.conf -extensions server_cert -days 3650 -notext -md sha256 -in certs/front-envoy/server.csr -out certs/front-envoy/server.crt -cert certs/CA/ca.crt -keyfile certs/CA/ca.key
创建front-proxy client证书
# openssl genrsa -out certs/front-envoy/client.key 2048
# openssl req -config openssl.conf -new -sha256 -key certs/front-envoy/client.key -out certs/front-envoy/client.csr -subj /CN=front-proxy
# openssl ca -batch -config openssl.conf -extensions client_cert -days 3650 -notext -md sha256 -in certs/front-envoy/client.csr -out certs/front-envoy/client.crt -cert certs/CA/ca.crt -keyfile certs/CA/ca.key
修改证书权限
# docker run -it --rm envoyproxy/envoy:v1.23-latest id envoy
uid=101(envoy) gid=101(envoy) groups=101(envoy)
# chown -R 101.101 certs/
front-envoy.yaml
查看代码
node:
id: front-envoy
cluster: front-envoy
admin:
profile_path: /tmp/envoy.prof
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
layered_runtime:
layers:
- name: admin
admin_layer: {}
static_resources:
secrets:
- name: server_cert
tls_certificate:
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
- name: client_cert
tls_certificate:
certificate_chain:
filename: "/etc/envoy/certs/client.crt"
private_key:
filename: "/etc/envoy/certs/client.key"
- name: validation_context
validation_context:
trusted_ca:
filename: "/etc/envoy/ca/ca.crt"
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: "/"
redirect:
https_redirect: true
port_redirect: 443
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
routes:
- match:
prefix: "/http/"
route:
prefix_rewrite: "/"
cluster: http
- match:
prefix: "/tls/"
route:
prefix_rewrite: "/"
cluster: tls
- match:
prefix: "/mtls/"
route:
prefix_rewrite: "/"
cluster: mtls
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket: # DownstreamTlsContext
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: server_cert
clusters:
- name: http
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: http
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: http
port_value: 80
- name: tls
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: tls
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: tls
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
validation_context_sds_secret_config:
name: validation_context
- name: mtls
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: mtls
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: mtls
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: client_cert
validation_context_sds_secret_config:
name: validation_context
envoy-sidecar-proxy_http.yaml
查看代码
admin:
profile_path: /tmp/envoy.prof
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 80 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: local_cluster }
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: local_cluster
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: local_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 127.0.0.1, port_value: 8080 }
envoy-sidecar-proxy_tls.yaml
查看代码
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
clusters:
- name: local_service
connect_timeout: 0.25s
type: STATIC
lb_policy: round_robin
load_assignment:
cluster_name: local_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
envoy-sidecar-proxy_mtls.yaml
查看代码
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
prefix: "/"
redirect:
https_redirect: true
port_redirect: 443
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_https
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
route_config:
name: https_route
virtual_hosts:
- name: https_route
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: local_service
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket: # DownstreamTlsContext
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates: # 基于DataSource，直接给出证书和私钥文件
certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
require_client_certificate: true # 强制验证客户端证书
clusters:
- name: local_service
connect_timeout: 0.25s
type: STATIC
lb_policy: round_robin
load_assignment:
cluster_name: local_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
docker-compose.yaml
查看代码
version: '3.3'
services:
envoy:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./front-envoy.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy:/etc/envoy/certs/
- ./certs/CA/:/etc/envoy/ca/
networks:
envoymesh:
ipv4_address: 172.31.7.2
aliases:
- front-proxy
expose:
- "80"
- "443"
- "9901"
ports:
- "8080:80"
- "8443:443"
- "9901:9901"
depends_on:
- webserver_http-01-sidecar
- webserver_http-02-sidecar
- webserver_tls-01-sidecar
- webserver_tls-02-sidecar
- webserver_mtls-01-sidecar
- webserver_mtls-02-sidecar
webserver_http-01-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_http.yaml:/etc/envoy/envoy.yaml
hostname: webserver_http-01
networks:
envoymesh:
ipv4_address: 172.31.7.11
aliases:
- webserver_http-01-sidecar
- front-proxy
- http
webserver_http-01:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=127.0.0.1
network_mode: "service:webserver_http-01-sidecar"
depends_on:
- webserver_http-01-sidecar
webserver_http-02-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_http.yaml:/etc/envoy/envoy.yaml
hostname: webserver_http-02
networks:
envoymesh:
ipv4_address: 172.31.7.12
aliases:
- webserver_http-02-sidecar
- front-proxy
- http
webserver_http-02:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=127.0.0.1
network_mode: "service:webserver_http-02-sidecar"
depends_on:
- webserver_http-02-sidecar
webserver_tls-01-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_tls.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver_tls-01
networks:
envoymesh:
ipv4_address: 172.31.7.13
aliases:
- webserver_tls-03-sidecar
- tls
webserver_tls-01:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=0.0.0.0
network_mode: "service:webserver_tls-01-sidecar"
depends_on:
- webserver_tls-01-sidecar
webserver_tls-02-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_tls.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver_tls-02
networks:
envoymesh:
ipv4_address: 172.31.7.14
aliases:
- webserver_tls-02-sidecar
- tls
webserver_tls-02:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=0.0.0.0
network_mode: "service:webserver_tls-02-sidecar"
depends_on:
- webserver_tls-02-sidecar
webserver_mtls-01-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_mtls.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver_mtls-01
networks:
envoymesh:
ipv4_address: 172.31.7.15
aliases:
- webserver_mtls-01-sidecar
- mtls
webserver_mtls-01:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=0.0.0.0
network_mode: "service:webserver_mtls-01-sidecar"
depends_on:
- webserver_mtls-01-sidecar
webserver_mtls-02-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_mtls.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver_mtls-02
networks:
envoymesh:
ipv4_address: 172.31.7.16
aliases:
- webserver_mtls-02-sidecar
- mtls
webserver_mtls-02:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=0.0.0.0
network_mode: "service:webserver_mtls-02-sidecar"
depends_on:
- webserver_mtls-02-sidecar
networks:
envoymesh:
driver: bridge
ipam:
config:
- subnet: 172.31.7.0/24
运行并测试
启动服务
docker-compose up
查看证书
查看代码
# curl 172.31.7.2:9901/certs
{
"certificates": [
{
"ca_cert": [
{
"path": "/etc/envoy/ca/ca.crt",
"serial_number": "51d6bb31f33491a0c776fb6534c4c2f7794aaee0",
"subject_alt_names": [],
"days_until_expiration": "3649",
"valid_from": "2022-08-20T03:10:25Z",
"expiration_time": "2032-08-17T03:10:25Z"
}
],
"cert_chain": [
{
"path": "/etc/envoy/certs/client.crt",
"serial_number": "1001",
"subject_alt_names": [],
"days_until_expiration": "3649",
"valid_from": "2022-08-20T03:17:16Z",
"expiration_time": "2032-08-17T03:17:16Z"
}
]
},
{
"ca_cert": [],
"cert_chain": [
{
"path": "/etc/envoy/certs/server.crt",
"serial_number": "1000",
"subject_alt_names": [],
"days_until_expiration": "3649",
"valid_from": "2022-08-20T03:16:15Z",
"expiration_time": "2032-08-17T03:16:15Z"
}
]
},
{
"ca_cert": [
{
"path": "/etc/envoy/ca/ca.crt",
"serial_number": "51d6bb31f33491a0c776fb6534c4c2f7794aaee0",
"subject_alt_names": [],
"days_until_expiration": "3649",
"valid_from": "2022-08-20T03:10:25Z",
"expiration_time": "2032-08-17T03:10:25Z"
}
],
"cert_chain": []
}
]
}
查看front-envoy启动的Listener
# curl 172.31.7.2:9901/listeners
listener_http::0.0.0.0:80
listener_https::0.0.0.0:443
测试访问服务
我们可在curl命令上为其提供私有CA证书文件，或者使用“-k”选项忽略提示的风险，从而访问https服务。
http请求自动跳转至https服务上
# curl -I 172.31.7.2
HTTP/1.1 301 Moved Permanently
location: https://172.31.7.2:443/
date: Sat, 20 Aug 2022 08:14:44 GMT
server: envoy
transfer-encoding: chunked
访问http
# curl -k https://172.31.7.2/http/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_http-01, ServerIP: 172.31.7.11!
# curl -k https://172.31.7.2/http/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_http-02, ServerIP: 172.31.7.12!
访问tls
# curl -k https://172.31.7.2/tls/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_tls-01, ServerIP: 172.31.7.13!
# curl -k https://172.31.7.2/tls/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_tls-02, ServerIP: 172.31.7.14!
访问mtls
# curl -k https://172.31.7.2/mtls/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_mtls-01, ServerIP: 172.31.7.15!
# curl -k https://172.31.7.2/mtls/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver_mtls-02, ServerIP: 172.31.7.16!
查看访问日志
tls-wgs-2-envoy-1 | [2022-08-20T09:24:43.445Z] "GET /http/ HTTP/1.1" 200 - 0 103 2 2 "-" "curl/7.68.0" "bc0b5abb-2be1-4aaf-b146-b223ce330bbf" "172.31.7.2" "172.31.7.11:80"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:45.006Z] "GET /http/ HTTP/1.1" 200 - 0 103 2 1 "-" "curl/7.68.0" "04b259d7-0e8a-45ff-aeea-023b6f8c0bf8" "172.31.7.2" "172.31.7.12:80"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:48.928Z] "GET /tls/ HTTP/1.1" 200 - 0 102 5 4 "-" "curl/7.68.0" "b71bd17d-9fb6-42ff-bd66-166a7366215d" "172.31.7.2" "172.31.7.14:443"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:51.585Z] "GET /tls/ HTTP/1.1" 200 - 0 102 3 2 "-" "curl/7.68.0" "076b3ec7-5a83-49b8-a663-e1bcaa560153" "172.31.7.2" "172.31.7.13:443"
tls-wgs-2-webserver_tls-01-sidecar-1 | [2022-08-20T09:24:51.586Z] "GET /tls/ HTTP/1.1" 200 - 0 102 2 1 "-" "curl/7.68.0" "076b3ec7-5a83-49b8-a663-e1bcaa560153" "172.31.7.2" "127.0.0.1:8080"
tls-wgs-2-webserver_mtls-01-sidecar-1 | [2022-08-20T09:24:54.662Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 3 2 "-" "curl/7.68.0" "8532ac7a-aaba-4e47-a5d0-63d444a7a6e8" "172.31.7.2" "127.0.0.1:8080"
tls-wgs-2-webserver_mtls-01-sidecar-1 | [2022-08-20T09:24:55.801Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 1 1 "-" "curl/7.68.0" "af41c174-c7a0-4227-8340-f161af0af4e1" "172.31.7.2" "127.0.0.1:8080"
tls-wgs-2-webserver_mtls-02-sidecar-1 | [2022-08-20T09:24:56.981Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 2 2 "-" "curl/7.68.0" "49e4cbfa-f13f-4085-8866-b03a29b611d9" "172.31.7.2" "127.0.0.1:8080"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:54.659Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 7 6 "-" "curl/7.68.0" "8532ac7a-aaba-4e47-a5d0-63d444a7a6e8" "172.31.7.2" "172.31.7.15:443"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:55.800Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 2 2 "-" "curl/7.68.0" "af41c174-c7a0-4227-8340-f161af0af4e1" "172.31.7.2" "172.31.7.15:443"
tls-wgs-2-envoy-1 | [2022-08-20T09:24:56.981Z] "GET /mtls/ HTTP/1.1" 200 - 0 103 3 3 "-" "curl/7.68.0" "49e4cbfa-f13f-4085-8866-b03a29b611d9" "172.31.7.2" "172.31.7.16:443"
openssl s_client命令
查看代码
# openssl s_client -connect 172.31.7.2:443
# 如下命令结果显示，tls传话已然能正常建立，但curl命令无法任何服务端证书的CA，除非我们给命令指定相应的私有CA的证书，以便于验证服务端证书
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = front-proxy
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = front-proxy
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:CN = front-proxy
i:CN = envoy-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEkDCCAnigAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwEzERMA8GA1UEAwwIZW52
b3ktY2EwHhcNMjIwODIwMDMxNjE1WhcNMzIwODE3MDMxNjE1WjAWMRQwEgYDVQQD
DAtmcm9udC1wcm94eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMcj
jvCXA7P9S02uI8rZSVqbOegtz39ssPQ9bGkbtZjjBUGysqYs1VKdtR00Gy+eJ55F
1LwGA3fs8nBFNIo2eSAEMkr8tOQFELCbj7kJbMKFqCKBO2i61T4cP++uUM8QzlYw
rQNaLdZcdPPk6uZffd5Gt5N2DoakMt5O3OFV+EGQgbSb9LBdju3zQCkfwZNpewhs
RpMNzdT4m4pbM1dBnhq04sOHwsjwgApeLeTxWrV6DGCBkx6TXOPadnxrI+I7bChZ
65EO4WKHADqlTvlHtMfU+qB6qCHIRxP/k1dYA95PINZ9dB0gqqesah8x5f3soC2V
mQgw/cO8xvbbj4kx7JECAwEAAaOB6jCB5zAJBgNVHRMEAjAAMBEGCWCGSAGG+EIB
AQQEAwIGQDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0ZWQgU2VydmVy
IENlcnRpZmljYXRlMB0GA1UdDgQWBBRsX1uAWFyCxOOKG+tuSOH+W7ssyjBOBgNV
HSMERzBFgBRDVByyc+OZZC6ivLr2ujfpTFgT2KEXpBUwEzERMA8GA1UEAwwIZW52
b3ktY2GCFFHWuzHzNJGgx3b7ZTTEwvd5Sq7gMA4GA1UdDwEB/wQEAwIFoDATBgNV
HSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOCAgEAE5OiviTeeelTRYSZ
w6ml0xiPEm9VGfGvgV/hhrVimlRoLV/rjz52z+ZO2hpvgdEBlaRIheKz38EieDgo
QGZAUIgMZA2F2iVnhtPX3R5lGVClzfPq1pvZb30WH5ukMY7MGY82bfoUgzuCz9jS
h5sNrsGgI/qI9yAp/memgPCmEhInHhnpsq/RHJuF+HnX9F8GvG3xy3TOAZl1Fz05
WqlXustuMy9y0fb+wAwmSGX82bWwbgsRlmSy3ZfnVAG5DaZ+/gvljg/rxR9n2Xuq
59SbTqjlmskN7SohpmAmgF6wClz7AgpaKkXN9+wJ9t9fn3IUUC3PD9ETSuB7FAlT
stGiKz1ZdGcSnCkFaqJT3XIfGeT5W4Ujj6oKzNHr9rZ43g5IRHIscEx010nVatX9
Zr+zNoVivhhthuiA7CJvkvHJIMFCeadRmugVu2O23XhAde4vcIXymeXFtrGSBUW3
GUtByhEpOG0KiInHqHTr+DaZuAiXcfb3MjaPqyJ6Onth5hHDhK2VWGXrsWlFjJ6P
10mZzwjX8rNXx78oMWdU4frZNBarSEoVbeBvR+SEt19vVwhXZZOED5EUb8jBsLpl
t48Natd11R05RT57EWCl+1QPrvBCO3R64sj3KmWl1PPzpyUtlP1yov1pdbfXt+aQ
L785fuyO0FHXha61ldCmAhnMYAE=
-----END CERTIFICATE-----
subject=CN = front-proxy
issuer=CN = envoy-ca
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1662 bytes and written 363 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
TLS环境TCP Proxy模式配置实例
环境说明
五个Service:
- envoy：Front Proxy,地址为172.31.9.2，监听于8443端口
- webserver01：第一个后端服务
- webserver01-sidecar：第一个后端服务的Sidecar Proxy,地址为172.31.9.11，监听于443端口
- webserver02：第二个后端服务
- webserver02-sidecar：第二个后端服务的Sidecar Proxy,地址为172.31.9.12, 监听于443端口
front-envoy.yaml
查看代码
admin:
profile_path: /tmp/envoy.prof
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
- filters:
- name: envoy.filters.network.tcp_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
cluster: web_cluster_01
stat_prefix: https_passthrough
clusters:
- name: web_cluster_01
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: web_cluster_01
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 172.31.9.11, port_value: 443 }
- endpoint:
address:
socket_address: { address: 172.31.9.12, port_value: 443 }
envoy-sidecar-proxy_https-passthrough.yaml
查看代码
admin:
profile_path: /tmp/envoy.prof
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 443 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: local_cluster }
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
clusters:
- name: local_cluster
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: local_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 127.0.0.1, port_value: 8080 }
docker-compose.yaml
查看代码
version: '3.3'
services:
envoy:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./front-envoy.yaml:/etc/envoy/envoy.yaml
networks:
envoymesh:
ipv4_address: 172.31.9.2
aliases:
- front-proxy
depends_on:
- webserver01-sidecar
- webserver02-sidecar
webserver01-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_https-passthrough.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver01
networks:
envoymesh:
ipv4_address: 172.31.9.11
aliases:
- webserver01-sidecar
webserver01:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=127.0.0.1
network_mode: "service:webserver01-sidecar"
depends_on:
- webserver01-sidecar
webserver02-sidecar:
image: envoyproxy/envoy:v1.23-latest
volumes:
- ./envoy-sidecar-proxy_https-passthrough.yaml:/etc/envoy/envoy.yaml
- ./certs/front-envoy/:/etc/envoy/certs/
hostname: webserver02
networks:
envoymesh:
ipv4_address: 172.31.9.12
aliases:
- webserver02-sidecar
webserver02:
image: ikubernetes/demoapp:v1.0
environment:
- PORT=8080
- HOST=127.0.0.1
network_mode: "service:webserver02-sidecar"
depends_on:
- webserver02-sidecar
networks:
envoymesh:
driver: bridge
ipam:
config:
- subnet: 172.31.9.0/24
运行和测试
创建
docker-compose up
测试服务
# curl -k https://172.31.9.2:8443/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver02, ServerIP: 172.31.9.12!
# curl -k https://172.31.9.2:8443/
iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.1, ServerName: webserver01, ServerIP: 172.31.9.11!
停止后清理
docker-compose down
参考文档
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/transport_socket/transport_socket
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#envoy-v3-api-msg-config-core-v3-transportsocket
https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#extension-envoy-transport-sockets-tls
https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-downstreamtlscontext
https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/tls.proto#envoy-v3-api-msg-extensions-transport-sockets-tls-v3-upstreamtlscontext
