Istio 1.1.11 not supporting http2? - https

Question

I recently asked this question on how to upgrade Istio 1.1.11 from using http1.1 to http2.
I followed the advice and my resultant services YAML looks like this.
##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
service: details
spec:
ports:
- port: 9080
name: http2
selector:
app: details
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: details-v1
labels:
app: details
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: details
version: v1
spec:
containers:
- name: details
image: istio/examples-bookinfo-details-v1:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
service: ratings
spec:
ports:
- port: 9080
name: http2
selector:
app: ratings
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ratings-v1
labels:
app: ratings
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: ratings
version: v1
spec:
containers:
- name: ratings
image: istio/examples-bookinfo-ratings-v1:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
service: reviews
spec:
ports:
- port: 9080
name: http2
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v1
labels:
app: reviews
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v1
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v1:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v2
labels:
app: reviews
version: v2
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v2
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v2:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v3
labels:
app: reviews
version: v3
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v3
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v3:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
service: productpage
spec:
ports:
- port: 9080
name: http2
selector:
app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productpage-v1
labels:
app: productpage
version: v1
spec:
replicas: 1
template:
metadata:
labels:
app: productpage
version: v1
spec:
containers:
- name: productpage
image: istio/examples-bookinfo-productpage-v1:1.13.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
I successfully followed this tutorial to curl the service using HTTPS.
curl before:
curl -o /dev/null -s -v -w "%{http_code}\n" -HHost:localhost --resolve
localhost:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt -HHost:localhost https://localhost:443/productpage
* Address in 'localhost:443:localhost' found illegal!
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: example.com.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [215 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [96 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [740 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=localhost; O=Localhost organization
* start date: Jan 13 05:22:09 2020 GMT
* expire date: Jan 12 05:22:09 2021 GMT
* common name: localhost (matched)
* issuer: O=example Inc.; CN=example.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe244006400)
> GET /productpage HTTP/2
> Host:localhost
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< content-type: text/html; charset=utf-8
< content-length: 4415
< server: istio-envoy
< date: Tue, 14 Jan 2020 03:22:30 GMT
< x-envoy-upstream-service-time: 1294
<
{ [4415 bytes data]
* Connection #0 to host localhost left intact
200
If I hit the service from a browser it works perfectly fine using url https://localhost/productpage
But, it stops working after I apply the above YAML. The browser just says
"upstream connect error or disconnect/reset before headers. reset reason: connection termination"
curl after:
curl -o /dev/null -s -v -w "%{http_code}\n" -HHost:localhost --resolve localhost:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt -HHost:localhost https://localhost:443/productpage
* Address in 'localhost:443:localhost' found illegal!
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: example.com.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [215 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [96 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [740 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=localhost; O=Localhost organization
* start date: Jan 13 05:22:09 2020 GMT
* expire date: Jan 12 05:22:09 2021 GMT
* common name: localhost (matched)
* issuer: O=example Inc.; CN=example.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe13a005200)
> GET /productpage HTTP/2
> Host:localhost
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< content-length: 95
< content-type: text/plain
< date: Tue, 14 Jan 2020 03:16:49 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 57
<
{ [95 bytes data]
* Connection #0 to host localhost left intact
503
My destination rules look like this
(Note: It fails only if I change the above YAML, designation rules seem to be working just fine):
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: productpage
spec:
host: productpage
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: ratings
spec:
host: ratings
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v2-mysql
labels:
version: v2-mysql
- name: v2-mysql-vm
labels:
version: v2-mysql-vm
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: details
spec:
host: details
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
---
Few questions:
1) What could be the cause? How can I fix this? Is this a bug in Istio?
2) I'm able to hit the service from the browser before making the changes and I've read here that modern browsers only support HTTP2. Does that mean I'm automatically compliant to HTTP2? How to verify this?
3) How to gather the relevant logs to track what protocol is being used and for inter-pod communication?

Answer 1

The issue here is that You are most likely trying to serve HTTP (bookinfo app) content via HTTP2 protocol deployment/cluster configuration.
The bookinfo sample application from istio documentation does not support HTTP2 in its base configuration.
You can verify if You web-server supports HTTP2 protocol with this web tool: http2-test
From the other case You linked it appears You are looking into switching internal cluster communication from HTTP to HTTP2.
If You chose to continue going this path I suggest deploying service like nginx with with HTTP2 configuration similar to this found in nginx documentation for debugging purposes.
This can have alternative approach as described in google cloud documentation. In this case You can use HTTP as internal protocol in Your cluster configuration and web-server and then translate the traffic to HTTP2 on istio gateway/external loadbalancer.