Different routes send traffic to wrong application in OpenShift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 3
  • 4
• HTTP/2
• Wildcard certificate

Issue

• After connecting to 'Application A', all the OpenShift routes show 'Application A' content or Page Not Found error for any application.
• Unable to connect to Openshift Web Console after connecting to Keycloak.
• Unable to connect to Openshift Web Console after connecting to a Service Mesh application.
• Unable to connect to Openshift Web Console after connecting to StackRox.

Resolution

This is caused by HTTP/2 connection reuse when the same certificate matches for different routes. This behavior depends on how the browser handles the connection. For example, if a new private window is opened, or a different browser used, the expected content will be shown.

Note: starting with OpenShift 4.16, the OpenShift router now checks when the same certificate is specified on more than one route or when a route specifies the default certificate as a custom certificate. When either one of these conditions is detected, the router configures the HAProxy load balancer so to not allow HTTP/2 client connections to any routes that use that certificate. Refer to This content is not included.OCPBUGS-29373 for additional information.

Workarounds

• The connection reuse can be avoided if a non-wildcard certificate is configured for the applications using HTTP/2. This is the preferred workaround.

• Configure a reencrypt route using custom application certificate to expose the application:

  $ oc create route reencrypt --service=<service-name> --cert=<configured-certificate> --key=<certificate-key> --dest-ca-cert=<ca-certificate> --ca-cert=<ca-certificate> --hostname=<hostname>
  

• Disabling HTTP/2: enable or disable HTTP/2 on Ingress Controller (this can be done for the entire cluster, or for specific Ingress Controller).

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

This behavior is called HTTP/2 connection coalescing and can happen when a few specific requirements are met:

• HTTP/2 is enabled
• Passthrough routes are in use

As it is described in Content from datatracker.ietf.org is not included.rfc7540#section-9.1.1, after establishing a secure HTTP/2 connection, this connection is reused for all the requests which match the certificate.

For example, a certificate with a subjectAltName of *.apps.example.com might permit the use of the same connection for requests to URIs starting with https://a.apps.example.com/ and https://b.apps.example.com/.

This is also described in HTTP/2.0 and SNI causing incorrect responses.

Diagnostic Steps

• Confirm that application uses HTTP/2 and shows a wildcard certificate *.apps.example.com.
• Connect to a secure HTTP/2 application which has a wildcard certificate configured and is exposed through a passthrough route, 'Application A' a.apps.example.com.
• Connect to other secure application whose url matches the certificate, 'Application B' b.apps.example.com, in a new browser tab.
• Browser shows an error or shows 'Application A' content.
• Repeat the same steps opening "Chrome browser" from command line with --disable-http2 option and check that each application shows the desired content.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.