DG 8 TLS SNI usage in Client server communication

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • Client to server arc via TLS

Issue

  • What is SNI and how it is used with client server communication?
  • What are the SNI implications and restrictions on DG 8's client and server?

Resolution

When connecting to a DG server (client outside OCP) connecting to the DG inside OCP via OCP Route, the SNI will be a limitation when using two server (main server and backup server).
The alternatives are: deploy everything in OCP, which will avoid Route / NP / LB and would improve the performance. Otherwise us service expose loadbalancer or nodeport.
Reason:
Previously there was a limitation on the hot rod client, which only allowed one SNI to be set, after SNI is fixed in the client, it would be possible to use Routes with direct access to each node (one Route per node, configured as its external host).

Currently, the hot rod client was updated so that the user does not need to manually configure SNI (clusterSniHostName) if you are on RHDG 8.4.6+, this means this line can be removed from the client code. The SNI is automatically validated by the server with the user connection.

For encryption discussion see solution Data Grid Operator Encryption types and implications. Final note that if DG 8.x is installed in RHEL 7, so then OpenSSL will force TLS 1.2, more details on SSL issues when starting Red Hat Data Grid 8 on RHEL 7.

Custom cert creation with SNI (Route's DNS)

  1. Create a custom certificate with SSL with Route's DNS: 
$ openssl req -x509 -newkey rsa:2048 -keyout my.key -out my.crt -days 7 -nodes -addext "subjectAltName = DNS:myhost.com" <--- the Route's DNS

This will create two files: my.crt and my.key. And for multiple: -addext "subjectAltName = DNS:myhost.com, DNS:otherhost.com"
This will result in:

#8: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: example.external.ocp.com
  1. Create the secret: 
$ oc create secret tls tls-secret --cert=path/to/tls.crt --key=path/to/tls.key <--- use the keystone from above --->
secret/tls-secret created
  1. Create the Infinispan's CR as below:
apiVersion: infinispan.org/v1
...
spec:
  replicas: 1
  service:
    type: DataGrid
  expose:
    type: Route
  security:
    endpointEncryption:
      certSecretName: tls-secret
      type: Secret <------------ The secret is the one provided

Application client access

In the client side, to access the Data Grid Cluster via route, create a truststore importing my.crt (or the valid name configured previously in step 1) :

keytool -importcert -keystore truststore.jks -alias server -file my.crt -ext SAN=dns:myhost.com

Note: myhost.com should be a valid domain name in the environment.

Root Cause

Red Hat recommends the usage of Hot Rod 8.4.6 or later.

About SNI

SNI (Server Name Indication) adds the server name to the initial handshake so it can make decisions based on it (i.e. send a different cert). Otherwise, the server doesn't know what user is connecting to until after the SSL handshake has finished.
For details on non-sni forwarding see details on on SNI and Ingress' fe_no_sni flag on this solution.
Ingres secure Route TLS Termination in OpenShift uses SNI (Server Name Indication) - which cause the issue with Hot Rod passing only one SNI Hostname.

Regarding using type Service outside OCP, see Data Grid Operator Guide - chapter 10:

NOTE
Service certificates use the internal DNS name of the Data Grid cluster as the common
name (CN), for example:
Subject: CN = infinispan_cr.mynamespace.svc
For this reason, service certificates can be fully trusted only inside OpenShift. If you want
to encrypt connections with clients running outside OpenShift, you should use custom
TLS certificates.
Service certificates are valid for one year and are automatically replaced before they
expire.

Hot Rod client limitation

The Hot Rod client does have a significant limitation – sniHostName is global to the CacheManager - meaning one SNI host name.

OCP 4 Route implications

Although an OCP 4 route is passthrough, the Route will impose SNI and there is a default cert being used.
This has little to do with which kind of cert (discussion above on type of cert: let's encrypt vs self-signed) is being used but rather with this fact above haproxy will use a certificate on the route connection. This is explained Content from www.densify.com is not included.here. Therefore this implicates in an issue with Route's SNI and exposes type Route for Hot Rod client access.

In more specific terms, this means that, although Data Grid's default route is passthrough (not edge, nor re-encrypt), the Ingress's Route will impose SNI and will use the default apps. certificates.
That is to say that non-SNI traffic will be routed to the secure port (default 443) and assigned a default certificate (apps. certificate) that most likely will impact the Hot Rod communication, resulting in an authentication error.
Also, Hot Rod's current communication only allows one SNI host to be passed as an argument.
As a corollary of the above, when using Hot Rod with two clusters (one as main and another as backup) a TLS issue can happen, and the sniHostName might need to be set: "No trusted certificate found".
The above issue has little to do with which kind of cert (discussion above on type of cert: let's encrypt vs self-signed) is being used but rather with this fact above, and therefore replacing the service certificate (spec.security.endpointEncryption.certSecretName) with another one, or using type Secret (instead of type Service) won't avoid this behavior.

The default Ingress's Route's certificate will be self-signed, consequently, the user needs to import it into the client's truststore - or buy a real certificate. Or use a company CA that you already trust.

Diagnostic Steps

To get a route:

$ oc get route
NAME                               HOST/PORT                                                                                          PATH   SERVICES                  PORT    TERMINATION   WILDCARD
infinispan_cr-no-enc-external   infinispan_cr-no-enc-external.com          infinispan_cr-route-no-enc   11222                 None
route-secret-external              route-secret-external-dg-test-nyc.apps.com                     route-secret              11222   passthrough   None

The "server name" TLS extension output below is SNI:

sh-4.4$ echo Q | openssl s_client -connect route-secret-external.com:443 2>/dev/null -tlsextdebug | grep "server name"
TLS server extension "server name" (id=0), len=0
sh-4.4$ exit
exit
....
$ oc rsh route-service-0
sh-4.4$ echo Q | openssl s_client -connect route-service-external.com:443 2>/dev/null -tlsextdebug | grep "server name"
TLS server extension "server name" (id=0), len=0

Get secrets and verify Issuer and Subject:

$ oc get secrets -A -o json | jq -r '.items | sort_by(.metadata.namespace,.metadata.name) |.[] |select((.type == "kubernetes.io/tls") or (.type == "SecretTypeTLS"))| "\(.metadata.namespace) \(.metadata.name) \(.data | to_entries[] | select(.key | test("key") or test("Key") | not)| .value)"' | while read namespace name cert; do echo -e "\n${namespace} - ${name}\n##################"; echo $cert | base64 -d | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin |openssl pkcs7 -print_certs -text -noout | grep -A4 Issuer:; done 
...
...
dg-test-nyc - dg-cluster-enc-credentials
##################
        Issuer: CN=openshift-service-serving-signer@1686163750 <------------
        Validity
            Not Before: Jun  7 19:38:44 2023 GMT
            Not After : Jun  6 19:38:45 2025 GMT
        Subject: CN=dg-cluster-enc.dg-test-nyc.svc
--
        Issuer: CN=openshift-service-serving-signer@1686163750 <-------------
        Validity
            Not Before: Jun  7 18:49:09 2023 GMT
            Not After : Aug  5 18:49:10 2025 GMT
        Subject: CN=openshift-service-serving-signer@1686163750 <------------

Verifying certificate chain:

To list which entity signed the certificate (on this case is self-certificate): echo Q | openssl s_client -connect $route:443

### Get inside the pod:
$ oc rsh example-infinispan-2-0
###
sh-4.4$ echo Q | openssl s_client -connect dg-cluster-nyc-both-external-dg-test-nyc.apps.com:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3 <---------
verify return:1
depth=0 CN = *.apps.rosa.openshiftapps.com
verify return:1
---
Certificate chain
 0 s:CN = *.apps.rosa.openshiftapps.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

To list each certificate: echo Q | openssl s_client -showcerts -connect $route:443

sh-4.4$ echo Q | openssl s_client -showcerts -connect dg-cluster-route-no-enc-external.com:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.apps.rosa.openshiftapps.com
verify return:1
---
Certificate chain
 0 s:CN = *.apps.rosa.openshiftapps.com
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
(...)
-----END CERTIFICATE-----
---
Server certificate
subject=CN = *.apps.rosa.openshiftapps.com
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4636 bytes and written 458 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.