Connect to Data Grid 8 using CLI generates Exception when enables SSL

Environment

• Red Hat Data Grid (RHDG)
  • 8.x

Issue

• Trying to connect to the Data Grid 8 cluster using cli.sh IP:PORT generates java.lang.IllegalStateException: No SSL TrustStore configuration in command output and below WARN trace in the server.log:

19:51:53,025 WARN  (SINGLE_PORT-ServerIO-4-1) [io.netty.handler.ssl.ApplicationProtocolNegotiationHandler] [id: 0x1783dae6, L:/192.168.0.10:11222 ! R:/192.168.0.10:43510] TLS handshake failed: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f726573742f76322f73657276657220485454502f312e310d0a486f73743a203132372e302e302e313a31313232320d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a4163636570742d456e636f64696e673a20677a69700d0a557365722d4167656e743a206f6b687474702f332e31342e382e7265646861742d30303030320d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
...

Resolution

Adding the --trustall option to the CLI command connection avoids any SSL-related ERROR making possible trust in the available SSL certificates. The command should be in the format ./cli.sh -c https://YOUR_HOST_ADDRESS:DG_PORT --trustall. Here is an example:

 ./cli.sh -c https://192.168.0.10:11222 --trustall
Username: your_user
Password:  your_password
[node1@cluster//containers/default]> 

Also, it is possible specify the truststore:

 ./cli.sh -c https://192.168.0.10:11222 --truststore /path/to/truststore --truststore-password ${TRUSTSTORE_PASSWORD}
Username: your_user
Password:  your_password
[node1@cluster//containers/default]> 

If the problem is related to FIPS, as in OpenSSL internal error: FATAL FIPS SELFTEST FAILURE", one option is to disable OpenSSL via -Dorg.infinispan.openssl=false as detailed on Data Grid 8 OpenSSL session objects native accumulation.

Root Cause

When enabling SSL in the Data Grid instances, should define the trust certificate or trust every available certificate each time when tries to connect using the CLI.

Using the flag generate-self-signed-certificate-host makes the DG instance generate a certificate at run time, in the example below it will generate a file called server.jks, the file will be generated on start up (or overwritten) and will have the password as set:

<server-identities>
                  <ssl>
                     <keystore path="server.jks"
                     relative-to="infinispan.server.config.path"
                     password="secret"
                     alias="server"
                     generate-self-signed-certificate-host="localhost"/>
                     <!-- Configures Data Grid Server to use specific TLS versions and cipher suites. -->
                     <engine enabled-protocols="TLSv1.2 TLSv1.3"/>
                     </ssl>
               </server-identities>

So then the following will happen:

### Before:
$ ls
groups.properties  infinispan-local.xml  infinispan.xml  infinispan-xsite.xml  log4j2.xml  users.properties
### Run DG 8
$ ls
groups.properties  infinispan-local.xml  infinispan.xml  infinispan-xsite.xml  log4j2.xml  server.jks   <----- file is created 
users.properties

Diagnostic Steps

1. Verify the trust store file: $ keytool -v -list -keystore /path/redhat-datagrid-8.3.0-server/server/conf/server.jks
2. The flag generate-self-signed-certificate-host can be a problem if the previous one is registered, the certificate (a jks file, p12 format) not necessarily will be accepted by the browser - so it might add a "not valid certificate at the corner"
3. To view the certificate in Firefox go on the url address, click on the locker (with an exclamation mark) > Connection not secure > More information > View Certificate; a new windows will open with the fingerprints of the certificate, just compare with the .p12 used by the server when starting.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.