Data Grid Operator Encryption types and implications

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • Operator

Issue

  • What are the Data Grid Operator Encryption types? What do they mean?
  • What are the implications of Encryption type Service?
  • If not set SNI Hostname then getting that error "No trusted certificate found"

Resolution

TypeHow it worksImplicationsUsageSNI details
Servicewith Service, the server certificate is generated by Openshift (and renew by it)The service, .e.g serv ice.beta.openshift.io, validates the TLSWithin OCPtype Service is the only variant of Route that needs SNI. Not using SNI: acts the same as Secret does (SNI or not).
Secretuser is entirely responsible for generating their certificates using their own CA and adding them to a Secret so that the Infinispan server can mount them on the podThe DG pod validates the certificateWithin or outside OCP-
NoneUser disables encryption, the Route created will have no terminationThe DG pod doesn't valid no certificate whatsoeverWithin and outside OCP-

For changing the Encryption's type at runtime, see the notes on Editing Encryption at Infinispan Custom Resource on the fly leads to misbehavior, where the main recommendation is to avoid changing at runtime.

Type Secret:

On type Secret, the user must provide their own certificates - and validate that the cert is correctly mounted on the pod later:

    security:
      endpointAuthentication: true
      endpointEncryption:
        certSecretName: infinispan_cr-cert-secret <------------------ secret
        clientCert: None
        type: Secret
      endpointSecretName: infinispan_cr-secret

Secret details:

  • if providing a .p12 keystore, the user needs to use "keystore.p12" as the key in the Secret.
  • if you're providing a public/private key combo, the user needs to use the keys "tls.crt" and "tls.key" respectively.

Using another name or name combination: it will fail - currently silently (without any message).

Type Service:

When the type is Service, the Secret specified by spec.security.endpointEncryption.certSecretName is used as an argument to the service annotation used to configure the This page is not included, but the link has been rewritten to point to the nearest parent document.Openshift certifates i.e. operator sets service.beta.openshift.io/serving-cert-secret-name: spec.security.endpointEncryption.certSecretName on the underlying service - on this case DG Operator doesn't generate the service but OCP itself generates after the DG add the service.beta.openshift.io annotation.
OCP 4 Generates and Renews the cert automatically, where OCP 4 automatically rotates the service CA certificate and although it is valid for 26 months, when there is less than 13 months validity left it is automatically refreshed. As a side note, OpenShift 3.11 it's the user's responsibility to ensure they get renewed before expiration in comparison to auto-managed in OCP 4.x fortunately.

    security:
      endpointAuthentication: true
      endpointEncryption:
        certSecretName: infinispan_cr-cert-secret <------------------ self signed
        certServiceName: service.beta.openshift.io
        clientCert: None
        type: Service
      endpointSecretName: infinispan_cr-generated-secret

By default the secret infinispan_cr-cert-secret will be self-signed. The user can set another one, which can be signed by another authority.

Certificate location

All TLS-related files will be mounted in the respective Infinispan Custom Resource's statefulset at the encryption location - /etc/encrypt/keystore:

$ cd /etc/encrypt/keystore
sh-4.4$ ls
tls.crt              ------------------> x509 certificate
tls.key              ------------------> private key

Given the .pem is a combination of the certificate and the private key so then DG Operator will combine them into a keystore.pem file: the /etc/security/conf/operator-security/keystore.pemis a combination of/etc/encrypt/keystoretls.keyand/etc/encrypt/keystore/tls.crt, consequently the keystore.pemonly exists at/etc/security/conf/operator-security/keystore.pem. The way the keystore configuration works is that it expects .pemfile with both.keyand.crt, hence the combination process. When the keystore element is not explicitly defined in the ConfigMap`, then no keystore is loaded at all by the server.

Comparing certs self-signed vs. third-party authentication

Self-signed:
When the *.apps cert for his is signed by the Ingress Operator's cert, which is just signed by itself. i.e. there's no chain of trust up to another trusted entity to verify it's legitimate. By default, Ingress Operator will be self-signed. And the service certificated created via annotation service (for instance DG Operator type Service) will be self-signed.

3rd party authentication
When the cert is signed by another authority, .e.g Let's Encrypt, which has a chain of trust up to one of the internet's root Certificate Authorities (CA).

Replacing default Ingress certificate

For Replacing the default Ingress certificate: This page is not included, but the link has been rewritten to point to the nearest parent document.OCP 4.12 Guide.

Issue with Route's SNI and expose type Route

As explained on the solution DG 8 TLS SNI usage in Client server communication, although Data Grid default route is passthrough (not edge, nor re-encrypt), the Ingress's Route will impose SNI and will use the default apps. certificates.

Root Cause

SNI

SNI (Server Name Indication) adds the server name to the initial handshake - see solution DG 8 TLS SNI usage in Client server communication for more details.
The SNI problem is/was that it was re-using the same SNI for different pods, causing issues. This isn't an issue with BASIC client intelligence since there is only one connection (unless cross-site).

Certificate rotation

If the cert is rotated then the Infinispan server will not pick up the change until the pod has been restarted as the Infinispan server is not able to dynamically load the cert because the JVM cannot just change the cert on the fly it needs the JVM to restart.

TypeBehavior
Service type EncryptionThe user can force the rotation by deleting the respective service (and configmap), which will force the Service-CA operator to create a new one.
Secret type EncryptionThe user has control over the certificate, but may need to drop the pods (custom resource for zero pods).

Secret vs Component

ComponentSecretUsage
Service type encryptioncertSecretName secretThis is used as an argument to the service annotation used to configure the This page is not included, but the link has been rewritten to point to the nearest parent document.Openshift certifates
endpointSecretName(Infinispan-CR)-generated-secretThis content is not included.Credentials for access to DG, used for cli access.
ConfigListener(Infinispan-CR)-config-listener-docker-id, (Infinispan-CR)-config-listener-token-idConfigListener tokens/secret
Operator(Infinispan-CR)-generated-operator-secretOperator secret
DG Cluster certificate(Infinispan-CR)-cert-secretDG cluster certificate to create an application connection, .e.g This content is not included.SpringBoot
Infinispan+Operator(Infinispan-CR)-infinispan-securityCombination of the tls pair for java usage.

Diagnostic Steps

Rotation:

In case automatic rotation fails, follow this procedure:

  1. Bring down the Infinispan CR's replicas to 0 then scale up your pod. This will ensure certs are properly applied. replicas: n --> replicas: 0 --then scale up to n.
  2. Otherwise, try to delete the (Infinispan-Cr)-infinispan-security and see if the secret is recreated.

To get a route:

$ oc get route
NAME                               HOST/PORT                                                                                          PATH   SERVICES                  PORT    TERMINATION   WILDCARD
infinispan_cr-no-enc-external   infinispan_cr-no-enc-external.com          infinispan_cr-route-no-enc   11222                 None
route-secret-external              route-secret-external-dg-test-nyc.apps.com                     route-secret              11222   passthrough   None

Keystore issues:

a - The configuration just doesn’t add the keystore. Not error is logged.
The problem here is that the keystore must be added as keystore.p12 - example:

  keystore.p12: >-

b- Caused by: java.security.KeyStoreException: ELY02035: KeyStore type could not be detected
The problem here is the password was not correct

c - Caused by: org.infinispan.commons.CacheConfigurationException: ISPN080069: Alias 'server' not in keystore '/etc/encrypt/keystore/keystore.p12'
The problem here is the alias was not on the keystone file.

Debug logs

Using logging.categories.org.wildfly.security: trace

Provide the following logs:

18:17:06,443 TRACE (main) [org.wildfly.security] Obtaining KeyStore instance of type pkcs12, providers: [WildFlyElytronPasswordProvider version 1.0]
18:17:06,444 TRACE (main) [org.wildfly.security] Obtaining KeyStore instance of type pkcs12, providers: [WildFlyElytronSaslGs2Provider version 1.0, WildFlyElytronSaslGssapiProvider version 1.0, WildFlyElytronSaslOAuth2Provider version 1.0, WildFlyElytronSaslLocalUserProvider version 1.0, WildFlyElytronSaslExternalProvider version 1.0, WildFlyElytronSaslScramProvider version 1.0, WildFlyElytronSaslDigestProvider version 1.0, WildFlyElytronSaslPlainProvider version 1.0, WildFlyElytronHttpSpnegoProvider version 1.0, WildFlyElytronHttpClientCertProvider version 1.0, WildFlyElytronHttpDigestProvider version 1.0, WildFlyElytronHttpBearerProvider version 1.0, WildFlyElytronHttpBasicProvider version 1.0, SUN version 17, SunRsaSign version 17, SunEC version 17, SunJSSE version 17, SunJCE version 17, SunJGSS version 17, SunSASL version 17, XMLDSig version 17, SunPCSC version 17, JdkLDAP version 17, JdkSASL version 17, SunPKCS11 version 17]
18:17:06,445 TRACE (main) [org.wildfly.security] Obtained KeyStore instance: java.security.KeyStore@2db2cd5
18:17:06,446 TRACE (main) [org.wildfly.security] KeyStoreCredentialStore: loading backing KeyStore /opt/infinispan/server/conf/internal-credentials.pfx external = false
18:17:06,661 INFO (main) [org.wildfly.openssl.SSL] WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1k FIPS 25 Mar 2021
18:17:06,663 INFO (main) [org.infinispan.SECURITY] ISPN000946: Using OpenSSL Provider
18:17:06,717 TRACE (main) [org.wildfly.security] KeyStoreCredentialStore: contains exact type <------------
18:17:06,896 DEBUG (main) [org.wildfly.security] Searching provider for: PKCS12               <------------ PKCS12
18:17:06,897 DEBUG (main) [org.wildfly.security] Provider found: SUN                          <------------ 
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.