Data Grid Operator Encryption types and implications
Environment
- Red hat OpenShift Container Platform (OCP)
- 4.x
- Red Hat Data Grid (RHDG)
- 8.x
- Operator
Issue
- What are the Data Grid Operator Encryption types? What do they mean?
- What are the implications of Encryption type Service?
- If not set SNI Hostname then getting that error "No trusted certificate found"
Resolution
| Type | How it works | Implications | Usage | SNI details |
|---|---|---|---|---|
| Service | with Service, the server certificate is generated by Openshift (and renew by it) | The service, .e.g serv ice.beta.openshift.io, validates the TLS | Within OCP | type Service is the only variant of Route that needs SNI. Not using SNI: acts the same as Secret does (SNI or not). |
| Secret | user is entirely responsible for generating their certificates using their own CA and adding them to a Secret so that the Infinispan server can mount them on the pod | The DG pod validates the certificate | Within or outside OCP | - |
| None | User disables encryption, the Route created will have no termination | The DG pod doesn't valid no certificate whatsoever | Within and outside OCP | - |
For changing the Encryption's type at runtime, see the notes on Editing Encryption at Infinispan Custom Resource on the fly leads to misbehavior, where the main recommendation is to avoid changing at runtime.
Type Secret:
On type Secret, the user must provide their own certificates - and validate that the cert is correctly mounted on the pod later:
security:
endpointAuthentication: true
endpointEncryption:
certSecretName: infinispan_cr-cert-secret <------------------ secret
clientCert: None
type: Secret
endpointSecretName: infinispan_cr-secret
Secret details:
- if providing a
.p12keystore, the user needs to use "keystore.p12" as the key in the Secret. - if you're providing a public/private key combo, the user needs to use the keys "tls.crt" and "tls.key" respectively.
Using another name or name combination: it will fail - currently silently (without any message).
Type Service:
When the type is Service, the Secret specified by spec.security.endpointEncryption.certSecretName is used as an argument to the service annotation used to configure the This page is not included, but the link has been rewritten to point to the nearest parent document.Openshift certifates i.e. operator sets service.beta.openshift.io/serving-cert-secret-name: spec.security.endpointEncryption.certSecretName on the underlying service - on this case DG Operator doesn't generate the service but OCP itself generates after the DG add the service.beta.openshift.io annotation.
OCP 4 Generates and Renews the cert automatically, where OCP 4 automatically rotates the service CA certificate and although it is valid for 26 months, when there is less than 13 months validity left it is automatically refreshed. As a side note, OpenShift 3.11 it's the user's responsibility to ensure they get renewed before expiration in comparison to auto-managed in OCP 4.x fortunately.
security:
endpointAuthentication: true
endpointEncryption:
certSecretName: infinispan_cr-cert-secret <------------------ self signed
certServiceName: service.beta.openshift.io
clientCert: None
type: Service
endpointSecretName: infinispan_cr-generated-secret
By default the secret infinispan_cr-cert-secret will be self-signed. The user can set another one, which can be signed by another authority.
Certificate location
All TLS-related files will be mounted in the respective Infinispan Custom Resource's statefulset at the encryption location - /etc/encrypt/keystore:
$ cd /etc/encrypt/keystore
sh-4.4$ ls
tls.crt ------------------> x509 certificate
tls.key ------------------> private key
Given the .pem is a combination of the certificate and the private key so then DG Operator will combine them into a keystore.pem file: the /etc/security/conf/operator-security/keystore.pemis a combination of/etc/encrypt/keystoretls.keyand/etc/encrypt/keystore/tls.crt, consequently the keystore.pemonly exists at/etc/security/conf/operator-security/keystore.pem. The way the keystore configuration works is that it expects .pemfile with both.keyand.crt, hence the combination process. When the keystore element is not explicitly defined in the ConfigMap`, then no keystore is loaded at all by the server.
Comparing certs self-signed vs. third-party authentication
Self-signed:
When the *.apps cert for his is signed by the Ingress Operator's cert, which is just signed by itself. i.e. there's no chain of trust up to another trusted entity to verify it's legitimate. By default, Ingress Operator will be self-signed. And the service certificated created via annotation service (for instance DG Operator type Service) will be self-signed.
3rd party authentication
When the cert is signed by another authority, .e.g Let's Encrypt, which has a chain of trust up to one of the internet's root Certificate Authorities (CA).
Replacing default Ingress certificate
For Replacing the default Ingress certificate: This page is not included, but the link has been rewritten to point to the nearest parent document.OCP 4.12 Guide.
Issue with Route's SNI and expose type Route
As explained on the solution DG 8 TLS SNI usage in Client server communication, although Data Grid default route is passthrough (not edge, nor re-encrypt), the Ingress's Route will impose SNI and will use the default apps. certificates.
Root Cause
SNI
SNI (Server Name Indication) adds the server name to the initial handshake - see solution DG 8 TLS SNI usage in Client server communication for more details.
The SNI problem is/was that it was re-using the same SNI for different pods, causing issues. This isn't an issue with BASIC client intelligence since there is only one connection (unless cross-site).
Certificate rotation
If the cert is rotated then the Infinispan server will not pick up the change until the pod has been restarted as the Infinispan server is not able to dynamically load the cert because the JVM cannot just change the cert on the fly it needs the JVM to restart.
| Type | Behavior |
|---|---|
| Service type Encryption | The user can force the rotation by deleting the respective service (and configmap), which will force the Service-CA operator to create a new one. |
| Secret type Encryption | The user has control over the certificate, but may need to drop the pods (custom resource for zero pods). |
Secret vs Component
| Component | Secret | Usage |
|---|---|---|
| Service type encryption | certSecretName secret | This is used as an argument to the service annotation used to configure the This page is not included, but the link has been rewritten to point to the nearest parent document.Openshift certifates |
| endpointSecretName | (Infinispan-CR)-generated-secret | This content is not included.Credentials for access to DG, used for cli access. |
| ConfigListener | (Infinispan-CR)-config-listener-docker-id, (Infinispan-CR)-config-listener-token-id | ConfigListener tokens/secret |
| Operator | (Infinispan-CR)-generated-operator-secret | Operator secret |
| DG Cluster certificate | (Infinispan-CR)-cert-secret | DG cluster certificate to create an application connection, .e.g This content is not included.SpringBoot |
| Infinispan+Operator | (Infinispan-CR)-infinispan-security | Combination of the tls pair for java usage. |
Diagnostic Steps
Rotation:
In case automatic rotation fails, follow this procedure:
- Bring down the Infinispan CR's replicas to 0 then scale up your pod. This will ensure certs are properly applied. replicas: n --> replicas: 0 --then scale up to n.
- Otherwise, try to delete the
(Infinispan-Cr)-infinispan-securityand see if the secret is recreated.
To get a route:
$ oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
infinispan_cr-no-enc-external infinispan_cr-no-enc-external.com infinispan_cr-route-no-enc 11222 None
route-secret-external route-secret-external-dg-test-nyc.apps.com route-secret 11222 passthrough None
Keystore issues:
a - The configuration just doesn’t add the keystore. Not error is logged.
The problem here is that the keystore must be added as keystore.p12 - example:
keystore.p12: >-
b- Caused by: java.security.KeyStoreException: ELY02035: KeyStore type could not be detected
The problem here is the password was not correct
c - Caused by: org.infinispan.commons.CacheConfigurationException: ISPN080069: Alias 'server' not in keystore '/etc/encrypt/keystore/keystore.p12'
The problem here is the alias was not on the keystone file.
Debug logs
Using logging.categories.org.wildfly.security: trace
Provide the following logs:
18:17:06,443 TRACE (main) [org.wildfly.security] Obtaining KeyStore instance of type pkcs12, providers: [WildFlyElytronPasswordProvider version 1.0]
18:17:06,444 TRACE (main) [org.wildfly.security] Obtaining KeyStore instance of type pkcs12, providers: [WildFlyElytronSaslGs2Provider version 1.0, WildFlyElytronSaslGssapiProvider version 1.0, WildFlyElytronSaslOAuth2Provider version 1.0, WildFlyElytronSaslLocalUserProvider version 1.0, WildFlyElytronSaslExternalProvider version 1.0, WildFlyElytronSaslScramProvider version 1.0, WildFlyElytronSaslDigestProvider version 1.0, WildFlyElytronSaslPlainProvider version 1.0, WildFlyElytronHttpSpnegoProvider version 1.0, WildFlyElytronHttpClientCertProvider version 1.0, WildFlyElytronHttpDigestProvider version 1.0, WildFlyElytronHttpBearerProvider version 1.0, WildFlyElytronHttpBasicProvider version 1.0, SUN version 17, SunRsaSign version 17, SunEC version 17, SunJSSE version 17, SunJCE version 17, SunJGSS version 17, SunSASL version 17, XMLDSig version 17, SunPCSC version 17, JdkLDAP version 17, JdkSASL version 17, SunPKCS11 version 17]
18:17:06,445 TRACE (main) [org.wildfly.security] Obtained KeyStore instance: java.security.KeyStore@2db2cd5
18:17:06,446 TRACE (main) [org.wildfly.security] KeyStoreCredentialStore: loading backing KeyStore /opt/infinispan/server/conf/internal-credentials.pfx external = false
18:17:06,661 INFO (main) [org.wildfly.openssl.SSL] WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1k FIPS 25 Mar 2021
18:17:06,663 INFO (main) [org.infinispan.SECURITY] ISPN000946: Using OpenSSL Provider
18:17:06,717 TRACE (main) [org.wildfly.security] KeyStoreCredentialStore: contains exact type <------------
18:17:06,896 DEBUG (main) [org.wildfly.security] Searching provider for: PKCS12 <------------ PKCS12
18:17:06,897 DEBUG (main) [org.wildfly.security] Provider found: SUN <------------
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.