How to configure Loki Object Storage CA certificate in RHOCP 4
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- Red Hat Loki Operator
- LokiStack
Issue
-
How to configure the Certificate Authority (CA) certificate for the
ObjectStorageendpoint in theLokiStack? -
After installing the Network Observability Operator using the SSL/TLS protocol, an error with the CA trust certificate occurs while Loki Operator attempts to save data on a S3 bucket using a customized certificate.
-
Errors while flushing to the
ObjectStoragebackend:msg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": tls: failed to verify certificate: x509: certificate signed by unknown authoritymsg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead -
Error message in the
LokiStackstatus:message: 'Invalid object storage CA configmap contents: key not present or data empty: service-ca.crt'
Resolution
When using S3 Compatible object storage or S3 object storage with custom certificate
-
Create a
ConfigMapcontaining the CA certificate necessary to connect to the Object Storage endpoint:# oc create cm loki-s3-ca -n <namespace> --from-file=service-ca.crt=./<CA certificate file name>.pem -
Modify the
LokiStackresource to add thespec.storage.tls.caNamefield referencing the name of theConfigMappreviously created, for example:# oc patch lokistack/<LokiStack CR name> -n <name> --type merge -p '{"spec":{"storage":{"tls":{"caName":"loki-s3-ca"}}}}'
When using ODF Object storage in the same OpenShift that the Loki pods
For OpenShift Data Foundation (ODF), if it uses the OCP Service CA, so it is not necessary to create an extra ConfigMap, the openshift-service-ca.crt ConfigMap existing in all the namespaces when created can be used and configured for providing the OpenShift CA:
$ oc patch lokistack/<LokiStack CR name> -n <namespace> --type merge -p '{"spec":{"storage":{"tls":{"caName":"openshift-service-ca.crt","caKey":"service-ca.crt"}}}}'
When using noobaa ObjectStorage backend, refer to This content is not included.how to configure Lokistack with noobaa ObjectStorage backend?
IMPORTANT NOTES
- the
caKeywhere theKeyis the data key of a Configmap containing a CA certificate. It needs to be in the same namespace as the LokiStack custom resource. If empty, it defaults toservice-ca.crt- The certificate needs to have the Subject Alternative Name (SAN) set as explained in Error x509: certificate relies on legacy Common Name field, use SANs instead in Openshift.
- For OpenStack Swift storage, this procedure does not work at the moment of writing this article. For OpenStack Swift object backend storage, read the article How to configure tls for accessing to OpenStack Swift object storage in Loki in RHOCP 4 .
- For Azure object storage this procedure does not work at the moment of writing this article as not implemented. Review RFE This content is not included."Support to configure .spec.storage.tls for Azure storage in Loki".
Root Cause
The Loki ingester, querier or compactor are not able to trust the certificate provided by the Object Storage endpoint.
Diagnostic Steps
-
Confirm the ingester pod is tracing
x509: certificate signed by unknown authorityorx509: certificate relies on legacy Common Name field, use SANs insteaderrors while flushing logs. In this example, the namespace isopenshift-loggingand the name of theLokiStackCustom Resource (CR) islogging-loki:$ oc logs logging-loki-ingester-0 -n openshift-logging | tail -1 2023-03-11T20:49:50.009849043+02:00 level=error ts=2023-03-11T18:49:49.968811906Z caller=flush.go:144 org_id=infrastructure msg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": x509: certificate signed by unknown authority, num_chunks: 1, labels: {kubernetes_container_name=\"registry-server\", kubernetes_host=\"worker0.example.com\", kubernetes_namespace_name=\"openshift-marketplace\", kubernetes_pod_name=\"community-operators-asdfg\", log_type=\"infrastructure\"}" -
Check if the
ConfigMapfor theservice-ca.crtis missing, or the data incorrect in theLokiStackresource:$ oc get lokistack logging-loki -n openshift-logging -o yaml [...] message: 'Invalid object storage CA configmap contents: key not present or data empty: service-ca.crt' reason: InvalidObjectStorageCAConfigMap status: "True" type: Degraded [...] -
Check if the
ConfigMapis created an has the keyservice-ca.crt(in this example, the name of theConfigMapisloki-s3-ca):$ oc get lokistack logging-loki -n openshift-logging -o yaml | grep caName caName: loki-s3-ca $ oc get cm loki-s3-ca -n openshift-logging -o yaml [...] data: service-ca.crt: | -----BEGIN CERTIFICATE----- [...] -
If the certificate is correctly configured in the
ConfigMap, but the error message istls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead, ensure the certificate has configured a Subject Alternative Name (SAN) as explained in Error x509: certificate relies on legacy Common Name field, use SANs instead in Openshift.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.