How to configure Loki Object Storage CA certificate in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat Loki Operator
  • LokiStack

Issue

  • How to configure the Certificate Authority (CA) certificate for the ObjectStorage endpoint in the LokiStack?

  • After installing the Network Observability Operator using the SSL/TLS protocol, an error with the CA trust certificate occurs while Loki Operator attempts to save data on a S3 bucket using a customized certificate.

  • Errors while flushing to the ObjectStorage backend:

    msg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": tls: failed to verify certificate: x509: certificate signed by unknown authority
    
    msg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead
    
  • Error message in the LokiStack status:

        message: 'Invalid object storage CA configmap contents: key not present or data
          empty: service-ca.crt'
    

Resolution

When using S3 Compatible object storage or S3 object storage with custom certificate

  1. Create a ConfigMap containing the CA certificate necessary to connect to the Object Storage endpoint:

    # oc create cm loki-s3-ca -n <namespace> --from-file=service-ca.crt=./<CA certificate file name>.pem
    
  2. Modify the LokiStack resource to add the spec.storage.tls.caName field referencing the name of the ConfigMap previously created, for example:

    # oc patch lokistack/<LokiStack CR name> -n <name> --type merge -p '{"spec":{"storage":{"tls":{"caName":"loki-s3-ca"}}}}'
    

When using ODF Object storage in the same OpenShift that the Loki pods

For OpenShift Data Foundation (ODF), if it uses the OCP Service CA, so it is not necessary to create an extra ConfigMap, the openshift-service-ca.crt ConfigMap existing in all the namespaces when created can be used and configured for providing the OpenShift CA:

$ oc patch lokistack/<LokiStack CR name> -n <namespace> --type merge -p '{"spec":{"storage":{"tls":{"caName":"openshift-service-ca.crt","caKey":"service-ca.crt"}}}}'

When using noobaa ObjectStorage backend, refer to This content is not included.how to configure Lokistack with noobaa ObjectStorage backend?

IMPORTANT NOTES

Root Cause

The Loki ingester, querier or compactor are not able to trust the certificate provided by the Object Storage endpoint.

Diagnostic Steps

  • Confirm the ingester pod is tracing x509: certificate signed by unknown authority or x509: certificate relies on legacy Common Name field, use SANs instead errors while flushing logs. In this example, the namespace is openshift-logging and the name of the LokiStack Custom Resource (CR) is logging-loki:

    $ oc logs logging-loki-ingester-0 -n openshift-logging | tail -1
    2023-03-11T20:49:50.009849043+02:00 level=error ts=2023-03-11T18:49:49.968811906Z caller=flush.go:144 org_id=infrastructure msg="failed to flush" err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"https://objectstorage.example.com:1234/loki-storage/backend\": x509: certificate signed by unknown authority, num_chunks: 1, labels: {kubernetes_container_name=\"registry-server\", kubernetes_host=\"worker0.example.com\", kubernetes_namespace_name=\"openshift-marketplace\", kubernetes_pod_name=\"community-operators-asdfg\", log_type=\"infrastructure\"}"
    
  • Check if the ConfigMap for the service-ca.crt is missing, or the data incorrect in the LokiStack resource:

    $ oc get lokistack logging-loki -n openshift-logging -o yaml
    [...]
        message: 'Invalid object storage CA configmap contents: key not present or data
          empty: service-ca.crt'
        reason: InvalidObjectStorageCAConfigMap
        status: "True"
        type: Degraded
    [...]
    
  • Check if the ConfigMap is created an has the key service-ca.crt (in this example, the name of the ConfigMap is loki-s3-ca):

    $ oc get lokistack logging-loki -n openshift-logging -o yaml | grep caName
            caName: loki-s3-ca
    $ oc get cm loki-s3-ca -n openshift-logging -o yaml
    [...]
    data:
      service-ca.crt: |
        -----BEGIN CERTIFICATE-----
    [...]
    
  • If the certificate is correctly configured in the ConfigMap, but the error message is tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead, ensure the certificate has configured a Subject Alternative Name (SAN) as explained in Error x509: certificate relies on legacy Common Name field, use SANs instead in Openshift.

Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.