How to configure LokiStack ObjectStorage CA certificate for OpenStack Swift and Azure object Storage in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
    • 6
  • Red Hat Loki Operator
  • LokiStack
  • OpenStack Swift as object storage
  • Azure Storage as object storage

Issue

  • How to configure the Certificate Authority (CA) certificate for the ObjectStorage endpoint in the LokiStack for using as object storage backend OpenStack Swift?

  • How to configure the Certificate Authority (CA) certificate for the ObjectStorage endpoint in the LokiStack for using as object storage backend Azure?

  • After configuring the CA as described in the article How to configure LokiStack ObjectStorage CA certificate in RHOCP 4 , it continues giving the error:

    2023-10-04T10:47:04.237909733Z level=error ts=2023-10-04T10:47:04.237661363Z caller=log.go:171 msg="error running loki" err="Get \"https://192.168.1.1:443/swift/v1/AUTH_XXXXXXXXXXXXXXXXX\": tls: failed to verify certificate: x509: certificate signed by unknown authority\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/remote-source/loki/app/pkg/storage/store.go:187\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/remote-source/loki/app/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/remote-source/loki/app/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/remote-source/loki/app/pkg/loki/modules.go:656\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/remote-source/loki/app/pkg/loki/loki.go:458\nmain.main\n\t/remote-source/loki/app/cmd/loki/main.go:110\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/remote-source/loki/app/pkg/loki/loki.go:458\nmain.main\n\t/remote-source/loki/app/cmd/loki/main.go:110\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598"

Resolution

Red Hat is aware of these feature requests, currently under review in:

For OpenStack Swift Storage

Red Hat reviewed the feature request in RFE This content is not included.OBSDA-535 - "Support to configure .spec.storage.tls for OpenStack Swift storage in Loki" and delivered a fix in RHOL 6.2 through errata RHBA-2025:2398.

For Azure Storage in Loki
RFE This content is not included.OBSDA-836 - "Support to configure .spec.storage.tls for Azure storage in Loki"

If these features are required, open a support case on the Red Hat Customer Portal referring to this solution.

Root Cause

Not possible to define spec.storage.tls.caName when the object storage backend is OpenStack Swift or Azure Object Storage.

When using OpenStack Swift or Azure Object Storage, if spec.storage.tls.caName is defined, it doesn't return an error, but it's not considered for the Loki configuration, failing to stablish a the network connection with the error:

tls: failed to verify certificate: x509: certificate signed by unknown authority [...]

Diagnostic Steps

Verify the LokiStack logging-loki custom resource has defined a secret containing the CA and also the type of Loki object storage used is swift or azure:

$ oc get LokiStack logging-loki -o yaml -n openshift-logging
...
  spec:
    managementState: Managed
    size: 1x.extra-small
    storage:
      schemas:
      - effectiveDate: "2022-06-01"
        version: v12
      secret:
        name: logging-loki-swift  <---- secret containing the CA
        type: swift               <---- type of storage used
      tls:
        caName: loki-swift-ca      <---- name of the field containing the CA for connecting to the OpenStack Swift object storage backend

Confirm that visible the error an error when accessing to the object storage because of the certificate is signed by an unknown authority:

$ oc logs logging-loki-ingester-0 -n openshift-logging |grep "x509: certificate signed by unknown authority"
2023-10-04T10:47:04.237716082Z Get "https://192.168.1.1:443/swift/v1/AUTH_XXXXXXXXXXXXXXXXX": tls: failed to verify certificate: x509: certificate signed by unknown authority
2023-10-04T10:47:04.237909733Z level=error ts=2023-10-04T10:47:04.237661363Z caller=log.go:171 msg="error running loki" err="Get \"https://192.168.1.1:443/swift/v1/AUTH_XXXXXXXXXXXXXXXXX\": tls: failed to verify certificate: x509: certificate signed by unknown authority\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(*store).chunkClientForPeriod\n\t/remote-source/loki/app/pkg/storage/store.go:187\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/remote-source/loki/app/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/remote-source/loki/app/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/remote-source/loki/app/pkg/loki/modules.go:656\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/remote-source/loki/app/pkg/loki/loki.go:458\nmain.main\n\t/remote-source/loki/app/cmd/loki/main.go:110\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/remote-source/loki/app/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/remote-source/loki/app/pkg/loki/loki.go:458\nmain.main\n\t/remote-source/loki/app/cmd/loki/main.go:110\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598"
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.