Workbench workaround for an object storage connection with a self-signed certificate

Solution Unverified - Updated

Environment

  • Red Hat OpenShift AI (RHOAI)
    • Version: < 2.8

Issue

Executing a pipeline from a Jupyter notebook by using the Elyra tool in a disconnected environment might fail because of an SSL certificate validation issue for the cloud object storage connection.

A failure due to cloud object storage SSL certificate validation is indicated by the following error in the Jupyter notebook logs:

file "/opt/app-root/lib64/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='minio-api-object-datastore.apps.example.com', port=443): Max retries exceeded with url: /demo-bucket?location= (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

Resolution

When using a data science pipeline in the data science project environment to create and execute pipelines from the Jupyter notebook, as described in This content is not included.Running a pipeline in JupyterLab, you might have SSL certificate validation issues.

This issue happens because the object storage certificates are not captured. Depending on your setup, you can resolve the issue in one of the following ways:

Method 1: Use object storage deployed on OpenShift and use self-signed certificate for OpenShift

Method 2: Use external object storage with custom certificates
This method is similar to the workaround documented in Data Science Pipelines workaround for an object storage connection with a self-signed certificate.

  1. If the custom certificates are present in a config map (for example, my-ca-config), mount the config map to the Notebook Custom Resource (CR), to use the custom certificate.

    (a) Use the oc client to edit the Notebook CR as follows:
    oc edit notebook <notebook-name> -n <data-science-project>

    (b) Find the container specification of the Notebook CR, and add the following updates:

    - In the `spec` > `container` > `notebook` section:

           env:  
       - name: SSL_CERT_FILE
         value: /etc/ssl/certs/ca-cert.crt
       volumeMounts:
         - mountPath: /etc/ssl/certs
           name: my-ca-config

    - In the `spec` > `volumes` section:

           volumes:
         - name: my-ca-config
           configMap:
             name: my-ca-config
             items:
               - key: ca.crt
                 path: ca.crt

Method 3: Build custom workbench images with the CA certificates embedded in them

  1. Update the workbench image to include the following text:

       # Use a multi-stage build to update CA certificates
   # Stage 1: Perform operations requiring root privileges
   FROM <base-image>

   USER root
   COPY ca-certificates.crt /etc/pki/ca-trust/source/anchors/
   RUN update-ca-trust extract

   # Copy updated CA certificates from the builder stage
   COPY --from=builder /etc/pki/ca-trust/extracted /etc/pki/ca-trust/extracted

   # Ensure the container runs as a non-root user
   USER 1001

  2. When using this custom image, edit the workbench as described in This content is not included.Updating a project workbench, and set the following environment variable to point to the SSL certificate from the embedded path:

    SSL_CERT_FILE: /etc/pki/ca-trust/extracted/ca-certificate.crt

See also:

Root Cause

To execute a pipeline from a notebook, you must use an object storage connection that uses either HTTP or HTTPS with a certificate signed by a publicly trusted certificate authority (commonly called a “valid certificate”). This document describes a workaround that enables you to use a self-signed certificate.

Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.