How to execute a pipeline from a Jupyter notebook in a disconnected environment

Solution Unverified - Updated

Environment

  • Red Hat OpenShift AI (RHOAI)
    • Version: <= 2.8

Issue

Executing a pipeline from a Jupyter notebook in a disconnected environment might fail because of an SSL certificate validation issue.

A failure due to pipeline SSL certificate validation is indicated by the following error in the Jupyter notebook logs:

  File "/opt/app-root/lib64/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ds-pipeline-pipelines-definition.apps.example.com', port=443): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

Resolution

When using a data science pipeline in the data science project environment to create and execute pipelines from the Jupyter notebook, as described in This content is not included.Running a pipeline in JupyterLab, you might have SSL certificate validation issues.

This issue happens because the pipeline certificates are not captured. Depending on your setup, you can resolve the issue in one of the following ways:

Method 1: Use the self-signed certificate provided from OpenShift

Method 2: Use a custom certificate for the pipeline server
This method is similar to the workaround documented in Data Science Pipelines workaround for an object storage connection with a self-signed certificate.

  1. If the custom certificates are present in a config map (for example, my-ca-config), mount the config map to the Notebook Custom Resource (CR), to use the custom certificate.

    (a) Use the oc client to edit the Notebook CR as follows:
    oc edit notebook <notebook-name> -n <data-science-project>

    (b) Find the container specification of the Notebook CR, and add the following updates:

    - In the `spec` > `container` > `notebook` section:

    env:  
  - name: PIPELINES_SSL_SA_CERTS
    value: /etc/ssl/certs/ca-cert.crt
volumeMounts:
  - mountPath: /etc/ssl/certs
    name: my-ca-config

    - In the `spec` > `volumes` section:

    volumes:
  - name: my-ca-config
    configMap:
      name: my-ca-config
      items:
        - key: ca.crt
          path: ca.crt

Method 3: Build custom workbench images with the CA certificates embedded in them

  1. Update the workbench image to include the following text:

       # Use a multi-stage build to update CA certificates
   # Stage 1: Perform operations requiring root privileges
   FROM <base-image>

   USER root
   COPY ca-certificates.crt /etc/pki/ca-trust/source/anchors/
   RUN update-ca-trust extract

   # Copy updated CA certificates from the builder stage
   COPY --from=builder /etc/pki/ca-trust/extracted /etc/pki/ca-trust/extracted

   # Ensure the container runs as a non-root user
   USER 1001

  2. When using this custom image, edit the workbench as described in This content is not included.Updating a project workbench, and set the following environment variable to point to the SSL certificate from the embedded path:

    PIPELINES_SSL_SA_CERTS: /etc/pki/ca-trust/extracted/ca-certificate.crt

See also:

Root Cause

To execute a pipeline from a notebook, you must use a connection that uses either HTTP or HTTPS with a certificate signed by a publicly trusted certificate authority (commonly called a “valid certificate”). This document describes a workaround that enables you to use a self-signed certificate.

Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.