Procedure

1. In the OpenShift web console, click User Management → Groups.

2. Click the name of the group containing the users that you want to view.

   • For administrative users, click the name of your administrator group. for example, rhoai-admins.

   • For normal users, click the name of your user group, for example, rhoai-users.

     The Group details page for the group appears.

Procedure

1. In the OpenShift web console, click User Management → Groups.

2. Click the name of the group you want to add users to.

   • For administrative users, click the administrator group, for example, rhoai-admins.

   • For normal users, click the user group, for example, rhoai-users.

     The Group details page for that group appears.

3. Click Actions → Add Users.

   The Add Users dialog appears.

4. In the Users field, enter the relevant user name to add to the group.
5. Click Save.

Procedure

1. From the OpenShift AI dashboard, click Settings → User management.
2. Select your OpenShift AI administrator groups: Under Data science administrator groups, click the text box and select an OpenShift group. Repeat this process to define multiple administrator groups.

3. Select your OpenShift AI user groups: Under Data science user groups, click the text box and select an OpenShift group. Repeat this process to define multiple user groups.

   Important

   The system:authenticated setting allows all users authenticated in OpenShift to access OpenShift AI.

4. Click Save changes.

Procedure

1. Obtain the location of the default image that you want to use as the base for your custom image.

   1. In the OpenShift console, select Builds → ImageStreams.
   2. Select the redhat-ods-applications project.
   3. From the list of installed imagestreams, click the name of the image that you want to use as the base for your custom image. For example, click pytorch.
   4. On the ImageStream details page, click YAML.

   5. In the spec:tags section, find the tag for the version of the image that you want to use.

      The location of the original image is shown in the tag’s from:name section, for example:

      name: 'quay.io/modh/odh-pytorch-notebook@sha256:b68e0192abf7d…'

   6. Copy this location for use in your custom image.
2. Create a standard Containerfile or Dockerfile.

3. For the FROM instruction, specify the base image location that you copied in Step 1, for example:

   FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0…

4. Optional: Install OS images:

   1. Switch to USER 0 (USER 0 is required to install OS packages).
   2. Install the packages.

   3. Switch back to USER 1001.

      The following example creates a custom workbench image that adds Java to the default PyTorch image:

       FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0…
      
       USER 0
      
       RUN INSTALL_PKGS="java-11-openjdk java-11-openjdk-devel" && \
          dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
          dnf -y clean all --enablerepo=*
      
       USER 1001

      Copy code to clipboard

5. Optional: Add Python packages:

   1. Specify USER 1001.
   2. Copy the requirements.txt file.

   3. Install the packages.

      The following example installs packages from the requirements.txt file in the default PyTorch image:

       FROM quay.io/modh/odh-pytorch-notebook@sha256:b68e0…
      
       USER 1001
      
       COPY requirements.txt ./requirements.txt
      
       RUN pip install -r requirements.txt

      Copy code to clipboard

6. Build the image file. For example, you can use podman build locally where the image file is located and then push the image to a registry that is accessible to OpenShift AI:

   $ podman build -t my-registry/my-custom-image:0.0.1 .
   $ podman push my-registry/my-custom-image:0.0.1

   Copy code to clipboard

   Alternatively, you can leverage OpenShift’s image build capabilities by using BuildConfig.

Procedure

1. From the OpenShift AI dashboard, click Settings → Notebook images.

   The Notebook images page appears. Previously imported images are displayed. To enable or disable a previously imported image, on the row containing the relevant image, click the toggle in the Enable column.

2. Optional: If you want to associate an accelerator and you have not already created an accelerator profile, click Create profile on the row containing the image and complete the relevant fields. If the image does not contain an accelerator identifier, you must manually configure one before creating an associated accelerator profile.

3. Click Import new image. Alternatively, if no previously imported images were found, click Import image.

   The Import Notebook images dialog appears.

4. In the Image location field, enter the URL of the repository containing the image. For example: quay.io/my-repo/my-image:tag, quay.io/my-repo/my-image@sha256:xxxxxxxxxxxxx, or docker.io/my-repo/my-image:tag.
5. In the Name field, enter an appropriate name for the image.
6. Optional: In the Description field, enter a description for the image.
7. Optional: From the Accelerator identifier list, select an identifier to set its accelerator as recommended with the image. If the image contains only one accelerator identifier, the identifier name displays by default.

8. Optional: Add software to the image. After the import has completed, the software is added to the image’s meta-data and displayed on the Jupyter server creation page.

   1. Click the Software tab.
   2. Click the Add software button.
   3. Click Edit ( ).
   4. Enter the Software name.
   5. Enter the software Version.
   6. Click Confirm ( ) to confirm your entry.
   7. To add additional software, click Add software, complete the relevant fields, and confirm your entry.

9. Optional: Add packages to the notebook images. After the import has completed, the packages are added to the image’s meta-data and displayed on the Jupyter server creation page.

   1. Click the Packages tab.
   2. Click the Add package button.
   3. Click Edit ( ).
   4. Enter the Package name. For example, if you want to include data science pipeline V2 automatically, as a runtime configuration, type odh-elyra.
   5. Enter the package Version. For example, type 3.16.7.
   6. Click Confirm ( ) to confirm your entry.
   7. To add an additional package, click Add package, complete the relevant fields, and confirm your entry.
10. Click Import.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Home → API Explorer.
3. In the search bar, enter OdhDashboardConfig to filter by kind.
4. Click the OdhDashboardConfig custom resource (CR) to open the resource details page.
5. Select the redhat-ods-applications project from the Project list.
6. Click the Instances tab.
7. Click the odh-dashboard-config instance to open the details page.
8. Click the YAML tab.
9. Edit the values of the options that you want to change.
10. Click Save to apply your changes and then click Reload to synchronize your changes to the cluster.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Home → API Explorer.
3. On the API Explorer page, search for the OdhApplication kind.
4. Click the OdhApplication kind to open the resource details page.
5. On the OdhApplication details page, select the redhat-ods-applications project from the Project list.
6. Click the Instances tab.
7. Click Create OdhApplication.

8. On the Create OdhApplication page, copy the following code and paste it into the YAML editor.

   apiVersion: dashboard.opendatahub.io/v1
   kind: OdhApplication
   metadata:
     name: examplename
     namespace: redhat-ods-applications
     labels:
       app: odh-dashboard
       app.kubernetes.io/part-of: odh-dashboard
   spec:
     enable:
       validationConfigMap: examplename-enable
     img: >-
       <svg width="24" height="25" viewBox="0 0 24 25" fill="none" xmlns="http://www.w3.org/2000/svg">
       <path d="path data" fill="#ee0000"/>
       </svg>
     getStartedLink: 'https://example.org/docs/quickstart.html'
     route: exampleroutename
     routeNamespace: examplenamespace
     displayName: Example Name
     kfdefApplications: []
     support: third party support
     csvName: ''
     provider: example
     docsLink: 'https://example.org/docs/index.html'
     quickStart: ''
     getStartedMarkDown: >-
       # Example
   
       Enter text for the information panel.
   
     description: >-
       Enter summary text for the tile.
     category: Self-managed | Partner managed | {org-name} managed

   Copy code to clipboard

9. Modify the parameters in the code for your application.

   Tip

   To see example YAML files, click Home → API Explorer, select OdhApplication, click the Instances tab, select an instance, and then click the YAML tab.

10. Click Create. The application details page appears.
11. Log in to OpenShift AI.
12. In the left menu, click Applications → Explore.
13. Locate the new tile for your application and click it.
14. In the information pane for the application, click Enable.

Procedure

1. Log in to the OpenShift console as a cluster administrator.

2. Open the dashboard configuration file:

   1. In the Administrator perspective, click Home → API Explorer.
   2. In the search bar, enter OdhDashboardConfig to filter by kind.
   3. Click the OdhDashboardConfig custom resource (CR) to open the resource details page.
   4. Select the redhat-ods-applications project from the Project list.
   5. Click the Instances tab.
   6. Click the odh-dashboard-config instance to open the details page.
   7. Click the YAML tab.
3. In the spec:dashboardConfig section, set the value of enablement to false to disable the ability for dashboard users to add applications to the dashboard.
4. Click Save to apply your changes and then click Reload to make sure that your changes are synced to the cluster.

Procedure

1. In the OpenShift web console, switch to the Administrator perspective.
2. Switch to the redhat-ods-applications project.
3. Click Operators → Installed Operators.
4. Click on the Operator that you want to uninstall. You can enter a keyword into the Filter by name field to help you find the Operator faster.

5. Delete any Operator resources or instances by using the tabs in the Operator interface.

   During installation, some Operators require the administrator to create resources or start process instances using tabs in the Operator interface. These must be deleted before the Operator can uninstall correctly.

6. On the Operator Details page, click the Actions drop-down menu and select Uninstall Operator.

   An Uninstall Operator? dialog box is displayed.

7. Select Uninstall to uninstall the Operator, Operator deployments, and pods. After this is complete, the Operator stops running and no longer receives updates.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Home → API Explorer.
3. On the API Explorer page, search for the OdhApplication kind.
4. Click the OdhApplication kind to open the resource details page.
5. On the OdhApplication details page, select the redhat-ods-applications project from the Project list.
6. Click the Instances tab.
7. Click Create OdhApplication.

8. On the Create OdhApplication page, copy the following code and paste it into the YAML editor.

   apiVersion: dashboard.opendatahub.io/v1
   kind: OdhApplication
   metadata:
     name: examplename
     namespace: redhat-ods-applications
     labels:
       app: odh-dashboard
       app.kubernetes.io/part-of: odh-dashboard
   spec:
     enable:
       validationConfigMap: examplename-enable
     img: >-
       <svg width="24" height="25" viewBox="0 0 24 25" fill="none" xmlns="http://www.w3.org/2000/svg">
       <path d="path data" fill="#ee0000"/>
       </svg>
     getStartedLink: 'https://example.org/docs/quickstart.html'
     route: exampleroutename
     routeNamespace: examplenamespace
     displayName: Example Name
     kfdefApplications: []
     support: third party support
     csvName: ''
     provider: example
     docsLink: 'https://example.org/docs/index.html'
     quickStart: ''
     getStartedMarkDown: >-
       # Example
   
       Enter text for the information panel.
   
     description: >-
       Enter summary text for the tile.
     category: Self-managed | Partner managed | Red Hat managed

   Copy code to clipboard

9. Modify the parameters in the code for your application.

   Tip

   To see example YAML files, click Home → API Explorer, select OdhApplication, click the Instances tab, select an instance, and then click the YAML tab.

10. Click Create. The application details page appears.
11. Log in to OpenShift AI.
12. In the left menu, click Applications → Explore.
13. Locate the new tile for your application and click it.
14. In the information pane for the application, click Enable.

Procedure

1. Log in to the OpenShift console as a cluster administrator.

2. Open the dashboard configuration file:

   1. In the Administrator perspective, click Home → API Explorer.
   2. In the search bar, enter OdhDashboardConfig to filter by kind.
   3. Click the OdhDashboardConfig custom resource (CR) to open the resource details page.
   4. Select the redhat-ods-applications project from the Project list.
   5. Click the Instances tab.
   6. Click the odh-dashboard-config instance to open the details page.
   7. Click the YAML tab.
3. In the spec:notebookController section, set the value of enabled to false to hide the Jupyter tile from the list of Enabled applications.
4. Click Save to apply your changes and then click Reload to make sure that your changes are synced to the cluster.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Workloads > Deployments.
3. From the Project drop-down list, select redhat-ods-applications.

4. In the Name column, click the name of the deployment for the component that you want to customize resources for.

   Note

   For more information about the deployment names for each component, see Overview of component resource customization.

5. On the Deployment details page that appears, click the YAML tab.
6. Find the .spec.template.spec.containers.resources section.

7. Update the value of the resource that you want to customize. For example, to update the memory limit to 500Mi, make the following change:

   containers:
           - resources:
               limits:
                   cpu: '2'
                   memory: 500Mi
               requests:
                   cpu: '1'
                   memory: 1Gi

   Copy code to clipboard
8. Click Save.
9. Click Reload.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Workloads > Deployments.
3. From the Project drop-down list, select redhat-ods-applications.

4. In the Name column, click the name of the deployment for the component to which you want to add the annotation.

   Note

   For more information about the deployment names for each component, see Overview of component resource customization.

5. On the Deployment details page that appears, click the YAML tab.
6. Find the metadata.annotations: section.

7. Add the opendatahub.io/managed: true annotation.

   metadata:
     annotations:
        opendatahub.io/managed: true

   Copy code to clipboard
8. Click Save.
9. Click Reload.

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. In the Administrator perspective, click Workloads > Deployments.
3. From the Project drop-down list, select redhat-ods-applications.
4. In the Name column, click the name of the deployment for the component for which you want to remove the annotation.
5. Click the Options menu .
6. Click Delete Deployment.

Procedure

1. To enable GPU support on an OpenShift cluster, follow the instructions here: Content from docs.nvidia.com is not included.NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

   Important

   After you install the Node Feature Discovery (NFD) Operator, you must create an instance of NodeFeatureDiscovery. In addition, after you install the NVIDIA GPU Operator, you must create a ClusterPolicy and populate it with default values.

2. Delete the migration-gpu-status ConfigMap.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate ConfigMap.
   3. Search for the migration-gpu-status ConfigMap.

   4. Click the action menu (⋮) and select Delete ConfigMap from the list.

      The Delete ConfigMap dialog appears.

   5. Inspect the dialog and confirm that you are deleting the correct ConfigMap.
   6. Click Delete.

3. Restart the dashboard replicaset.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Click Workloads → Deployments.
   3. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate deployment.
   4. Search for the rhods-dashboard deployment.
   5. Click the action menu (⋮) and select Restart Rollout from the list.
   6. Wait until the Status column indicates that all pods in the rollout have fully restarted.

1. Install the latest version of the Intel Gaudi AI Accelerator Operator from OperatorHub.
2. Create and configure a custom workbench image for Intel Gaudi AI accelerators. A prebuilt workbench image for Gaudi accelerators is not included in OpenShift AI.
3. Manually define and configure an accelerator profile for each Intel Gaudi AI device in your environment.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy code to clipboard

2. Create an empty Kueue resource flavor, as follows:

   1. Create a file called default_flavor.yaml and populate it with the following content:

      Empty Kueue resource flavor

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ResourceFlavor
      metadata:
        name: default-flavor

      Copy code to clipboard

   2. Apply the configuration to create the default-flavor object:

      $ oc apply -f default_flavor.yaml

      Copy code to clipboard

3. Create a cluster queue to manage the empty Kueue resource flavor, as follows:

   1. Create a file called cluster_queue.yaml and populate it with the following content:

      Example cluster queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ClusterQueue
      metadata:
        name: "cluster-queue"
      spec:
        namespaceSelector: {}  # match all.
        resourceGroups:
        - coveredResources: ["cpu", "memory", "nvidia.com/gpu"] # If you use AMD GPUs, substitute "nvidia.com/gpu" with "amd.com/gpu"
          flavors:
          - name: "default-flavor"
            resources:
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
            - name: "nvidia.com/gpu" # If you use AMD GPUs, substitute "nvidia.com/gpu" with "amd.com/gpu"
              nominalQuota: 5

      Copy code to clipboard

   2. Replace the example quota values (9 CPUs, 36 GiB memory, and 5 NVIDIA GPUs) with the appropriate values for your cluster queue. The cluster queue will start a distributed workload only if the total required resources are within these quota limits.

      You must specify a quota for each resource that the user can request, even if the requested value is 0, by updating the spec.resourceGroups section as follows:

      • Include the resource name in the coveredResources list.
      • Specify the resource name and nominalQuota in the flavors.resources section, even if the nominalQuota value is 0.

   3. Apply the configuration to create the cluster-queue object:

      $ oc apply -f cluster_queue.yaml

      Copy code to clipboard

4. Create a local queue that points to your cluster queue, as follows:

   1. Create a file called local_queue.yaml and populate it with the following content:

      Example local queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: LocalQueue
      metadata:
        namespace: test
        name: local-queue-test
        annotations:
          kueue.x-k8s.io/default-queue: 'true'
      spec:
        clusterQueue: cluster-queue

      Copy code to clipboard

      The kueue.x-k8s.io/default-queue: 'true' annotation defines this queue as the default queue. Distributed workloads are submitted to this queue if no local_queue value is specified in the ClusterConfiguration section of the data science pipeline or Jupyter notebook or Microsoft Visual Studio Code file.

   2. Update the namespace value to specify the same namespace as in the ClusterConfiguration section that creates the Ray cluster.
   3. Optional: Update the name value accordingly.

   4. Apply the configuration to create the local-queue object:

      $ oc apply -f local_queue.yaml

      Copy code to clipboard

      The cluster queue allocates the resources to run distributed workloads in the local queue.

Procedure

1. In the OpenShift console, click Workloads → ConfigMaps.
2. From the Project list, select redhat-ods-applications.
3. Search for the codeflare-operator-config config map, and click the config map name to open the ConfigMap details page.
4. Click the YAML tab to show the config map specifications.

5. In the data:config.yaml:kuberay section, you can edit the following entries:

   ingressDomain

   This configuration option is null (ingressDomain: "") by default. Do not change this option unless the Ingress Controller is not running on OpenShift. OpenShift AI uses this value to generate the dashboard and client routes for every Ray Cluster, as shown in the following examples:

   Example dashboard and client routes

   ray-dashboard-<clustername>-<namespace>.<your.ingress.domain>
   ray-client-<clustername>-<namespace>.<your.ingress.domain>

   Copy code to clipboard

   mTLSEnabled

   This configuration option is enabled (mTLSEnabled: true) by default. When this option is enabled, the Ray Cluster pods create certificates that are used for mutual Transport Layer Security (mTLS), a form of mutual authentication, between Ray Cluster nodes. When this option is enabled, Ray clients cannot connect to the Ray head node unless they download the generated certificates from the ca-secret-_<cluster_name>_ secret, generate the necessary certificates for mTLS communication, and then set the required Ray environment variables. Users must then re-initialize the Ray clients to apply the changes. The CodeFlare SDK provides the following functions to simplify the authentication process for Ray clients:

   Example Ray client authentication code

   from codeflare_sdk import generate_cert
   
   generate_cert.generate_tls_cert(cluster.config.name, cluster.config.namespace)
   generate_cert.export_env(cluster.config.name, cluster.config.namespace)
   
   ray.init(cluster.cluster_uri())

   Copy code to clipboard

   rayDashboardOauthEnabled

   This configuration option is enabled (rayDashboardOAuthEnabled: true) by default. When this option is enabled, OpenShift AI places an OpenShift OAuth proxy in front of the Ray Cluster head node. Users must then authenticate by using their OpenShift cluster login credentials when accessing the Ray Dashboard through the browser. If users want to access the Ray Dashboard in another way (for example, by using the Ray JobSubmissionClient class), they must set an authorization header as part of their request, as shown in the following example:

   Example authorization header

   {Authorization: "Bearer <your-openshift-token>"}

   Copy code to clipboard

6. To save your changes, click Save.

7. To apply your changes, delete the pod:

   1. Click Workloads → Pods.
   2. Find the codeflare-operator-manager-<pod-id> pod.
   3. Click the options menu (⋮) for that pod, and then click Delete Pod. The pod restarts with your changes applied.

1. In the OpenShift console, click Workloads → Deployments.
2. Search for the codeflare-operator-manager deployment, and then click the deployment name to open the deployment details page.
3. Click the Pods tab. When the status of the codeflare-operator-manager-<pod-id> pod is Running, the pod is ready to use. To see more information about the pod, click the pod name to open the pod details page, and then click the Logs tab.

Procedure

1. Log in to the OpenShift as a cluster administrator.
2. Click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
3. Click the DSC Initialization tab.
4. Click the default-dsci object.
5. Click the YAML tab.

6. In the spec section, update the .spec.devFlags.logmode flag with the log level that you want to set.

   apiVersion: dscinitialization.opendatahub.io/v1
   kind: DSCInitialization
   metadata:
     name: default-dsci
   spec:
     devFlags:
       logmode: development

   Copy code to clipboard
7. Click Save.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy code to clipboard
2. To access the full content of the changed custom resources, set the OpenShift audit log policy to WriteRequestBodies or a more comprehensive profile. For more information, see About audit log policy profiles.

3. Fetch the audit log files that are available for the relevant control plane nodes. For example:

   oc adm node-logs --role=master --path=kube-apiserver/ \
     | awk '{ print $1 }' | sort -u \
     | while read node ; do
         oc adm node-logs $node --path=kube-apiserver/audit.log < /dev/null
       done \
     | grep opendatahub > /tmp/kube-apiserver-audit-opendatahub.log

   Copy code to clipboard

4. Search the files for the DSC and DSCI custom resources. For example:

   jq 'select((.objectRef.apiGroup == "dscinitialization.opendatahub.io"
                   or .objectRef.apiGroup == "datasciencecluster.opendatahub.io")
                 and .user.username != "system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager"
                 and .verb != "get" and .verb != "watch" and .verb != "list")' < /tmp/kube-apiserver-audit-opendatahub.log

   Copy code to clipboard