How to grant permission to create Ray clusters and distributed workloads in RHOAI

Solution Verified - Updated

Environment

  • Red Hat OpenShift Data Science
    • Version: 2.4
  • Red Hat OpenShift AI
    • Version: 2.5

Starting from version 2.5, Red Hat OpenShift Data Science is renamed to Red Hat OpenShift AI. For more details, see the This content is not included.2.5 Release Notes.

Note: This solution provides a workaround for an issue that is now fixed. If you are using Red Hat OpenShift AI 2.6 or later, this solution is not necessary.

Issue

Users cannot create Ray clusters or distributed workloads in namespaces where they have admin or edit permissions.

Resolution

To grant the appropriate permissions, create a ClusterRole for the resources created by the KubeRay Operator and CodeFlare Operator, and specify the admin and edit aggregation labels.

Note that the KubeRay Operator and CodeFlare Operator are included in the Red Hat OpenShift AI product, and should not be installed separately.

Complete the following steps as a user with cluster-admin permissions:

  1. Create a file named dw-roles.yaml with the following content:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kuberay-edit-role
  labels:
	rbac.authorization.k8s.io/aggregate-to-edit: "true"
	rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups:
  - ray.io
  resources:
  - rayjobs
  - rayclusters
  - rayservices
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - workload.codeflare.dev
  resources:
  - appwrappers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  1. Run the YAML file:
oc apply -f dw-roles.yaml

Users with admin or edit privileges in the namespace associated with their data science project can now create Ray clusters and distributed workloads.

This fix changes the permissions for all namespaces in the OpenShift cluster; therefore, you apply this fix only once.

Root Cause

Red Hat OpenShift AI does not provide an access role for the resources created by the KubeRay Operator and CodeFlare Operator.

Diagnostic Steps

To verify that your changes were successfully applied, confirm that the edit and admin cluster roles include the ray and workload entries, as follows:

$ oc describe clusterrole edit | grep ray.io
> rayclusters.ray.io                []             []    [create delete get list patch update watch]
> rayjobs.ray.io                    []             []    [create delete get list patch update watch]
> rayservices.ray.io                []             []    [create delete get list patch update watch]

$ oc describe clusterrole admin | grep ray.io
> rayclusters.ray.io                []             []    [create delete get list patch update watch]
> rayjobs.ray.io                    []             []    [create delete get list patch update watch]
> rayservices.ray.io                []             []    [create delete get list patch update watch]

$ oc describe clusterrole edit | grep workload.codeflare.dev
> appwrappers.workload.codeflare.dev                []             []    [create delete get list patch update watch]

$ oc describe clusterrole admin | grep workload.codeflare.dev
> appwrappers.workload.codeflare.dev                []             []    [create delete get list patch update watch]
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.