How to create a ValidatingAdmissionPolicy to ensure pods have a valid scheduler set in OpenShift

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• Pods created with an invalid schedulerName are stuck in Pending status.
• Is it possible to restrict the usage of the scheduler to an allowed list?
• How can a ValidatingAdmissionPolicy be created to control pod creation?

Resolution

Note: The following procedure applies to all pods, independently of their origin (manual pod creation, deployments, replicasets, daemonsets...).

The following steps are meant to create a ValidatingAdmissionPolicy that restricts pod creation ensuring the schedulerName is one of the ones in a parameters list.

It also creates a cluster CustomResourceDefinition that could be used by other ValidatingAdmissionPolicy resources.

1. Create a CustomResourceDefinition cluster-scoped that will be used to be able to create the resource ClusterParameter. This parameter will contain data accessible by other resources such as the ValidatingAdmissionPolicy.

   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     name: clusterparameter.parameters.com
   spec:
     group: parameters.com
     versions:
       - name: v1
         served: true
         storage: true
         schema:
           openAPIV3Schema:
             type: object
             properties:
               allowed:
                 type: array
                 items:
                   type: string
     scope: Cluster
     names:
       plural: clusterparameter
       singular: clusterparameter
       kind: ClusterParameter
       shortNames:
         - clusterparam
   

2. Create a ClusterParameter resource that contains a list with the schedulerName values that are going to be accepted. This resource is cluster-scoped, so there's no need to define a namespace for it.

   apiVersion: parameters.com/v1
   kind: ClusterParameter
   metadata:
     name: allowed-schedulers
   allowed:
     - "default-scheduler"
     - "alternative-scheduler"
   

3. Create a ValidatingAdmissionPolicy that monitors creation and updates of pods. In the example below, the ValidatingAdmissionPolicy checks if the pod schedulerName value is one of the values in the ClusterParameter list. It also skips pods belonging to namespaces starting with "openshift-".

   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingAdmissionPolicy
   metadata:
     name: "forbid-invalid-schedulers"
   spec:
     failurePolicy: Fail
     paramKind:
       apiVersion: parameters.com/v1
       kind: ClusterParameter
     matchConstraints:
       resourceRules:
       - apiGroups:   [""]
         apiVersions: ["v1"]
         operations:  ["CREATE", "UPDATE"]
         resources:   ["pods"]
     validations:
       - expression: object.metadata.namespace.startsWith("openshift-") || params.allowed.exists(a, a == object.spec.schedulerName)
         messageExpression: "'Invalid scheduler set in pod.spec.schedulerName ('+ string(object.spec.schedulerName) + '), schedulerName must be one of the following: ' + params.allowed.join(', ')"
         reason: Invalid"
   

4. Create a ValidatingAdmissionPolicyBinding that binds the ValidatingAdmissionPolicy to the ClusterParameter and denies the creation of those pods if the expression in the ValidatingAdmissionPolicy returns a false value.

   apiVersion: admissionregistration.k8s.io/v1
   kind: ValidatingAdmissionPolicyBinding
   metadata:
     name: restrict-schedulername-binding
   spec:
     validationActions: [Deny, Audit]
     policyName: forbid-invalid-schedulers
     paramRef:
       name: allowed-schedulers
       parameterNotFoundAction: Deny
   

Diagnostic Steps

To check if the Resolution Steps worked, follow these steps:

1. Create a namespace.

   $ oc new-project test-fakescheduler
   

2. Create a pod with the following YAML definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: example-mustfail
     namespace: test-fakescheduler
   spec:
     schedulerName: test
     containers:
       - name: httpd
         image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest'
   

If the pod creation returns the following error, the procedure was followed successfully:

An error occurred
Error "ValidatingAdmissionPolicy 'forbid-invalid-schedulers' with binding 'restrict-schedulername-binding' denied request: Invalid scheduler set in pod.spec.schedulerName (test), schedulerName must be one of the following: default-scheduler, alternative-scheduler" for field "undefined".

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.