How do I install a three-node OpenShift Data Foundation compact cluster using ACM policy on RHACM connected clusters on IBM Power?

Solution Unverified - Updated

Resolution

Prerequisites

  • A Red Hat Advanced Cluster Management (RHACM) Hub cluster into which the managed clusters have been imported.
  • Managed clusters on which you will set up OpenShift Data Foundation.
  • Label the managed clusters on which you want to set up OpenShift Data Foundation. This label is used to determine the clusters on which the policies will be applied.
  • A minimum of 3 disks on each managed cluster that will be used for OpenShift Data Foundation.
  • Set the cluster.ocs.openshift.io/openshift-storage='' label on each node of the managed clusters.

Procedure

This is a two-step process. First, you need to install and configure the Local Storage Operator on the managed clusters. The Local Storage Operator will provision Persistent Volumes (PVs) using the local devices on the managed cluster. The OpenShift Data Foundation Operator uses these PVs to set up the storage cluster.

  1. Log into the RHACM Hub cluster using the command-line interface.

  2. Create a namespace in which to create the policies:

# oc create ns odf-policy-ns
  1. Install and configure the Local Storage Operator.

Install and configure the Local Storage Operator using a policy similar to the following. You need to change the storageClassDevices and node selector fields to select the disks you want to use for OpenShift Data Foundation.

i) Save the following policy to a file, for example, lso-policy.yaml:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  annotations:
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
    policy.open-cluster-management.io/standards: NIST SP 800-53
  name: policy-local-storage-operator
spec:
  disabled: false
  remediationAction: enforce
  policy-templates:
  - objectDefinition:
      apiVersion: policy.open-cluster-management.io/v1
      kind: ConfigurationPolicy
      metadata:
        name: policy-local-storage-operator
      spec:
        remediationAction: enforce
        severity: low
        object-templates:
        - complianceType: musthave
          objectDefinition:
            apiVersion: v1
            kind: Namespace
            metadata:
              labels:
                openshift.io/cluster-monitoring: "true"
              name: openshift-local-storage
        - complianceType: musthave
          objectDefinition:
            apiVersion: operators.coreos.com/v1
            kind: OperatorGroup
            metadata:
              name: openshift-storage-operatorgroup
              namespace: openshift-local-storage
              annotations:
                olm.providedAPIs: LocalVolume.v1.local.storage.openshift.io
            spec:
              targetNamespaces:
              - openshift-local-storage
        - complianceType: musthave
          objectDefinition:
            apiVersion: operators.coreos.com/v1alpha1
            kind: Subscription
            metadata:
              name: local-storage-operator
              namespace: openshift-local-storage
            spec:
              channel: stable
              installPlanApproval: Automatic
              name: local-storage-operator
              source: optional-operators
              sourceNamespace: openshift-marketplace
        - complianceType: musthave
          objectDefinition:
            apiVersion: local.storage.openshift.io/v1
            kind: LocalVolume
            metadata:
              name: localblock
              namespace: openshift-local-storage
            spec:
              logLevel: Normal
              managementState: Managed
              nodeSelector:
                nodeSelectorTerms:
                  - matchExpressions:  
                     - key: kubernetes.io/hostname
                       operator: In
                       values:
                          - hostname1
                          - hostname2
                          - hostname3
              storageClassDevices:      # <-- Modify this field to match the devices that the Local Storage Operator should use to provision the PVs.
                - devicePaths:
                   - /dev/sdc
                  storageClassName: localblock
                  volumeMode: Block
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: placement-policy-local-storage-operator
spec:
  clusterConditions:
  - status: "True"
    type: ManagedClusterConditionAvailable
  clusterSelector:
    matchExpressions:       # <-- Modify this field to match the labels set on the managed clusters.
    - key: environment
      operator: In
      values:
      - dev
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-policy-local-storage-operator
placementRef:
  apiGroup: apps.open-cluster-management.io
  kind: PlacementRule
  name: placement-policy-local-storage-operator
subjects:
- apiGroup: policy.open-cluster-management.io
  kind: Policy
  name: policy-local-storage-operator

ii) Create the policy on the ACM hub cluster:

# oc create -f lso-policy.yaml -n odf-policy-ns

iii) Wait until the policy is compliant. Check the status of the policy:

# oc get policy -n odf-policy-ns
  1. Install and configure the OpenShift Data Foundation Operator.

i) Save the following policy to a file, for example, odf-policy.yaml:

# This policy governs the installation of the official version of
# the OpenShift Data Foundation on the managed clusters.
#
# If set to "enforce" it'll install the operator.
# Used APIs: OLM, OCS, ODF
# https://github.com/operator-framework/operator-lifecycle-manager
# https://github.com/openshift/ocs-operator
# https://github.com/red-hat-storage/odf-operator
 
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
 annotations:
   policy.open-cluster-management.io/categories: CM Configuration Management
   policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
   policy.open-cluster-management.io/standards: NIST SP 800-53
 name: policy-odf-operator
spec:
 disabled: false
 policy-templates:
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: policy-odf-namespace
       spec:
         object-templates:
           - complianceType: musthave
             objectDefinition:
               apiVersion: v1
               kind: Namespace
               metadata:
                 name: openshift-storage
         remediationAction: inform
         severity: high
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: policy-odf-operator-operatorgroup
       spec:
         object-templates:
           - complianceType: mustonlyhave
             objectDefinition:
               apiVersion: operators.coreos.com/v1alpha2
               kind: OperatorGroup
               metadata:
                 name: openshift-storage-operatorgroup
                 namespace: openshift-storage
               spec:
                 targetNamespaces:
                   - openshift-storage
         remediationAction: inform
         severity: high
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: policy-odf-operator-subscription
       spec:
         object-templates:
           - complianceType: mustonlyhave
             objectDefinition:
               apiVersion: operators.coreos.com/v1alpha1
               kind: Subscription
               metadata:
                 name: odf-operator
                 namespace: openshift-storage
               spec:
                 installPlanApproval: Automatic
                 name: odf-operator
                 source: redhat-operators
                 sourceNamespace: openshift-marketplace             
         remediationAction: inform
         severity: high
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: policy-storagesystem
       spec:
         object-templates:
           - complianceType: mustonlyhave
             objectDefinition:
               apiVersion: odf.openshift.io/v1alpha1
               kind: StorageSystem
               metadata:
                 name: odf-storagecluster-storagesystem
                 namespace: openshift-storage
               spec:
                 kind: storagecluster.ocs.openshift.io/v1
                 name: ocs-storagecluster
                 namespace: openshift-storage
         remediationAction: inform
         severity: low
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: policy-storagecluster
       spec:
         object-templates:
           - complianceType: mustonlyhave
             objectDefinition:
               apiVersion: ocs.openshift.io/v1
               kind: StorageCluster
               metadata:
                 annotations:
                   uninstall.ocs.openshift.io/cleanup-policy: delete
                   uninstall.ocs.openshift.io/mode: graceful
                   cluster.ocs.openshift.io/local-devices: 'true'
                 name: ocs-storagecluster
                 namespace: openshift-storage
               spec: 
                 flexibleScaling: true
                 manageNodes: false
                 monDataDirHostPath: /var/lib/rook
                 storageDeviceSets:
                   - count: 3       # <-- Modify the count to match the number of devices that are set up by the Local Storage Operator on the managed cluster. The minimum value is 3.
                     dataPVCTemplate:
                       spec:
                         accessModes:
                           - ReadWriteOnce
                         resources:
                           requests:
                             storage: 100Gi
                         storageClassName: localblock #  <-- Modify the storageClassName to match the one created by the Local Storage Operator.
                         volumeMode: Block
                     name: ocs-deviceset-localblock
                     portable: false
                     replica: 1
                     placement: {}
                     resources: {}
         remediationAction: inform
         severity: low
 remediationAction: enforce

 
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
 name: binding-policy-odf-operator
placementRef:
 apiGroup: apps.open-cluster-management.io
 kind: PlacementRule
 name: placement-policy-odf-operator
subjects:
 - apiGroup: policy.open-cluster-management.io
   kind: Policy
   name: policy-odf-operator
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
 name: placement-policy-odf-operator
spec:
 clusterConditions:
   - status: 'True'
     type: ManagedClusterConditionAvailable
 clusterSelector:
   matchExpressions:    # <-- Modify this field to match the labels on the managed clusters.
     - key: environment
       operator: In
       values:
         - dev

ii) Create the policy in the namespace on the ACM Hub cluster:

# oc create -f odf-policy.yaml -n odf-policy-ns

iii) Wait until the policy is compliant. Check the status of the policy:

# oc get policy -n odf-policy-ns

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.