Changes to Target Ratio Settings in OpenShift Data Foundation (ODF) v4.21 Through ODF v5.x

Solution Verified - Updated

Environment

Red Hat OpenShift Data Foundation (RHODF) v4.21
Red Hat OpenShift Data Foundation (RHODF) v4.22
Red Hat OpenShift Data Foundation (RHODF) v4.23
Red Hat OpenShift Data Foundation (RHODF) v5+

Issue

The solution will address how to remove the manual targetRatio setting for optimal performance in ODF v4.21+.

If the environment implemented the Storage optimization using Ceph Autoscaler and a Target Size Ratio for PG Increase/Placement on Large Pools - OpenShift Data Foundation (ODF) v4.19+ solution, it's important to understand the upcoming changes when upgrading to ODF v4.21. In summary, the changes are as follows:

  • In ODF v5.0/v4.23, the targetRatio will be disabled for all existing pools.

  • For all new installs deployed on ODF v4.21 or higher, targetRatio is 0; no manual adjustments are needed.

  • For already configured ODF clusters that upgraded to v4.21+, Ceph pools with a manually set targetRatio. These configurations may remain in place, but it is not optimal with respect to the default autoscaling already provided. To revert back to default, follow the resolution to patch the storagecluster for optimal autoscaling.

Resolution

NOTE: This resolution will cause PG movement (Recovery/Rebalance). If wanting to minimize the impact on client IO, and restrict recovery I/O, please review the Ceph/ODF: Slow backfill and slow scrub/deep-scrub under mClock I/O scheduler solution and configure Ceph for high_client_ops.

  1. If the implementation was accomplished via the now-archived v4.18 method of setting the pool to reconcileStrategy: ignore in the storagecluster.yaml, and NOT the method covered in the official Product Documentation, this configuration will need to be reverted.

a. To validate, execute the following command:

$ oc get storagecluster -n openshift-storage ocs-storagecluster -o yaml | grep -B5 reconcileStrategy

b. If the above output DOES NOT return any output, proceed to step 2. If the above output DOES show Ceph pool(s) is/are configured in the storagecluster.yaml with a reconcileStrategy: ignore, and your environment DOES NOT fall in the category of actively disabling the pool/storageclass as shown in the Flexibility in OpenShift Container Storage component deployment solution, this should be reverted.

c. To revert this configuration, execute the following on the respective pool:

oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{"spec": {"managedResources": {"cephBlockPools": {"reconcileStrategy": null}}}}'

oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{"spec": {"managedResources": {"cephFilesystems": {"reconcileStrategy": null}}}}'

oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{"spec": {"managedResources": {"cephObjectStores": {"reconcileStrategy": null}}}}'

After the reconcileStrategy: ignore has been removed, the operators should reconcile the Ceph pool CRs back to their default settings and PG movement may occur.

  1. For ODF v4.21 and above clusters. To remove the targetRatio on default Ceph pools, execute the following:
  • For default CephBlockPool:
oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{
  "spec": {
    "managedResources": {
      "cephBlockPools": {
        "poolSpec": {
          "parameters": {
            "target_size_ratio": "0"
          }
        }
      }
    }
  }
}'
  • For default CephFileSystems:
oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{
  "spec": {
    "managedResources": {
      "cephFilesystems": {
        "dataPoolSpec": {
          "parameters": {
            "target_size_ratio": "0"
          }
        }
      }
    }
  }
}'
  • For default Rados Gateway (RGW) Pool:
oc patch storagecluster ocs-storagecluster -n openshift-storage --type=merge -p '{
  "spec": {
    "managedResources": {
      "cephObjectStores": {
        "dataPoolSpec": {
          "parameters": {
            "target_size_ratio": "0"
          }
        }
      }
    }
  }
}'
  1. For ODF v4.21 and above, to remove the targetRatio on any custom/non-default CephBlockPool, change the pool name in the command below to match the pool name and execute the following:
oc patch cephblockpool <pool-name> -n openshift-storage --type=json -p='[
  {"op": "replace", "path": "/spec/parameters/target_size_ratio", "value": "0"},
  {"op": "remove", "path": "/spec/replicated/targetSizeRatio"}
]'
  1. For ODF v4.21 and above, to remove the targetRatio on any custom/non-default CephFileSystems pool, execute the following:
oc patch storagecluster ocs-storagecluster -n openshift-storage --type=json -p='[
  {
    "op": "add",
    "path": "/spec/managedResources/cephFilesystems/additionalDataPools/0/parameters",
    "value": { "target_size_ratio": "0" }
  }
]'

Root Cause

This issue is being tracked in the following bug cases:

Content from redhat.atlassian.net is not included.DFBUGS-2665

Content from redhat.atlassian.net is not included.DFBUGS-5871

Clusters upgraded to 4.21 no longer have a requested target size ratio. The target ratio is, in fact, removed from the CephBlockPool, CephFilesystem, and CephObjectStore CRs.

In ODF v4.21, Rook does not remove the target size ratio. If the setting is not reverted in the Custom Resource (CR) in ODF v4.20 or below, whatever value the target size ratio had before the upgrade will be retained.

The Ceph PG autoscaler has been found to work more effectively when the target size ratio in it's default state.

Removing the target size ratio causes immediate data movement across the cluster, so we want to be very careful and intentional about when we cause data movement for PG autoscaler. This should be planned during off-peak time to allow the Ceph balancer time to remap Placement Groups (PGs).

Diagnostic Steps

ODF v4.18 and below: Method that should be reverted:

$ oc get storagecluster -n openshift-storage -o yaml

apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1
  kind: StorageCluster
<omitted-for-space>
    managedResources:
      cephBlockPools:
        reconcileStrategy: ignore <----
<omitted-for-space>
      cephFilesystems:
        activeMetadataServers: 1
        reconcileStrategy: ignore <----
<omitted-for-space>
      cephObjectStores:
        disableRoute: true
        reconcileStrategy: ignore <----

ODF v4.19+:

$ oc get storagecluster -n openshift-storage -o yaml

apiVersion: v1
items:
- apiVersion: ocs.openshift.io/v1
  kind: StorageCluster
<omitted-for-space>
    managedResources:
      cephBlockPools:
        poolSpec:
          replicated:
            size: 3
            targetSizeRatio: 0.02 <-----
<omitted-for-space>
      cephFilesystems:
        activeMetadataServers: 1
        dataPoolSpec:
          replicated:
            size: 3
            targetSizeRatio: 0.98 <-----
<omitted-for-space>
      cephObjectStores:
        dataPoolSpec:
          replicated:
            size: 3
            targetSizeRatio: 0.8 <-----
SBR
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.