Deploying two MGR pods in active/standby - Dev Preview ODF 4.13

Updated

Background

Currently, ocs-operator deploys a single Ceph mgr daemon without any standby. There is a dependency on the Kubernetes health checks to keep the pod running and when the liveness probe fails, the pod is restarted. Deploying two mgrs one as active and the other as standby helps in better reliability and availability.

Overview

With the support for two mgr pods, when an active mgr goes down, Ceph updates the passive mgr to be active. However, there is a delay of 15 to 30 seconds for Ceph to promote the passive mgr to active. Rook queries Ceph at an interval of 20 seconds to find out if there is a new active mgr. When Rook finds a new active mgr, it updates the required services to direct the traffic to the new active mgr.

CAUTION: Before you enable the deployment of two mgrs, be aware of the required resources as the second mgr consumes 3Gi memory and 1 CPU.

Enabling active-passive mgr

Procedure

  1. Use one of the following methods to enable:

    • When you create a new storagecluster, add this in the storagecluster spec:
spec:
    mgr:
        enableActivePassive: true
  • When you use an existing storagecluster, patch the storagecluster to enable the feature:
$ oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/mgr", "value": {"enableActivePassive" : true} }]'
  1. Wait for the storagecluster to change to Ready state.
$ oc get storagecluster
NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   14m   Ready              2023-05-30T15:31:14Z   4.13.0

Verifying active-passive mgr pods

Checking the two mgr pods

  • Run the following command to display the mgr pods.
$ oc get pods | grep mgr
rook-ceph-mgr-a-5d4b6ffb76-dpjtp                                  3/3     Running     0          37m
rook-ceph-mgr-b-56d6cffc8f-zvgtt                                  3/3     Running     0          36m

Checking the two mgr daemons using Ceph toolbox

  1. Enable Ceph toolbox and rsh into it.

  2. Check if the two mgr daemons are present and their metadata.

sh-5.1$ ceph mgr metadata
[
    {
        "name": "a",
        "addr": "10.129.2.112",
        "addrs": "10.129.2.112:0/2658460754",
        "arch": "x86_64",
        "ceph_release": "quincy",
        "ceph_version": "ceph version 17.2.6-50.el9cp (c202ddb5589554af0ce43432ff07cd7ce8f35243) quincy (stable)",
        "ceph_version_short": "17.2.6-50.el9cp",
        "container_hostname": "rook-ceph-mgr-a-5f8f4c55dc-mkjrr",
        "container_image": "quay.io/rhceph-dev/rhceph@sha256:fa6d01cdef17bc32d2b95b8121b02f4d41adccc5ba8a9b95f38c97797ff6621f",
        "cpu": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
        "distro": "rhel",
        "distro_description": "Red Hat Enterprise Linux 9.2 (Plow)",
        "distro_version": "9.2",
        "hostname": "ip-10-0-207-60.ec2.internal",
        "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Apr 12 10:45:03 EDT 2023",
        "kernel_version": "5.14.0-284.11.1.el9_2.x86_64",
        "mem_cgroup_limit": "3221225472",
        "mem_swap_kb": "0",
        "mem_total_kb": "65029304",
        "os": "Linux",
        "pod_name": "rook-ceph-mgr-a-5f8f4c55dc-mkjrr",
        "pod_namespace": "openshift-storage"
    },
    {
        "name": "b",
        "addr": "10.131.0.145",
        "addrs": "10.131.0.145:0/3284700333",
        "arch": "x86_64",
        "ceph_release": "quincy",
        "ceph_version": "ceph version 17.2.6-50.el9cp (c202ddb5589554af0ce43432ff07cd7ce8f35243) quincy (stable)",
        "ceph_version_short": "17.2.6-50.el9cp",
        "container_hostname": "rook-ceph-mgr-b-954655dc4-44l58",
        "container_image": "quay.io/rhceph-dev/rhceph@sha256:fa6d01cdef17bc32d2b95b8121b02f4d41adccc5ba8a9b95f38c97797ff6621f",
        "cpu": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
        "distro": "rhel",
        "distro_description": "Red Hat Enterprise Linux 9.2 (Plow)",
        "distro_version": "9.2",
        "hostname": "ip-10-0-143-54.ec2.internal",
        "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Apr 12 10:45:03 EDT 2023",
        "kernel_version": "5.14.0-284.11.1.el9_2.x86_64",
        "mem_cgroup_limit": "3221225472",
        "mem_swap_kb": "0",
        "mem_total_kb": "65029304",
        "os": "Linux",
        "pod_name": "rook-ceph-mgr-b-954655dc4-44l58",
        "pod_namespace": "openshift-storage"
    }
]
  1. Check the active and standby mgrs.
sh-5.1$ ceph mgr stat
{
    "epoch": 15,
    "available": true,
    "active_name": "a",
    "num_standby": 1
}

Demonstrating the switch over from passive to active when mgr fails

Fail the active mgr and check if the standby mgr takes over.

For example, when mgr a is failed:

sh-5.1$ ceph mgr fail a

mgr b becomes active. Another mgr daemon becomes the standby:

sh-5.1$ ceph mgr stat
{
    "epoch": 17,
    "available": true,
    "active_name": "b",
    "num_standby": 1
}

when mgr b is failed:

sh-5.1$ ceph mgr fail b

mgr a becomes active. Another mgr daemon becomes standby

sh-5.1$ ceph mgr stat
{
    "epoch": 26,
    "available": true,
    "active_name": "a",
    "num_standby": 1
}

Disabling active-passive mgr

In case you do not want to use the feature and revert to single mgr configuration, perform the following steps.

Procedure

  1. Patch the storagecluster to change the value of the variable to false.
$ oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/mgr", "value": {"enableActivePassive" : false} }]'
  1. Wait for the storagecluster to change to Ready state.
$ oc get storagecluster
NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   68m   Ready              2023-05-30T15:31:14Z   4.13.0

  1. Verify if active-passive mgr is disabled.

    • Check if only one mgr pod is present.
$ oc get pods | grep mgr
rook-ceph-mgr-a-786b4d75d5-l7mtn                                  2/2     Running     0          10m
  • Check if only one mgr daemon is present.
sh-5.1$ ceph mgr metadata
[
    {
        "name": "a",
        "addr": "10.129.2.121",
        "addrs": "10.129.2.121:0/1098299305",
        "arch": "x86_64",
        "ceph_release": "quincy",
        "ceph_version": "ceph version 17.2.6-50.el9cp (c202ddb5589554af0ce43432ff07cd7ce8f35243) quincy (stable)",
        "ceph_version_short": "17.2.6-50.el9cp",
        "container_hostname": "rook-ceph-mgr-a-5dfdd8c94d-4x5nc",
        "container_image": "quay.io/rhceph-dev/rhceph@sha256:fa6d01cdef17bc32d2b95b8121b02f4d41adccc5ba8a9b95f38c97797ff6621f",
        "cpu": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
        "distro": "rhel",
        "distro_description": "Red Hat Enterprise Linux 9.2 (Plow)",
        "distro_version": "9.2",
        "hostname": "ip-10-0-207-60.ec2.internal",
        "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Apr 12 10:45:03 EDT 2023",
        "kernel_version": "5.14.0-284.11.1.el9_2.x86_64",
        "mem_cgroup_limit": "3221225472",
        "mem_swap_kb": "0",
        "mem_total_kb": "65029304",
        "os": "Linux",
        "pod_name": "rook-ceph-mgr-a-5dfdd8c94d-4x5nc",
        "pod_namespace": "openshift-storage"
    }
]
  1. Check active and standby mgr.
sh-5.1$ ceph mgr stat
{
    "epoch": 35,
    "available": true,
    "active_name": "a",
    "num_standby": 0
}
Product(s)
Category
Article Type