How do I start and manage clvmd and cmirrord in a pacemaker cluster?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 7 with the Resilient Storage Add On
  • Red Hat Enterprise Linux (RHEL) 6 with the Resilient Storage Add On
  • pacemaker
  • A need exists for LVM2 logical volumes shared by multiple cluster nodes
    • Optionally: A need exists for these volumes to be mirrored across redundant storage devices

Issue

  • How do I start clvmd and cmirrord?
  • I'm not able to find any systemd services for clvmd or cmirrord
# systemctl start cmirror.service
Failed to issue method call: Unit cmirror.service failed to load: No such file or directory.
# systemctl start clvmd.service
Failed to issue method call: Unit clvmd.service failed to load: No such file or directory.
  • RHEL 7 is missing startup scripts for daemons like clvmd and cmirrord that existed in RHEL 5 and 6
  • Unable to start clvmd in pacemaker cluster

Resolution

Package Installation


To use `clvmd`, the `lvm2-cluster` package must be installed from the [Resilient Storage repository](https://access.redhat.com/site/solutions/45930) on all nodes.

To use cmirrord, the cmirror package must be installed from the same repository.

Daemon Configuration


To use clustered logical volumes through `clvmd`, `locking_type` must be set to 3 in `/etc/lvm/lvm.conf`. This can be done using `lvmconf`: 
# lvmconf --enable-cluster

Daemon Start/Stop

RHEL 7

The `clvmd` and `cmirrord` daemons can both be started and managed using the `ocf:heartbeat:clvm` resource agent, and do not need to be started during boot with `systemd`. This agent depends on DLM, so first a [`controld` resource and clone-set should be created to run throughout the cluster](https://access.redhat.com/site/solutions/878043)

Now the clvm resource and clones can be created:

# pcs resource create clvmd ocf:heartbeat:clvm clone interleave=true ordered=true

Or if clvmd and cmirrord are to be used (i.e. if there will be mirrored logical volumes in clustered volume groups):

# pcs resource create clvmd ocf:heartbeat:clvm with_cmirrord=true clone interleave=true ordered=true

NOTE: ensure that with_cmirrord=true occurs before the clone parameter in the command so it is assigned to the correct attribute section

Because this resource depends on DLM, ordering and colocation constraints should also be created to enforce the dependency:

# pcs constraint order start dlm-clone then clvmd-clone
# pcs constraint colocation add clvmd-clone with dlm-clone

These should automatically start throughout the cluster and allow for the use of clustered volumes whenever nodes have joined the cluster.

NOTE: It is important that any dependent LVM resources be ordered to start after this clvm resource. There should either be an order constraint dictating that clvm starts before any LVM resource or the group it is in, or some other mechanism should be used to ensure this ordering. Dependencies between LVM and clvm resources, or between controld and clvm resources can be created in a few different ways.

For more information on configuring the cluster to manage GFS2 resources, see the RHEL 7 Global File System 2 documentation.

RHEL 6

Simply enable clvmd in chkconfig, and cmirrord if cluster-mirrored volumes will be used:

# chkconfig clvmd on
# # cmirror Optional - only if using cluster mirrored volumes
# chkconfig cmirrord on

Now when starting and stopping pacemaker or the cluster through normal means using pcs cluster start, pcs cluster stop, service pacemaker start, or service pacemaker stop, clvmd and cmirrord will be started and stopped as needed. There is no need to configure resources in the cluster to manage these daemons.

Root Cause

Although a SysVinit-style clvmd startup script is provided in RHEL 7, it is not supported by Red Hat in that release. The correct method for configuring clvmd to start and be managed is through an ocf:heartbeat:clvm resource within pacemaker.

In RHEL 6, clvmd and cmirrord are automatically started and stopped by the pacemaker init service when it is called, either on boot-up or shutdown if pacemaker is chkconfig'd on, or when started manually with service pacemaker start or service pacemaker stop. Using pcs cluster start or pcs cluster stop to start and stop cluster nodes will also cause clvmd and cmirrord to be started.the SysVinit scripts for clvmd and cmirrord are the correct way to start and stop these daemons.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.