How can I freeze a cluster service or stop single resources of a service for maintenance in RHEL?

Environment

• Red Hat Enterprise Linux (RHEL) 5 Update 3 or later with the High Availability Add On
• Red Hat Enterprise Linux Server (RHEL) 6 with the High Availability Add On (rgmanager or pacemaker)
• Red Hat Enterprise Linux Server RHEL 7
• Red Hat Enterprise Linux Server RHEL 8
• Red Hat Enterprise Linux Server RHEL 9
• One or more services in /etc/cluster/cluster.conf

Issue

• How can I temporarily disable rgmanager status checks against a service so that I can perform maintenance on the resources without causing a service failure?
• For maintenance it is needed to temporarily stop a resource that is part of a cluster service, but the rest of the service should remain running
• When stopping a resource that is part of a cluster service, rgmanager automatically starts the resource again when it detects it has stopped
• How to freeze cluster services during network outages or maintenance?

Resolution

rgmanager:

Freeze complete service/resource group:

A cluster service can be frozen with clusvcadm -Z. When a service is frozen, the resources will no longer be doing status checks until the service is unfrozen. The main reason for freezing a service is to do maintenance on the cluster. When a clustered service is frozen, there will not be any failover if one of the resources fails(recovery would begin after the service is unfrozen if it is still in failed state).

• To freeze the cluster service, use the -Z option.

# clusvcadm -Z myService

While the service is frozen, no status checks will be executed against the resources it contains. This means that individual resources can be stopped and started and rgmanager will not detect it or cause a failover.

• To unfreeze the cluster service, use the -U option.

# clusvcadm -U myService

Manage (Start/Stop/Status) Particular Resource:

If individual resources must be stopped or started while the service is frozen, this can be done with rg_test:

Warning: rg_test is able to start, stop, and status resources outside of the normal policies enforced by rgmanager. For instance, if a service is running on node 1, rg_test is able to start that service on node 2 causing corruption. This command should be used with extreme caution.

# rg_test test /etc/cluster/cluster.conf stop mysql mysql-resource

When all necessary resources have been shutdown, the maintenance can be done. After finishing the maintenance, start the resource with rg_test again:

# rg_test test /etc/cluster/cluster.conf start mysql mysql-resource

Until the service is unfrozen it will not be monitored by rgmanager. Before unfreezing, it is generally recommended to check the status of the service. Otherwise, if a resource was not started back up properly, as soon as the service is unfrozen and a status check is run, rgmanager would detect a failure and perform recovery. To check the status of the service, again use rg_test:

# rg_test test /etc/cluster/cluster.conf status service myService

*Note: The rg_test utility will stop all instances of a resource on a given node, potentially causing undesired results if multiple services on a single node are sharing the same resource. Do not perform these steps on resources which have multiple instances within cluster.conf unless all relevant services have been frozen.

pacemaker:

To put the cluster in maintenance mode or freeze it:

# pcs property set maintenance-mode=true

To unfreeze the cluster:

# pcs property unset maintenance-mode

Root Cause

• Freezing and unfreezing a cluster-service is possible with rgmanager-2.0.46-1.el5 and newer, which is available since Red Hat Enterprise Linux 5 Update 3.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.