Troubleshooting the fence_rhevm Fence Agent in a Red Hat Enterprise Linux High Availability environment

Solution Verified - Updated

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.0
  • Red Hat Enterprise Linux (RHEL) 6 with High Availability Add-On
  • Red Hat Enterprise Linux (RHEL) 5.7+(with ISCSI) with High Availability Add-On
  • Red Hat Enterprise Linux (RHEL) 5.9+ with High Availability Add-On

Issue

  • Is it possible to fence a cluster node using the agent "fence_rhevm"?
  • The error received is "Failed: Unable to obtain correct plug status or plug is not available".

Resolution

  • To test the fence_rhevm agent use the fence_rhevm command as follows, and replace the values shown as described below:
# time fence_rhevm -o status -z -a <rhev_Manager> -u <sslPort> -l '<user>@<domain>' -p '<password>' -n <vm_name>
  • -o status - This will check whether the node can be contacted and used by fence_rhevm without actually fencing. Remove this to execute the fence against the node.
  • <rhev_host> - This should be the IP address or resolvable hostname of the RHEV-M system which is running the virtual machines that make up the cluster. Be sure to ping test between the virtual machines and the RHEV-M host and modify network settings if necessary.
  • <sslPort> - This is the SSL port number set during initial install of RHEV-M. The default values are 8443 for RHEV 3.0 and 443 for RHEV 3.1+. When using the SSL default of 443, you could leave this out of the command.
  • <user> - This is the domain username, which corresponds to a user that has permission to power on and off the RHEV virtual guests forming the cluster. This would be the same username used when signing in o the RHEV user interface.
  • <domain> - This is the authentication domain associated with that domain user. This value is required, and it corresponds with the domain selected when that users signs in to the RHEV user interface.
  • <password> - This is the domain user's password. Again, this is the same password that user uses to sign in to the RHEV user interface.
  • <vm_name> - This is the name of the virtual machine in RHEV that corresponds to the node you are trying to fence.

Example status check:

# time fence_rhevm -o status -z -a rhevmanager.example.com -u 8443 -l 'admin@internal' -p 'password' -n VM1

When you are encountering errors, you can use additional options to increase the verbose output for better troubleshooting, and to capture the output in a file. The following can be used when SSL is over 8443 as in RHEV 3.0:

# time fence_rhevm -o status -a <rhev_hostname> -z -u 8443 -n <vm_name> -l '<user>@<domain>' -p '<password>' -v
# time fence_rhevm -o status -a <rhev_hostname> -z -u 8443 -n <vm_name> -l '<user>@<domain>' -p '<password>' -v -D /tmp/fence_rhevm-debug.txt
# time fence_rhevm -o status -a <rhev_hostname> -z -u 8443 -n <vm_name> -l '<user>@<domain>' -p '<password>' -vvv  | tee /tmp/fence_rhevm-debug.log

If the location of the REST API directory is not using the default then you should specify the location: How to specify the API directory used with the fencing agent fence_rhevm on RHEL 7?

# time fence_rhevm -o status -a <rhev_hostname> -z -u 8443 -n <vm_name> -l '<user>@<domain>' -p '<password>'  --api-path=/api -v

In addition the name of the virtual machine is case sensitive. To check if the name used with fence_rhevm is the same that is on the host then run the following command to get a list of virtual machines. Then verify that virtual machine names are the same(including the case of characters).

# curl -H "Accept: application/xml" -H "Content-Type: application/xml" -k -u <user>:<domain> https://<rhev-m hostname>/api/vms | grep "\<name\>"

When testing fencing a node in accordance with your cluster configuration, use pcs stonith fence in Pacemaker-based clusters, orfence_node in RGmanager-based clusters instead:

Pacemaker (RHEL 7, 8, and possibly RHEL 6.5 onwards)
# pcs stonith fence <node>
## Where <node> is the node name as known to Pacemaker
RGmanager (RHEL 6)
# fence_node -vv <node_name_from_cluster_config>
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.