Why does the fencing among the VMs in RHV 4.1 environment is not working with fence_rhevm?

Solution Verified - Updated

Environment

  • Red Hat Virtualization 4.1
  • Red Hat Enterprise Linux 7 with High-Availability or Resilient Storage Add-on
  • fence-agents-rhevm-4.0.11-47.el7_3.5.x86_64

Issue

  • fence_rhevm command fails to provide correct status of agents for High Availability.

Resolution

    1. Check the api path is correct:

      fence_rhevm --api-path=/ovirt-engine/api/vms -a -l admin@internal -p -z -o list --ssl-insecure -v

  • Check for permissions of user:

    • Explicitly add to the relevant virtual machines the UserRole role from RHV GUI.

    • Double check that the filter+search is working correctly using following script:

        #!/bin/bash -ex

  url="https://myrhv.example.com/ovirt-engine/api"
  user="admin@internal"
  password="mypassword"
  vm="myvm"

  curl \
  --verbose \
  --cacert "/etc/pki/ovirt-engine/ca.pem" \
  --request GET \
  --header "Version: 3" \
  --header "Filter: true" \
  --header "Accept: application/xml" \
  --user "${user}:${password}" \
  "${url}/vms?search=name%3D${vm}"
    1. With fence-agents-rhevm-4.0.11-66.el7 or later, new --disable-http-filter option has been added, so that RHEV admin user can disable the default filtering.

    • With --disable-http-filter - RHEV admin user can access all VMs without explicitly granted permissions needed.

        # fence_rhevm -a 10.XX.XX.XX -l admin@internal -p password -z --ssl-insecure -o status -n vm1 --disable-http-filter
  Status: ON
    1. To use --disable-http-filter in pacemaker cluster stonith device, update the fence device configuration using command below. (below will disable the default filtering)

pcs stonith update fence_device disable_http_filter=1

    1. To use --disable-http-filter in cman fencedevice, update /etc/cluster/cluster.conf and propagate changes to all the nodes in cluster
        <fencedevice agent="fence_rhevm" disable_http_filter="1" ipaddr="$RHV-Manager" login="$USER@$DOMAIN" name="$FENCE-NAME" passwd="***" power_wait="10" shell_timeout="20" ssl="on" ssl_insecure="on"/>

Root Cause

A new header and option was added to fence_rhevm. The header Filter: True. The behaviour is related to permissions where the use of the Filter: true means that the user, in this case admin@internal, drops its super user privileges, so she will see only the virtual machines that she has explicitly been granted permissions (by default super users see all virtual machines).In addition, the the use of the Filter: true header wasn't compatible with search. So when the agent tries to search for a particular virtual machine, like below the server will used to be ignored. That was fixed in version 3.6 of RHV.

vms?search=name%3Dcvs3tci02

Diagnostic Steps

  • Following outputs can be seen when trying to list VMs and trying to fence them:

    • Listing VMs produces one of the following outputs

        # fence_rhevm -a <manager> -l admin@internal -p <password> -z -o list --ssl-insecure -v
  vms

  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>404 Not Found</title>
  </head><body>
  <h1>Not Found</h1>
  <p>The requested URL /api/vms was not found on this server.</p>
  </body></html>

      or

        # fence_rhevm -a <manager> -l admin@internal -p <password> -z -o list --ssl-insecure -v
  vms

  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <vms/>

    • Fencing the VM produces following output

        # fence_rhevm -a  <manager> -l admin@internal -p <password> -o status -n node_1 -z --ssl-insecure --api-path=/ovirt-engine/api/v3
  Failed: Unable to obtain correct plug status or plug is not available

  • Tried to change fence_rhevm script where fencing worked after commenting the below two lines.

       #"Prefer: persistent-auth",
   #"Filter: true",

  • The use of the Filter: true means that the user, in this case admin@internal, drops its super user privileges, so she will see only the virtual machines that she has explicitly been granted permissions (by default super users see all virtual machines).

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.