fence_vmware_rest monitor fails with error: "Exception: 400: Too many virtual machines. Add more filter criteria to reduce the number."

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 7, 8 with the High Availability Add-on
  • vSphere 6.5 or above
  • fence_vmware_rest

Issue

  • fence_vmware_rest monitor or list operation fails when more than 1000 virtual machines are found.

Resolution

Red Hat Enterprise Linux 7
  • The issue (bz1654058) has been resolved with errata RHBA-2020:3850 with the following package(s): fence-agents-4.2.1-41.el7 or later. A new option was added called filter.
    #####Red Hat Enterprise Linux 8
  • The issue (bz1827652) has been resolved with errata RHBA-2020:4622 with the following package(s): fence-agents-4.2.1-53.el8 or later. A new option was added called filter.

Workaround


Disable the monitor operation of the `fence_vmware_rest` stonith device. Refer to Method 2 of the following Knowledgebase solution: [How can I leave a resource in the configuration but no longer monitor it or have the cluster manage it in a RHEL 6 or 7 High Availability cluster with pacemaker?](/solutions/1259763). The device can still be used to fence an unhealthy node but will not run its periodic status checks.

With the errata above a new option was added that can be used to filter which VMs are listed when querying VMware VCenter called: filter. The option can filter which VMs are listed and in most cases you will want to filter with filter.names. This can be exact names or a substring used in multiple VM names. The filter option was added because VMware rest API will error out if there are more than a 1000 VMs that will be returned when listing the VMs. The filter option provides us with the ability to limit which VMs are listed in order
to keep the VMs listed under 1000.

For example:

# fence_vmware_rest  --filter="filter.names=vmware-node21&filter.names=vmware-node22" .....
For more information about using filters on VMware then see:

Root Cause

This is a limitation of the VMware REST API. According to the API documentation, the Content from code.vmware.com is not included./vcenter/vm GET call supports "at most 1000 visible... virtual machines in vCenter matching the VM.FilterSpec."

Diagnostic Steps

# fence_vmware_rest -a <address> -l <login> -p <password> -o list
Traceback (most recent call last):
  File "/usr/sbin/fence_vmware_rest", liine 189, in <module>
    main()
  File "/usr/sbin/fence_vmware_rest", line 184, in main
    result = fence_action(conn, options, set_power_status, get_power_status, get_list)
  File "/usr/share/fence/fencing.py", line 1045, in fence_action
    outlets = get_outlet_list(tn, options)
  File "/usr/sbin/fence_vmware_rest", line 42, in get_list
    res = send_command(conn, "vcenter/vm")
  File "/usr/sbin/fence_vmware_rest", line 129, in send_command
    raise Exception("{}: {}".format(rc, result(["value"]["messages"][0]["default_message"]))
Exception: 400: Too many virtual machines. Add more filter criteria to reduce the number.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.