How can I collect data from all my ESXi hosts with the PCP VMWARE agent?

Environment

• Red Hat Enterprise Linux (RHEL) 7
• VMware ESXi 6.5 (and others)

Issue

• How can I collect data from all my ESXi hosts

Resolution

On RHEL releases where the package is available:

• The PCP VMware PMDA from the pcp-pmda-vmware RPM package contains a perl script that calls the vSphere SDK API, as per https://www.vmware.com/support/developer/viperltoolkit/ and so should work with versions of ESXi that supports that API.
• The PMDA is written to monitor only one ESXi server. The PCP 'domain' is 90, as defined in /var/lib/pcp/pmns/stdpmid. The easiest way to monitor more than one ESXi server would be to deploy a dedicated guest running RHEL7 for each ESXi host to be monitored, and install PCP and the needed PMDAs on that guest. Each 'monitoring guest' can of course be hosted by the ESXi host they are monitoring, and would typically have a hostname that identifies that ESXi host (e.g. by appending 'mon' to the ESXi host name, or some such.
• The code of pcp-pmda-vmware code is not currently able to monitor more than one ESXi host. Use the dedicated monitoring guest approach described above
• Once you have all the dedicated monitoring guests up and running, you can also enable PCP archive logging by enabling the pmlogger service on each guest. It is also easy to set up centralized logging, where you have a single RHEL system running multiple pmloggers (each remotely monitoring one of the dedicated monitoring guests). In a centralized logging configuration, the advantage is that all of the PCP logs will then be available for analysis in the one location (below /var/log/pcp/pmlogger/<HOSTNAME> for each HOSTNAME).
• If you want to set up centralized logging with multiple pmlogger daemons running, the recommended service to use is pmmgr(1), although you can use the regular pmlogger(1) service if desired by adding additional control files, one for each remote host to be monitored. These control files live in /etc/pcp/pmlogger/control.d and by default we only ship 'local', which you can use as a template for others. In such a centralized logging deployment, you would NOT need to run the 'local' pmlogger service on each guest - all logging would be remote to the centralized logging server.

Root Cause

On older RHEL versions (RHEL6.6 and later, RHEL7 and 8) you can use the PCP VMware agent. Later it was removed, details from the commit message:

commit 70ebec85e9ac144be3da7d1962be2645158a89a8
Author: Mark Goodwin <mgoodwin@redhat.com>
Date:   Thu Oct 28 13:34:18 2021 +1100

    packaging: obsolete the disfunctional pcp-pmda-vmware package
    
    Remove the VMware PMDA - it is non-functional because it uses the
    deprecated viperl API. It is obsoleted by pcp-pmda-openmetrics,
    where we now ship a vmware.url that works with the metrics end-point
    served by vmware_exporter. See previous commits and comments in
    src/pmdas/openmetrics/vmware.url.  For more details on vmware_exporter,
    see https://github.com/pryorda/vmware_exporter#readme
    
    Note RHBZ#1758277 asks for vmguestlib, which provides more detailed
    metrics that vmware_exporter. A future PCP release may provide a new
    PMDA based on that API and in general other tools and libraries included
    as part of open-vm-tools. For further details, see
    https://github.com/vmware/open-vm-tools/blob/master/README.md
    
    With the vmware_exporter end-point, we have about 85 new metrics,
    which is better than none at all from the completely disfunctional
    out-going pcp-pmda-vmware package.
    
    Resolves:RHBZ#1485102
    Related:RHBZ#1758277

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.