How to install and gather performance logging using Performance Co-Pilot ?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 7, 8, 9 and 10
  • Performance Co-Pilot (PCP)
  • pcp-zeroconf

Issue

  • Need to install and begin logging metrics in RHEL 7/8/9/10 ?
  • Environment stating performance degradation/slowness

Resolution

For RHEL 7/8/9, installing the pcp-zeroconf package will pull in package pcp and further packages via dependencies.

  • Install the pcp-zeroconf package:
# yum install pcp-zeroconf -y
  • Change default pmlogging time from 60 seconds to 10 seconds. See below section regarding "Change the default logging interval"
  • Upon installing the pcp-zeroconf package the PCP daemons pmlogger and pmcd are started automatically. You can chkconfig these on by issuing the below commands:
# systemctl enable pmcd
# systemctl enable pmlogger
  • Once the zero-conf package is installed, we can wait till the next event occurs. When this new event completes, tar the current logger files and to provide to Red Hat:
# tar czf /tmp/pcp-logs.tgz /var/log/pcp

  • Change the default logging interval
    The default logging (sampling) interval is 60 seconds. This interval was chosen as a balance between logging volume and the level of detail needed for long term monitoring. For storage related performance monitoring (in particular), you may want to increase the resolution of the data by sampling more frequently, for example every 10 seconds. To do that, edit /etc/pcp/pmlogger/control.d/local and append -t 10s to the line for LOCALHOSTNAME, and restart the pmlogger service (systemctl restart pmlogger or the equivalent service pmlogger restart command for RHEL 5/6). Note that on older versions of PCP, the file to edit is /etc/pcp/pmlogger/control. For further details see How can I change the default logging interval used by Performance Co-Pilot (PCP)?.

  • To completely stop the archive logs from being produced, you have to stop the cron job. If the pmlogger service is stopped only, it will be restarted by cron after being checked every 30 minutes. The cron file that is to be edited to completely stop the pmlogger from restarting is /etc/cron.d/pcp-pmlogger. Please comment the lines out of this file and re-run crontab so the entry is non-existent.

  • Please note that unlike some other tools, the Performance Co-Pilot data gathering tool is not lightweight and depending on the amount of processes running the logfiles in pmlogger may grow to be large. It is recommended to monitor the logging directory /var/log/pcp/pmlogger/hostname/ when the pmcd and pmlogger services are actively running to assure you do not have an issue with /var filling up. For more information on changing the logging configuration to keep archive files less days, please see the below knowledge article:
    How to configure maximum archive size and retention period for Performance Co-Pilot?

  • It is also to be noted that Performance Co-Pilot does collect data on previous version of Red Hat Enterprise Linux (RHEL 7.4-) but the output has been shown to not always be accurate. For performance data gathering for RHEL version lower than 7.5 please utilize collectl for performance metric gathering. For more information on collectl please see the knowledge article:
    How to use the collectl utility to troubleshoot performance issues in Red Hat Enterprise Linux

Below are some commonly used commands to monitor real time performance and their pcp equivalents.

Legacy Command PCP Command Common Options
iostat -d sda sdb 2 or iotop pmiostat -t 2 -R '^sd(a|b)$' The '-t 2' is the update interval.
Instead of the -R regex you can use '-x dm' for device-mapper
logical devices instead of scsi devices
top or pidstat 2 pcp atop Update interval is controlled in ineractive mode using 'i'.
The default 'g' for generic info is displayed.
Use 'm' for memory stats.
Use 'd' for disk usage.
Use 'c' for CPU usage of the command line of the running process
See pcp-atop(1) for more options
vmstat 2 pcp vmstat 2 The default update interval is 10 seconds. Adding '2' changes that to a 2 second update.
mpstat 2 pcp mpstat 2 Default interval is 10 seconds, adding '2' changes the update interval.
For a graphical view use 'pmchart -c CPU
dstat dstat is a link to /usr/libexec/pcp/bin/pcp-dstat pcp-dstat is a general performance analysis tool
that lets you view multiple system resources.
Common uses are:
--top-cpu
--disk-wait
--page
--load
--top-mem
--filesystem

For more information on Performance Co-Pilot's capabilities please see the Side-by-side comparison of PCP tools with legacy tools article.

Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.