How can I use Performance Co-Pilot (PCP) to capture a "once-off" performance data archive log for a specific interval of time

Solution Verified - Updated

Environment

  • RHEL5
  • RHEL6
  • RHEL7
  • other platforms, e.g. Fedora, Debian, etc.

Issue

  • the pmlogger service (or pmmgr) is normally enabled in order to capture long term performance data archive logs, however in some circumstances higher frequency sampled data is required on a short term basis
  • how can I use the pmlogger(1) utility to capture a once-off performance data archive log for a short time, e.g. whilst my benchmark runs, or for an hour at a certain time of the day

Resolution

  • first install pcp. On most versions of RHEL and Fedora, this is simply yum install pcp, since PCP is in the base distribution. For more details, see How do I install Performance Co-Pilot (PCP) on my RHEL server to capture performance logs. You can choose not to enable the pmlogger service, but the pmcd service needs to be enabled and running.
  • create or choose the pmlogger configuration file to specify the performance metrics that are to be logged. In the vast majority of cases, the global default configuration will be suitable. This configuration will log all of the performance metrics needed for most PCP tools (apart from per-process metrics, see below). Check if /var/lib/pcp/config/pmlogger/config.default exists. If it doesn't then the pmlogger service has never been started on this system. In that case, you can run pmlogconf to generate a suitable config file as follows:
# pmlogconf -c -r -q ~/myconfig.default
  • this will create myconfig.default in your home directory. You can use this config as-is, or modify it if you need to add or remove metrics to be logged.
  • The big exception to using config.default is if you need to log per-process "proc" metrics, in which case additional steps are required (see below). Per-process metrics are not logged by default by PCP due to the overheads and log volumes involved.
  • now start the workload (if necessary) - note that PCP collects system level performance data. If you need to capture performance data specific to a single application instead of at the system level, then you should use perf-record(1), which is NOT part of the PCP suite. Note also that PCP has APIs for instrumenting applications to incorporate application specific performance data into the PCP framework - it is often extremely beneficial to monitor this application specific performance data along side the system level performance data. See pmdammv(1) and the Content from www.performancecopilot.org is not included.PCP programmers Guide for details.
  • Now run the pmlogger utility to capture the performance data archive log, as follows :
pmlogger -c config.default -t 2s -s 30m /tmp/myarchive
  • the above command will use config.default (in the default pmlogger config search path /var/lib/pcp/config/pmlogger/), with a sampling interval of 2 seconds and will run for 30 minutes, to create the PCP archive log /tmp/myarchive.*. The -t and -s options support sensible suffixes, e.g. m for minutes, h for hours, etc. See the pmlogger(1) man page for details.
  • instead of running for a specified duration (-s 30m), one can tie pmlogger to a particular process (-p $pid), so that pmlogger shuts itself down automatically when the given process exits. This is handy when you just want to gather system level performance data whilst a benchmark runs.
  • wait for pmlogger to terminate (or interrupt it with ctl-c). You can now use any PCP tool to examine the data in /tmp/myarchive, e.g. :

# pminfo -a /tmp/myarchive will list the metrics in myarchive
# pmiostat -a /tmp/myarchive will report block device i/o statistics
# pmstat -a /tmp/myarchive will report CPU and VM statistics
# pmchart -c CPU -O-0 -a /tmp/myarchive will use the pmchart(1) graphical tool to replay myarchive, initially starting with the 'CPU' view, positioned at the end of the archive (-O-0).
.. and many other commands. You will need to install the pcp-monitor meta-package to gain access to these tools. This can be done on a different system, e.g. your workstation.

  • if required, the PCP archive can be attached to a support case by creating a compressed tarball, e.g. tar czf /tmp/myarchive.tgz /tmp/myarchive.*
  • note there are three files in a PCP archive: *.meta, *.index and *.0 for the metadata, temporal index and the data volume(s) respectfully.

To include per-process data in the archive

  • the easiest way is enable pcp-atop logging, as follows: edit /var/lib/pcp/config/pmlogconf/tools/atop-proc and change force available to force include. Obviously you'll need to do this before the step above where you ran pmlogconf to generate the pmlogger config file.
  • see also the man page for pmdaproc(1). This is an optional PCP agent that collects per-process data, with some powerful filtering mechanisms to help reduce data volume.
  • to use optional PCP agents, you may need to also install the pcp-collector package, along with it's dependencies.

Remote logging

See Also

Root Cause

  • most sites would just use the pmlogger or pmmgr service to capture performance data on an on-going basis. PCP logs will be rolled over periodically, see pmlogger_daily(1) and related commands.
  • in some circumstances, higher frequency sampled data is required on a short term basis - that's when the pmlogger(1) utility should be used manually as described in this knowledge base solution.
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.