How can I customize the Performance Co-Pilot logging configuration

Updated 11 Aug 2021

Default configuration

The default logging configuration is generally sufficient for most performance analysis tasks. This configuration includes over 1000 metrics by default, and has been tuned to include sufficient metrics to allow most PCP client tools to replay performance data from an archive as part of a retrospective performance analysis. The actual default set of metrics is platform dependent - after enabling and starting the logging service (the service is named pmlogger), you can use the pminfo tool on one of the resulting PCP archives to list the set of metrics in that archive.

As such, logging configuration changes and customization are normally not needed and only required if there is s specific need, e.g.

if one or more non-default PCP Performance Metrics Domain Agent (PMDA) has been enabled, or
the logging frequency for certain metrics needs tuning to reduce the daily data volume, or
the logging frequency for certain metrics needs to be increased for greater resolution when investigating difficult performance problems
to add logging support for any additional metrics exported by a new PMDA or needed by a new or custom monitoring tool

It is also possible to dynamically adjust the metrics logged and/or the logging frequency using the Performance Metrics Inference Engine pmie(1) and the pmlc(1) tool, see their respective manual pages and the Content from pcp.readthedocs.io is not included.Performance Co-Pilot User's and Administrator's Guide. For example, if a certain performance condition is detected, pmie can use pmlc to add metrics and/or increase logging frequencies dynamically to allow more detailed analysis.

Logging Management

The pmlogger(1) service under the control of pmlogger_daily(1) is the most commonly used service for logging performance data on the local host. It's included in the base pcp RPM package and is available by enabling the pmlogger service (for RHEL6 chkconfig pmlogger on; service pmlogger start, or for RHEL7 and later: systemctl enable pmlogger; systemctl start pmlogger). The resulting archives are located by default in the directory /var/log/pcp/pmlogger/<HOSTNAME>.

The pmlogextract(1) and pmlogrewrite(1) tools can be used to merge multiple archives for the same host into a single archive and perform other log manipulation tasks.

Logging Configuration

The pmloggerlog control mechanisms are by default configured using pmlogconf(1), which uses files in the /var/lib/pcp/config/pmlogconf directory as it's source of configuration files.

$ cd /var/lib/pcp/config/pmlogconf
$ ls
apache  disk        kernel  memory  netfilter   platform    sgi     sqlserver 
cpu     filesystem  mailq   mysql   networking  postgresql  shping  tools      zimbra

Each of these directories contains a number of pmlogconf(1) configuration files, for example: /var/lib/pcp/config/pmlogconf/tools/free :

#pmlogconf-setup 2.0
ident	metrics used by the pcp-free(1) command
probe	kernel.uname.sysname ~ Linux ? include : exclude
	mem.util.free
	mem.util.shared
	mem.util.bufmem
	mem.util.cached
	mem.util.highFree
	mem.util.highTotal
	mem.util.lowFree
	mem.util.lowTotal
	mem.util.swapFree
	mem.util.swapTotal

The above configuration specifies that pmlogconf should probe the target system and if the returned value of the kernel.uname.sysname metric contains the string "Linux", then the following list of memory related metrics should be added the the current logging configuration. The pmlogconf(1) man page describes syntax other than "probe", e.g. to force logging without probing, and other options. This configuration syntax is extremely flexible and is designed to be portable and platform agnostic as shipped.

Enabling per-process logging

Per-process logging is not enabled by default - it's considered too voluminous. To enable logging for a reasonable subset of the proc.* metrics, e.g. sufficient to replay with the pcp-atop(1) command (and more generally, any other PCP command), use the following steps:

install the pcp-zeroconf package, which configures this to be enabled by default

edit /var/lib/pcp/config/pmlogconf/tools/atop-proc
change force available to force include
then restart the pmlogger service with: service pmlogger restart

Other points to note

the final set of logged metrics is the union of all pmlogconf configuration files. If a particular metric (or all metrics in a subtree) is named in more than one pmlogconf configuration file, it is only actually logged once per logging interval (not multiple times) by pmlogger.
the resulting pmlogger(1) configuration file is saved in /var/lib/pcp/config/pmlogger/config.default for the primary (localhost) pmlogger instance.
custom pmlogconf(1) configuration files can be created, e.g. /var/lib/pcp/config/pmlogconf/LOCAL which can then be incorporated into the system wide configuration by simply restarting the pmlogger service with systemctl restart pmlogger (or on RHEL6 with the appropriate equivalent service command).
it is simple to set up a "pmlogger farm", where multiple pmlogger processes will be managed, with each using the same input configuration but logging a different (remote) host. The default is just to log metrics for the local host.
See pmlogger_daily(1) and the Content from pcp.readthedocs.io is not included.Archive Logging section of the Content from pcp.readthedocs.io is not included.PCP Administrators Guide for more details.