How can I visually inspect Performance Co-Pilot or SAR archives?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 8, 9 and 10
  • Performance Co-Pilot (PCP)
  • sar (package sysstat)

Issue

  • How can I visually inspect Performance Co-Pilot archives?
  • How can I visually inspect sar archives?
  • I have a sosreport, which contains pmlogger archives in /var/log/pcp/pmlogger. How can I analyse these?

Resolution

Various supported methods exist to analyse/inspect pmlogger archive files, for example:

  • pmchart, an X based tool which is part of package pcp-gui. man pmchart for details.
  • pmrep, pminfo and further commandline tools

If sar data should be inspected, it first needs to be converted to PCP archive files. Package sysstat in newer RHEL versions contains the sadf utility. This tool can be used to convert to PCP archives, for example using this command:

$ sadf sa01 -l -O pcparchive=archivefile-sa01

For graphical inspection or PCP archive data, the following container can be used which will fetch a container with redis and grafana, import the archive files and make them accessible via webbrowser. Please note that the following container based inspection procedure is provided 'as is', and at the moment not supported.

The container is right now just built for the x86_64 architecture. It can be run for example like this:

$ podman run -t --security-opt label=disable -p 3000:3000 \
      -v /location/to/pcp/archives:/archives \
      quay.io/performancecopilot/archive-analysis

This command starts the container, which runs Redis and Grafana inside the container and loads all PCP archives of the selected directory into Redis. After executing the command, http://localhost:3000/dashboards can be accessed, and from there the analysis dashboard. This URL can be accessed with a webbrowser. Although the webaccess is already active, the import of the archive files might still be running. The console output will inform once all archives are imported, for example:

[..]
[23103.978779] import[35]: Importing archive /archives/20220905... [14/16]
[23114.152909] import[35]: Successfully imported archive in 0:10m.
[23114.153156] import[35]: Importing archive /archives/20220906.06.51... [15/16]
[23114.957352] import[35]: Successfully imported archive in 0:0m.
[23114.957757] import[35]: Importing archive /archives/20220828... [16/16]

When accessing the URL, the default time frame of 1 year is displayed, and the actual time frame covered by the archive files is typically just a small part of this. As first action, it is recommended to look at the graph, and use the mouse to select the time frame with data, to zoom into that time frame.

By default, a dashboard is displayed showing metrics like Load average, Memory utilization and so on. The archive files typically contain many more metrics, which can also visualized and used for analysis. To access these, the add panel button from the top can be used, or existing matric graphs like Load average can be edited to visualize different metrics.

More details:

Root Cause

After installation of the pcp-zeroconf package or configuration of pmlogger, collection of archives files to /var/log/pcp/pmlogger starts. PCP archive files are also part of generated sosreports.

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.