[Troubleshooting] How to check Smart Array for logged hardware problems

Updated
The following references HPE provided utilities ssacli, ssaducli, hpssacli, hpacucli (hpssacli and hpacucli are predecessors to ssacli utility, these older utilities and now deprecated), and hpasmcli. These command utilities are neither shipped nor supported by Red Hat.
DISCLAIMER: The following information has been provided by Red Hat, but is outside the scope of the posted This content is not included.Service Level Agreements and This content is not included.support coverage. The information is provided as-is without representations or warranties about the suitability or accuracy of the information provided. The intent is to provide information to accomplish the system's needs. Use of the information below is at the user's own risk.

Issue

* How to collect the IML output from Smart Array * How to tell Smart Array adapter has had hardware issues * How to collect Smart Array adapter configuration information * How to tell the type of hardware raid configuration setup for Smart Array logical volumes * How to tell the current state of both logical and physical disks attached to the Smart Array storage controller * How to see if cciss controller has logged hardware events * How to see if hpsa controller has logged hardware events * How to collect an HPE AHS (Active Health System) log to share with HPE

Environment

* Red Hat Enterprise Linux 5, 6, 7, 8 * Smart Array Adapter

Resolution

There are two main logs (items 2 & 3) plus three other pools of information (1, 4, and 6) to share with HPE for a hardware event review of the Smart Array. Include at least items 2 & 3 with any Smart Array related support case, or all if so asked. It is very common that when we engage HPE for assistance on an issue they require an AHS (item 6), for example.
  1. IML: the IML log, (hpasmcli show iml),
  2. ADU: the ADU log (hpssacli ctrl all diag file=./adudreport.dat ris=on xml=off),
  3. CFG: the basic controller configuration information, and
  4. EVT: the event log(s), if available
  5. TAR: combine the files from 1-4 and submit.

  6. AHS: HPE's AHS (Active Health System) log, submit separately
      7. NOTE: The AHS report is a binary file that requires an HPE proprietary viewer. Red Hat does not review AHS reports, but simply passes them on to HPE. An AHS log is only needed when an issue is being worked directly with HPE -- typically via a bugzilla. So while providing one when requested or on a pro-active basis just be aware that Red Hat does not have the means to decode or review a submitted AHS log.

Submit gathered IML, ADU, basic configuration (config_info.txt), and event logs (hpsa_event_log.tar, if available) files to HPE and, if requested, to Red Hat support case. An HPE support case may be needed as much of the information within the gather logs is in a vendor specific format that only HPE can decode. An as-is/unsupported bash script is available to help with gathering the data from the controller. It can be This content is not included.[downloaded] (use right click and "save link as" option), and then perform commands:
      # chmod +x infostor-smartarray.bsh ; ./infostor-smartarray.bsh
and upload the resulting file.
NOTE: the script does not capture the AHS log as this requires login into iLO.

1. IML

 There are a couple ways to check the hardware IML (Integrated Management Logs) output from Smart Arrays, including via iLO on HPE Servers.

To use iLO to download an IML, see HPE Documentation such as

Contact HPE for further assistance on iLO use.

A common terminal-based method is to install and use HPE's hpasmcli utility. If you don't already have this utility installed, it is available from HPE within their PSP (ProLiant Support Pack) kit off their website (a search of hpasmcli will eventually lead to a PSP download page).

You don't have to, or necessarily want to, install the whole PSP kit. To install just the hpasmcli utility, de-tar the PSP kit and then only install the health rpm which contains the hpsasmcli.

For example, after downloading the PSP kit from HPE's website: 


# tar xvf psp-9.10.rhel5.x86_64.en.tar.gz
# rpm -iv hp-health-9.1.0.42-52.rhel5.x86_64.rpm
# /sbin/hpasmcli
HPE management CLI for Linux (v2.0)
Copyright 2008 Hewlett-Packard Development Group, L.P.

--------------------------------------------------------------------------
NOTE: Some hpasmcli commands may not be supported on all Proliant servers.
      Type 'help' to get a list of all top level commands.
--------------------------------------------------------------------------
hpasmcli> show IML
:
<IML events>
.

HPE should be engaged to review entries within the IML log, especially if events include things like:

POST Error: 1719 - A controller failure event occurred prior to this power-up

Should be able to collect this into a file via:

# hpasmcli show IML > hpsa-config-IML.txt

2. ADU

 Use HPE's `ssaducli` utility to extract and ADU report. The current version of the utility as of 11/2020 is:
# ./ssaducli -V

Smart Storage Diagnostics 4.21.7.0 2020-07-15

# ssaducli -adu -f ./hpsa-config-adureport.dat

Contact HPE for the latest version of this vendor supplied utility or visit HPE's support website and download/install the correct and latest HPE command utility for your model Smart Array card: typically this will be the ssaducli utility. For example Content from support.hpe.com is not included."HPE Smart Storage Administrator Diagnostic Utility (HPE SSADU) CLI for Linux 64-bit".

3. CFG: Basic Controller Configuration Info

Use the vendor utility ssacli to gather basic controller configuration information. The current version of this utility as of 11/2020 is:

# ssacli version

SSACLI Version: 4.21.7.0 2020-07-15

SOULAPI Version: 4.21.7.0 2020-07-15

Contact HPE for the latest version of this vendor supplied utility or visit HPE's support website and download/install the correct and latest HPE command utility for your model Smart Array card: typically this will be the ssaducli utility. For example Content from downloads.linux.hpe.com is not included."HPE Repository Listing".contains the ssacli rpm with the above version available for download.

As root, run the following ssacli commands and submit the resulting config-info.txt file to both HPE and Red Hat support cases.

echo $(date)                                > hpsa-config-info.txt
uname -a                                   >> hpsa-config-info.txt
echo "ssacli version"                      >> hpsa-config-info.txt ; ssacli version              &>> hpsa-config-info.txt
echo "ssacli ctrl all show status"         >> hpsa-config-info.txt ; ssacli ctrl all show status &>> hpsa-config-info.txt
echo "ssacli ctrl all show detail"         >> hpsa-config-info.txt ; ssacli ctrl all show detail &>> hpsa-config-info.txt
echo "ssacli ctrl all show config"         >> hpsa-config-info.txt ; ssacli ctrl all show config &>> hpsa-config-info.txt
echo "ssacli ctrl all show config detail"  >> hpsa-config-info.txt
      ssacli ctrl all show config detail  &>> hpsa-config-info.txt
echo "ssacli ctrl all diag"                >> hpsa-config-info.txt
      ssacli ctrl all diag  file=./hpsa-config-diag.dat ris=on xml=off &>> hpsa-config-info.txt
      tar --remove-files -Jcf hpsa-config-info.tar.xz hpsa-config-info.txt hpsa-config-diag.dat*

NOTE: Substitute hpssacli or hpacucli for the above ssacli command if running these older, deprecated versions of HPE's Smart Array utility.

4. EVT: Event (controller slot) Log(s)

 There also may be HPE event capture agents already installed and running. If so, then timestamped event logs will be available at location: 
/var/spool/compaq/hpasm/registry/serial_output/slot<X>

where <X> is a 0..n number corresponding the the Smart Array instance within the system. Create a tar file of any/all files at the above path, if present.

# tar -cvf hpsa-config-event-log.tar /var/spool/compaq/hpasm/registry/serial_output/*

 

 

5. TAR: Combine gathered data & submit

For example:

# tar -Jcvf hpsa-info.tar.xz hpsa-config*
hpsa-config-adudreport.dat
hpsa-config-event-log.tar
hpsa-config-IML.txt
hpsa-config-info.tar.xz

 

Submit file `hpsa-info.tar.xz` to appropriate support case(s).

 

6. AHS: HPE's Active Health System (AHS)

There also may be HPE AHS information that can be captures via iLO for later generation systems. When Red Hat works issues in conjunction with the hardware vendor, a copy of the AHS is often requested. If available on your model server, please gather and submit. The general steps to collect an AHS log should be:

  1. Log into iLO GUI
    2. Click on the "Active Health System Log" button
    • From there you can select to dlownload/get an AHS log covering a specified date range
  2. Select "Show Advanced Settings" and select download of the entire AHS log available from the system<.li>

For example, see HPE documentation:

Or contact HPE directly for more detailed instructions regarding collection of AHS logs.

Product(s)
Category
Article Type