[Engineering Notes] scsi INQUIRY and REPORT TARGET PORT GROUPS commands with regards to path state and priority

Updated

Overview

There are several different methods used to determine path priority, but the standard one using scsi commands relies on scsi inquiry device identification page 0x83 and the output from scsi report target port groups (RTPG) commands.

So inquiry page 83 defines which port group the current device/path is located in, and the report target port groups defines that status of that port group. If either of those pieces of information are not available, then the "prio alua" path priority method cannot be specified within multipath for the associated storage device(s).

The RTPG output defines port groups, their current state and what path states they support. Each port group within RTPG lists a target port group id along with a number of relative target port ids contained within the port group. This creates, in essense, a set of paired numbers of {port id, relative id} for each port group. The information from scsi inquiry defines a {port id, relative id} for the lun. Using that as a search key into the RTPG data, the kernel can determine what the path state and in turn the path priority is of the lun. Examples will be shown more fully below, but a condenced example:


device's Page 83                device's RTPG table
(group,relative ids) ------+    (g,r): path state
                           |    (g,r): path state
                           +--> (g,r): path state
                                (g,r): path state

SCSI NQUIRY PG 83 from device
# sg_inq -p 0x83 devicename
:
      Relative target port: 0x3
:
      Target port group: 0x1

>> Using 3,1 from above, look-up the current path state:

SCSI RTPG from device:    
# sg_rtpg devicename
:
    target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x00  <== alua state is active/optimized as defined by SCSI standard
    Relative target port ids:
      0x1
      0x2
      0x3
      0x3

So in the above case this device's path will be assigned a path priority of '50'. To change that priority, say to '10', then the configuration within storage would have to be modified by the storage vendor or storage admin.

While SCSI Asymmtric Logical Unit Access (ALUA) depends upon valid information from scsi inquiry and rtpg commands to determine which paths are active/optimized vs active/non-optimized, these same scsi commands can be used for determine path priority in Active/Passive configurations or just all Active path situations. However, since these commands post-date most Active/Passive storage firmware releases they are not typically used or available for use with many vendor's Active/Passive storage models. Instead, most Active/Passive storage models will use a vendor specific means of determining path state. For example, "prio rdac" is used for some Active/Passive IBM storage models and "prio emc" is used for some Active/Passive EMC storage models. Moreover, for all Active path situations, use of "multibus" plus "prio const" multipath attributes are equivalent to invoking "group by prio" and "prio alua" attributes so in cases of all Active path states many storage boxes do not bother to implement RTPG or target/relative port ids within inquiry page 83 output and just specify the use of "multibus" plus "prio alua" instead.

Implicit vs Explicit ALUA

You will likely see events within the /var/log/messages of:

 .... scsi 0:0:0:1: alua: supports implicit TPGS
or
 .... scsi 0:0:0:1: alua: supports explicit TPGS

The TPGS within the event refers to Target Port Groups Support - that is the LUN supports ALUA via SCSI commands. Devices that support ALUA can be in one of two modes: implicit (read-only) or explicit (read/write).

  • implicit: REPORT TPGS is supported

    • the host is able to read the current TPG state (Target Port Group state) back from storage
      • Implicit support just means that the controller only supports reporting the target port group status via SCSI INQUIRY and REPORT TPG commands
      • The 'SET TPG' (Set Target Port Group) task management commands are not supported by storage
      • the host is unable to change or force ALUA state changes -- state changes are performed only by the storage controller's normal operating modes and methods

  • explicit: REPORT TPGS and SET TPGS are supported

    • the host is able to read the current TPG state back from storage
    • the host is able to set the current TPG state though the use of the 'Set Target Port Group' task management command -- the host can explicitly set the path state.
      • for example, this might be used a vendor specific multipath hardware handler to force failback or to keep all paths enabled/active on one controller to prevent split service (some luns served on one controller, while others served on a different controller)

The determination of whether a LUN is implicit or explicit TPGS is determined by examining the TPGS field within returned data from the SCSI INQUIRY command.

In conjunction with the "supports {implicit|explicit} TGPS" event in /var/log/messages there will be a second following event that identifies both the current path status and which path status are supported by the LUN. For example:

 .... scsi 0:0:0:1: alua: supports implicit TPGS
  .... sd 0:0:0:1: alua: port group 01 state N non-preferred supports tolusna

or
.... scsi 0:0:0:1: alua: supports explicit TPGS

.... sd 0:0:0:1: alua: port group 02 state A preferred supports tolUsNA

The highlighted line presents four key pieces of information about the LUN:

  • the port group this LUN in contained in (from INQUIRY PG83:'Target port group' field)
  • the current path state (from RTPG:'target port group asymmetric access state' field)
  • whether the current target port group is the preferred primary target port group or not (from RTPG:pref field)
  • which path states this LUN supports (from RPTG:T_SUP field, other path access states supported)
    • note: in some cases RTPG will return a path access state for the target port group, but not claim support of any specific path access states.
    • 'tolusna' indicates no path states will be returned, although the 'target port group asymmetric access state' will include a valid path access state
    • 'tolUsNA' indicates 'Unavailable', 'Active/Non-Optimal', and 'Active/Optimal' path states are supported.
    • Each upper case letter indicates the LUN supports the associated path access state, while lower case means it does not support that access state.
    • Description of the field in the SCSI specification "Beyond the current state reported in 'asymmetric access state' field, the RTPG also returns a set of state bit flags indicating which other access status are supported and can be reported back within the current 'asymmetric access state' field by this target port group".

Overview of ALUA SCSI commands

The scsi commands to request page 83 and report target port groups information from devices are available from the sg3_utils package:

# sg_inq  -p      /dev/sdN
# sg_inq  -p 0x0  /dev/sdN
# sg_inq  -p 0x83 /dev/sdN
# sg_rtpg -d      /dev/sdN

All commands need to return appropriate data in order to be supported within multipath ("prio alua").
If the sg_rtpg command, for example, returns:

# sg_rtpg -v /dev/sda
    report target port groups cdb: a3 0a 00 00 00 00 00 00 04 00 00 00 
report target port group:  Fixed format, current;  Sense key: Illegal Request
 Additional sense: Invalid field in cdb
bad field in Report Target Port Groups cdb including unsupported service action

then the device does not support the scsi command REPORT TARGET PORT GROUPS which is required for multipath prio alua use. Therefore, the storage box in which the device lives does not support scsi alua capability. It may be active/active (all paths same priority) and therefore doesn't need RTPG or uses a vendor specific method for determining path priority.

SCSI Inquiry (sg_inq)

To determine if the device supports Target Port Groups (TPGS), check the standard output of scsi inquiry command:

# sg_inq          /dev/sdN
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
  SCCS=1  ACC=0  TPGS=3  3PC=1  Protect=0  BQue=0
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
    length=36 (0x24)   Peripheral device type: disk
 Vendor identification: LIO-ORG 
 Product identification: buddisk4        
 Product revision level: 4.0
 Unit serial number: 4e4fc8de-391e-4b07-be9a-c9f0caf6823d

The scsi specification defines the values within the TPGS field as follows:

TPGS Field
00b The logical unit does not support asymmetric logical unit access of supports a form
    of asymmetric access that is vendor sepcific.  Neither the RTPG command nor the
    Set Report Target Port Groups command is supported.
01b The logical unit supports only implicit asymmetric logical unit access.  The logical
    unit is capable of changing target port asymmetric access states without a SET TARGET
    PORT GROUPS command.  The RTPG command is supported.  The SET TARGET PORT GROUPS
    command is not supported.
10b The logical unit supports only explicit asymmetric logical unit access. The logical
     unit only changes target port asymmetric access states as requested with the SET TARGET
     PORT GROUPS command. Both the RTPG command and the SET TARGET PORT GROUPS command
     are supported.
11b The logical unit supports both explicit and implicit asymmetric logical unit access.
    Both the RTPG command and the SET TARGET PORT GROUPS commands are supported.

So in the above case, TPGS=3 indicates both the RTPG (Report Target Port Groups) and SET TARGET PORT GROUPS commands are supported. Within the multipath alua code, the standard inquiry field TPGS is checked. If that field is zero, the event message "alua not supported" will be output from multipath code into /var/log/messages.

Not all devices support inquiry page 0x83, to determine if a device does, you can use page 0x0:

# sg_inq -p 0x0 /dev/sdN
 Only hex output supported
VPD INQUIRY, page code=0x00:
   [PQual=0  Peripheral device type: disk]
   Supported VPD pages:
     0x0        Supported VPD pages
     0x80       Unit serial number
     0x83       Device identification

In the above example, page 0x83 is supported. Page 0x83 contains an array of various device identifiers, but the main two of interest is the device wwid and ones associated with port target information. Just because the storage device supports the Device Identification page (0x83) does not automatically mean it will return target port group and relative target port id record types as part of that page's information.

Page 0x83 can contain a world wide unique identifier, if an identifier record includes an NAA format id. Other types of ids are available, including vendor specific for example. The NAA id is defined to be unique if in IEEE format. This id is commonly referred to as the device's world wide identifier or wwid.

The port target information defines a group and relative port id within that group for the device when accessed via the path we sent this command down (specifically its associated with the scsi target). Whereas the wwid for a device will be the same irregardless of which path the device is accessed on, the target port information is unique to the access path (target port) the inquiry command is sent to.

But first lets take a look at the whole output from a typical device which includes both wwid and target port information.

# sg_inq -p 0x83 /dev/sdN

VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    id_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x6016
      Vendor Specific Identifier: 0x47f02a00
      Vendor Specific Identifier Extension: 0x6ef3fad97224e011
      [0x6006016047f02a006ef3fad97224e011]
  Designation descriptor number 2, descriptor length: 20
    id_type: vendor specific [0x0],  code_set: Binary
    associated with the addressed logical unit
 00     00 01 00 00 00 00 00 00  00 8c 00 00 00 00 00 00    ................
  Designation descriptor number 3, descriptor length: 8
    id_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x5
  Designation descriptor number 4, descriptor length: 8
    id_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x1

There can be multiple identifier records returned within page 0x83. In the case above there are four:

  1. a NAA device identifier (wwid in this case: 0x6006016047f02a006ef3fad97224e011),
  2. a vendor specific identifier,
  3. a relative target port identifier (0x5), and
  4. a target port group identifier (0x1).

The last two fields, {target port group,relative target port} ids, are used as part of ALUA and serve as a lookup key into RTPG port information. The REPORT TARGET PORT GROUPS (RTPG) output will include port groups organized by those two numbers and specify the group path state as we'll see below.

Device Identifier (wwid)

Lets take a separate look at the NAA identifier and then the two port identifiers.
NOTE: the numbers below, like 7.7.3.3, identifier the scsi specification clause that describes this field/value combination. In this case from "SCSI Primary Commands - 4 (SPC-4)" draft, dated 18 February 2009.
  Designation descriptor number 1, descriptor length: 20
    id_type: NAA,  code_set: Binary                          (7.7.3.6)
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x6016
      Vendor Specific Identifier: 0x47f02a00
      Vendor Specific Identifier Extension: 0x6ef3fad97224e011
      [0x6006016047f02a006ef3fad97224e011]

The above identifier record contains an NAA identifier. The Identifier field contents and description are defined within the standard. See the specification for more detailed information. The type of identifier can be one of the following per scsi specification:
  • /* 0h */ Vendor specific (7.7.3.3)
  • /* 1h */ T10 vendor ID based (7.7.3.4)
  • /* 2h */ EUI-64 based (7.7.3.5)
  • /* 3h */ NAA (7.7.3.6)
  • /* 4h */ Relative target port identifier (7.7.3.7) {aka Port # ;1=A, 2=B}
  • /* 5h */ Target port group (7.7.3.8)
  • /* 6h */ Logical unit group (7.7.3.9)
  • /* 7h */ MD5 logical unit identifier (7.7.3.10)
  • /* 8h */ SCSI name string (7.7.3.11)
  • /* 9h */ -
    /* Fh */ Reserved

With NAA identifier type records, there are several different formats possible for the identifer itself to be in. Again these are defined by the standard:

  • 0x2, 2h IEEE Extended (7.7.3.6.2)
  • 0x3, Locally Assigned (7.7.3.6.3)
  • 0x5, IEEE Registered (7.7.3.6.4)
  • 0x6, IEEE Registered Extended (7.7.3.6.5)
  • all others, reserved
Links contained herein to external websites are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to a website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

NOTE: The IEEE name formats are described in the following document: Content from standards.ieee.org is not included.http://standards.ieee.org/regauth/oui/tutorials/fibreformat.html

The format number refers to the leading nibble of the identifier. The most common formats used are 5h and 6h. In the above case the wwid starts with a 6, so is "6h -- IEEE Registered Extended" format. When linux displays wwids, it often prepends a letter/digit to the wwid. With wwids coming from page 83, it prepends the identifier type, in this case a '3' to denote an NAA type. 

$ ls -l /dev/disk/by-id/*6006016047f02a006ef3fad97224e011
lrwxrwxrwx. 1 root root 9 Apr 24 13:53 /dev/disk/by-id/scsi-36006016047f02a006ef3fad97224e011 -> ../../sdf
lrwxrwxrwx. 1 root root 9 Apr 24 13:53 /dev/disk/by-id/wwn-0x6006016047f02a006ef3fad97224e011 -> ../../sdf
NOTE: linux actually is displaying a unique identifier which sometimes includes the wwid if it was available. If page x83 isn't available, it reverts to using the serial number plus other inquiry fields.

Each IEEE identifer includes an OUI, and Organizationally Unique Identifier. See the following links for more information on OUI:

" An OUI / "company_id" is a 24-bit globally unique assigned number referenced by various standards. The OUI is usually concatenated with 24 or 40 bits by an Organization to create a 48-bit or 64-bit number that is unique to a particular piece of hardware. The OUI can be used to create MAC Addresses, Bluetooth Device Addresses or Ethernet Addresses. There are other uses of the OUI as well, such as its use as a company identifier in the SNAP protocol. "

... and in this case a world wide unique identifier for a scsi device. This also means you can decode a wwid and lookup the company using the OUI within the wwid.... as in: 

$ ./wwn 0x6006016047f02a006ef3fad97224e011
                  Vendor    Vendor
 NAA     OUI      Specific  Specific Ext.
   6     00-60-16 047F02A00.6EF3FAD97224E011 CLARIION

In summary, the wwid of a device is available from scsi inquiry page 0x83 as shown above. This wwid will be the same independent of which path the device is accessed from (since the same lun is at the end of each of those paths). Multipath software can then group sdN disks returning the same wwid together as this signifies they are the same device.

Target Port Information

Now lets turn to the target port identifier records within page 0x83, namely: 
  Designation descriptor number 3, descriptor length: 8
    id_type: Relative target port,  code_set: Binary      (7.7.3.7) {aka Port #}
    associated with the target port
      Relative target port: 0x5
  Designation descriptor number 4, descriptor length: 8
    id_type: Target port group,  code_set: Binary        (7.7.3.8)
    associated with the target port
      Target port group: 0x1

The first record is a relative target port identifier, 5 in this case.

The second record is a port group, 1 in this case.

Its easier to think of these two together as port group,port ident so (1,5). Using the scsi report target port groups command we can retrieve the defining information for target port groups for this device and determine what that specific path to this device, above, represents in terms of access status. The two pieces of information combined, that is the path to device port group/port ident info from inquiry page 83 and the report target port groups information, are used to yield a path priority. So, whereas the returned wwid record for a device is the same independent of path used to retrieve the data, the returned (port group,port ident) can be different on each path of a device. To determine the path status for each (path group, port identifier) we'll need to retrieve the RTPG information for the device.

Report Target Port Groups (sg_rtpg)

The report target port groups command defines the target port groups for the accessed device. The returned information is the same independent of which path is used to access the same device. The target port group information defines whether and which groups are optimized or non-optimized (associated with ALUA), etc. Also the target port group number and port identifier from the inquiry command can be used to determine which target port group that path belongs to and help derive a path priority.

For example, the following is the output for report target port groups for the same device as above.

# sg_rtpg -d /dev/sdN
Report target port groups:
  target port group id : 0x1 , Pref=0
    target port group asymmetric access state : 0x01 (active/non optimized)        current path access state
    T_SUP : 0, O_SUP : 0, L_SUP: 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1   supported path access states
    status code : 0x02 (target port asym. state changed by implicit lu behaviour)
    vendor unique status : 0x00
    target port count : 08
    Relative target port ids:
      0x01
      0x02
      0x03
      0x04
      0x05
      0x06
      0x07
      0x08
  target port group id : 0x2 , Pref=1
    target port group asymmetric access state : 0x00 (active/optimized)
    T_SUP : 0, O_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
    status code : 0x02 (target port asym. state changed by implicit lu behaviour)
    vendor unique status : 0x00
    target port count : 08
    Relative target port ids:
      0x09
      0x0a
      0x0b
      0x0c
      0x0d
      0x0e
      0x0f
      0x10

The full set of asymmetric access state values defined by the SCSI specification are:
  • 0h Active/Optimized        {multipath will show path as 'active'}
  • 1h Active/Non-optimized {multipath will show path as 'active'}
  • 2h Standby                          {multipath will show path as 'ghost'}
  • 3h Unavailable
  • 4h-Eh Reserved
  • Fh Transitioning between states

The full list of access states supported defined by the SCSI specification are:

  • T_SUP: state transitioning supported
  • O_SUP: offline state supported
  • U_SUP: unavailable state supported
  • L_SUP: logical block dependent (in later SPC-4 specs)
  • S_SUP: standby state supported
  • AN_SUP: non-optimized state supported
  • AO_SUP: optimized state supported

A '1' within the RTPG output indicates the state is supported, a '0' indicates it is not supported by this lun.

The above indicates the target port group 1 includes relative identifiers 1-8 (1,1, 1,2, ... 1,8) -- 1,5 being the device info from the sg_inq page 0x83 commmand previously shown). This port is the non-preferred port (Pref=0) and has an access state of active/non-optimized. This is the access state of "off-speed" ALUA paths. As such the calculated path priority for devices in this target port group (1,1 ... 1,8) will be less than the calculated priority for devices in target port group 2. If multipath is grouping paths by priority, then the non-optimized paths will be grouped together in the one group and the optimized paths in another group.

The other port group is preferred port (Pref=1) and its access state is active/optimized. This being the active/optimized path group, all devices in this group (2,9 ... 2,16) will be the current active/in-use path group within multipath if paths are grouped by priority.

The device has Asymmetric Logical Unit Access (ALUA) access if the sg_rtpg command output includes both types of active access within different access groups:

# sg_rtpg  -d /dev/sdN | grep "access state"
    target port group asymmetric access state : 0x01 (active/non-optimized)
    target port group asymmetric access state : 0x00 (active/optimized)

The device is active/active if the sg_rtpg command output includes just active/optimized type of access within all access groups:

# sg_rtpg  -d /dev/sdN | grep "access state"
    target port group asymmetric access state : 0x00 (active/optimized)

A caveat for active/active is that such devices may not support sg_rtpg commands.

In summary, if the system appears to be creating path priorities or grouping paths incorrectly, then reviewing the inquiry data from page 83 and the report target port groups information can be done to determine if the problem is more likely a storage or host-side configuration specific issue.

/var/log/messages event: "port group 02 state A supports ToUsNA"

This string is output from the following code when the alua path priority code is being run within native multipath:

/*
 * alua_rtpg - Evaluate REPORT TARGET GROUP STATES
 * @sdev: the device to be evaluated.
 *
 * Evaluate the Target Port Group State.
 * Returns SCSI_DH_DEV_OFFLINED if the path is
 * found to be unuseable.
 */
static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)
{
:
                 sdev_printk(KERN_INFO, sdev,
                    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
                    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),
                    pg->pref ? "preferred" : "non-preferred",
                    valid_states&TPGS_SUPPORT_TRANSITION?'T':'t',
                    valid_states&TPGS_SUPPORT_OFFLINE?'O':'o',
                    valid_states&TPGS_SUPPORT_LBA_DEPENDENT?'L':'l',      << RHEL6,7
                    valid_states&TPGS_SUPPORT_UNAVAILABLE?'U':'u',
                    valid_states&TPGS_SUPPORT_STANDBY?'S':'s',
                    valid_states&TPGS_SUPPORT_NONOPTIMIZED?'N':'n',
                    valid_states&TPGS_SUPPORT_OPTIMIZED?'A':'a');

Note that an uppercase letter indicates the state is supported and a lower case that it is not a state supported by storage per RTPG command output.

The "state A" part therefore comes from here:

static char print_alua_state(int state)
{
        switch (state) {
        case TPGS_STATE_OPTIMIZED:
                return 'A';                    << Active/optimized
        case TPGS_STATE_NONOPTIMIZED:
                return 'N';                    << Active/Non-optimized
        case TPGS_STATE_STANDBY:
                return 'S';                    << aka passive
        case TPGS_STATE_UNAVAILABLE:
                return 'U';
        case TPGS_STATE_OFFLINE:
                return 'O';
        case TPGS_STATE_TRANSITIONING:
                return 'T';
        default:
                return 'X';
        }
}

So RTPG data can express not only Active/Optimized and Active/Non-optimized but Standby (passive) and other states.

In the specific example case of "port group 02 state A supports ToUsNA":

  • this device is Active/Optimized state
  • it supports Transistioning, Unavailable, Active/Non-optimized and Active/Optimized states (upper case letters).
  • it does not support Offline and Standby states (lower case letters).
Tags
Article Type