How do I tell if ALUA is enabled on a LUN?
Environment
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 5
- SAN storage
Issue
- How do I tell if ALUA is enabled on a LUN?
- Apart from checking the SAN configuration, how can I tell if a LUN is in active-active or active-passive mode?
Resolution
Note: For brevity, herein we will use the following:
group:portas a short hand for referring to the pair of values associated with:These data fields are returned within both VPD page 83 via the SCSI INQUIRY command and within the returned data from the SCSI RTPG (Report Target Port Groups) command.
Target port group:Relative target portpath:lunas a short hand for referring to a specific path to a lun. For example the set of scsi addresses 1:0:0:20, 1:0:2:20, 2:0:1:20,2:0:2:20 may all be paths to the same device. The shorthand of path:lun is simply used to indicate we're talking about a specific path to a specific lun.
There are two parts to the answer of whether storage supports ALUA:
-
Are the necessary commands supported,
- From a technical standpoint, all that is required to support ALUA are the necessary SCSI commands. This means that the returned scsi version supported by storage must be 03h [SPC/SCSI-3] or greater. However, VPD (Vital Product Data) page 83 must also be present and available. VPD pages needed are not mandatory until scsi version 04h [SPC-2]. See the data section for more details.
-
The returned data is available and consistent, allowing the necessary group:port pair matching.
- VPD page 83 must be available. For RHEL 7 and later, this means either a scsi version of 04h [SPC-2] or later is returned by storage within INQUIRY or there is "special" entry in the kernel's device info table from the hardware vendor.
- From a practical standpoint, the commands must return data where matching group:port pairs in VPD and RTPG data is present.
- While the RTPG data returned from storage is common to the set of all path:lun for the same lun, the VPD page 83 data is typically specific to a given path:lun.
- When group:port pairs of data are inconsistent between VPD and RTPG, this can result in the following event being output:
... alua: No target port descriptors found ... alua: not attached
Verify Required Commands Supported
For ALUA to be supported, the following is required:
- SCSI RTPG (Report Target Port Groups) command is supported.
- SCSI version 03h [SPC], or greater must be returned by storage, this indicates that SCSI-3 command set is supported, which includes RTPG.
- It is unusual for storage to return a scsi version value less than 05h [SPC-3] these days.
- SCSI INQUIRY VPD page 83 is present,
- VPD page 83 contains a group:port pair of values.
Use the following steps to confirm storage supports the necessary commands.
Install the SCSI command utilities:
yum install sg3_utils
Send a "Report Target Port Groups" query to the path:
sg_rtpg -d /dev/sdX
The following output indicates a path does not support ALUA:
# sg_rtpg /dev/sdX
Report Target Port Groups command not supported
The following output indicates a path supports, is configured for, ALUA:
target port group asymmetric access state : 0x00 (active/optimized)
target port group asymmetric access state : 0x01 (active/non optimized)
The full set of asymmetric access state values defined by the SCSI specification are:
- 0h Active/Optimized
- 1h Active/Non-optimized
- 2h Standby
- 3h Unavailable
- 4h-Eh Reserved
- Fh Transitioning between states
Example
# sg_rtpg -vvd /dev/sda
open /dev/sdg with flags=0x802
report target port groups cdb: a3 0a 00 00 00 00 00 00 04 00 00 00
report target port group: requested 1024 bytes but got 116 bytes
Report list length = 116
Report target port groups:
target port group id : 0x1 , Pref=0
target port group asymmetric access state : 0x01 (active/non optimized)
T_SUP : 0, O_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
status code : 0x01 (target port asym. state changed by SET TARGET PORT GROUPS command)
vendor unique status : 0x00
target port count : 0c
Relative target port ids:
0x01
0x02
0x03 << target port id 1,3 is active/non optimized alua pass (offspeed path)
:
target port group id : 0x2 , Pref=1
target port group asymmetric access state : 0x00 (active/optimized)
T_SUP : 0, O_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1
status code : 0x01 (target port asym. state changed by SET TARGET PORT GROUPS command)
vendor unique status : 0x00
target port count : 0c
Relative target port ids:
0x0d
0x0e
:So this storage is configured in ALUA mode. Looking at a specific device's SCSI Inquiry VPD page 0x83, multipath can determine if the path is an optimized path or non-optimized (offspeed) path. All optimized paths would be grouped together. This is done by locating the group:port id pair from VPD page 83 and matching it to the same group:port id pair within the returned RTPG data.
# sg_inq -vv -p 0x83 /dev/sda : Designation descriptor number 3, descriptor length: 8 designator_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x3 Designation descriptor number 4, descriptor length: 8 designator_type: Target port group, code_set: Binary associated with the target port Target port group: 0x1So group:port key pair is 0x1:0x3 and that key pair
is present in the RTPG data. This will prevent the "No target port descriptor found" message.
So /dev/sda has group,port pair of 1,3. Looking up group id 1, port id 3 in the above sg_rtpg output we see that group is currently in the active/non-optimized state. Typically active/optimized path groups have a calculated priority of something list 50-80 and active/non-optimized have a lesser calculated priority such as:
mpath2 (3600601602df02d00aae0129e5c11e111) dm-60 DGC,VRAID
[size=30G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 1:0:3:20 sds 70:944 [active][ready]
\_ 0:0:1:20 sdk 67:288 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 0:0:2:20 sda 8:592 [active][ready]
\_ 1:0:2:20 sdx 128:592 [active][ready]
Verify Required Data Present
For ALUA to be supported, the following data is required to be available:
- From SCSI INQUIRY VPD page 83: a group:port pair
- VPD pages needed are not mandatory until scsi version 04h [SPC-2]. See This content is not included.notes below for why this matters for RHEL7 and later kernels.
- From SCSI RTPG data includes a matching group:port pair.
- First check that the VPD page 83 is available.
# # sg_inq -p 0 /dev/sda VPD INQUIRY: Supported VPD pages page Supported VPD pages: 0x0 Supported VPD pages 0x80 Unit serial number 0x83 Device identification 0xc1 vendor: Date code (seagate)
The following code from RHEL 7+ determines if VPD pages like page 83, will be read in at lun discovery time. So either 'try_vpd_pages' is set via a special vendor specific entry in the device info table or the scsi version returned by the INQUIRY command is SPC-2 or greater.
static inline int scsi_device_supports_vpd(struct scsi_device *sdev) { /* Attempt VPD inquiry if the device blacklist explicitly calls * for it. */ if (sdev->try_vpd_pages) return 1; /* * Although VPD inquiries can go to SCSI-2 type devices, * some USB ones crash on receiving them, and the pages * we currently ask for are mandatory for SPC-2 and beyond */ if (sdev->scsi_level >= SCSI_SPC_2 && !sdev->skip_vpd_pages) return 1; return 0; }
1a) Deterimine if a group:port key pair is available within VPD page 83.
1b) For RHEL7 or later kernels, check whether the returned SCSI version indicates VPD pages are mandatory.
- Case 1: storage only supports SCSI-3 commands via version=0x03 [SPC] per INQUIRY data.
- ALUA is not supported unless there is a "special" override entry for this storage vendor:model within the kernel. Otherwise you will see the "No target port descriptor found" events logged in messages.
- Case 2: storage supports SCSI-3 and mandatory VPD page 83 as it returns version=0x04 [SPC-2], or later within INQUIRY data.
- This is nominal case for the majority of storage models.
# sg_inq /dev/sda standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x03 [SPC] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=1 Resp_data_format=2 SCCS=0 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 [SPI: Clocking=0x0 QAS=0 IUS=0] length=252 (0xfc) Peripheral device type: disk Vendor identification: HPE Product identification: OPEN-V Product revision level: 9001 Unit serial number: 999999
In the above case, this storage only supports SCSI-3 commands -- which includes RTPG -- but is not compliant with providing mandatory VPD page access. While this does not impact RHEL 6 or earlier kernels, it does impact RHEL 7 and later kernels. Within RHEL6 and earlier kernels, multipath read in VPD pages as they were needed. Within RHEL7 and later, VPD pages are read in and saved at lun discovery time. Multipath code in RHEL7 then uses these cached VPD pages to perform the group:port id pair match. However, VPD pages are only read in if the device's scsi version is 04h [SPC-2] or later or if there is a "special" override entry present within the kernel.
NOTE: Setting host mode 68 within HP OPEN* storage eliminates the need for a special blacklist entry related to ALUA support. The change when host mode 68 is enabled includes returning scsi version 04h. the minimum version number needed for ALUA compliance. Please check with HPE to verify that this host mode is enabled to prevent issues.
There is a table within the kernel with "special" instruction entries that vendors can specify to change the kernel's behavior when interacting with a vendor's storage hardware. One such "special" instruction is a flag that overrides the scsi version and forces VPD pages to be read in. You can access this table via the following command:
# cat /proc/scsi/device_info | grep HP 'HP' 'C1750A' 0x1 'HP' 'C1790A' 0x1 'HP' 'C2500A' 0x1 'HP' 'A6189A' 0x240 'HP' 'OPEN-' 0x10020000 'HP' 'NetRAID-4M' 0x2 'HP' 'HSV100' 0x21000 'HP' 'C1557A' 0x2 'HP' 'C3323-300' 0x20 'HP' 'C5713A' 0x40000 'HP' 'DISK-SUBSYSTEM' 0x20000
There is a manual entry present for HP:OPEN with a couple flags set:
#define BLIST_REPORTLUN2 0x20000 /* try REPORT_LUNS even for SCSI-2 devs
(if HBA supports more than 8 LUNs) */
#define BLIST_TRY_VPD_PAGES 0x10000000 /* Attempt to read VPD pages */
So in this case HP:OPEN devices have "special handling" flags that include the flag specifying VPD pages should be read in, even though this storage indicates a compliance with a scsi specification that does not require VPD pages be present. The only issue is you'll note that the entry is "HP:OPEN-" and not "HPE:OPEN-". The vendor name change as of Feb, 2021 still had not been submitted upstream nor has the vendor opened a bugzilla with Red Hat to have the vendor name corrected. However, there is a method to manually add a corrective entry directly at boot time. In this case you would add the following to the kernel boot line:
... scsi_mod.dev_flags="HPE:OPEN-V:0x10020000"
You can use the same command above to verify the entry was added:
# cat /proc/scsi/device_info | grep HPE 'HPE ' 'OPEN-V ' 0x10020000
Note that when manually adding entries the full and exact vendor and model strings must be specified. Only static entries within the kernel can use a shortened notation for models such as specifying "OPEN-" to match all models of "OPEN-". If there are multiple and diffierent OPEN- model names present in the configuration, each one needs to be added separately. Contact your storage vendor for additional guidance.
For storage that returns scsi version 03h or less, if there is no "special" try vpd flags present within RHEL 7 (or later) kernels then ALUA support will not work. This is because the VPD page 83 will not be read in at lun discovery time and therefore no group:port key pair is available for the multipath alua kernel modules. The result will be the same "No target port descriptor found." event.
# sg_inq /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x04 [SPC-2] [AERC=0] [TrmTsk=0] NormACA=1 HiSUP=1 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=1 Protect=0 [BQue=0] EncServ=0 MultiP=1 (VS=1) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 [SPI: Clocking=0x0 QAS=0 IUS=0] length=193 (0xc1) Peripheral device type: disk Vendor identification: EMC Product identification: Invista Product revision level: 6200 Unit serial number: AAAAAAAAAAAA
In the above case, storage indicates a scsi version (04h) where VPD pages are mandatory. This is the nominal case with almost all storage. No "special" override flags are required for VPD behavior, although other flags may be present if so specified by the storage vendor.
# cat /proc/scsi/device_info | grep EMC | grep Invista 'EMC' 'Invista' 0x240#define BLIST_SPARSELUN 0x040 /* Non consecutive LUN numbering */
define BLIST_LARGELUN 0x200 /* LUNs past 7 on a SCSI-2 device */
- Match VPD group:port key pair to VPD group:port key pair.
When the RTPG command is supported, but a group:port pair match within the RTPG data cannot be found an "alua: No {matching} target port descriptors found" is output into messages file.
File: ./drivers/scsi/device_handler/scsi_dh_alua.c/*
alua_check_vpd - Evaluate INQUIRY vpd page 0x83
@sdev: device to be checked
Extract the relative target port and the target port group
descriptor from the list of identificators.
*/
static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,
int tpgs)
{
int rel_port = -1, group_id;
struct alua_port_group *pg, *old_pg = NULL;group_id = scsi_vpd_tpg_id(sdev, &rel_port); if (group_id < 0) { /* * Internal error; <b>TPGS supported but required</b> * <b>VPD identification descriptors not present.</b> * <b><u>Disable ALUA support</u></b> */ sdev_printk(KERN_INFO, sdev, "%s: <b><font color="red">No target port descriptors found</font></b>\n", <<<<<<<<<<<< here. ALUA_DH_NAME); return SCSI_DH_DEV_UNSUPP; }</code></pre></blockquote>As previously noted above, the SCSI INQUIRY VPD page 83 can have a group:port pair defined within the returned data., In the following case, the group:port pair is 1:4. The only acceptable form of these fields is 'Binary'.
Designation descriptor number 3, descriptor length: 8 designator_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x4 Designation descriptor number 4, descriptor length: 8 designator_type: Target port group, code_set: Binary associated with the target port Target port group: 0x1That same pair of values must exist within the returned data. While the VPD data provides a key data pair associated with that specific path to that specific lun within storage, the RTPG data provides both status and capability of the path group the page:lun belong to. Without the two key data pairs being present in both sets of data, a match cannot be performed and alua for at least this specific path:lun cannot be used.
Again, as we saw within the above section, the returned data from RTPG includes sections like the following. In this case the set of group:port pairs defined are 1:1, 1:2, and 1:3. If this was the only set of data returned, then the group:port pair of 1:3 above is not found and the "No {matching} target port descriptions found" is output -- alua is disabled for that device. There must be a match between the group:port key pair within a path:lun's VPD page 83 and within the set of group:port key pairs returned by RTPG. The configuration data returned by storage must be consistent and complete in order for alua to be usable by multipath for this path:lun device.
target port group id : 0x1 , Pref=0 target port group asymmetric access state : 0x01 (active/non optimized) T_SUP : 0, O_SUP : 0, U_SUP : 1, S_SUP : 0, AN_SUP : 1, AO_SUP : 1 status code : 0x01 (target port asym. state changed by SET TARGET PORT GROUPS command) vendor unique status : 0x00 target port count : 0c Relative target port ids: 0x01 <-- group:port pair = 1:1 0x02 <-- group:port pair = 1:2 0x03 <-- group:port pair = 1:3Note: RTPG data records consist of a "target port group id" and 1 or more "relative target port ids". It is common to have more than one port ids defined within a group.
Note: The VPD page 83 data for this path:lun indicated it was group:port pair 1:4 and that key value pair is not present within the above returned RTPG data. Therefore, alua is not available for this path:lun device.
Root Cause
The LUN can be queried with the
sg_rtpgcommand, available in thesg3_utilspackage.A SCSI "Report Target Port Groups" (RTPG) command is sent to the specified device and the returned data is decoded.
This is the same command/data used within multipath for ascertaining port status within ALUA configurations.
Diagnostic Steps
See Engineering Notes - scsi INQUIRY and REPORT TARGET PORT GROUPS commands for more information.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.