How do you map between scsi address <0 0 0 0> (or 0:0:0:0) and scsi device name (sda)?
Environment
- Red Hat Enterprise Linux 5
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
Issue
- How do you interpret 0:0:0:0?
- What is 0:0:0:0 and what does it mean?
- What does <0 0 0 0> mean with relationship to an sdN scsi disk?
- How do I backmap the 0:0:0:0 from error or event messages to an sd device?
- How do I map a scsi hctl address to a scsi device name (hctl -> sd)?
- How do I map a scsi device name to a scsi hctl address (sd -> hctl)?
- How do I map an sd device to 0:0:0:0 type reference?
- How to backmap from hbtl->sd device?
Resolution
- what is <0 0 0 0>, 0:0:0:0, hctl, h:c:t:l, hbtl, and h:b:t:l?
- how to map from sdN scsi device to hctl scsi address?
- how to map from hctl scsi address to sdN scsi device?
Note: the sg_map command is from sg3_utils package. The lsscsi command from the lsscsi package can also be used and provides similar output/data.
1. map hctl to sd deviceRHEL6,7::
$ ls -d /sys/bus/scsi/devices/4:0:0:0/block/sd* /sys/bus/scsi/devices/4:0:0:0/block/sdb-OR-
$ /usr/bin/sg_map -x /dev/sg0 3 0 0 0 5 /dev/scd0
/dev/sg1 3 0 1 0 0 /dev/sda
/dev/sg2 4 0 0 0 0 /dev/sdb-OR-
$ /usr/bin/lsscsi -g [3:0:0:0] storage HP HSV300 1000 - /dev/sg0
[3:0:0:1] disk HP HSV300 1000 /dev/sda /dev/sg1
[4:0:0:0] disk HP HSV300 1000 /dev/sdb /dev/sg2
RHEL5:
2. map sd to hctl$ ls -l /sys/bus/scsi/devices/0:0:0:0/block:* lrwxrwxrwx 1 root root 0 Jul 16 11:16 /sys/bus/scsi/devices/0:0:0:0/block:sda -> ../../../../../../block/sda-OR-
$ /usr/bin/sg_map -x /dev/sg0 0 0 0 0 0 /dev/sda /dev/sg1 2 0 0 0 5 /dev/scd0
/dev/sg2 10 0 0 0 0 /dev/sdb
RHEL6,7:
$ ls -l /sys/block/sdb/device/scsi_device/ total 0
drwxr-xr-x. 3 root root 0 Jul 16 11:14 4:0:0:0-OR-
$ /usr/bin/sg_map -x /dev/sg0 3 0 0 0 5 /dev/scd0
/dev/sg1 3 0 1 0 0 /dev/sda
/dev/sg2 4 0 0 0 0 /dev/sdb
RHEL5:
3. What is the 0:0:0:0 or <0 0 0 0> reference in error messages like:$ ls -l /sys/block/sda/device lrwxrwxrwx 1 root root 0 Jul 16 11:16 /sys/block/sda/device -> ../../devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0-OR-
$ /usr/bin/sg_map -x /dev/sg0 0 0 0 0 0 /dev/sda /dev/sg1 2 0 0 0 5 /dev/scd0
/dev/sg2 10 0 0 0 0 /dev/sdb
kernel: SCSI error : <2 0 5 0> return code = 0x20000 kernel: sd 1:0:0:14: SCSI error: return code = 0x08000002 kernel: SCSI disk error : host 0 channel 0 id 0 lun 0 return code = 25040001
Those numbers all reference a specific scsi device via a scsi address. This scsi "address" defines the route to the device via:
<h c t l> or sometimes <h b t l>. I think hctl is less confusing as 'b' or bus is more overloaded than the 'c' channel. H C T L <0 0 0 0> 0:0:0:0 | | | | | | | +---- 'l' : lun number | | | | | +------ 't' : target number | | | +-------- 'c' : channel number -- or | 'b' : bus number | +---------- 'h' : host adapter number
Physical Route
H C T L
[ host adapter ] [channel] +--- bus transport/interconnect -+----+----+---------->
| | |
| | :
| | .
| |
| +--------------[ target ] --+--+--+--+-->
| | | | | :
| | | | +---> [lun 3] [B]
| | | +------> [lun 2]
| | +---------> [lun 1]
| +------------> [lun 0]
| +------------> [lun 0]
|
+-------------------[ target ] --+--+--+---->
| | | :
| | +------> [lun 2] [A]
| +---------> [lun 1]
+------------> [lun 0]
Lets assume that the above host adapter is the 4th scsi adapter discovered in the system, so is scsi3. The adapter doesn't support multiple channels, so the channel number is 0, and the two targets shown are the first and second targets discovered on the bus/interconnect connected to this adapter.
In the above, the scsi route or address to [A] is host 3, channel 0, target 0, lun 2 - 3:0:0:2.
The route or scsi route to, or address of, [B] is host 3, channel 0, target 1, lun 3 - 3:0:1:3.
Just as scsi host numbers are assigned sequentially in as discovered order, the sdNN names are also assigned to devices in as discovered order. Lets assume for the moment that the 3:0:0:0 - 3:0:0:2 [A] devices were names sdg, sdh, and sdi. You can see the list of LUNs behind a given target by using the sg_luns command against any of the luns behind a given target. The sg_luns command sends a scsi REPORT LUNS command to the device, although the target is the one that answers the
command, not the lun itself.
NOTE: The sg_luns and sg_map commands are from the optional sg3utils package.
$ sg_luns -d /dev/sdg
Lun list length = 96 which imples 12 lun entry
Report luns [select_report=0]:
0000000000000000
Peripheral device addressing: lun=0
0001000000000000
Peripheral device addressing: lun=1
0002000000000000
Peripheral device addressing: lun=2
0003000000000000
Peripheral device addressing: lun=3
0004000000000000
Peripheral device addressing: lun=4
0005000000000000
Peripheral device addressing: lun=5
0006000000000000
Peripheral device addressing: lun=6
0007000000000000
Peripheral device addressing: lun=7
0008000000000000
Peripheral device addressing: lun=8
0009000000000000
Peripheral device addressing: lun=9
000A000000000000
Peripheral device addressing: lun=10
000B000000000000
Peripheral device addressing: lun=11
Performing the same command on sdh and sdi should by scsi definition result in the same data given that the target
is the same in all three cases.
sg_luns /dev/sdh
Lun list length = 96 which imples 12 lun entry
Report luns [select_report=0]:
0000000000000000
Peripheral device addressing: lun=0
0001000000000000
Peripheral device addressing: lun=1
0002000000000000
Peripheral device addressing: lun=2
0003000000000000
Peripheral device addressing: lun=3
0004000000000000
Peripheral device addressing: lun=4
0005000000000000
Peripheral device addressing: lun=5
0006000000000000
Peripheral device addressing: lun=6
0007000000000000
Peripheral device addressing: lun=7
0008000000000000
Peripheral device addressing: lun=8
0009000000000000
Peripheral device addressing: lun=9
000A000000000000
Peripheral device addressing: lun=10
000B000000000000
Peripheral device addressing: lun=11
Host (Adapter) Number As scsi host bus adaters (HBAs) are discovered, they are assigned a the next available scsi host number available. So the first scsi adapter discovered becomes scsi host 0 (scsi0). Sometimes this is reference as the scsi bus number -- which is confusing because in hctl nomenclature the scsi0 is the host adapter number, not the bus number.
For example:
Jun 1 08:08:22 de0119006lb006 kernel: scsi0 : ioc0: LSISAS1064, FwRev=011a5a00h, Ports=1, MaxQ=478, IRQ=50
Jun 1 08:08:22 de0119006lb006 kernel: scsi1 : qla2xxx
Jun 1 08:08:23 de0119006lb006 kernel: scsi2 : qla2xxx
Jun 1 08:08:23 de0119006lb006 kernel: scsi3 : qla2xxx
Jun 1 08:08:23 de0119006lb006 kernel: scsi4 : qla2xxx
Within the lspci output, we see dual-channel PCI QLogic cards:
03:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
03:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
04:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
04:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
The .0/.1 is the function address on the card. So 03:00.* refers to the same physical card and the trailing .0 and .1 to separate pci functions on that card (aka dual-channels or dual-ports). Just note that the product description of the card as being dual-channel has nothing to do with the channel nomenclature within linux hctl -- it just happens to be the same word. The overloaded use of the same term could be confusing.
Channel (bus) number As scsi channels that are attached to the scsi host are discovered, they are assigned a the next available scsi channel number available for that host. So the first scsi channel discovered becomes scsi channel 0 on the given host.
The preference is to reference this as channel number as its less confusing than using the alternate of bus in hbtl -- bus is often used to reference the physical interconnect and can lead to confusion with cards that support more than one channel (although most scsi adapters support just 1 channel per host instance) as to which you are referencing. Moreover, sometimes folks refer to scsi0 as being scsi bus 0 adding to the confusion surrounding hctl nomenclature. But the word 'channel' although also overloaded in that we have dual-channel fibre channel adapters too, is less confusing. In the context of dual-channel FC adapter, the 'channel' refers to having two separate scsi hosts on the same pci card. Sometimes a dual-channel fibre channel card is referred to as a dual-port card because it has two FC ports.
Many (most?) scsi host bus adapters have a 1:1 relationship between scsi host and channel -- that is there is only one channel per scsi host supported by the hardware. That is why its most common to see the 'c' field in hctl with a zero value.
As a counter example, here's an LSI card having multiple channels per scsi host, in that the assigned channel number is '1' in this case for devices sda and sdb.
Jan 31 08:52:18 kernel: scsi0 : ioc0: LSISAS1068E B3, FwRev=00192f00h, Ports=1, MaxQ=266, IRQ= Jun 1 08:08:22 kernel: sd 0:1:0:0: Attached scsi disk sda Jan 31 08:52:19 kernel: sd 0:1:2:0: Attached scsi disk sdb Jan 31 08:52:19 kernel: scsi1 : ioc1: LSI53C1030 C0, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=1 Jan 31 08:52:19 kernel: sd 1:0:0:0: Attached scsi disk sdc
Note that
- sda is scsi0, channel 1, target 0, lun 0 (0:1:0:0).
- sdb is scsi0, channel 1, target 1, lun 0 (0:1:1:0).
- sdc is scsi1, channel 0, target 0, lun 0 (1:0:0:0) - different scsi adapter.
Target Number As scsi targets that are attached to the scsi host are discovered, they are assigned the next available scsi target number available for that host. So the first scsi target discovered becomes scsi target 0 on the given host. On fibre channel, scsi targets are near enough synonomous with storage controller for many storage boxes.
Within fibre channel, the available targets are found be either interogating the fibre channel switch (fabric) or polling loop ids (loop). Similarly, on parallel scsi buses targets are found by polling a range of target values to see if any devices respond.
In the following example, there are two PCI Qlogic dual-channel cards which result in four scsi hosts being discovered. There are two targets detected attached to each host. Behind each target are multiple logical units (disks).
Jun 1 08:08:22 de0119006lb006 kernel: scsi1 : qla2xxx Jun 1 08:08:23 de0119006lb006 kernel: scsi2 : qla2xxx Jun 1 08:08:23 de0119006lb006 kernel: scsi3 : qla2xxx Jun 1 08:08:23 de0119006lb006 kernel: scsi4 : qla2xxx Jun 1 08:08:24 de0119006lb006 kernel: sd 1:0:0:0: Attached scsi disk sdb Jun 1 08:08:24 de0119006lb006 kernel: sd 1:0:0:1: Attached scsi disk sdc Jun 1 08:08:24 de0119006lb006 kernel: sd 1:0:0:2: Attached scsi disk sdd : . Jun 1 08:08:45 de0119006lb006 kernel: sd 1:0:1>:12: Attached scsi disk sdaz Jun 1 08:08:45 de0119006lb006 kernel: sd 1:0:1:13: Attached scsi disk sdba Jun 1 08:08:46 de0119006lb006 kernel: sd 1:0:1:14: Attached scsi disk sdbb : . etc.
Logical Unit Numbers (LUNs) There can be multiple LUNs (disks for instance) behind each scsi target. To determine which luns exist, the scsi REPORT LUNS command is used. The lun 0 must exists per scsi specification so the REPORT LUNS is sent to that lun number behind discovered targets. In the rare case that the device does not support REPORT LUNS, the kernel is capable of polling a range of lun numbers to see if any devices respond.
For example here is a snippet of some luns that were found behind a given target:
kernel: sd 3:0:1:32: Attached scsi disk sdep kernel: sd 3:0:1:33: Attached scsi disk sdeq kernel: sd 3:0:1:34: Attached scsi disk sder kernel: sd 3:0:1:35: Attached scsi disk sdes kernel: sd 3:0:1:36: Attached scsi disk sdet kernel: sd 3:0:1:37: Attached scsi disk sdeu kernel: sd 3:0:1:38: Attached scsi disk sdev kernel: sd 3:0:1:39: Attached scsi disk sdew
Diagnostic Steps
For example, if I've received errors on device sdcc and wish to issue the scsi remove command as part of replacing it:
$ echo "scsi remove-single-device h c t l" > /proc/scsi/scsi
But I don't know the hctl scsi address associated with disk sdcc, then I can use the sg_map command to find that information:
$ sg_map -x | grep sdcc /dev/sg62 4 0 1 20 0 /dev/sdcc $ echo "scsi remove-single-device 4 0 1 20" > /proc/scsi/scsi
In the above example, the device could also have been deleted via:
- echo 1 > /sys/block/sdcc/device/delete, or
- echo 1 > /sys/class/scsi_device/4:0:1:20/device/delete, or
- echo 1 > /sys/class/scsi-generic/sg62/device/delete
Any and all of the above result in the device being deleted. See How do I rescan the SCSI bus to add or remove a SCSI device without rebooting the computer? for additional information and examples for removing devices from the configuration.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.