How to temporarily (unbind) disable HBA (Fibre Channel) ports (controllers) on RedHat for testing DMP & vxattachd interoperability
Description
The following article describes how you can temporarily (unbind) disable HBA (Fibre Channel) ports (controllers) on RedHat for testing Veritas Dynamic Multi-pathing (DMP) behaviour and vxattachd interoperability.
vxattachd is a Volume Manager daemon which interacts with vxnotify to retrieve notification events.
The vxattachd script is located in /opt/VRTS/bin/vxattachd.
vxnotify is a notification vehicle that can retrieve events from vxconfigd. Once the event has been processed by vxnotify, the vxattachd daemon has something to process.
The Veritas Volume Manager (VxVM) vxattachd daemon will online previously offlined disks.
To list available HBA ports, type:
# ls -l /sys/class/fc_host
total 0
lrwxrwxrwx. 1 root root 0 Jun 24 13:52 host28 -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.0/host28/fc_host/host28
lrwxrwxrwx. 1 root root 0 Jun 24 14:15 host29 -> ../../devices/pci0000:80/0000:80:02.0/0000:81:00.1/host29/fc_host/host29
The "lspci" command can be used to display the available Fibre Channel (FC) Adapters:
# lspci | grep Fibre
81:00.0 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)
81:00.1 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)
The Fibre Channel (FC) HBA port access on RedHat can be temporarily disabled using the unbind string against the pci-device-id bus address.
Syntax:
# echo "PCI-DEVICE-ID" > /sys/bus/pci/drivers/qla2xxx/unbind
To disconnect access for a specific PCI device (81:00:0), the command would be as follows:
# echo "0000:81:00.0" > /sys/bus/pci/drivers/qla2xxx/unbind
To display the filename & associated WWN details for all FC HBA controllers:
# awk '{print substr(FILENAME, $0)" \"" $0"\" "}' /sys/class/fc_host/host*/port_name
/sys/class/fc_host/host28/port_name "0x2001000e1ec2f9ca"
/sys/class/fc_host/host29/port_name "0x2001000e1ec2f9cb"
To display the filename & HBA port status:
# awk '{print substr(FILENAME, $0)" \"" $0"\" "}' /sys/class/fc_host/host*/port_state
/sys/class/fc_host/host28/port_state "Online"
/sys/class/fc_host/host29/port_state "Online"
# vxdmpadm getctlr all
LNAME PNAME VENDOR CTLR-ID
=====================================================================================================c0 c0 - -
c11 c11 - -
c28 c28 QLogic Corporation 20:00:00:0e:1e:c2:f9:ca
c29 c29 QLogic Corporation 20:00:00:0e:1e:c2:f9:cb
To disconnect access to specific PCI devices using PCI bus and device driver name
# host_name=`vxdctl list | grep -i hostid: | sed 's/^.*: //' | cut -d "-" -f1`
# echo $host_name
fred
# vxdmpadm getsubpaths | egrep -v "disk|other_disk|${host_name}"
NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-NAME CTLR ATTRS PRIORITY
=================================================================================================
sdd ENABLED(A) - hitachi_vspgx0_01d2 hitachi_vspgx0 c28 - -
sde ENABLED(A) - hitachi_vspgx0_01d3 hitachi_vspgx0 c28 - -
sdf ENABLED(A) - hitachi_vspgx0_01d4 hitachi_vspgx0 c28 - -
sdh ENABLED(A) - hitachi_vspgx0_01d4 hitachi_vspgx0 c28 - -
sdg ENABLED(A) - hitachi_vspgx0_01d5 hitachi_vspgx0 c28 - -
sdi ENABLED(A) - hitachi_vspgx0_01d5 hitachi_vspgx0 c28 - -
sdo ENABLED(A) - hitachi_vspgx1_0150 hitachi_vspgx1 c28 - -
sdz ENABLED(A) - hitachi_vspgx1_0150 hitachi_vspgx1 c29 - -
sdaa ENABLED(A) - hitachi_vspgx1_0151 hitachi_vspgx1 c29 - -
sdp ENABLED(A) - hitachi_vspgx1_0151 hitachi_vspgx1 c28 - -
sdab ENABLED(A) - hitachi_vspgx1_0152 hitachi_vspgx1 c29 - -
sdq ENABLED(A) - hitachi_vspgx1_0152 hitachi_vspgx1 c28 - -
sdac ENABLED(A) - hitachi_vspgx1_0153 hitachi_vspgx1 c29 - -
sdr ENABLED(A) - hitachi_vspgx1_0153 hitachi_vspgx1 c28 - -
sdad ENABLED(A) - hitachi_vspgx1_0154 hitachi_vspgx1 c29 - -
sds ENABLED(A) - hitachi_vspgx1_0154 hitachi_vspgx1 c28 - -
# echo "0000:81:00.0" > /sys/bus/pci/drivers/qla2xxx/unbind
# echo "0000:81:00.0" > /sys/bus/pci/drivers/qla2xxx/unbind
The Veritas daemon "vxattachd" will process the events raised by UDEV to vxesd.
In the latest (enhanced) version of vxattachd, stale DMPNODEs will now be marked for removal (vxdisk rm <disk-access-name>) & deleted by running "vxdisk scandisks" which destroys any STALE (removed) DMPNODE entries.
Once the unbind operations have completed, no SAN attached storage will be connected, until the PCI addresses are bound again using the bind syntax.
Neither Veritas DMP or the OS will report any SAN attached storage:
# vxdmpadm getsubpaths | egrep -v "disk|other_disk|${host_name}"
NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-NAME CTLR ATTRS PRIORITY
=================================================================================================
# systool -vc fc_host | grep Device
Only internal storage will be visible to DMP
# vxdmpadm getsubpaths | egrep "disk|other_disk|${host_name}"
sda ENABLED(A) - fred_disk_0 disk c0 - -
sdb ENABLED(A) - fred_1 disk c0 - -
sdc ENABLED(A) - sdc other_disks c11 - -
The
UDEV framework has also deleted the OS device handles.
# lsscsi
[0:2:0:0] disk DELL PERC H730 Mini 4.29 /dev/sda
[0:2:1:0] disk DELL PERC H730 Mini 4.29 /dev/sdb
[10:0:0:0] cd/dvd HL-DT-ST DVD+-RW GU90N A3B0 /dev/sr0
[11:0:0:0] disk DELL IDSDM 0001 /dev/sdc
# vxdmpadm listctlr all
CTLR_NAME ENCLR_TYPE STATE ENCLR_NAME PATH_COUNT
=========================================================================
c0 Disk ENABLED disk 2
c11 OTHER_DISKS ENABLED other_disks 1
The block devices can be viewed with the corresponding PCI bus address information
# ls -l /dev/disk/by-path/ | grep -v part
total 0
lrwxrwxrwx. 1 root root 9 Jun 22 16:50 pci-0000:00:1a.0-usb-0:1.3:1.0-scsi-0:0:0:0 -> ../../sdc
lrwxrwxrwx. 1 root root 9 Jun 22 16:50 pci-0000:00:1f.2-ata-6.0 -> ../../sr0
lrwxrwxrwx. 1 root root 9 Jun 22 16:50 pci-0000:03:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 9 Jun 22 16:50 pci-0000:03:00.0-scsi-0:2:1:0 -> ../../sdb
How to review what Fibre Channel Controller are available to the host
# lspci | grep Fibre
81:00.0 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)
81:00.1 Fibre Channel: QLogic Corp. ISP8324-based 16Gb Fibre Channel to PCI Express Adapter (rev 02)
To reconnect (bind) access for a specific PCI device via an available PCI bus, type:
In this instance the PCI-address-id is "
81:00.0" as shown above:
# echo "0000:81:00.0" > /sys/bus/pci/drivers/qla2xxx/bind
/var/log/messages
<snip>
Jun 24 13:52:50 fred kernel: qla2xxx [0000:81:00.0]-011c: : MSI-X vector count: 32.Jun 24 13:52:50 fred kernel: qla2xxx [0000:81:00.0]-001d: : Found an ISP2031 irq 76 iobase 0xffffa71041d74000.
Jun 24 13:52:50 fred kernel: qla2xxx [0000:81:00.0]-0075:28: ZIO mode 6 enabled; timer delay (200 us).
Jun 24 13:52:50 fred kernel: qla2xxx [0000:81:00.0]-ffff:28: FC4 priority set to NVMe
Jun 24 13:52:52 fred kernel: scsi host28: qla2xxx
Jun 24 13:52:52 fred kernel: qla2xxx [0000:81:00.0]-00fb:28: QLogic QLE2662 - QLogic QLE2662 Dual Port FC16 HBA.
Jun 24 13:52:52 fred kernel: qla2xxx [0000:81:00.0]-00fc:28: ISP2031: PCIe (8.0GT/s x8) @ 0000:81:00.0 hdma+ host#=28 fw=8.07.11 (d0d5).
Jun 24 13:52:52 fred kernel: qla2xxx [0000:81:00.0]-500a:28: LOOP UP detected (16 Gbps).
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdd(8/48) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sde(8/64) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdg(8/96) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdf(8/80) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdh(8/112) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdj(8/144) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdi(8/128) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdk(8/160) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdm(8/192) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdl(8/176) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdn(8/208) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdp(8/240) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdo(8/224) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdq(65/0) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdr(65/16) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sdt(65/48) is added.
Jun 24 13:52:54 fred vxesd[6142]: vxesd: Device sds(65/32) is added.
Jun 24 13:53:04 fred kernel: VxVM vxdmp V-5-0-34 [Info] added disk array 413010, datype = Hitachi_VSPGx
Jun 24 13:53:04 fred kernel: VxVM vxdmp V-5-0-34 [Info] added disk array 420445, datype = Hitachi_VSPGx
<snip>
The OS will update the /dev/disk/by-path directory contents with the newly discovered sd# devices:# ls -l /dev/disk/by-path/ | grep -v part
total 0
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d204-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d204-lun-1 -> ../../sde
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d204-lun-2 -> ../../sdf
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d204-lun-3 -> ../../sdg
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d205-lun-0 -> ../../sdh
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e801232d205-lun-1 -> ../../sdi
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-1 -> ../../sdk
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-10 -> ../../sdt
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-2 -> ../../sdl
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-3 -> ../../sdm
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-4 -> ../../sdn
lrwxrwxrwx. 1 root root 9 Jun 24 13:52 fc-0x2001000e1ec2f9ca-0x50060e80124fdd01-lun-5 -> ../../sdo
Bind the other HBA as follows:
# echo "0000:81:00.1" > /sys/bus/pci/drivers/qla2xxx/bind
The Veritas "vxnotify" utility can be used to display the event messages that are processed by vxnotify, and to be actioned by vxattachd.
# vxnotify
connected
Reconfiguration in progress
Reconfiguration finished
enabled controller c29 belonging to disk array hitachi_vspgx1
added path sdaa(65/160) belonging to dmpnode hitachi_vspgx1_0151(201/80)
added path sdab(65/176) belonging to dmpnode hitachi_vspgx1_0152(201/112)
added path sdz(65/144) belonging to dmpnode hitachi_vspgx1_0150(201/96)
added path sdac(65/192) belonging to dmpnode hitachi_vspgx1_0153(201/128)
added path sdad(65/208) belonging to dmpnode hitachi_vspgx1_0154(201/144)
Reconfiguration in progress
Reconfiguration finished
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
hitachi_vspgx0_01d2 auto - - offline
hitachi_vspgx0_01d3 auto - - offline
hitachi_vspgx0_01d4 auto - - offline
hitachi_vspgx0_01d5 auto:cdsdisk - (testdg) online thinrclm
hitachi_vspgx1_0150 auto:cdsdisk - (datadg) online thinrclm
hitachi_vspgx1_0151 auto:cdsdisk - (datadg) online thinrclm
hitachi_vspgx1_0152 auto:cdsdisk - (datadg) online thinrclm
hitachi_vspgx1_0153 auto:cdsdisk - (datadg) online thinrclm
hitachi_vspgx1_0154 auto:cdsdisk - (datadg) online thinrclm
fred_disk_0 auto:LVM - - LVM
fred_disk_1 auto:none - - online invalid
sdc auto - - nolabel
NOTES:
The lspci command, can be found in the pciutils rpm package and can be installed by running
# yum install pciutils
The "systool" command is available with the sysfsutils rpm package and can be used to display HBA related details:
# yum install sysfsutils
To display the details for a specific HBA adapter, i.e. host28:
# systool -c fc_host -v -d host28
Class = "fc_host"
Class Device = "host28"
Class Device path = "/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host28/fc_host/host28"
dev_loss_tmo = "30"
fabric_name = "0x100050eb1a9d06f2"
issue_lip = <store method only>
max_npiv_vports = "254"
node_name = "0x2000000e1ec2f9ca"
npiv_vports_inuse = "0"
port_id = "0x051400"
port_name = "0x2001000e1ec2f9ca"
port_state = "Online"
port_type = "NPort (fabric via point-to-point)"
speed = "16 Gbit"
supported_classes = "Class 3"
supported_speeds = "4 Gbit, 8 Gbit, 16 Gbit"
symbolic_name = "QLE2662 FW:v8.07.11 DVR:v10.01.00.22.07.9-k"
system_hostname = ""
tgtid_bind_type = "wwpn (World Wide Port Name)"
uevent =
vport_create = <store method only>
vport_delete = <store method only>
Device = "host28"
Device path = "/sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host28"
fw_dump =
issue_logo = <store method only>
nvram = "ISP "
optrom_ctl = <store method only>
optrom =
reset = <store method only>
sfp = ""
uevent = "DEVTYPE=scsi_host"
vpd = "▒!"
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.