Description
This document attempts to explain the essential steps required to unpresent(mask) existing storage devices controlled by DMP on Linux.
In this instance, EMC storage is going to be used to illustrate the requirements outlined by Veritas.
The recommended Veritas Volume Manager version’s are:
5.0 MP4
5.1 SP1 RP1 P1 HF3 ( 5.1 SP1 RP2 )
6.0
LUN Removal Pre-Checks
Check the /etc/vx/disk.info file contents.
# grep "0xfff" /etc/vx/disk.info
Example of a LUN that has not been removed correctly, hence the “0xffffffff” entry EMC%5FSYMMETRIX%5F000290300822%5F2200264000 sdaf 0xffffffffffffffff 0x2
emc0_0264 EMC 000290300822
Refresh VxVM & DMP.
Refresh DDL layer and dev_t list.
1.] vxdisk scandisks
2.] vxddladm assign names
3.] grep "0xffff" /etc/vx/disk.info
LUN Removal overview
1.] Filesystem(s) unmounted Disk removed removed diskgroup
2.] vxdisk rm <da-name> vxdisk -e list
3.] Mask LUN from Server Ensure LUN is masked using storage interface
4.] Clean-up O/S handles Unconfigure and delete O/S handles
Final Step: Refresh vxVM and DMP details
In this instance, the EMC storage (SYMDEV) to be removed is 0264.
# vxprint -qhtg datadg
dg datadg default default 22000 1311937139.32.rdgv20z01
dm emc0_0264 emc0_0264 auto 65536 20947216 -
# vxdisk -qe list emc0_0264
emc0_0264 auto:cdsdisk emc0_0264 datadg online sdaf std
# vxdg destroy datadg
# vxdisk -qe list emc0_0264
emc0_0264 auto:cdsdisk - - online sdaf std
Close the EMC storage (SYMDEV) device at the VxVM layer.
# vxdisk rm emc0_0264
# vxdmpadm getsubpaths dmpnodename=emc0_0264
NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS
================================================================================
sdaf ENABLED(A) - c1 EMC emc0 -
sdl ENABLED(A) - c1 EMC emc0 -
# lsscsi | egrep -w '(sdaf|sdl)'
[1:0:4:268] disk EMC SYMMETRIX 5773 /dev/sdl
[1:0:5:249] disk EMC SYMMETRIX 5773 /dev/sdaf
# rpm -qa | grep -i SYM
symcli-symcli-V6.5.1-0
symcli-storbase-V6.5.1-0
symlinks-1.2-24.2.2
symcli-datacore-V6.5.1-0
symcli-storfull-V6.5.1-0
symcli-symrecover-V6.5.1-0
symcli-datastorbase-V6.5.1-0
symcli-srmbase-V6.5.1-0
symcli-core-V6.5.1-0
symcli-star_perl-V6.5.1-0
Veritas Volume Manager 5.0 MP3 introduced the new “-v” option with vxdmpadm:
# vxdmpadm -v getdmpnode dmpnodename=emc0_0264
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME SERIAL-NO ARRAY_VOL_ID
========================================================================================================
emc0_0264 ENABLED EMC 2 2 0 emc0 2200264000 0264
AVID (Array Volume Identifier) with PowerPath (pseudo) names is not supported. We don’t discover AVID content in PowerPath (PP) environments, hence the “vxdmpadm”
CLI related commands will not list the AVID content.
Veritas Volume Manager 5.0 MP3 introduced LUN_COUNT:
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
=======================================================================================
emc0 EMC 000290300822 CONNECTED A/A 19
disk Disk DISKS CONNECTED Disk 1
emc_clariion0 EMC_CLARiiON CK200053300424 CONNECTED CLR-A/PF 1
Note: Veritas disk access name emc0_0264 is associated with enclosure emc0, which relates to Symmetrix ID (SID) “00290300822”, “822” shortened.
EMC storage (SYMDEV) to be removed is 0264 related to SID “822”.
EMC DMX STORAGE OVERVIEW
HBA and WWN content
symmask discover hba refresh the HBA details for the local server
symmask list hba display the WWN and HBA details for the local server
vxdmpadm getctlr all display WWN and HBA details using DMP
vxlist ctlr display WWN and HBA details using DMP
LUN masking commands
Sample syntax shown below for each command:
symmaskdb -sid <SID> list devs –wwn <WWN>
# symmaskdb -sid 822 list devs -wwn 210000e08b1c1df0
symmask –sid <SID> list assign –dev <SYMDEV>
# symmask –sid 822 list assign –dev 0264
symmask –sid <SID> -wwn <WWN> remove devs <SYMDEV> -dir <director> -p <port number>
# symmask -sid 822 -wwn 210000e08b1c1df0 remove devs 0264 -dir 16c -p 0
# symmask -sid 822 -wwn 210000e08b1c1df0 remove devs 0264 -dir 1c -p 1
symmask refresh and symcfg discover# symmask –sid 822 refresh –noprompt
# symcfg –sid 822 discover
SAMPLE SYMMASK HBA OUTPUT
The SYMCLI command “symmask discover hba” refreshes the HBA content:
# symmask discover hba
Symmetrix ID : 000290300822
Device Masking Status : Success
Identifier Type User-generated Name
---------------- ----- ---------------------------------
210000e08b1c1df0 Fibre pluto/210000e08b1c1df0
Symmetrix ID : 000290301414
Device Masking Status : Success
Identifier Type User-generated Name
---------------- ----- ---------------------------------
210000e08b1c1df0 Fibre pluto/210000e08b1c1df0
EMC enclosure emc0, relates to Symmetrix ID (SID) “00290300822”, which is masked to WWN “210000e08b1c1df0”.
LIST WWN CONTENT
The SYMCLI command “symmask list hba” displays the WWN related content:
# symmask list hba
Identifier Type Adapter Physical Device Path Dir:P
---------------- ----- ---------------- ------------------------ -----
210000e08b1c1df0 Fibre 210000e08b1c1df0 /dev/sdj 16C:0
/dev/sdad 16C:1
/dev/sdax 16C:0
/dev/sday 16C:1
# /opt/VRTSsfmh/bin/vxlist ctlr
TY NAME TYPE STATE WWN PNAME
hba c0 FC online - c0
hba c1 FC online 210000E08B1C1DF0 c1
Note: EMC enclosure emc0, relates to Symmetrix ID (SID) “00290300822”, which is masked to WWN “210000E08B1C1DF0”.
LIST LINUX HBA WWN CONTENT
How to determine the HBA WWN related content for a Linux based server using systool :
# systool -c fc_host -v | egrep '(Device|name)'
Class Device = "host1"
Class Device path = "/sys/class/fc_host/host1"
fabric_name = "0x100500021e7c11"
node_name = "0x200000e08b1c1df0"
port_name = "0x210000e08b1c1df0"
symbolic_name = "QLA2340 FW:v3.03.26 DVR:v8.02.00.06.05.03-k"
system_hostname = ""
Device = "host1"
Device path = "/sys/devices/pci0000:00/0000:00:0a.0/0000:02:05.0/host1"
Note: EMC enclosure emc0, relates to Symmetrix ID (SID) “00290300822”, which is masked to WWN “210000e08b1c1df0”.
VIEW SYMDEV ASSIGNMENT PER SID
The SYMCLI command “symmaskdb –sid <SID> list assignment –dev <SYMDEV>” displays the related WWN content for the specified SYMDEV.
# symmaskdb -sid 822 list assignment -dev 0264
Symmetrix ID : 000290300822
Device Identifier Type Dir:P
------ ---------------- ----- ----------------
0264 210000e08b1c1df0 FIBRE FA-16C:0,FA-16C:1
In this instance, the EMC storage (SYMDEV) to be removed is 0264.
EMC enclosure emc0, relates to Symmetrix ID (SID) “00290300822”, which is masked to WWN “210000e08b1c1df0”.
LIST LUN ASSIGNMENT PER WWN
The SYMCLI command “symmaskdb –sid <SID> list devs –wwn <WWN>” displays the related LUN assignment per WWN.
# symmaskdb -sid 822 list devs -wwn 210000e08b1c1df0
Symmetrix ID : 000290300822
Originator Port wwn : 210000e08b1c1df0
User-generated Name : pluto/210000e08b1c1df0
Sym Dev LUN
Name Dir:P Physical Device Name VBUS TID SYMM HOST Attr Cap(MB)
------ ----- ----------------------- ---- --- ---- ---- ---- -------
0263 16C:0 /dev/sdk 0 0 10b b 10312
16C:1 /dev/sdk 0 0 f8 f8 10312
0264 16C:0 /dev/sdl 0 0 10c c 10312
16C:1 /dev/sdl 0 0 f9 f9 10312
0265 16C:0 /dev/sdm 0 0 10d d 10312
16C:1 /dev/sdm 0 0 fa fa 10312
0266 16C:0 /dev/sdn 0 0 10e e 10312
16C:1 /dev/sdn 0 0 fb fb 10312
0267 16C:0 /dev/sdo 0 0 10f f 10312
16C:1 /dev/sdo 0 0 fc fc 10312
0268 16C:0 /dev/sdp 0 0 110 10 10312
16C:1 /dev/sdp 0 0 fd fd 10312
0269 16C:0 /dev/sdq 0 0 111 11 10312
16C:1 /dev/sdq 0 0 fe fe 10312
026A 16C:0 /dev/sdr 0 0 112 12 10312
16C:1 /dev/sdr 0 0 ff ff 10312
026B 16C:0 /dev/sds 0 0 113 13 10312
16C:1 /dev/sds 0 0 100 0 10312
026C 16C:0 /dev/sdt 0 0 114 14 10312
16C:1 /dev/sdt 0 0 101 1 10312
0283 16C:0 /dev/sdu 0 0 115 15 2005
16C:1 /dev/sdu 0 0 102 2 2005
0284 16C:0 /dev/sdv 0 0 116 16 2005
16C:1 /dev/sdv 0 0 103 3 2005
0285 16C:0 /dev/sdw 0 0 117 17 2005
16C:1 /dev/sdw 0 0 104 4 2005
0287 16C:0 /dev/sdx 0 0 119 19 2005
16C:1 /dev/sdx 0 0 106 6 2005
0288 16C:0 /dev/sdy 0 0 11a 1a 2005
16C:1 /dev/sdy 0 0 107 7 2005
0289 16C:0 /dev/sdz 0 0 11b 1b 2005
16C:1 /dev/sdz 0 0 108 8 2005
028A 16C:0 /dev/sdaa 0 0 11c 1c 2005
16C:1 /dev/sdaa 0 0 109 9 2005
028B 16C:0 /dev/sdab 0 0 11d 1d 2005
16C:1 /dev/sdab 0 0 10a a 2005
028C 16C:0 /dev/sdac 0 0 11e 1e 2005
16C:1 /dev/sdac 0 0 10b b 2005
REMOVE LUN ASSIGNMENT FOR WWN
The SYMCLI command “symmask –sid <SID> -wwn <WWN> remove devs <SYMDEV> -dir <director> -p <port number>” is used to mask the LUN from the specific WWN.
# symmask -sid 822 -wwn 210000e08b1c1df0 remove devs 0264 -dir 16C -p 0
# symmask -sid 822 -wwn 210000e08b1c1df0 remove devs 0264 -dir 16C -p 1
REFRESH EMC DMX DATABASE
The SYMCLI commands “symmask –sid <SID> refresh” and “symcfg –sid 822 discover” update the Symmaskdb database content.
# symmask -sid 822 refresh -noprompt
Symmetrix FA directors updated with contents of SymMask Database 000290300822
# symcfg -sid 822 discover
Attempting discovery of Symmetrix 000290300822
This operation may take up to a few minutes. Please be patient...
Optional: Confirm the EMC SYMDEV “0264” masking has been removed.
# symmaskdb -sid 822 list assignment -dev 0264
No device masking database records could be found for the specified input parameters
# symmaskdb -sid 822 list devs -wwn 210000e08b1c1df0
Symmetrix ID : 000290300822
Originator Port wwn : 210000e08b1c1df0
User-generated Name : rdgv20z01/210000e08b1c1df0
Sym Dev LUN
Name Dir:P Physical Device Name VBUS TID SYMM HOST Attr Cap(MB)
------ ----- ----------------------- ---- --- ---- ---- ---- -------
0263 16C:0 /dev/sdk 0 0 10b b 10312
16C:1 /dev/sdk 0 0 f8 f8 10312
CLEAN-UP OS DEVICE HANDLES
Various utilities can be used to display the current Linux OS device handle content. In this instance, the “lsscsi” utility will be used:
# vxdmpadm getsubpaths dmpnodename=emc0_0264
NAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS
================================================================================
sdaf DISABLED - c1 EMC emc0 -
sdl DISABLED - c1 EMC emc0 -
# lsscsi | egrep -w '(sdaf|sdl)'
[1:0:4:268] disk EMC SYMMETRIX 5773 /dev/sdl
[1:0:5:249] disk EMC SYMMETRIX 5773 /dev/sdaf
DELETE OS DEVICE HANDLE
The OS device handle for the removed lun, can be deleted using the echo string format shown below:
# echo 1 > /sys/block/<sd#-name>/device/delete
# lsscsi | grep -w <sd#-name>
Example:
# echo 1 > /sys/block/sdl/device/delete
# echo 1 > /sys/block/sdaf/device/delete
# lsscsi | egrep -w '(sdaf|sdl)’
Note: The OS device handle(s) should no longer be reported by “lsscsi”.
Refresh controller content
# vxdmpadm getctlr all
LNAME PNAME VENDOR CTLR-ID
=============================================================================================
c1 c1 Qlogic-Corp. 21:00:00:e0:8b:1c:1d:f0
c0 c0 - -
# systool -c fc_host -v | egrep '(Device|name)'
Class Device = "host1"
Class Device path = "/sys/class/fc_host/host1"
fabric_name = "0x100500021e7c11"
node_name = "0x200000e08b1c1df0"
port_name = "0x210000e08b1c1df0"
symbolic_name = "QLA2340 FW:v3.03.26 DVR:v8.02.00.06.05.03-k"
system_hostname = ""
Device = "host1"
Device path = "/sys/devices/pci0000:00/0000:00:0a.0/0000:02:05.0/host1"
# echo "- - -" > /sys/class/scsi_host/host1/scan
# lsscsi | egrep -w '(sdaf|sdl)'
REFRESH VxVM & DMP
The “0xffffffff” /etc/vx/disk.info file entry denotes that the LUN has been marked for LUN removal. DMP will not expect the LUN back.
# vxdisk scandisks
# grep 0xfff /etc/vx/disk.info
EMC%5FSYMMETRIX%5F000290300822%5F2200264000 sdaf 0xffffffffffffffff 0x2 emc0_0264 EMC 000290300822
# vxddladm assign names (refreshes the contents of the /etc/vx/disk.info file )
# grep 0xfff /etc/vx/disk.info
# vxdmpadm getsubpaths dmpnodename=emc0_0264
VxVM vxdmpadm ERROR V-5-1-2268 emc0_0264 is not a valid dmp node name
VERIFY ENCLOSURE LUN_COUNT
The emc0 enclosure LUN_COUNT was 19 prior to masking the LUN from the server:
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
=======================================================================================
emc0 EMC 000290300822 CONNECTED A/A 19
disk Disk DISKS CONNECTED Disk 1
emc_clariion0 EMC_CLARiiON CK200053300424 CONNECTED CLR-A/PF 1
Revised LUN_COUNT is now 18 following the removal process:
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
=======================================================================================
emc0 EMC 000290300822 CONNECTED A/A 18
disk Disk DISKS CONNECTED Disk 1
emc_clariion0 EMC_CLARiiON CK200053300424 CONNECTED CLR-A/PF 1
Process complete.
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.