Description
This document outlines how you can provision a LUN to a running Primary (Control) LDOM configuration when the LUN is presented at the storage array layer
This is PHASE #3 of LDOM LUN reconfiguration process
The same process can be performed on an alternate (Service) domain
IMPORTANT
If the correct steps are not followed, it will be more complex to clean-up the whole stack
Unwanted operational impact may also be experienced, i.e. vxconfigd core dumps or system panics
It is crucial that the Storage Admin inform the sysadmin prior to removing the LUN from the assigned server
This reduces the chances of the cfgadm (Leadville stack) reporting the access path in the unwanted “failing” state
KEY POINTS
Manual discovery may be required to discover newly presented devices by probing for devices using the cfgadm or fcinfo remote-port interfaces
-
VxVM must be informed that the intended disk is being removed to ensure the cfgadm (Leadville stack) correctly reflects the access path state: unusable
-
Adding and removing devices must be done at different times, and not under the same reconfiguration event
-
Prior to adding or removing devices, the cfgadm -alo show_FCP_dev output should always be checked for existing failing or unusable access path states
It is essential that the /dev/rdsk contents do not contain any stale OS device handles , as the Veritas Device Discovery Layer (DDL) will build its DMP structures based on these device handles
If the cfgadm stack is clean of failing and unusable entries, then the OS device handles can be safely removed using:
# devfsadm -Cvc disk
The VxVM and DMP structures can then be refreshed using stable OS device tree information by running:
# vxdisk scandisks
# vxddladm assign names
If I/O is ongoing to a LUN in the same storage enclosure, devices are detected automatically, or if a single LUN is already presented to the host from the same enclosure
If this is not the case or there is no traffic or access to the same storage enclosure, then a manual probe will be required
MULTI-STEP ADDITION PROCESS:
The presentation of LUNs is also a multi-step process and requires manual intervention to detected newly presented devices
1. In some cases, a manual probe (rescan) of the devices may be required to detect newly provisioned LUNs, which can be achieved by running:
# cfgadm -alo show_FCP_dev
It is recommended that the above cfgadm output be checked prior to presenting new LUNs to the system
If the output is checked for failing or unusable path states beforehand, the optimized fcinfo remote-port approach can be used, if everything is reported as unknown:
# cat oracle.sh
#! /bin/ksh
CTRLS="c15" ;
for ctrl in $CTRLS ; do
luxadm -e dump_map /dev/cfg/$ctrl |
nawk '{if($1~"[0-9]+"){if($0~"Disk"){TPWWN=TPWWN" "$4}else if($0~"Adapter"){HBAPWWN=$4}}}END{print HBAPWWN" "TPWWN}' |
while read HBAPWWNT TGTPWWNS ; do
for tgtpwwn in $TGTPWWNS ; do
CMD="fcinfo remote-port -ls -p $HBAPWWNT $tgtpwwn" ;
echo "Ctrl "$ctrl": "$CMD ;
eval $CMD ; done ;
done ;
done
NOTE: Controller c15 is specified in the script as this related to the controller associated with the EMC_CLARiiON (VNX) enclosure
# vxdmpadm listctlr
CTLR_NAME ENCLR_TYPE STATE ENCLR_NAME PATH_COUNT
=========================================================================
c4 Disk ENABLED disk 1
c5 Disk ENABLED disk 1
c15 EMC_CLARiiON ENABLED emc_clariion0 6
c15 EMC_CLARiiON ENABLED emc_clariion1 32
2. Prior to running the fcinfo command, present the storage device through the required Storage Array interface:
# /opt/Navisphere/bin/naviseccli -h <IP-ADDRESS> storagegroup -addhlu -gname testgrp -hlu 2 -alu 46
3. Discover the newly presented devices using the oracle.sh script
In this instance, the CTRLS variable has been defined to scan controller c15
# sh -x oracle.sh
+ CTRLS=c15
+ luxadm -e dump_map /dev/cfg/c15
+ read HBAPWWNT TGTPWWNS
+ nawk '{if($1~"[0-9]+"){if($0~"Disk"){TPWWN=TPWWN" "$4}else if($0~"Adapter"){HBAPWWN=$4}}}END{print HBAPWWN" "TPWWN}'
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016c08604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
Remote Port WWN: 5006016c08604623
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: 6c3f00
Port Symbolic Name: CLARiiON::::SPB42::FC::::::
Node WWN: 5006016088604623
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 4
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 256
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d2s2
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d3s2
LUN: 4
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d4s2
LUN: 5
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d5s2
LUN: 6
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d6s2
LUN: 7
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C08604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016408604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
Remote Port WWN: 5006016408604623
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: 6c4000
Port Symbolic Name: CLARiiON::::SPA42::FC::::::
Node WWN: 5006016088604623
Link Error Statistics:
Link Failure Count: 26
Loss of Sync Count: 21
Loss of Signal Count: 26
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 2040
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d2s2
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d3s2
LUN: 4
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d4s2
LUN: 5
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d5s2
LUN: 6
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d6s2
LUN: 7
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016408604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016d08604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
Remote Port WWN: 5006016d08604623
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: db3400
Port Symbolic Name: CLARiiON::::SPB43::FC::::::
Node WWN: 5006016088604623
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 4
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 255
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d2s2
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d3s2
LUN: 4
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d4s2
LUN: 5
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d5s2
LUN: 6
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d6s2
LUN: 7
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016D08604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016508604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
Remote Port WWN: 5006016508604623
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: db3700
Port Symbolic Name: CLARiiON::::SPA43::FC::::::
Node WWN: 5006016088604623
Link Error Statistics:
Link Failure Count: 64
Loss of Sync Count: 1024
Loss of Signal Count: 7
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 1785
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d2s2
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d3s2
LUN: 4
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d4s2
LUN: 5
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d5s2
LUN: 6
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d6s2
LUN: 7
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016508604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
Remote Port WWN: 5006016c096000b4
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: db3c00
Port Symbolic Name: CLARiiON::::SPB42::FC::::::
Node WWN: 50060160896000b4
Link Error Statistics:
Link Failure Count: 59
Loss of Sync Count: 516
Loss of Signal Count: 6
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 1020
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C096000B4d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C096000B4d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: Unknown
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t5006016C096000B4d3s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 50060164096000b4
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
+ fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
Remote Port WWN: 50060164096000b4
Active FC4 Types: SCSI
SCSI Target: yes
Port ID: db3e00
Port Symbolic Name: CLARiiON::::SPA42::FC::::::
Node WWN: 50060160896000b4
Link Error Statistics:
Link Failure Count: 52
Loss of Sync Count: 562
Loss of Signal Count: 10
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 1275
Invalid CRC Count: 0
LUN: 0
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t50060164096000B4d0s2
LUN: 1
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t50060164096000B4d1s2
LUN: 2
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t50060164096000B4d2s2
LUN: 3
Vendor: DGC
Product: VRAID
OS Device Name: /dev/rdsk/c15t50060164096000B4d3s2
+ read HBAPWWNT TGTPWWNS
4. The Veritas vxdiskconfig command can used to execute two commands in a single operation:
The manual execution of 'devfsadm -c disk' should not be necessary, as the devfsadmd daemon will perform this operation when new devices are detected
Although the devfsadm command will not cause any harm, it may take a whle to complete, especially in larger configurations
NOTE: When removing devices, it is essential that 'devfsadm -Cvc disk' be executed
# devfsadm -c disk
# vxdctl enable
5. The newly presented disk should now be visible to VxVM
# vxdisk list
# vxdisk -eo alldgs list | grep _46
emc_clariion0_46 auto:cdsdisk - (slidg) online thinrclm c15t5006016C096000B4d2 tprclm RAID_6
6. The Virtual backend devices can now be defined using the DMPNODE device
In this instance, the dmpnode is presented to both the Primary and alternate service I/O domains
The device discovery process outlined in steps 1-5 should also be performed on the alternate service I/O domain
# ldm add-vdsdev /dev/vx/dmp/emc_clariion0_46s2 scoobycds@altio-vds0
# ldm add-vdsdev /dev/vx/dmp/emc_clariion0_46s2 scoobycds@primary-vds0
7. The virtual disk devices can now also be created
# ldm add-vdisk timeout=25 scoobycds-pri scoobycds@primary-vds0 scooby
# ldm add-vdisk timeout=25 scoobycds-alt scoobycds@altio-vds0 scooby
8. The GUEST domain "scooby" should now be able to see the exported devices, once the devices are discovered within the GUEST domain
# vxdiskconfig
DMPDR TOOL
The process can be automated using the DMPDR tool:
# /usr/lib/vxvm/voladm.d/bin/dmpdr -o refresh