How to remove access for a path to a LUN - SOLARIS

Description

This document outlines the required multi-step PATH removal guidelines when removing access for a path(s) to a LUN from a Solaris Sparc environment
 

KEY POINTS:

• VxVM must be informed that the intended path(s) are being removed to ensure the cfgadm (Leadville stack) correctly reflects the access path state: unusable
   
• It is vital that sufficient active paths remain available, so the DMPNODE is not impacted
  
  
• Adding and removing device paths must be done at different times, and not under the same reconfiguration event
   
• The cfgadm -alo show_FCP_dev output should always be checked for existing failing or unusable access path states prior to adding or removing device paths

MULTI-STEP REMOVAL PROCESS:

When intentionally removing paths to LUN from a server, VxVM must be informed prior to any storage activity

For illustration purposes, paths from controller c3 associated with Veritas disk access name "emc_clariion0_5" will be removed at the storage layer:

1.       Define the LUN variable, in this instance pointing to Veritas disk access (da) / DMPNODE name "emc_clariiion0_5":
 

LUN="emc_clariion0_5"

2.        Confirm the paths to be removed are associated with the required DMPNODE:

# vxdisk path | egrep "(${LUN})"
c4t500601613EA07595d1s2     emc_clariion0_5      -            -            ENABLED
c3t500601693EA07595d1s2     emc_clariion0_5      -            -            ENABLED
c3t500601613EA07595d1s2     emc_clariion0_5      -            -            ENABLED
c4t500601693EA07595d1s2     emc_clariion0_5      -            -            ENABLED

NOTE: It is vital that sufficient active paths remain available, so the DMPNODE is not impacted

3.       The DMP paths for the Veritas access name should also be disabled using:
 

           # vxdmpadm disable path=<c#t#d#>
 

           In this instance, the paths c3t500601693EA07595d1s2 & c3t500601613EA07595d1s2

           # vxdmpadm disable path=c3t500601693EA07595d1s2

           # vxdmpadm disable path=c3t500601613EA07595d1s2

4.        Confirm the DMP paths associated with controller c3 now reflect the intended DISABLED state:

# vxdisk path | egrep "(${LUN})"
c4t500601613EA07595d1s2     emc_clariion0_5      -            -            ENABLED
c3t500601693EA07595d1s2     emc_clariion0_5      -            -            DISABLED
c3t500601613EA07595d1s2     emc_clariion0_5      -            -            DISABLED
c4t500601693EA07595d1s2     emc_clariion0_5      -            -            ENABLED

5.       Perform Storage array activity to remove LUN from Solaris server

The Peripheral Qualifier/Device Type (PQ) attribute will report a non-zero value when the PATH has been intentionally removed at the Storage Layer:
 

The script "vxdmpadm_remove_list" can be used to expose the Peripheral Qualifier/Device Type (PQ) attribute content:

# ./vxdmpadm_remove_list |  grep $LUN
c3t500601613EA07595d1s2 DISABLED(M) PRIMARY      emc_clariion0_5 emc_clariion0 c3        -      20
c3t500601693EA07595d1s2 DISABLED(M) SECONDARY    emc_clariion0_5 emc_clariion0 c3        -      20
c4t500601613EA07595d1s2 ENABLED(A) PRIMARY      emc_clariion0_5 emc_clariion0 c4        -      0
c4t500601693EA07595d1s2 ENABLED(A) SECONDARY    emc_clariion0_5 emc_clariion0 c4        -      0

SCRIPT

# cat vxdmpadm_remove_list

#! /bin/ksh

details=`vxdmpadm getsubpaths all | egrep -v "NAME|="`

IFS=$'\n'

headerline=`vxdmpadm getsubpaths all | ggrep -a NAME`

append="Peripheral Qualifier/ Device Type"

echo "$headerline     $append"

echo "==================================================================================================================================="

for Line in $details

do

path=`echo $Line | awk '{print $1}'`

append=`/etc/vx/diag.d/vxscsiinq -d $path | ggrep -a Peripheral | awk '{print $5}'`

echo "$Line      $append"

done

OIFS=$IFS
 

# End of Script

6.        Extract the c#::<WWN> details for the disabled paths associated with the DMPNODE(s) defined in the LUN variable earlier:

          # ACCESS=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | grep -i disabled | awk '{ print $1","$2 }' | tr '[:upper:]' '[:lower:]' | tr -s "\n" "|"`

7.        Extract the access path details to specify with the cfgadm unconfigure operation later:

           # CFGADM=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | grep -i disabled | awk '{ print $1 }' | tr '[:upper:]' '[:lower:]'`

8.       Verify the cfgadm -alo show_FCP_dev output reports the intended unusable access path state

           # cfgadm -alo show_FCP_dev

          or   

           # cfgadm -alo show_FCP_dev | egrep "${ACCESS}"

NOTE: If the cfgadm output reports a failing state for any path, something has gone wrong with the process

The luxadm -e offline should be used to correct any access paths reporting a failing state 
 

NOTE: The luxadm interface has been marked as obsolete in Solaris 11.4

The next sequence consists of two commands which perform different actions against the OS devices nodes and /dev links

9.       The OS device node handles (representation of devices in the kernel) must be manually deleted when a LUN is removed from a server

          # cfgadm -c unconfigure -o unusable_SCSI_LUN -c unconfigure c#::<wwn>    
 

          or 
 

          # cfgadm -o unusable_FCP_dev -c unconfigure $CFGADM

 

10.        The “devfsadm –Cvc disk” command should now be used to clean-up any stale /dev links 

           # devfsadm -Cvc disk

11.       Refresh the VxVM device tree using “vxdisk scandisks” command

When the VxVM vxdisk scandisks CLI command is triggered, VxVM will perform a reconfiguration event

• During the reconfiguration process, VxVM will compare the old DMP/VxVM database (which contains stale entries) with the newly discovered database content
  
• If device paths have been removed or added, DMP will act accordingly
  
• A reconfiguration event will then be triggered which will delete (remove) the DMPNODE or the paths associated with the DMPNODE

# vxdisk scandisks

12.       If the vxnotify file contains “regeneration of device names is required” then execute the command “vxddladm assign names”.

          # vxddladm assign names

The reconfiguration events are recorded in the /etc/vx/dmpevents.log, if the vxesd (eventsource) daemon is running

 

CLEAN-UP COMMANDS
 

The following command sequence can be used to clean-up the cfgadm stack when the access paths report the unexpected failing state
 

1.     Define the LUN variable, in this instance pointing to Veritas disk access (da) / DMPNODE name "emc_clariiion0_5":
 

LUN="emc_clariion0_5"

2.    Confirm the OS native (c#t#d#) path(s) to be removed reflect the expected disabled state in the vxdisk path command output for the LUN(s) defined:

# vxdisk path | egrep "(${LUN})"
c4t500601613EA07595d1s2     emc_clariion0_5      -            -            ENABLED
c3t500601693EA07595d1s2     emc_clariion0_5      -            -            DISABLED
c3t500601613EA07595d1s2     emc_clariion0_5      -            -            DISABLED
c4t500601693EA07595d1s2     emc_clariion0_5      -            -            ENABLED

NOTE: Confirm the DMP paths associated with controller c3 reflect the expected DISABLED state

3.       Extract the c#::<WWN> details for the disabled paths associated with the DMPNODE(s) defined in the LUN variable earlier:

          # ACCESS=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | grep -i disabled | awk '{ print $1","$2 }' | tr '[:upper:]' '[:lower:]' | tr -s "\n" "|"`

4.        Extract the access path details to specify with the cfgadm unconfigure operation later:

           # CFGADM=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | grep -i disabled | awk '{ print $1 }' | tr '[:upper:]' '[:lower:]'`

5.    Extract the DMP disabled OS native path (c#t#d#) details from the vxdisk path command output for the required DMPNODE(s):
 

# OFFLINE=`vxdisk path | grep $LUN | grep -i disabled | awk '{ print "luxadm -e offline /dev/rdsk/"$1}'`
 

# echo "$OFFLINE"
luxadm -e offline /dev/rdsk/c3t500601693EA07595d1s2
luxadm -e offline /dev/rdsk/c3t500601613EA07595d1s2

6.    Offline the DMP disabled OS native device paths using luxadm -e offline:

# echo "$OFFLINE" | sh -x
+ luxadm -e offline /dev/rdsk/c3t500601693EA07595d1s2
+ luxadm -e offline /dev/rdsk/c3t500601613EA07595d1s2
 

7.    Verify the cfgadm access paths now report an unusable path state for the disabled DMP path(s) to be removed:
 

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
c3::500601613ea07595,1         disk         connected    configured   unusable
c3::500601693ea07595,1         disk         connected    configured   unusable
 

8.    Unconfigure the intended unusable access paths
 

The OS device node handles (representation of devices in the kernel) must be manually deleted when a LUN is removed from a server

# cfgadm -o unusable_FCP_dev -c unconfigure $CFGADM
 

9.    Verify the access paths have been removed by the previous cfgadm unconfigure operation
 

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"

10.    The “devfsadm –Cvc disk” command should now be used to clean-up any stale /dev links 
 

# devfsadm -Cvc disk
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s0
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s1
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s2
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s3
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s4
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s5
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s6
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601613EA07595d1s7
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s0
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s1
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s2
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s3
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s4
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s5
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s6
devfsadm[6761]: verbose: removing file: /dev/dsk/c3t500601693EA07595d1s7
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s0
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s1
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s2
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s3
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s4
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s5
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s6
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601613EA07595d1s7
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s0
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s1
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s2
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s3
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s4
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s5
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s6
devfsadm[6761]: verbose: removing file: /dev/rdsk/c3t500601693EA07595d1s7



It is essential that the /dev/rdsk contents do not contain any stale OS device handles or /dev links, as the Veritas Device Discovery Layer (DDL) will build its DMP structures based on these device handles

11.    The ‘vxdisk scandisks” command is responsible for destroying the removed DMPNODE
 

# vxdisk scandisks

12.    Refresh the DDL device tree  and /etc/vx/disk.info file contents

# vxddladm assign names

DMPDR TOOL:

The process can be automated using the DMPDR tool:

# /usr/lib/vxvm/voladm.d/bin/dmpdr -o refresh