LDOM: How to remove a disk from a Solaris service I/O domain

Description
 

This document outlines how you can remove a LUN from a running Primary (Control) LDOM configuration when the LUN is to be intentionally removed at the storage array layer

This is PHASE #2 of LDOM LUN reconfiguration process

The same process can be performed on an alternate (Service) domain

In addition, the manual process of converting the cfgadm failing state for a given access path to the intended unusable path state is also documented

 

IMPORTANT
 

If the correct steps are not followed, it will be more complex to clean-up the whole stack

Unwanted operational impact may also be experienced, i.e. vxconfigd core dumps or system panics

It is crucial that the Storage Admin inform the sysadmin prior to removing the LUN from the assigned server

This reduces the chances of the cfgadm (Leadville stack) reporting the access path in the unwanted “failing” state
 

KEY POINTS
 

•     The Virtual disk and Virtual backend objects must be removed from the Primary (Control) LDOM domain
   
•     VxVM must be informed that the intended disk is being removed to ensure the cfgadm (Leadville stack) correctly reflects the access path state: unusable
   
•     Adding and removing devices must be done at different times, and not under the same reconfiguration event
   
•     Prior to adding or removing devices, the cfgadm -alo show_FCP output should always be checked for existing failing or unusable access path states
  
   

LUN REMOVAL STEPS

When intentionally removing a LUN from a server, VxVM must be informed prior to any storage activity

1.    The following shell variables are defined to make the removal of Veritas disk access name “emc_clariion0_46” is easier as the procedure is performed:

# LUN="emc_clariion0_46"

# ACCESS=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | awk '{ print $1","$2 }' | tr '[:upper:]' '[:lower:]' | tr -s "\n" "|"`

# CFGADM=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | awk '{ print $1 }' | tr '[:upper:]' '[:lower:]'`
 

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
c15::50060164096000b4,2        disk         connected    configured   unknown
c15::5006016c096000b4,2        disk         connected    configured   unknown
 

2.    The following command will inform Veritas Volume Manager (VxVM) and DMP that Veritas disk access name “emc_clariion0_46” is intended to be removed:

The Veritas disk access (da) name should be offlined in the event that the entire LUN is being removed

      # vxdisk offline $LUN

The Veritas disk access name should be marked for removal using:
 

       # vxdisk rm $LUN

The above command informs & prepares VxVM to remove the specified Veritas disk access (da) name from the system
 

NOTE: The above commands are not to be performed if we are only removing paths of a LUN and not the entire LUN

 

Visual Illustration:
 

3.    In addition, the DMP paths for the Veritas access name should also be disabled using:
 

# vxdmpadm -f disable dmpnodename=$LUN

4.    Perform Storage array activity to remove LUN from Solaris server

#  /opt/Navisphere/bin/naviseccli -h <IP-ADDRESS> storagegroup -removehlu -gname testgrp  -hlu 2 -o

5.    Verify the cfgadm -alo show_FCP_dev output reports the intended unusable access path state
 

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
c15::50060164096000b4,2        disk         connected    configured   unusable
c15::5006016c096000b4,2        disk         connected    configured   unusable
 

NOTE: If the cfgadm output reports a failing state for any device, something has gone wrong with the process


The luxadm -e offline should be used to correct any access paths reporting a failing state

The next sequence consists of two commands which perform different actions against the OS devices nodes and /dev links
 

6.     The OS device node handles (representation of devices in the kernel) must be manually deleted when a LUN is removed from a server

# cfgadm -c unconfigure -o unusable_SCSI_LUN -c unconfigure c#::<wwn>  
 

To make things easier, the $CFGADM variable can be used if defined earlier:

# cfgadm -o unusable_FCP_dev -c unconfigure $CFGADM
 

7.    The “devfsadm –Cvc disk” command should now be used to clean-up any stale /dev links 

# devfsadm -Cvc disk
devfsadm[15539]: verbose: instances marked for removal:
devfsadm[15539]: verbose:       /pci@380/pci@1/pci@0/pci@5/SUNW,emlxs@0,1/fp@0,0/ssd@w50060164096000b4,2
devfsadm[15539]: verbose:       /pci@380/pci@1/pci@0/pci@5/SUNW,emlxs@0,1/fp@0,0/ssd@w5006016c096000b4,2
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s0
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s1
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s2
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s3
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s4
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s5
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s6
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s0
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s1
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s2
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s3
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s4
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s5
devfsadm[15539]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s6
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s0
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s1
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s2
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s3
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s4
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s5
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s6
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s0
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s1
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s2
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s3
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s4
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s5
devfsadm[15539]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s6

8.    Refresh the VxVM device tree using “vxdisk scandisks” command

When the VxVM vxdisk scandisks CLI command is triggered, VxVM will perform a reconfiguration event
 

•    During the reconfiguration process, VxVM will compare the old DMP/VxVM database (which contains stale entries) with the newly discovered database content
   
•     If device paths have been removed or added, DMP will act accordingly
   
•     A reconfiguration event will then be triggered which will delete (remove) the DMPNODE or the paths associated with the DMPNODE
   

        # vxdisk scandisks

 

9.    If the vxnotify file contains “regeneration of device names is required” then execute the command “vxddladm assign names”.
 

# vxddladm assign names

The reconfiguration events are recorded in the /etc/vx/dmpevents.log, if the vxesd (eventsource) daemon is running

LUN REMOVAL CORRECTION STEPS
 

The following steps provide a procedure to correct the unintended failing cfgadm state back to the required unusable state
 

In this instance, the required vxdisk rm <disk-access-name> is not executed:

1.    The following shell variables are defined to make the removal of Veritas disk access name “emc_clariion0_46” is easier as the procedure is performed:

# LUN="emc_clariion0_46"

# ACCESS=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | awk '{ print $1","$2 }' | tr '[:upper:]' '[:lower:]' | tr -s "\n" "|"`

# CFGADM=`vxdisk path | sed "s/t/::/g;s/d/ /g;s/s2//g" | egrep "(${LUN})" | awk '{ print $1 }' | tr '[:upper:]' '[:lower:]'`
 

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
c15::50060164096000b4,2        disk         connected    configured   unknown
c15::5006016c096000b4,2        disk         connected    configured   unknown

2.    Perform Storage array activity to remove LUN from Solaris server
 

#  /opt/Navisphere/bin/naviseccli -h <IP-ADDRESS> storagegroup -removehlu -gname testgrp -hlu 2 -o

3.    Verify the cfgadm -alo show_FCP_dev output reports the intended unusable access path state

# cfgadm -alo show_FCP_dev
 

NOTE: If the cfgadm output reports a failing state for any device, something has gone wrong with the process

The luxadm -e offline should be used to correct any access paths reporting a failing state

# cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
c15::50060164096000b4,2        disk         connected    configured   failing
c15::5006016c096000b4,2        disk         connected    configured   failing


4.    The failing cfgadm access paths cannot be unconfigured until the above Veritas disk-access (da) name is closed and corresponding luxadm -e offline commands are executed against the underlying paths

       # vxdisk list | grep 46
       emc_clariion0_46 auto:cdsdisk    -            (slidg)      online thinrclm

       # OFFLINE=`vxdisk path | grep $LUN | awk '{ print "luxadm -e offline /dev/rdsk/"$1}'`

       # vxdisk rm $LUN
 

5.    The cfgadm path state will not reflect the required unusable state until the appropriate luxadm -e offline commands are issued:

        # cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
       c15::50060164096000b4,2        disk         connected    configured   failing
       c15::5006016c096000b4,2        disk         connected    configured   failing

        # echo "$OFFLINE" | sh -x
       + luxadm -e offline /dev/rdsk/c15t5006016C096000B4d2
       + luxadm -e offline /dev/rdsk/c15t50060164096000B4d2
 

       # cfgadm -alo show_FCP_dev | egrep "${ACCESS}"
       c15::50060164096000b4,2        disk         connected    configured   unusable
       c15::5006016c096000b4,2        disk         connected    configured   unusable

6.    Now cfgadm reflects the required unusable state, the paths can be unconfigured:


In this instance the paths on the c15 controller will be unconfigured
 

       # cfgadm -o unusable_FCP_dev -c unconfigure c15::50060164096000b4
       # cfgadm -o unusable_FCP_dev -c unconfigure c15::5006016c096000b4

       Or

       # cfgadm -o unusable_FCP_dev -c unconfigure c15

       Or

       # cfgadm -o unusable_FCP_dev -c unconfigure $CFGADM

7.    Verify the cfgadm access path have been deleted (unconfigured)
 

       # cfgadm -alo show_FCP_dev | egrep "${ACCESS}"

8.    The “devfsadm –Cvc disk” command should now be used to clean-up any stale /dev links 

# devfsadm -Cvc disk
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s0
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s1
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s2
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s3
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s4
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s5
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t50060164096000B4d2s6
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s0
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s1
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s2
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s3
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s4
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s5
devfsadm[13208]: verbose: removing file: /dev/dsk/c15t5006016C096000B4d2s6
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s0
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s1
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s2
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s3
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s4
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s5
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t50060164096000B4d2s6
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s0
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s1
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s2
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s3
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s4
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s5
devfsadm[13208]: verbose: removing file: /dev/rdsk/c15t5006016C096000B4d2s6

9.    Refresh the VxVM device tree using “vxdisk scandisks” command

When the VxVM vxdisk scandisks CLI command is triggered, VxVM will perform a reconfiguration event
 

•     During the reconfiguration process, VxVM will compare the old DMP/VxVM database (which contains stale entries) with the newly discovered database content
   
•     If device paths have been removed or added, DMP will act accordingly
   
•     A reconfiguration event will then be triggered which will delete (remove) the DMPNODE or the paths associated with the DMPNODE
   

The ‘vxdisk scandisks” command is responsible for destroying the removed DMPNODE
 

         # vxdisk scandisks
 

10.    Refresh the DDL device tree and /etc/vx/disk.info file contents

         # vxddladm assign names
 

11.    Confirm the DMPNODE is no longer visible to VxVM

         # vxdisk list
 

12.    If the vxnotify file contains “regeneration of device names is required” then execute the command “vxddladm assign names”.
 

         # vxddladm assign names

The reconfiguration events are recorded in the /etc/vx/dmpevents.log, if the vxesd (eventsource) daemon is running