LDOM: How to remove a Virtual disk presented to a Solaris GUEST domain

Description

This document outlines how you can remove a Virtual disk from a running GUEST domain prior to LUN being intentionally removed at the storage array layer

 

This is PHASE #1 of LDOM LUN reconfiguration process

The Virtual Disk Client (VDC) is responsible for managing the disk communication and not the SCSI driver in the LDOM GUEST domains

The cfgadm interface does not exist in the GUEST domain

 

IMPORTANT
 

If the correct steps are not followed, it will be more complex to clean-up the whole stack

Unwanted operational impact may also be experienced, i.e. vxconfigd core dumps or system panics
 

It is crucial that the Storage Admin inform the sysadmin prior to removing the LUN from the assigned server

This reduces the chances of the underlying cfgadm (Leadville stack) reporting the access path in the unwanted “failing” state

 

KEY POINTS
 

•     VxVM must be informed that the intended disk is being removed to ensure the Virtual disk can be removed correctly from the Primary (Control) LDOM domain
   
•     Adding and removing devices must be done at different times, and not under the same reconfiguration event
   
•     The LDOM related disk objects need to be removed using Solaris ldm related command from the Control LDOM domain
   

LUN REMOVAL STEPS
 

From the Primary (Control) LDOM domain:

1.    Establish which devices are exported from the I/O domains to any GUEST domain(s):

In this instance, we are interested in emc_clariion0_46s2 (aka scoobycds)
 

# ldm list -o disk
NAME
primary

VDS
    NAME         VOLUME         OPTIONS          MPGROUP        DEVICE
    primary-vds0 iso_vol                                        /sol-11_3-text-sparc.iso
                 scoobyboot                                     /dev/vx/dmp/emc_clariion0_45s2
                 scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2
                 scooby255                                      /dev/vx/dmp/emc_clariion1_255s2
                 iso_sol10                                      /sol-10-u11-ga-sparc-dvd.iso

------------------------------------------------------------------------------
NAME
altio

VDS
    NAME         VOLUME         OPTIONS          MPGROUP        DEVICE
    altio-vds0   scoobyboot                                     /dev/vx/dmp/emc_clariion0_45s2
                 scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2
                 scooby255                                      /dev/vx/dmp/emc_clariion1_255s2

 

2.    From the Control Domain, Veritas disk access (da) name emc_clariion0_46s2 is visible

# vxdisk -eo alldgs list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
emc_clariion0_44 auto           -            -           nolabel              c15t5006016C096000B4d0s2 tprclm RAID_6
emc_clariion0_45 auto:ZFS       -            -           ZFS                  c15t50060164096000B4d1 tprclm RAID_6
emc_clariion0_46 auto:cdsdisk   -            (cdsdg)     online thinrclm      c15t50060164096000B4d2 tprclm RAID_6
emc_clariion0_47 auto:ZFS       -            -           ZFS                  c15t50060164096000B4d3s2 tprclm RAID_6

3.    Virtual disks may exist are then presented to the GUEST LDOM domain
 

# ldm list -o disk scooby
NAME
scooby

DISK
            NAME         VOLUME                 TOUT ID   DEVICE  SERVER         MPGROUP
            isoscooby    iso_vol@primary-vds0        6    disk@6  primary
            scoobydisk-pri scoobyboot@primary-vds0 10   0    disk@0  primary
            scoobydisk-alt scoobyboot@altio-vds0  10   1    disk@1  altio
            scoobycds-alt scoobycds@altio-vds0   10   2    disk@2  altio
            scoobycds-pri scoobycds@primary-vds0 10   3    disk@3  primary
            scooby255-pri scooby255@primary-vds0 10   4    disk@4  primary
            scooby255-alt scooby255@altio-vds0   10   5    disk@5  altio

From the GUEST domain “scooby”:

The Virtual disk needs to be closed in the GUEST domain, so the Virtual disk object can be removed from the Control domain:

1.    Verify you can see the Virtual disk access (da) name from the GUEST domain:
 

# vxdisk -eo alldgs list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
emc_clariion0_45 auto:ZFS       -            -           ZFS                  c1d0             tprclm RAID_6
emc_clariion0_46 auto:cdsdisk   emc_clariion0_46  cdsdg       online thinrclm      c1d2             tprclm RAID_6
emc_clariion1_255 auto:cdsdisk   emc_clariion1_255  slidg       online thinrclm      c1d5s2           tprclm RAID_6

NOTE: In this instance, we are concerned with emc_clariion0_46
 

2.    Ensure the file system for the related disk group is unmounted:

       # df -F vxfs
       /vol01             (/dev/vx/dsk/slidg/vol01): 2059602 blocks   257447 files
       /vol02             (/dev/vx/dsk/cdsdg/vol02): 2059602 blocks   257447 files

      # umount /vol01

3.    Ensure the required disk group associated with the Virtual disk to be removed has been deported:

       # vxdg list
       NAME         STATE           ID
       cdsdg        enabled,cds          1524639113.8.scooby
       slidg        enabled              1524663399.12.scooby

       # vxdg deport cdsdg
 

4.    Extract the underlying DMP paths for the Virtual disk to be removed:

       # vxdisk path | grep -w emc_clariion0_46
       c1d3                        emc_clariion0_46     -            -            ENABLED
       c1d2                        emc_clariion0_46     -            -            ENABLED
 

5.    The DMP paths for the Veritas access name should also be disabled using:
 

       # vxdmpadm -f disable dmpnodename=emc_clariion0_46

6.    To enable the Virtual disk to be removed from the Control domain, the Virtual disk must be closed at the VxVM layer
 

       # vxdisk rm emc_clariion0_46

From the Primary (Control) LDOM domain again:

1.    Display the Virtual backend device name details to be deleted in relation to emc_clariion0_46s2

# ldm list -o disk primary | grep emc_clariion0_46s2
                 scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2

# ldm list -o disk altio | grep emc_clariion0_46s2
                 scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2
 

2.    Now extract the Virtual disk name details to be deleted:

# ldm list -o disk | grep "scoobycds@"
        scoobycds-pri scoobycds@primary-vds0 10   2    disk@2  primary
        scoobycds-alt scoobycds@altio-vds0   10   3    disk@3  altio

3.    Remove the Virtual disk details
 

       # ldm rm-vdisk scoobycds-pri scooby

       # ldm rm-vdisk scoobycds-alt scooby

 

4.    Verify the Virtual disk objects have been deleted:

       # ldm list -o disk | grep "scoobycds@"

5.    Now remove the Virtual backend device entries related to the previously deleted Virtual disks:

# ldm list -o disk primary | grep -w  scoobycds
                         scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2

# ldm list -o disk altio | grep -w  scoobycds
                         scoobycds                                      /dev/vx/dmp/emc_clariion0_46s2
 

# ldm rm-vdsdev scoobycds@primary-vds0
# ldm rm-vdsdev scoobycds@altio-vds0
 

6.    Confirm the Virtual backend device entries have been deleted

       # ldm list -o disk altio | grep -w  scoobycds
       # ldm list -o disk primary | grep -w  scoobycds

The physical underlying LUNs can now be removed

This process is covered in an additional document.