How to remove/replace an encapsulated Solaris bootdisk (boot device aka rootdisk, rootmirror) controlled by the cfgadm (leadville) stack with Veritas Volume Manager (VxVM)

Description

The following article attempts to explain the process required to replace an encapsulated bootdisk (boot device) controled by the Solaris cfgadm (leadville) stack with Veritas Volume Manager (VxVM) on a Solaris host.
 

1.] Confirm that the system is booted from an alternate disk to the disk to be replaced.

# prtconf -vp | grep boot

NEW to VxVM 6.0

With VxVM 6.0 onwards, it will be possible to display the bootpath (disk the server is actually booted from) using the VxVM vxeeprom CLI command:


Sample output


# vxeeprom bootpath
/pci@1c,600000/scsi@2/disk@1,0:a


This saves the need to run O/S specific commands (SOLARIS SPARC only) such as prtconf.

 
2.] Using vxdiskadm menu option 4 or VxVM CLI command vxdg, mark the disk for replacement:


The following command is equivalent to running "vxdiskadm menu option 4":

# vxdg -g <diskgroup-name> -k rmdisk <disk-media-name>
 

In this instance, the rootmirror (aka c1t1d0s2) will be removed, so the syntax would be:

# vxdg -g rootdg -k rmdisk rootmirror
 

Steps


1.] Perform "vxdiskadm" option 4                        (Remove a disk for replacement)

Volume Manager Support Operations
Menu: VolumeManager/Disk

 1      Add or initialize one or more disks
 2      Encapsulate one or more disks
 3      Remove a disk
 4      Remove a disk for replacement               <<<<<<     Select this option
 5      Replace a failed or removed disk
 6      Mirror volumes on a disk
 7      Move volumes from a disk
 8      Enable access to (import) a disk group
 9      Remove access to (deport) a disk group
 10     Enable (online) a disk device
 11     Disable (offline) a disk device
 12     Mark a disk as a spare for a disk group
 13     Turn off the spare flag on a disk
 14     Unrelocate subdisks back to a disk
 15     Exclude a disk from hot-relocation use
 16     Make a disk available for hot-relocation use
 17     Prevent multipathing/Suppress devices from VxVM's view
 18     Allow multipathing/Unsuppress devices from VxVM's view
 19     List currently suppressed/non-multipathed devices
 20     Change the disk naming scheme
 21     Get the newly connected/zoned disks in VxVM view
 22     Change/Display the default disk layouts
 list   List disk information


 ?      Display help about menu
 ??     Display help about the menuing system
 q      Exit from menus

Select an operation to perform:

 
Note: Once the disk has been marked for replacement, the "vxdiskadm" interface will run "vxdisk rm <da-name>" to close the disk at the VxVM layer.

In this instance the Veritas disk access (da) name is c1t1d0.
 

2.] Prior to removing the faulty disk, ensure that the disk to be replaced is not listed by the VxVM command "vxdisk list".

# vxdisk  list
<snippet>
DEVICE       TYPE           DISK        GROUP        STATUS    OS_NATIVE_NAME   ATTR
c1t0d0s2     auto:sliced    rootdisk     rootdg      online   c1t0d0s2         -
                              c1t1d0s2 rootmirror        removed was: c1t1d0s2     <<<< this is expected, ignore this line entry
<snippet>


If VxVM does report the Veritas disk access (da) name to be replaced, run "vxdisk rm <da-name>" to prepare the disk for removal.


Example


# vxdisk rm c1t1d0s2                                     ( In this instance, it refers to the rootmirror disk )



3.] Using the Solaris luxadm interface, mark all the paths to the disk as offline.
 

# luxadm -e offline /dev/rdsk/c#t#d#s2

ie

# luxadm -e offline /dev/rdsk/c1t1d0s2


Note: In this instance there is only a single path to c1t1d0.


4.] Remove (mask) the bootdisk from the Solaris server/SAN. Do not present the new replacement disk at this time.


5.] In this instance, the Solaris cfgadm (leadville) stack is reporting the access point for the removed LUN (c1t1d0).


Refresh the cfgadm output, by running "cfgadm -alo show_FCP_dev".

# cfgadm -alo show_FCP_dev
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             fc-private   connected    configured   unknown
c1::500000e013f04cb1,0         disk         connected    configured   unknown
c1::5000cca004ac2c85,0         disk         connected    configured   unusable          <<<< reference to c1t0d0


c1t1d0 refers to WWN "5000cca004ac2c85".


# ls -la /dev/rdsk/c1t1d0*
<snippet>
lrwxrwxrwx   1 root     root          74 Jan 26 05:32 /dev/rdsk/c1t1d0s0 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:a,raw
lrwxrwxrwx   1 root     root          74 Jan 26 05:33 /dev/rdsk/c1t1d0s1 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:b,raw
lrwxrwxrwx   1 root     root          74 Jan 26 05:33 /dev/rdsk/c1t1d0s2 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:c,raw
lrwxrwxrwx   1 root     root          74 Jan 26 05:33 /dev/rdsk/c1t1d0s3 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:d,raw
lrwxrwxrwx   1 root     root          74 Jan 26 05:33 /dev/rdsk/c1t1d0s4 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:e,raw
lrwxrwxrwx   1 root     root          74 Jan 26 05:33 /dev/rdsk/c1t1d0s5 -> ../../devices/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w5000cca004ac2c85,0:f,raw
.
<snippet>


6.] Using cfgadm, unconfigure the access point for "5000cca004ac2c85"


# cfgadm -o unusable_FCP_dev -c unconfigure c1::5000cca004ac2c85



7.] Clean-up the stale OS device handles using Solaris command "devfsadm -Cvc disk".

Sample output

# devfsadm -Cvc disk
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/dsk/c1t1d0s7
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s0
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s1
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s2
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s3
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s4
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s5
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s6
devfsadm[5935]: verbose: removing file: /dev/rdsk/c1t1d0s7

8.] Present the replacement disk to the server


9.]  Force Fibre Channel SAN disk rescan using the Solaris "luxadm" interface.


a.] Establish the port reference to be rescanned.


In this instance the output is shown as follows:


# luxadm -e port
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl                     CONNECTED


b.] Force the rescan using the port reference. In this instance, it would be:

# luxadm -e forcelip /devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl


7.] Refresh the cfgadm (leadville) stack


#  cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0                 CD-ROM       connected    configured   unknown
c1                             fc-private   connected    configured   unknown
c1::500000e0139f7621,0         disk         connected    configured   unknown                 <<<< new reference to c1t1d0
c1::500000e013f04cb1,0         disk         connected    configured   unknown
usb0/1                         unknown      empty        unconfigured ok
usb0/2                         unknown      empty        unconfigured ok
usb0/3                         unknown      empty        unconfigured ok
usb0/4                         unknown      empty        unconfigured ok


8.] Refresh Solaris, VxVM and the /etc/vx/disk.info file contents.


# vxdiskconfig

The above command will execute the commands "devfsadm" and "vxdctl enable" as a single transaction.


# ls -la /dev/rdsk/c1t1d0*


The following command will refresh the contents of the /etc/vx/disk.info file.

# vxddladm assign names   ( Introduced in 5.0 MP3 onwards)


9.] Ensure the new disk is presented to VxVM correctly.

# vxdisk  list
<snippet>
DEVICE       TYPE           DISK        GROUP        STATUS    OS_NATIVE_NAME   ATTR
c1t0d0s2     auto:sliced    rootdisk     rootdg      online   c1t0d0s2         -
c1t1d0s2     auto           -            -           nolabel   c1t1d0s2         -   <<< new disk presented to VxVM
<snippet>


Note: Veritas disk access name "c1t1d0s2" contains a single line entry now, therefore it can be safely labelled and initialized for VxVM use.