LDOM: How to provision a disk to a Solaris service I/O domain

Article: 100044071
Last Published: 2021-07-22
Ratings: 7 0
Product(s): InfoScale & Storage Foundation

Description

 

This document outlines how you can provision a LUN to a running Primary (Control) LDOM configuration when the LUN is presented at the storage array layer


This is PHASE #3 of LDOM LUN reconfiguration process


The same process can be performed on an alternate (Service) domain


 

IMPORTANT
 

If the correct steps are not followed, it will be more complex to clean-up the whole stack


Unwanted operational impact may also be experienced, i.e. vxconfigd core dumps or system panics


It is crucial that the Storage Admin inform the sysadmin prior to removing the LUN from the assigned server

This reduces the chances of the cfgadm (Leadville stack) reporting the access path in the unwanted “failing” state
 

KEY POINTS
 

Manual discovery may be required to discover newly presented devices by probing for devices using the cfgadm or fcinfo remote-port interfaces
 

  • VxVM must be informed that the intended disk is being removed to ensure the cfgadm (Leadville stack) correctly reflects the access path state: unusable
     
  • Adding and removing devices must be done at different times, and not under the same reconfiguration event
     
  • Prior to adding or removing devices, the cfgadm -alo show_FCP_dev output should always be checked for existing failing or unusable access path states


It is essential that the /dev/rdsk contents do not contain any stale OS device handles , as the Veritas Device Discovery Layer (DDL) will build its DMP structures based on these device handles

 

If the cfgadm stack is clean of failing and unusable entries, then the OS device handles can be safely removed using:

# devfsadm -Cvc disk
 

The VxVM and DMP structures can then be refreshed using stable OS device tree information by running:


# vxdisk scandisks

# vxddladm assign names
 

 

If I/O is ongoing to a LUN in the same storage enclosure, devices are detected automatically, or if a single LUN is already presented to the host from the same enclosure
 

If this is not the case or there is no traffic or access to the same storage enclosure, then a manual probe will be required

 


MULTI-STEP ADDITION PROCESS:


The presentation of LUNs is also a multi-step process and requires manual intervention to detected newly presented devices
 

1.    In some cases, a manual probe (rescan) of the devices may be required to detect newly provisioned LUNs, which can be achieved by running:
 

# cfgadm -alo show_FCP_dev


It is recommended that the above cfgadm output be checked prior to presenting new LUNs to the system
 

If the output is checked for failing or unusable path states beforehand, the optimized fcinfo remote-port approach can be used, if everything is reported as unknown:


# cat oracle.sh

#! /bin/ksh
CTRLS="c15" ;
for ctrl in $CTRLS ; do
      luxadm -e dump_map /dev/cfg/$ctrl |
      nawk '{if($1~"[0-9]+"){if($0~"Disk"){TPWWN=TPWWN" "$4}else if($0~"Adapter"){HBAPWWN=$4}}}END{print HBAPWWN" "TPWWN}' |
      while read HBAPWWNT TGTPWWNS ; do
            for tgtpwwn in $TGTPWWNS ; do
                  CMD="fcinfo remote-port -ls -p $HBAPWWNT $tgtpwwn" ;
                  echo "Ctrl "$ctrl": "$CMD ;
                  eval $CMD ; done ;
      done ;
done


NOTE: Controller c15 is specified in the script as this related to the controller associated with the EMC_CLARiiON (VNX) enclosure


# vxdmpadm listctlr
CTLR_NAME       ENCLR_TYPE      STATE        ENCLR_NAME      PATH_COUNT
=========================================================================
c4              Disk            ENABLED      disk                 1
c5              Disk            ENABLED      disk                 1
c15             EMC_CLARiiON    ENABLED      emc_clariion0        6
c15             EMC_CLARiiON    ENABLED      emc_clariion1        32


2.    Prior to running the fcinfo command, present the storage device through the required Storage Array interface:


#  /opt/Navisphere/bin/naviseccli -h <IP-ADDRESS> storagegroup -addhlu -gname testgrp -hlu 2 -alu 46



3.    Discover the newly presented devices using the oracle.sh script

In this instance, the CTRLS variable has been defined to scan controller c15

 

# sh -x oracle.sh
+ CTRLS=c15
+ luxadm -e dump_map /dev/cfg/c15
+ read HBAPWWNT TGTPWWNS
+ nawk '{if($1~"[0-9]+"){if($0~"Disk"){TPWWN=TPWWN" "$4}else if($0~"Adapter"){HBAPWWN=$4}}}END{print HBAPWWN" "TPWWN}'
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016c08604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c08604623
Remote Port WWN: 5006016c08604623
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: 6c3f00
        Port Symbolic Name: CLARiiON::::SPB42::FC::::::
        Node WWN: 5006016088604623
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 4
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 256
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d2s2
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d3s2
        LUN: 4
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d4s2
        LUN: 5
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d5s2
        LUN: 6
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d6s2
        LUN: 7
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C08604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016408604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016408604623
Remote Port WWN: 5006016408604623
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: 6c4000
        Port Symbolic Name: CLARiiON::::SPA42::FC::::::
        Node WWN: 5006016088604623
        Link Error Statistics:
                Link Failure Count: 26
                Loss of Sync Count: 21
                Loss of Signal Count: 26
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2040
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d2s2
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d3s2
        LUN: 4
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d4s2
        LUN: 5
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d5s2
        LUN: 6
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d6s2
        LUN: 7
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016408604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016d08604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016d08604623
Remote Port WWN: 5006016d08604623
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: db3400
        Port Symbolic Name: CLARiiON::::SPB43::FC::::::
        Node WWN: 5006016088604623
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 4
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 255
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d2s2
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d3s2
        LUN: 4
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d4s2
        LUN: 5
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d5s2
        LUN: 6
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d6s2
        LUN: 7
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016D08604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016508604623
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016508604623
Remote Port WWN: 5006016508604623
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: db3700
        Port Symbolic Name: CLARiiON::::SPA43::FC::::::
        Node WWN: 5006016088604623
        Link Error Statistics:
                Link Failure Count: 64
                Loss of Sync Count: 1024
                Loss of Signal Count: 7
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 1785
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d2s2
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d3s2
        LUN: 4
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d4s2
        LUN: 5
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d5s2
        LUN: 6
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d6s2
        LUN: 7
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016508604623d7s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
+ fcinfo remote-port -ls -p 10000090fa9b1d49 5006016c096000b4
Remote Port WWN: 5006016c096000b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: db3c00
        Port Symbolic Name: CLARiiON::::SPB42::FC::::::
        Node WWN: 50060160896000b4
        Link Error Statistics:
                Link Failure Count: 59
                Loss of Sync Count: 516
                Loss of Signal Count: 6
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 1020
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C096000B4d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C096000B4d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: Unknown
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t5006016C096000B4d3s2
+ CMD='fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4'
+ echo 'Ctrl c15: fcinfo' remote-port -ls -p 10000090fa9b1d49 50060164096000b4
Ctrl c15: fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
+ eval fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
+ fcinfo remote-port -ls -p 10000090fa9b1d49 50060164096000b4
Remote Port WWN: 50060164096000b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port ID: db3e00
        Port Symbolic Name: CLARiiON::::SPA42::FC::::::
        Node WWN: 50060160896000b4
        Link Error Statistics:
                Link Failure Count: 52
                Loss of Sync Count: 562
                Loss of Signal Count: 10
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 1275
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t50060164096000B4d0s2
        LUN: 1
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t50060164096000B4d1s2
        LUN: 2
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t50060164096000B4d2s2
        LUN: 3
          Vendor: DGC
          Product: VRAID
          OS Device Name: /dev/rdsk/c15t50060164096000B4d3s2
+ read HBAPWWNT TGTPWWNS



4.    The Veritas vxdiskconfig command can used to execute two commands in a single operation:

 

The manual execution of 'devfsadm -c disk' should not be necessary, as the devfsadmd daemon will perform this operation when new devices are detected

Although the devfsadm command will not cause any harm, it may take a whle to complete, especially in larger configurations

NOTE: When removing devices, it is essential that 'devfsadm -Cvc disk' be executed

 


# devfsadm -c disk
# vxdctl enable

5.    The newly presented disk should now be visible to VxVM


# vxdisk list
 

# vxdisk -eo alldgs list | grep _46
emc_clariion0_46 auto:cdsdisk   -            (slidg)     online thinrclm      c15t5006016C096000B4d2 tprclm RAID_6


6.     The Virtual backend devices can now be defined using the DMPNODE device


In this instance, the dmpnode is presented to both the Primary and alternate service I/O domains

 

The device discovery process outlined in steps 1-5 should also be performed on the alternate service I/O domain

 

         # ldm add-vdsdev /dev/vx/dmp/emc_clariion0_46s2 scoobycds@altio-vds0
         # ldm add-vdsdev /dev/vx/dmp/emc_clariion0_46s2 scoobycds@primary-vds0

 

7.    The virtual disk devices can now also be created
 

          # ldm add-vdisk timeout=25 scoobycds-pri scoobycds@primary-vds0 scooby
          # ldm add-vdisk timeout=25 scoobycds-alt scoobycds@altio-vds0 scooby

 

8.     The GUEST domain "scooby" should now be able to see the exported devices, once the devices are discovered within the GUEST domain

 

         # vxdiskconfig


DMPDR TOOL

The process can be automated using the DMPDR tool:


# /usr/lib/vxvm/voladm.d/bin/dmpdr -o refresh
 

 

Was this content helpful?