SFHA 5.1 Virtualization (Solaris) IBM SVC managed devices require the vdc_timeout value to be set in the GUEST domain to ensure that I/O service is maintained by DMP when rebooting/restarting the LDOM service domain server

Problem

 
In the event that the SERVICE domain were to panic, has hung, or has been restarted intentionally, the filesystem read/write operations within the GUEST domain may hang in regards to I/O service.


Sample configuration

In this instance, the SERVICE domain configuration consists of two Solaris Logical Domains (LDOMs) for which will be referred to as the "Primary" and "Alternate" domain servers.

Both SERVICE domains each host the virtual disk backend via 2 paths each, the virtual disk backend component is then presented as a virtual disk to the GUEST domain, which is where the data is actually stored.


GUEST domain hosts the "virtual disk"

# echo | format
<snippet of paths to a single virtual disk with 4 paths>
               1. c0d1 <IBM-2145-0000 cyl 10238 alt 2 hd 64 sec 256>
                     /virtual-devices@100/channel-devices@200/disk@1
               2. c0d2 <IBM-2145-0000 cyl 10238 alt 2 hd 64 sec 256>
                     /virtual-devices@100/channel-devices@200/disk@2
               3. c0d3 <IBM-2145-0000 cyl 10238 alt 2 hd 64 sec 256>
                     /virtual-devices@100/channel-devices@200/disk@3
               4. c0d4 <IBM-2145-0000 cyl 10238 alt 2 hd 64 sec 256>
                     /virtual-devices@100/channel-devices@200/disk@4
<end of snippet>


Veritas Volume Manager (VxVM) view of the virtual disk backend content on the "Primary" SERVICE domain

In this instance the "virtual disk" provisioned presents VxVM diskgroup "datadg" is imported in the GUEST domain.

# vxdisk path | grep -w datadg
c0d4s2                      san_vc0_1            disk01       datadg    ENABLED
c0d3s2                      san_vc0_1            disk01       datadg    ENABLED
c0d2s2                      san_vc0_1            disk01       datadg    ENABLED
c0d1s2                      san_vc0_1            disk01       datadg    ENABLED

# vxdmpadm listenclosure all output
ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
=======================================================================================
san_vc0           SAN_VC         020063c081b2XX00     CONNECTED    A/A-A-IBMSVC   1
 

Veritas Volume Manager (VxVM) view of the virtual disk backend content on the "Primary" SERVICE domain

# vxdisk -o alldgs list | grep -w datadg
san_vc0_10   auto:cdsdisk    -            (datadg) online

# vxdisk path | grep -w san_vc0_1               
c2t50050768012042C2d18s2    san_vc0_10     -            -   ENABLED                
c2t50050768012040D9d18s2    san_vc0_1     -            -   ENABLED     

Veritas Volume Manager (VxVM) view of the virtual disk backend content on the "Alternate" SERVICE domain

# vxdisk -o alldgs list | grep -w datadg
san_vc0_10   auto:cdsdisk    -            (datadg) online

# vxdisk path | grep -w san_vc0_1
c0t50050768011042C2d18s2    san_vc0_10     -            -  ENABLED
c0t50050768011040D9d18s2    san_vc0_10     -            -  ENABLED

If the Primary SERVICE domain server were to be rebooted, 2 paths will be lost from the Primary server, leaving 2 remaining paths from the Alternate SERVICE domain server.
As a direct loss of one of the SERVICE domain servers, potentially some of the writes will now hang within the GUEST domain.

Without manual configuration of the virtual disk client (VDC) implementation from the default design, the VDC driver will block I/O. Normal service is automatically resumed when the impacted "SERVICE" returns to a normal operational service.

Error Message

To iosolate the issue a system crash dump is obtained from the Solaris based GUEST domain server, whilst a simple "cp" command line operation is in progress on the GUEST server when the Primary domain server is rebooted.

In this instance, the Solaris crash dump is analyzed using the Solaris SCAT utility.


Sample output

 

Cause

Virtual Disk Timeout

By default, if the "SERVICE" domain providing access to a virtual disk backend is down, all I/O from the guest domain to the corresponding virtual disk is blocked. The I/O automatically is resumed when the "SERVICE" domain is operational and is servicing I/O requests to the virtual disk backend.

However, there are some cases when file systems or applications might not want the I/O operation to block, but for it to fail and report an error if the service domain is down for too long. It is now possible to set a connection timeout period for each virtual disk, which can then be used to establish a connection between the virtual disk client on a guest domain and the virtual disk server on the service domain. When that timeout period is reached, any pending I/O and any new I/O will fail as long as the service domain is down and the connection between the virtual disk client and server is not reestablished.

This timeout can be set by doing one of the following: 

• Using the ldm add-vdisk command.

  ldm add-vdisk timeout=seconds disk-name volume-name@service-name ldom

• Using the ldm set-vdisk command.

  ldm set-vdisk timeout=seconds disk-name ldom

Specify the timeout in seconds. If the timeout is set to 0, the timeout is disabled and I/O is blocked while the service domain is down (this is the default setting and behavior).

Alternatively, the timeout can be set by adding the following line to the /etc/system file on the guest domain.

Reference: http://docs.oracle.com/cd/E19604-01/821-0406/virtualdisktimeout/index.html

Note: If this tunable is set, it overwrites any timeout setting done using the ldm CLI. Also, the tunable sets the timeout for all virtual disks in the guest domain.

Solution

In an effort to resolve the hung state in the GUEST domain environment, the vdc_timeout kernel tunable can be used to define the timeout value for all disks residing in the GUEST domain by adding an entry to the /etc/system file.
 

Note: The vdc_timeout setting equates to the number of seconds before the timeout is reached.

If vdc_timeout is 0 then no timeout is set.
 

  To set the vdc_timeout, add the following line entry in the /etc/system file on the GUEST domain and reboot the GUEST domain for the amendment to take effect:

   set vdc:vdc_timeout=30
 

  In the above instance, the timeout is set to 30 seconds.

 
Once the "vdc_timeout" setting has been defined and the server rebooted, the hung state should no longer occur in the GUEST domain.

Applies To

Array configuration:
The IBM SVC arrays must be configured in ALUA mode as specified in the Hardware article:
000031529

LDOMs
Solaris Logical Domains (LDoms) is a virtualization technology on the Solaris SPARC platform that enables the creation of independent virtual machine environments on the same physical system.

LDoms are a virtualized computing environment abstracted from all physical devices, which allow you to consolidate and centrally manage your workloads on one system.

The logical domains can be specified roles such as a Control domain, Service domain, I/O domain , and Guest domain.
Each domain is a full virtual machine where the operating systems can be started, stopped, and rebooted independently.

https://sort.veritas.com/public/documents/sf/5.1/solaris/pdf/sfha_virtualization.pdf 
 

Figure 1.0

Storage Foundation (SF) stack model with Solaris Logical Domains





Note: The SERVICE domain may be referred to as the CONTROL domain.

VDC design
The default behavior for the virtual disk client (VDC) is that I/O will be blocked in the event of loss of connectivity/standard operational performance to a SERVICE domain. If the VDC timeout property can be set to have the I/O timed out. When this property is not present or set to 0 then there's no timeout, and I/O will not timeout.

A virtual disk contains two components:

The virtual disk itself will appear in the GUEST domain, and the virtual disk backend, which is where the data is stored and where virtual I/O ends up.

The virtual disk backend is exported from a SERVICE domain by the virtual disk server (vds) driver. The vds driver communicates with the virtual disk client (vdc) driver in the GUEST domain through the hypervisor using a logical domain channel (LDC).

A virtual disk appears as /dev/[r]dsk/cXdYsZ devices in the GUEST domain.

 

Note: The virtual disk backend can be a physical disk, a physical disk slice, a file, a volume from a volume management framework, such as the Zettabyte File System (ZFS), Solaris Volume Manager (SVM), Veritas Volume Manager (VxVM), or any disk pseudo device accessible from the service domain.