VCS VMwareDisks 7.4.2.3101 Agent enhanced to scan and check VMware VMDK disk visibility with Veritas DMP prior to onlining VCS DiskGroup resource

Problem

The VCS DiskGroup agent is failing to import a Veritas Volume Manager (VxVM) managed disk group for recently attached VMware VMDK disks in a VMware virtual environment.

VMware virtual environments are able to access VMware VMDK (Virtual Machine Disks) and RDM (Raw Device Mapping) disks that reside on a shared datastore. This datastore is accessible from multiple virtual machines.

The virtual disks are only attached to a single virtual machine at a time.

VCS provides the VMwareDisks agent that communicates with the VMware ESX/ESXi hosts to perform the disk detach and attach related operations to move the virtual disk(s) between the VCS clustered virtual machines, 
 

Error Message
 

/var/log/messages

Feb 22 17:53:59 server101 vxvm:vxconfigd: V-5-1-16253 Disk group import of testdg failed with error 150 - No valid disk found containing disk group
Feb 22 17:54:02 server101 vxvm:vxconfigd: V-5-1-16253 Disk group import of testdg failed with error 150 - No valid disk found containing disk group
Feb 22 17:54:02 server101 vxvm:vxconfigd: V-5-1-16253 Disk group import of testdg failed with error 150 - No valid disk found containing disk group
Feb 22 17:54:02 server101 AgentFramework[6796]: VCS ERROR V-16-10031-1503 DiskGroup:testdg:online:** ERROR: vxdg import (force) failed on Disk Group testdg.

 

Cause

The Veritas Volume Manager (VxVM) import operation is being called too early and reports multiple vxdg import failures as the VMwareDisks related tasks have not fully completed.

The required DMPNODEs have not been created when the vxdg import process is started to import the DiskGroup resource.

VxVM and DMP both require additional time to create the corresponding Veritas DMPNODE and Veritas disk access (da) name.
 

disk.EnableUUID


For each Virtual Machine (VM) participating in the cluster, the disk.EnableUUID parameter must be set to "TRUE" for each virtual machine.

The disk.EnableUUID parameter is necessary to ensure the VMDK disk always presents a consistent UUID to all virtual machines.

Follow the steps below from the vSphere client to enable the disk UUID on each Virtual Machine:
 

Enabling disk UUID on virtual machines
 

1. Power off the guest

2. Select the guest and select Edit Settings

3. Select the Options tab on top

4. Select General under the Advanced section

5. Select the Configuration Parameters on right-hand side

6. Check to see if the parameter disk.EnableUUID is set, if it is there then make sure it is set to TRUE
If the parameter is not there, select Add Row and add it

7. Power on the guest

 

Solution

The VCS VMwareDisks agent has been enhanced to scan and check for the VMware VMDK disks recently attached to the Virtual Machine(s), prior to attempting the import operation for the VxVM related disk group.

Sample: /etc/VRTSvcs/conf/config/main.cf

group websg1 (
        SystemList = { server101 = 0, server102 = 1 }
        AutoStartList = { server101 }
        )

        DiskGroup webdg1 (
                DiskGroup = testdg
                )

        IP webip1 (
                Critical = 0
                Device = ens256
                Address = "xxx.xxx.xx.xxx"
                NetMask = "255.255.255.0"
                )

        NIC webnic1 (
                Critical = 0
                Device = ens256
                NetworkHosts = { "xxx.xxx.xx.x" }
                )

        VMwareDisks webVMwaredisk (
                ESXDetails = {
                         "xxx.xxx.xx.xx" = "<username>@vsphere.local=<Encryoted password>" }
                DiskPaths = {
                         "6000C296-eb5f-e7f2-3e4a-8ff7b52db5ec:[datastore-name] server101/server101_1.vmdk" = "1:0",
                         "6000C29e-8b65-611c-0ff9-d639e55d83e5:[datastore-name>] server101/server101_2.vmdk" = "1:1" }
                )

        webdg1 requires webVMwaredisk
        webip1 requires webnic1



Notes:

The sample UDIDs "6000C296-eb5f-e7f2-3e4a-8ff7b52db5ec" and "6000C29e-8b65-611c-0ff9-d639e55d83e5" shown for illustration purposes only.


The ESXDetails attribute contains a list of hostnames and credentials of the ESX hosts or vCenter on which the virtual machines are configured. The key is ESX or vCenter hostname or IP address.

The is format 'User name'='Encrypted password'.

The DiskPaths attributes contains a list of disks paths to be managed by the VMwareDisks resource.

The vmdk format is  'Disk_UUID: [Data store name] Disk_path', followed by 'SCSI controller key: Target ID (unit number of the disk)'

In case of RDM disks, add the RDM prefix to the DIskPaths resource 'RDM:Disk_UUID:[Data store name] Disk_path', followed by 'SCSI controller key: Target ID (unit number of the disk)'.

If the disk UUID is not provided, the agent will discover the UUID and modify the attribute to have UUID of the disk.

VRTSvcsag  Private hotfix Disclaimer

A private hotfix is now available for both RHEL7 and RHEL8 via VRTSvcsag rpm for InfoScale 7.4.2.

vcsag-rhel7_x86_64-HotFix-7.4.2.3101.tar.gz

vcsag-rhel8_x86_64-HotFix-7.4.2.3101.tar.gz

 

Patch ID: 7.4.2.3101

• 4112578 (4113151) VMwareDisksAgent reports resource online before VMware disk to be online is present into vxvm/dmp database.

DETAILS OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
This patch fixes the following incidents:

Patch ID: 7.4.2.3101

• 4112578 (Tracking ID: 4113151)

SYMPTOM:
Dependent DiskGroupAgent fails to get its resource online due to disk group import failure.

DESCRIPTION:
VMwareDisksAgent reports its resource online just after VMware disk is attached to virutal machine, if dependent DiskGroup resource starts to online at the moment it must fail because VMware disk is not yet present into vxdmp database due to VxVM transaction latency. Customer used to add retry times to work around this problem but cannot apply the same to every environment.

RESOLUTION:
Added a finite period of wait for VMware disk is present into vxdmp database before online is complete.

The updated VRTSvcsag 7.4.2.3101 rpm now has improved logging functionality, recording when the required LUN Serial Numbers (LSNs) have been created for the VMDKs attached to the Virtual Machine


Sample Output

# tail -10 /var/VRTSvcs/log/VMwareDisks_A.log
2023/04/19 07:50:55 VCS INFO V-16-10061-22563 VMwareDisks:webVMwaredisk:online:Fetching VMware HA information from IsVMHAEnabled attribute
2023/04/19 07:51:01 VCS INFO V-16-10061-22560 VMwareDisks:webVMwaredisk:online:Disk scan for LUN Serial Number: 6000C296EB5FE7F23E4A8FF7B52DB5EC successful
2023/04/19 07:51:01 VCS INFO V-16-10061-22560 VMwareDisks:webVMwaredisk:online:Disk scan for LUN Serial Number: 6000C29E8B65611C0FF9D639E55D83E5 successful
2023/04/19 07:51:12 VCS INFO V-16-10061-22561 VMwareDisks:webVMwaredisk:online:LUN Serial Number: 6000C296EB5FE7F23E4A8FF7B52DB5EC added into dmp
2023/04/19 07:51:12 VCS INFO V-16-10061-22561 VMwareDisks:webVMwaredisk:online:LUN Serial Number: 6000C29E8B65611C0FF9D639E55D83E5 added into dmp

Please contact Veritas Technical Support to obtain the private hotfix. This hotfix has not yet gone through any extensive Q&A testing.

Consequently, if you are not adversely affected by this problem and have a satisfactory temporary workaround in place, we recommend that you wait for the public release of this hotfix.

Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.