DELL EMC BCV (Business continuance volume) in NR (Not-Ready) synchronized state can cause vxconfigd to hang with VRTSaslapm 7.4.2.3200

Problem

DELL "EMC" provide Business Continuance Volumes (BCV) which provide a hardware clone image of an existing source EMC Symmetrix/VMAX disk. The BCV can be an independent image (mirror) of standard disk or hardware replicated disk using EMC Symmetrix Remote Data Facility (SRDF).

When EMC BCV disks are placed in a Not-Ready (NR) synchronized state, it can cause vxconfigd to hang when Array Support Library (ASL) VRTSaslapm 7.4.2.3200 is installed on Linux.
 

Error Message
 

Sample output

After the system has booted, the systemctl command shows the following vxvm-boot service errors:

# systemctl status vxvm-boot
● vxvm-boot.service - VERITAS Volume Manager Boot service
   Loaded: loaded (/etc/systemd/system/vxvm-boot.service; enabled; vendor preset: disabled)
   Active: activating (start) since Wed 2023-01-04 13:49:01 IST; 1min 3s ago
  Control: 14861 (vxvm-boot)
    Tasks: 4
   CGroup: /system.slice/vxvm-boot.service
           ├─14861 /bin/sh /etc/vx/vxvm-boot start
           ├─15277 /bin/sh - /etc/vx/vxvm-startup
           ├─15698 vxconfigd -x syslog
           └─15699 vxconfigd -x syslog

Jan 04 13:49:01 fred systemd[1]: Starting VERITAS Volume Manager Boot service...
Jan 04 13:49:01 fred vxvm-boot[14861]: Starting up VxVM:
Jan 04 13:49:01 fred vxvm-boot[14861]: Refreshing key links for APMs
Jan 04 13:49:01 fred vxvm-boot[14861]: VxVM general startup...
Jan 04 13:49:36 fred vxvm:vxconfigd[15699]: V-5-1-11551 array_da_to_disk: Cannot find active path for dmpnode emc0_05b0
Jan 04 13:49:36 fred vxvm:vxconfigd[15699]:

# vxdctl mode

Note: The "vxdctl mode" command hangs.

 

# vxdisk -eo alldgs list | grep "bcv nr"
<snippet>
emc0_05b0    auto           -            -           error                sdtq             bcv nr tprclm
emc0_05b1    auto           -            -           error                sdts             bcv nr tprclm
emc0_05b2    auto           -            -           error                sdtu             bcv nr tprclm
<snippet>
 

The pstack output from the vxconfigd PID (process id) was captured to a file (named stack) and it shows vxconfigd is waiting for IO to complete:
 

# ps -ef | grep vxconfig
root     23647     1  1 19:46 ?        00:00:02 vxconfigd -k -m disable -x syslog
root     23985 23872  0 19:48 pts/3    00:00:00 grep --color=auto vxconfig

# cat stack
[<ffffffffc0eb1013>] dmp_biowait+0xc3/0xf0 [vxdmp]
[<ffffffffc0ec38f8>] dmp_indirect_io+0x538/0x690 [vxdmp]
[<ffffffffc0eb0568>] gendmpioctl+0x3f8/0x950 [vxdmp]
[<ffffffffc0eb0d69>] dmpioctl+0x39/0x80 [vxdmp]
[<ffffffffc0ed801a>] dmp_ioctl+0x3a/0x70 [vxdmp]
[<ffffffffaf767b1a>] blkdev_ioctl+0x28a/0xa20
[<ffffffffaf68e7f1>] block_ioctl+0x41/0x50
[<ffffffffaf6632e0>] do_vfs_ioctl+0x3a0/0x5b0
[<ffffffffaf663591>] SyS_ioctl+0xa1/0xc0
[<ffffffffafb93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff

Cause

The Array Support Library (ASL) interacts with the Device Discovery Layer (DDL) at a user level layer of VxVM. The Array Policy Module (APM) interacts with Veritas Dynamic Multi-pathing (DMP) kernel component of VxVM.

A design change was done to EMC detected devices within the VRTSaslapm to handle the EMC Metro/Geo feature. This resulted in a disk handling change for Not-Ready (NR) devices, causing vxconfigd hangs, boot issues and multiple unexpected DMP related events.

The issue has only been reported on Linux at this time, although other platforms may be impacted.
 

Solution

Veritas engineering has addressed this issue by adapting the VRTSaslapm for EMC BCV devices on Linux platforms only.

A supported VRTSaslapm (7.4.2.3601) hotfix has only been made available for InfoScale 7.4.2.x on Linux platforms: RHEL7, RHEL8, SLES12 and SLES15 only.

Download Link:

VRTSaslapm 7.4.2.3601 is available for download via sort.veritas.com as follows:
https://sort.veritas.com/asl/details/892

Please contact Veritas Technical Support if you have problems downloading the VRTSaslapm  hotfix. 

The issue has not yet been verified on other platforms such as Solaris 11 Sparc or AIX for InfoScale versions 7.4.2 or 8.0.x.

Note: This VRTSaslapm hotfix has not yet gone through any extensive Q&A testing.

Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Veritas plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.
 

Veritas will be releasing a public patch release for this hotfix solution in a future patch release update. The solution will also be ported to InfoScale 8.0.x for all impacted platforms.

Consequently, if you are not adversely affected by this problem and have a satisfactory temporary workaround in place, we recommend that you wait for the next public patch. Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.