NetBackup for VMware backups fail with Status Code 12 and "Error 18000 (listener error GVmomiFaultInvalidResponse)"

Problem

Backups of multiple virtual machine intermittently failing with error 12 (file open failed) .

NetBackup version 8.1.2

The errors in vxms provider are similar to the ones reported in

VMware Backups may fail with "Error 18000 (listener error GVmomiFaultInvalidResponse)" 

When many VMs are backed up concurrently, VDDK can crash occasionally.

When many virtual machines are backed up in parallel, VixDiskLib might crash. The issue is not consistently reproducible, but was determined to be a problem with null entries in libgvmomi. This issue has been fixed in this VDDK version 6.0.2

Reference : Virtual Disk Development Kit 6.0.2 Release Notes

However NetBackup 8.1.2 has VDDK 6.5.3

Error Message

VXMS Provider logs:-

01/22/2019 11:04:11 : g_vixInterfaceLogger:libvix.cpp:1805 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: VixDiskLibVim_FreeNfcTicket: Free NFC ticket. 
01/22/2019 11:04:11 : g_vixInterfaceLogger:libvix.cpp:1805 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: Get NFC ticket completed. 
01/22/2019 11:04:11 : g_vixInterfaceLogger:libvix.cpp:1805 <DEBUG> : [VFM_ESINFO] VixDiskLib: Error occurred when obtaining NFC ticket for [xxx__vSAN_01] xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/test1_1-000003.vmdk. Error 18000 (Cannot connect to the host) (fault (null), type GVmomiFaultInvalidResponse, reason: (none given), translated to 18000) at 2220. 
01/22/2019 11:04:11 : g_vixInterfaceLogger:libvix.cpp:1805 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_OpenEx: Cannot open disk [xxxx_vSAN_01] xxxx-xxxx-xxxx-xxxx-xxxxxxx/test1_1-000003.vmdk. Error 18000 (Cannot connect to the host) at 5021. 
01/22/2019 11:04:11 : g_vixInterfaceLogger:libvix.cpp:1805 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_Open: Cannot open disk [xxxx_xxxx_xxxx_vSAN_01] xxxxx-xxxx-xxxx-xxxx-xxxxxxx/test1_1-000003.vmdk. Error 18000 (Cannot connect to the host) at 5059. 
01/22/2019 11:04:11 : vdOpen:VixInterface.cpp:462 <DEBUG> : Done with Open(): 0 
01/22/2019 11:04:11 : vdOpen:VixInterface.cpp:486 <WARN> : Open() error: 18000 
01/22/2019 11:04:11 : openVixDiskImage:VixGuest.cpp:3852 <ERROR> : vdOpen() for [[xxxx_xxxx_xxxx_vSAN_01] xxxxxx-xxxx-xxxx-xxxx-xxxxxxx/test1_1-000003.vmdk] returned error code [18000] 
01/22/2019 11:04:11 : openFileName:VixGuest.cpp:3762 <ERROR> : Exception: VERIFY_VXMS 
01/22/2019 11:04:11 : log:Error.cpp:249 <WARN> : Error: 0x00000019 occured in file ../VixGuest.cpp, at line 3762 
01/22/2019 11:04:11 : log:Error.cpp:249 <WARN> : Error: 0x00000019 occured in file ../VixGuest.cpp, at line 3762 
01/22/2019 11:04:11 : openFileName:VixGuest.cpp:3804 <WARN> : Returning: 25 
01/22/2019 11:04:11 : vixMapOpenFileName:VixCoordinator.cpp:1105 <TRACE> : rflags returned: 0xdff7fb2c 
01/22/2019 11:04:11 : vixMapOpenFileName:VixCoordinator.cpp:1110 <WARN> : Returning: 25 
01/22/2019 11:04:11 : hv_vmware_map_open_file_name:libvix.cpp:1434 <WARN> : Returning: 25

Cause

Analysis of vxms provider logs shows that vdOpen call failed to open the disk and it goes in for retry .Which confirms that the connectivity issue is transient in nature

Solution

Engineering has provided an EEB (Emergency Engineering Binary) for version 8.1.2 to address this issue. Please call support and request the EEB related to this issue (ET3969083).

Netbackup services need to be stopped before installing the EEB. EEB should be installed on the backup host server.