VMware backup may be partially successful due to snapshot deletion timed out in VixDiskLib_EndAccess() VDDK call.
Problem
VMware snapshot deletion intermittently returns Status1 although the snapshot is deleted successfully. In rare cases, VMware backup may exit Status 13.
Error Message
When VMware backup exited Status 1 (the requested operation was partially successful), VMware snapshot deletion timed out in VixDiskLib_EndAccess() VDDK call.
Messages similar to the following may appear in ncfnbcs log.
2018/03/26 01:34:12.368 [Application] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] [Warning] V-309-28 WRN - Calling EndAccess
2018/03/26 01:34:12.369 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLib: VixDiskLib_EndAccess: Disk access completed.
(snip)
2018/03/26 01:34:12.390 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Enable VMotion.
(snip)
2018/03/26 01:49:21.022 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLibVim: Wait too long time (900000000 sec) for the completion of the whole task processing. Tring to abort it!
When VMware backup exited status 13 (file read failed), bpbkar crashed as a result of time-out in GetNfcTicket() VDDK call.
Messages similar to the following may appear in VxMS provider log.
03:32:07.0480 : openLeafSnapshotDisks:VixGuest.cpp:457 <DEBUG> : Calling vdOpen(Disk: [xxx] xxx.vmdk, Flags: 0x4)
03:32:07.0480 : vdOpen:VixInterface.cpp:426 <INFO> : Calling Open()
03:32:07.0480 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_OpenEx: Open a disk.
03:32:07.0480 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [xxx] xxx.vmdk.
(snip)
03:47:06.0683 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: Wait too long time (900000000 sec) for the completion of the whole task processing. Tring to abort it!
Cause
The issues are caused by the known issue in VDDK6.0 and 6.5.
Intermittent hang observed when opening virtual disks.
A customer observed intermittent hanging while opening virtual disks. The hang was found to occur inside the VixDiskLibVim module, in VixDiskLib_PrepareForAccess, VixDiskLib_EndAccess, or GetNfcTicket, and may be caused by network problems. Fixes have been identified and will appear in a future release. One fix is a timeout for HTTP requests, and another is a timeout for the open operation.
VDDK 6.0 -
Virtual Disk Development Kit 6.0 Release Notes
Release date: 12 MAR 2015 | Builds: ESXi 2494585, VDDK 2498720
VDDK 6.5 -
Virtual Disk Development Kit 6.5 Release Notes
Release date: 15 NOV 2016 | Builds: ESXi 4564106, VDDK 4604867
Solution
VDDK6.7 resolved the issue. The issue has been resolved in NetBackup 8.2 which includes VDDK6.7.2.
Due to the fact that NetBackup 8.1.2 and earlier versions do not support VDDK6.7, please let the jobs retry and confirm they work successfully as a workaround of Status 13. As to Status 1, no further action is required.