VMware backup may be partially successful due to snapshot deletion timed out in VixDiskLib_EndAccess() VDDK call.

Article: 100044690
Last Published: 2019-01-17
Ratings: 0 0
Product(s): Appliances, NetBackup

Problem

VMware snapshot deletion intermittently returns Status1 although the snapshot is deleted successfully. In rare cases, VMware backup may exit Status 13.

Error Message

When VMware backup exited Status 1 (the requested operation was partially successful), VMware snapshot deletion timed out in VixDiskLib_EndAccess() VDDK call.
Messages similar to the following may appear in ncfnbcs log.

2018/03/26 01:34:12.368 [Application] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] [Warning] V-309-28 WRN - Calling EndAccess
2018/03/26 01:34:12.369 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLib: VixDiskLib_EndAccess: Disk access completed.
(snip)
2018/03/26 01:34:12.390 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Enable VMotion.
(snip)
2018/03/26 01:49:21.022 [Diagnostic] NB 51216 ncf 309 PID:57664 TID:58408 File ID:366 [No context] 1 V-309-28 [VDDK LOG] INF - VixDiskLibVim: Wait too long time (900000000 sec) for the completion of the whole task processing. Tring to abort it!

When VMware backup exited status 13 (file read failed), bpbkar crashed as a result of time-out in GetNfcTicket() VDDK call. 
Messages similar to the following may appear in VxMS provider log.

03:32:07.0480 : openLeafSnapshotDisks:VixGuest.cpp:457 <DEBUG> : Calling vdOpen(Disk: [xxx] xxx.vmdk, Flags: 0x4)
03:32:07.0480 : vdOpen:VixInterface.cpp:426 <INFO> : Calling Open()

03:32:07.0480 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_OpenEx: Open a disk.

03:32:07.0480 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [xxx] xxx.vmdk.
(snip)
03:47:06.0683 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLibVim: Wait too long time (900000000 sec) for the completion of the whole task processing. Tring to abort it!

Cause

The issues are caused by the known issue in VDDK6.0 and 6.5.

Intermittent hang observed when opening virtual disks.
A customer observed intermittent hanging while opening virtual disks. The hang was found to occur inside the VixDiskLibVim module, in VixDiskLib_PrepareForAccess, VixDiskLib_EndAccess, or GetNfcTicket, and may be caused by network problems. Fixes have been identified and will appear in a future release. One fix is a timeout for HTTP requests, and another is a timeout for the open operation.

VDDK 6.0 - 
Virtual Disk Development Kit 6.0 Release Notes
Release date: 12 MAR 2015 | Builds: ESXi 2494585, VDDK 2498720

VDDK 6.5 - 
Virtual Disk Development Kit 6.5 Release Notes
Release date: 15 NOV 2016 | Builds: ESXi 4564106, VDDK 4604867

Solution

VDDK6.7 resolved the issue.  The issue has been resolved in NetBackup 8.2 which includes VDDK6.7.2.
Due to the fact that NetBackup 8.1.2 and earlier versions do not support VDDK6.7, please let the jobs retry and confirm they work successfully as a workaround of Status 13. As to Status 1, no further action is required.

References

Etrack : 3946284

Was this content helpful?