VMware snapshot creation fails - Unable to access file since it is locked

Problem

Snapshot creation fails with Status 156.

Error Message

From NetBackup job details:

9/3/2014 1:31:33 PM - Info bpbrm(pid=4908) INF - vmwareLogger: WaitForTaskCompleteEx: Unable to access file <unspecified filename> since it is locked <232>
9/3/2014 1:31:33 PM - Info bpbrm(pid=4908) INF - vmwareLogger: WaitForTaskCompleteEx: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE
9/3/2014 1:31:33 PM - Info bpbrm(pid=4908) INF - vmwareLogger: ConsolidateVMDisks: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE
9/3/2014 1:31:33 PM - Info bpbrm(pid=4908) INF - vmwareLogger: ConsolidateVMDisksAPI: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE
9/3/2014 1:31:36 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with 36: SYM_VMC_TASK_REACHED_ERROR_STATE
9/3/2014 1:31:36 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - VMware error received: Unable to access file <unspecified filename> since it is locked
9/3/2014 1:31:36 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
9/3/2014 1:31:36 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL -
9/3/2014 1:31:37 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
9/3/2014 1:31:37 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL -
9/3/2014 1:31:37 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - snapshot processing failed, status 156
9/3/2014 1:31:37 PM - Critical bpbrm(pid=4908) from client 10.8.92.9: FTL - snapshot creation failed, status 156
9/3/2014 1:31:38 PM - Warning bpbrm(pid=4908) from client 10.8.92.9: WRN - ALL_LOCAL_DRIVES is not frozen
9/3/2014 1:31:38 PM - Info bpfis(pid=4248) done. status: 156
9/3/2014 1:31:38 PM - end VMware, Create Snapshot; elapsed time: 0:01:10
9/3/2014 1:31:38 PM - Info bpfis(pid=4248) done. status: 156: snapshot error encountered
9/3/2014 1:31:38 PM - end writing
Status 156

Cause

Identified that this virtual machine had orphaned snapshot files.

Identified that the VMware hotadd mechanism had not unmounted this virtual machine's disks (VMDK files) from the backup host, causing the snapshot consolidations to fail.

This also prevented future backup attempts to fail during the snapshot creation attempt with the errors reported above in the job details.

Solution

To clean this up we did the following:

1. In the VMware vSphere client we located the virtual machine that was the backup host and using "Edit Settings" removed the unmounted disk from the failing client.
2. Once removed, using VMware vSphere client we located the virtual machine that was the client and used VMware's snapshot manager to consolidate its disks.

Note: The above steps worked in this particular instance.  For production systems it is recommended to contact your local VMware administrator or VMware support for assistance in unmounting the hotadded disks from the backup and consolidating orphaned snapshots.

Suggest reviewing 3rd Party VMware documentation on this same topic:

• Unable to consolidate VMware virtual machine snapshots due to file lock (2151402) - https://kb.vmware.com/s/article/2151402

Applies To

- NetBackup 7.6 or newer with the VMware agent using the hotadd transport type.
- VDDK 5.5.1 or newer
- VMware vCenter server version 5.1.0 (build 1364037) or newer
- hotadd is a transport type that can be used when the backup host is itself a virtual machine that resides on a ESX/ESXi host that has access to all datastores via a SAN.  During a virtual machine backup using the hotadd transport type the client virtual machine's disks (VMDK) files are mounted (hotadded) on the backup host as additional disks. When the backup is complete these mounted disks should be released by the VMware hotadd mechanism and subsequently a snapshot  consolidation of the client virtual machine should take place.