VMware backups fail with "snapshot creation failed, status 156 WRN - ALL_LOCAL_DRIVES is not frozen " due to Quiescing Issue

Problem

VMware VSS Snapshot provider can not quiesce VM

Error Message

5/16/2011 8:10:17 AM - begin Create Snapshot
5/16/2011 8:10:21 AM - snapshot backup of client <client_name> using method VMware
5/16/2011 8:10:50 AM - Critical bpbrm(pid=7952) from client <client_name>: FTL - snapshot creation failed, status 156  
5/16/2011 8:10:50 AM - Warning bpbrm(pid=7952) from client <client_name>: WRN - ALL_LOCAL_DRIVES is not frozen

Task Event from vCenter console:
Cannot  create a  quiesced  snapshot  because the  create  snapshot  operation  exceeded  the time  limit for  holding off I /O in the  frozen  virtual  machine.

Cause

Vstorage process overview:

1. Master initiates the backup
2. The NetBackup client on the VMware backup host logs into the vcenter and initiates a snapshot
3. VMware VMtools calls  VSS to synchronize the file system and creates a snapshot on the datastore.
4. The NetBackup client reads the data off the datastore.

Virtual machine quiesce :

Note: By default, this option is enabled. In the great majority of cases, you should accept the default. I/O on the virtual machine is quiesced before NetBackup creates the snapshot.
Without quiescing file activity, data consistency in the snapshot cannot be guaranteed. If not consistent, the backed up data may be of little or no value. If this option is disabled, the snapshot is created without quiescing I/O on the virtual machine. This would be analogous to a power off. In this case, you must perform your own checks for data
consistency in the backup data.
Caution: Veritas does not recommend that you disable quiesce. In most cases, this option should be enabled.

When NetBackup initiates the snapshot (via VCB or vStorage), the virtual machine quiescence is done by ESX/ESXi using VMware tools. VMware tools framework is the way VMkernel can get into guest operating system. The tools framework interacts with pre-freeze script (if configured) to put the application in backup mode and once the script is run, the VMware VSS Provider (also available from VMware tools) will create the VSS snapshot so that writes to the volumes are tracked. Now VMkernel creates the VM snapshot. Once the VM snapshot creation is complete, VMkernel calls the VMware VSS provider inside the guest via tools framework and removes the VSS snapshot. After this post-thaw script is called to let the application resume the normal operation mode. NetBackup really uses just the VM snapshot. The pre-freeze/post-thaw scripts and VSS snapshots inside the guests are a way to create a consistent VM snapshot which is not controlled by NetBackup.

Solution

Troubleshooting Volume Shadow Copy (VSS) quiesce issue

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Creating a quiesced snapshot in Windows 2008 and Windows 2008 R2 fails with the error: Snapshot guest failed.

1. If Server disk is busy with applications such as SQL, Install NetBackup client and use the SQL agent for Application level backup.
2. Contact VMware regarding these issues.
3. Create a new Policy disable quiescing. This is not recommended because it will backup the disk as though there was a power off.
4. On a Windows VM ,please try using Veritas VSS provider instead. It may resolve the issue.

NetBackup 7.x , NetBackup 8.x

Windows 2008 R2, Windows 2012, Windows 2016