Linux Server VMware virtual machine backup fails with "An error occurred while quiescing the virtual machine".

Problem

Snapshots are failing for VMware backups on a single Linux VM (virtual machine) with status 156 as it appears to be unable to quiesce the system.

Linux Server VMware virtual machines may fail to generate a quiesced snapshot, causing the NetBackup job to fail with an Operation Status: 156 (Snapshot error encountered).

Error Message

Jul 19, 2021 9:54:07 PM - Info bpfis (pid=9001) Backup started
Jul 19, 2021 9:54:08 PM - snapshot backup of client vmclient5 using method VMware_v2
Jul 19, 2021 9:54:08 PM - Info bpbrm (pid=8301) INF - vmwareLogger:  Creating snapshot for vCenter server vcenter1.domain.local, ESX host esxsvr6.domain.local, BIOS UUID 123ab4c5-d678-e901-fg2h-786j5k6lm78n, Instance UUID 786a654b-32c1-098d-7e65-432fg10hijk9, Display Name vmclient5, Hostname vmclient5 
Jul 19, 2021 9:54:08 PM - Info bpbrm (pid=8301) INF - vmwareLogger:  Connection state of virtual machine: connected.
Jul 19, 2021 9:54:15 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - vSphere_freeze: Snapshot task for virtual machine vmclient5 (/vmmor/vm-38298) failed, 10 retries remaining, error type: 237, error message: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details..
...(snip)...
Jul 19, 2021 9:56:45 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - vSphere_freeze: Snapshot task for virtual machine vmclient5 (/vmmor/vm-38298) failed, 0 retries remaining, error type: 237, error message: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details..
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - 
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - 
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - snapshot processing failed, status 156
Jul 19, 2021 9:56:57 PM - Critical bpbrm (pid=8301) from client vmclient5: FTL - snapshot creation failed, status 156
Jul 19, 2021 9:56:57 PM - Warning bpbrm (pid=8301) from client vmclient5: WRN - ALL_LOCAL_DRIVES is not frozen
Jul 19, 2021 9:56:57 PM - Info bpfis (pid=9001) done. status: 156
...(snip)...
Operation Status: 156

Cause

Attempts to login to the Linux VM, to see the virtual machine's event log for details, indicate that the client is not responsive.

Example:
# ssh vmclient5
ssh_exchange_identification: read: Connection reset by peer

Additionally, for this same backup error, the virtual machine vendor suggests the cause is related to:

• The VMtools version is too low, meaning it could be impacted by performance issues, or other known issues, which are fixed in newer versions.
• The backup being application-consistent, while not using customer pre-freeze and post-thaw scripts.

Solution

Possible solutions include:

1. Rebooting the VM client.

2. Upgrading the VMtools, as it provides the ability to take quiesced snapshots of the guest OS.

3. Create customer pre-freeze and post-thaw scripts, to allow application-consistent backups.