VMware backups failing with Status 23 and Status 6

Problem

All VMware backups using either nbd transport mode or san transport mode are failing with Status 6 with backup job details in activity monitor showing message

"ERR - Error opening the snapshot disks using given transport mode: Status 23."

The snapshots appear to be successful but the bpfis log and the snapshot job details showing the following messages -

09:13:09.051 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_Connect: Establish connection.
09:13:09.051 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Disable VMotion.
09:13:10.205 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: Licenses could not be retrieved for supplied user credentials.
09:13:10.439 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: VixDiskLib_PrepareForAccess returned:3014 [Insufficient permissions in the host operating system]
09:14:00.219 [9028.8020] <2> send_keep_alive: INF - sending keep alive
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: Calling VixDiskLib_PrepareForAccess
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_PrepareForAccess: Prepare to access disk.
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLib: VixDiskLib_Connect: Establish connection.
09:14:10.453 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: VixDiskLibVim_AllowVMotion: Disable VMotion.
09:14:11.685 [9028.8948] <2> onlfi_vfms_logf: INF - VixDiskLibVim: Licenses could not be retrieved for supplied user credentials.
09:14:11.919 [9028.8948] <2> onlfi_vfms_logf: INF - lockVM: VixDiskLib_PrepareForAccess returned:3014 [Insufficient permissions in the host operating system]

Error Message

Status 6

ERR - Error opening the snapshot disks using given transport mode: Status 23

Cause

It was determined that -

1. Several permissions were missing in the vCenter role used by the account specified for vCenter server credentials on the master server.

2. This account was also not added to vCenter level permissions tab with propagate down checked.

Note: This may be encountered in a working environment that was upgraded to NetBackup 7.6.0.2 or higher due to the change to the VMware VDDK to VDDK 5.5 or higher.   This VDDK version requires additional permissions.

Solution

Note: Making changes to vCenter roles, permissions, and user/groups is not a NetBackup process. The steps below are provided as an example but these changes should be made in coordination with a VMware administrator who understands the process and the potential impact of making these changes.

1. Locate the account specified on the NetBackup master server as follows:

Open the NetBackup Administration Console and go to "Media and Device Management -> Virtual Machine Servers -> Credentials"

Double-click on the vCenter in question.

Identify the username specified for this vCenter server.

2. Log into vCenter server using vSphere client.

3. In vSphere client go to "Home -> Administration -> Roles"

4. Identify the vCenter role used for the VMware backup agent account from (1) above.

5. Confirm this vCenter role has all of the permissions identified in the following NetBackup document linked to in the "Related Articles" section below -

What are the minimum permissions needed to properly backup and restore using vStorage api?
 
6. If any required permissions are missing, add them to this role and save the changes.

7. Next, in vSphere client navigate to one of the vCenter inventory views under "Home -> Inventory." Either "Host and Clusters" or "VMs and Templates" will work.

8. In the left-hand pane, locate and click on the vCenter server in the topology.

9. Click on the vCenter server's "Permissions" tab.

10. Confirm that the account specified in (1) above is visible in the User/Group column.

11. If not this user should be added. Or be confirmed to be in a group that is added.

12. Double-click on the user or group.

13. Confirm it is in the correct "Role" that has the permissions defined in (5) above.

14. Confirm the "Propagate" box is checked.