Nutanix backups fail with Status 1, status 4276, and status 4201

Problem

Nutanix backups frequently fail with various status codes, including Status 1, Status 4276, and Status 4201 when using Active Directory credentials.

1.   Getting status 1 due to snapshot delete failing with ACCESS_DENIED message in BPFIS log.
2.  Getting status 4726, status 4201, and status 156 due to ATHENTICATION_REQUIRED in BPFIS log.

Error Message

The following errors can be seen in the BPFIS log from the backup host:

10:17:29.868 [21156] <2> Nutanix: : ntnxahv_RestRequest::ntnxpgnGetSnapshotChangedRegions : ntnxpgnGetSnapshotChangedRegions output: {"state": "ERROR", "code": 401, "message_list": [{"reason": "AUTHENTICATION_REQUIRED", "message": "Authentication required.", "details": Authentication tokens in API request are invalid"api_version": "3.1"} ,HTTP code: 401 return value: 114

OR

17:45:21.750 [1337191] <2> Nutanix: : ntnxpgnDeleteVmSnapshot output: { "api_version": "3.1",  "code": 403,  "message_list": [ { "message": "Cannot delete DR policy based snapshot b759b5b8-6960-4ca9-95a4-a288145f0dd3",  "reason": "ACCESS_DENIED" } ],  "state": "ERROR" } HTTP code: 403 return value: 114
17:45:21.750 [1337191] <16> deleteVmSnapshot: NTNXAHVManagement_HMI::deleteVmSnapshot Unable to delete vm snapshot with uuid: b759b5b8-6960-4ca9-95a4-a288145f0dd3
17:45:21.750 [1337191] <16> delete_vmsnapshot_and_statefiles: Snapshot deletion failed for vm_uuid: (8ad104d1-7877-496c-9f63-189336b5137a), snap_uuid: (b759b5b8-6960-4ca9-95a4-a288145f0dd3)
17:45:21.750 [1337191] <16> delete_statefiles_snapshot: delete_statefiles_snapshot: Snapshot deletion failed for vm_uuid 8ad104d1-7877-496c-9f63-189336b5137a and Snapshot ID is b759b5b8-6960-4ca9-95a4-a288145f0dd3
17:45:21.750 [1337191] <16> handle_delete_previous_snapshot_statefiles: Failed to delete state files and Snapshot from server
17:45:21.750 [1337191] <16> do_thaw: handle_delete_previous_snapshot_statefiles failed for vm_uuid: (8ad104d1-7877-496c-9f63-189336b5137a)
17:45:21.751 [1337191] <16> do_thaw: do_thaw: Hypervisor failed to delete the VM snapshot.

Cause

Nutanix Support confirms an issue where AD recursive search caused API response time issues and so then it's recommended that a local account to be used.

Nutanix referenced internal article KB 5497 which says:

"If UPN suffixes have bee configured in the AD domain, we must disable UPN suffix for the backup account because v2 APIs require specifying the fully-qualified domain name and don not understand UPN suffixes.  (v3 APIs, on the other hand, will not accept fully-qualified domain name when UPN suffixes are configured.)  So, to have a single AD account access APIs in both v2 and v3 namespaces, we must make sure that the UPN suffix for the backup account matches the fully-qualified domain name.  Be aware that in some cases, AD recursive search for a user account used for authentication of backup appliance can result in a high values of API response times.  In these cases, it is recommended that a local account is used in order to avoid timeouts."

Solution

Credential NetBackup to use a Nutanix local account (with admin rights) rather than using Active Directory.
After doing this the failures noted above do not occur.