Hyper-V backups fail with status 25

Problem

Hyper-V Cluster fails with status 25 when cluster name resolves to Node 1 but  is successful when cluster name resolves to node 2.

Error Message

Increase debug logging levels for the following:

vxlogcfg -p 51216 -o 464 -a -s DebugLevel=6
vxlogcfg -p 51216 -o 457 -a -s DebugLevel=6
vxlogcfg -p 51216 -o 463 -a -s DebugLevel=6
vxlogcfg -p 51216 -o 377 -a -s DebugLevel=6
vxlogcfg -p 51216 -o 366 -a -s DebugLevel=6

Collect the ncfnbcs log after reproducing the issue.  Found the following error:

7/30/2020 16:28:46.911 [Debug] NB 51216 libhyperv 463 PID:11348 TID:11836 File ID:366 [No context] 4 [getHostDetails] powershell command = Get-NBHostDetails | Select-Object -Property Name, Fqdn, OsVersion, OsCaption, IsAbove2008R2, HasHypervModule, HasClusterModule, HasScvmmModule  (../PowershellCommand.cpp:131)

7/30/2020 16:28:47.556 [Debug] NB 51216 libhyperv 463 PID:11348 TID:11836 File ID:366 [No context] 1 [CHypervVmList::verifySupportForHypervServer] Failed to get any valid server configuration for Node 1 (../CHypervVm.cpp:1698)

7/30/2020 16:28:47.556 [Debug] NB 51216 libhyperv 463 PID:11348 TID:11836 File ID:366 [No context] 1 [CHypervVmList:doConnect] Error in supporting specified server. (../CHypervVm.cpp:930)

7/30/2020 16:28:47.556 [Debug] NB 51216 libhyperv 463 PID:11348 TID:11836 File ID:366 [No context] 1 [CHypervVmList:connect] Failed to connect to HYPERV Server during full discovery2 (../CHypervVm.cpp:514)

7/30/2020 16:28:47.557 [Application] NB 51216 ncfhypervpi 464 PID:11348 TID:11836 File ID:366 [No context] [Error] V-464-8 Failed to establish connection.

7/30/2020 16:28:47.557 [Application] NB 51216 libhyperv 463 PID:11348 TID:11836 File ID:366 [No context] [Error] V-463-7 Unsupported Hyper-V server configuration for server [Node 1].

The above shows that the Powershell command Get-NBHostDetails is failing.

Cause

The Get-NBHostDetails is custom cmdlet and is a wrapper containing other Powershell commands. One or more of the contained Powershell commands is failing.

Solution

To identify the Powershell command that is failing, it is necessary to run each one individually from a Powershell prompt.  The following Powershell commands are identified from the NCFNBCS log from a node that performs successful backups:

FailoverClusters\Get-Cluster | Select-Object -Property Domain, EnableSharedVolumes, Id, Name, SharedVolumesRoot

FailoverClusters\Get-ClusterNode | Select-Object -Property Id, NodeInstanceID, Name, Cluster, State, MajorVersion, MinorVersion, Description | where State -eq "Up"

Hyper-V\Get-Vm -ComputerName Antimony,Bismuth | Select-Object -Property Name, Id, ComputerName, State, Notes, IsClustered, Version

In this particular case the first two commands were successful, but the third command failed.
It was discovered that the Powershell tools for Hyper-V were not installed on the cluster node 1 but were on cluster node 2.

After installing the Powershell tools for Hyper-V on Node 1 the backups were successful.