Backups to Remote Node in GCO NetBackup Cluster Fail with Status Code 37, Not Authorised

Problem

A three node GCO cluster sees backup failures when the cluster is failed over to the remote node

Error Message

Backups start but eventually fail with status code 37, not authorized.

Nbcert logs on the remote master with nbcurl logging enabled shows the following where .104 is it the virtual name IP, and .105 is the node name:

01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::helper_check_connect_status: nbcurl CONNECT FROM 192.102.31.105.38744 TO 192.102.31.104.1556 fd = 8
01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::helper_check_connect_status: Returning VN_STATUS_SUCCESS
01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::nbio_connect: Returning VN_STATUS_SUCCESS
01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::bio_connect: RC [0] STAT [0] MAXFD [0] TIMEOUT [-1].
01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::bio_connect: Returning VN_STATUS_SUCCESS
01:57:53.402 [31765.31765] <2> nbclnt_curl_prefnet::operator std::string(): resolved hostname => IP [remotemaster.nbu.net:1556:192.102.31.104].
01:57:53.402 [31765.31765] <2> NBClientCURL::resetCurlResolver: Hostname resolution set to [remotemaster.nbu.net:1556:192.102.31.104].
01:57:53.402 [31765.31765] <2> NBClientCURL::performCurlOperation: CONNECTION RETRY with current time: 1529891872 connectionStartTime: 1529891843
Snip
01:57:53.413 [31765.31765] <2> curl_debug_logger(): == Info: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
01:57:53.413 [31765.31765] <2> curl_debug_logger(): == Info: Closing connection 30
01:57:53.413 [31765.31765] <2> NBClientCURL::performCurlOperation: CONNECTION RETRY timeout: 30 seconds reached with current time: 1529891873 connectionStartTime: 1529891843
01:57:53.413 [31765.31765] <16> NBClientCURL::performCurlOperation: Failed to perform operation: SSL connect error
01:57:53.413 [31765.31765] <2> mapCurlError: Translating CURL status [35] to NetBackup status [26]
01:57:53.413 [31765.31765] <2> NBClientCURL::performCurlOperation: Fetched httpcode = 0
01:57:53.413 [31765.31765] <16> LoginWithCertManager::getNewToken: POST failed with error: 26, URL : netbackup/loginwithcert
01:57:53.413 [31765.31765] <2> NBClientCURL:~NBClientCURL: Performing curl_easy_cleanup()
01:57:53.413 [31765.31765] <2> NBClientCURL:~NBClientCURL: Performing curl_global_cleanup()
01:57:53.413 [31765.31765] <2> nbclnt_curl_prefnet::reset: Returning VN_STATUS_SUCCESS
01:57:53.413 [31765.31765] <16> LoginWithCertManager::getNewToken: Exception encountered, status code: 26
01:57:53.413 [31765.31765] <16> LoginWithCertManager::getToken: get new token failed with error: 26
01:57:53.413 [31765.31765] <16> LoginWithCertManager::getToken: Exception encountered, status code: 26
01:57:53.413 [31765.31765] <16> NBClientCURL::performCurlOperation: Failed to fetch authorization token. Actual Error code 26return code :5930

Cause

When the DR node was installed, it was installed as primary master server rather then part an existing cluster so generated its own host identity, CA certificate and host ID certificate for the node name. When this server was then added to the existing nodes, a  new host id certificate was generated from the  DB on the primary node had as it had no record of the DR node. The host database on the existing nodes did not have a record of the host id certificate that was originally assigned to the node because this was assigned from another master server. This resulted in a mismatch between the host id record the node name had and the host id record that the database had.

Solution

On the primary  node:

1. Generate a reissue token for the node name either via the admin console or on the command line:
   nbcertcmd -createToken -name token_name -reissue -host host_name

On the remote node without failing over

1. Rename the certmapinfo,json file in /usr/openv/var/vxsss or <install_path>\NetBackup\var\vxss\certmapinfo.json
    
2. Run nbcertcmd -getCAcertificate
    
3. Run nbcertcmd -getceritificate -token token_name -force
    
4. Fail over to the remote node and test the backups