Problem
Backup jobs are failing with status 7641 and 61
Error Message
Error in job details:
Nov 21, 2020 5:19:16 PM - Error bpbrm (pid=272953) [PROXY] Received status: 7641 with message Failed to find a common CA Root to complete secure handshake: Connector CAs(["12f7cd25-ca23-474f-98a8-abcdefghijkl"]), Acceptor CAs(["5ge74756-9b9a-47eb-916d-bc5123456789"]).
Nov 21, 2020 5:19:16 PM - Error bpbrm (pid=272953) [PROXY] Encountered error (CERT_PROTOCOL_SELECT_COMMON_CA_ROOT) while processing(CertProtocol).
Nov 21, 2020 5:19:16 PM - Error bpbrm (pid=272953) cannot send mail because BPCD on nbuclient exited with status 61: the vnetd proxy encountered an error
Nov 21, 2020 5:19:16 PM - Info bpbkar (pid=0) done. status: 7641: Failed to find a common CA Root for secure handshake
Entry for "masterHostId" does not match in /usr/openv/var/vxss/certmapinfo.json
file of client and master
From Master:
# cat /usr/openv/var/vxss/certmapinfo.json
type c:\program files\veritas\netbackup\var\vxss\certmapinfo.json
[
{
"hostID": "12f7cd25-ca23-474f-98a8-abcdefghijkl",
"serverName": "nbumaster.example.com",
"issuerName": "nbumaster.example.com",
"certType": 1,
"isServerMaster": 1,
"issuedBy": "/CN=broker/OU=root@nbumaster.example.com/O=vx",
"crlPath": "/usr/openv/var/vxss/crl/3006a182.crl",
"securityLevel": 1,
"crlNextRefreshTime": 1606084681,
"crlLastRefreshTime": 1606070281,
"masterHostId": "12f7cd25-ca23-474f-98a8-abcdefghijkl"
}
From client:
root@nbuclient:/usr/openv/netbackup# cat /usr/openv/var/vxss/certmapinfo.json
type c:\program files\veritas\netbackup\var\vxss\certmapinfo.json
[
{
"hostID": "1234defe-bcf6-45ac-932b-fdc987654321",
"serverName": "nbumaster.example.com",
"issuerName": "nbumaster.example.com",
"certType": 1,
"isServerMaster": 1,
"issuedBy": "/CN=broker/OU=root@nbumaster.example.com/O=vx",
"crlPath": "/usr/openv/var/vxss/crl/3006a182.crl",
"securityLevel": 2,
"crlNextRefreshTime": 1606108191,
"crlLastRefreshTime": 1606079391,
"masterHostId": "5ge74756-9b9a-47eb-916d-bc5123456789"
}
When
/usr/openv/netbackup/bin/nbcertcmd -getCertificate
c:\program files\veritas\netbackup\bin\nbcertcmd -getCertificate
command is ran on the client machine:
masterHostId RECEIVED: 5ge74756-9b9a-47eb-916d-bc5123456789
masterHostId ACTUAL: 12f7cd25-ca23-474f-98a8-abcdefghijkl
Cause
Master server is sending the wrong hostID as "masterHostId" in certmapinfo.json file
The incorrect host id (5ge74756-9b9a-47eb-916d-bc5123456789) belongs to 1 client: nbu-client-bad
nbu-client-bad nbu-client-bad 8.1.2 RedHat Linux (3.10.0-1127.19.1.el7.x86_64) 5ge74756-9b9a-47eb-916d-bc5123456789 UNIX x86_64 Yes GenuineIntel Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, 32 CPUs
It should also be noted that when we do a bptestbpcd to client, it detects client as "NB_MACHINE_TYPE = MASTER_SERVER"
Emm aliases for master server shows that client nbu-client-bad is added as an alias of the master server, "Which is not right"
nbumaster# /usr/openv/netbackup/bin/admincmd/nbemmcmd -machinealias -getaliases -machinename nbumaster -machinetype master
c:\program files\veritas\netbackup\bin\nbemmcmd -machinealias -getaliases -machinename nbumaster -machinetype master
NBEMMCMD, Version: 8.1.2
The following aliases were found for the alias: "nbumaster"
nbumaster-bkp
nbu-client-bad <<<<<<<< This is not right
nbumaster.example.com
nbumaster
Command completed successfully.
Solution
Confirm that the nbu-client-bad is not an intended alias of the master server.
1> Delete the incorrect machine alias using below command:
/usr/openv/netbackup/bin/admincmd/nbemmcmd -machinealias -deletealias -alias nbu-client-bad -machinetype master
c:\program files\veritas\netbackup\bin\admincmd\nbemmcmd -machinealias -deletealias -alias nbu-client-bad -machinetype master
2> Stop Netbackup services on master server
/usr/openv/netbackup/bin/goodies/netbackup stop
c:\program files\veritas\netbackup\bin\bpdown -f -v
3> Stop Veritas Private Branch Exchange service:
/opt/VRTSpbx/bin/vxpbx_exchanged stop
net stop "Veritas Private Branch Exchange"
4> Rename the /usr/openv/var/host_cache directory
mv /usr/openv/var/host_cache /usr/openv/var/host_cache_old
cd c:\program files\veritas\netbackup\var
ren host_cache host_cache_old
5> Start pbx service:
/opt/VRTSpbx/bin/vxpbx_exchanged start
net start "Veritas Private Branch Exchange"
6> Start Netbackup services:
/usr/openv/netbackup/bin/goodies/netbackup start
c:\program files\veritas\netbackup\bin\bpdown -f -v
7> Log on to a client which is having connectivity issue
8> Delete the existing certificate on "CLIENT MACHINE ONLY"
/usr/openv/netbackup/bin/nbcertcmd -deleteallcertificates
c:\program files\veritas\netbackup\bin\nbcertcmd -deleteallcertificates
9> Get a new certificate on client machine:
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
c:\program files\veritas\netbackup\bin\nbcertcmd -getCertificate -force
If a re-issue token is requested, please create and provide the token with -token switch
10> Check if you got "masterHostId" as "12f7cd25-ca23-474f-98a8-abcdefghijkl" in /usr/openv/var/vxss/certmapinfo.json
file
c:\program files\veritas\netbackup\var\vxss\certmapinfo.json
11> Test the connection and backups now.