Troubleshooting client certificate deployment

Step 

Action 

1

Verify communication with NBWMC on the Primary server:

nbcertcmd -ping

If this succeeds, skip to step 10

2

Check the same command on the Primary Server itself (or if command line access to the Primary is not available, check from any other NetBackup Media Server or Client):

nbcertcmd -ping

If this fails, NBWMC is not working correctly see: https://www.veritas.com/content/support/en_US/article.100073385 for troubleshooting steps to resolve NBWMC issues.

If this step succeeds, the issue is on the client side, proceed with step 3.

3

Test the network connectivity.

Ping the primary server by name:

ping <nbmaster>

If this succeeds, skip to step 7

4

If ping of primary server by name fails, get the IP address of the Primay Server (or all IP addresses if there are multiple interfaces) and ping by IP:

ping 10.11.12.13
ping 192.111.22.33

If this succeeds, skip to step 6

5

If ping of Primary Server by IP also fails, test this from another client or media server. Some environments have the ICMP protocol, which the "ping" command uses, disabled for security reasons.

If there is a chance that this might be the case, continue to the next steps. Otherwise, if pinging the Primary Server works from other hosts, there are networking issues on the client that need to be resolved.

6

If pinging the Primary Server is successful by IP but failed by hostname:

This may indicate a need to update the DNS (Domain Name Service) entries for the Primary server, or validate the DNS server used by the problem client. If DNS is not in use then updating the hosts file on the client can be used to resolve the hostname of the Primary Server.

The hosts file can be found at:

Linux/Unix:
/etc/hosts

Windows:
C:\Windows\system32\drivers\etc\hosts

After updating the hosts file, clear the NetBackup host cache:

Linux/Unix:
/usr/openv/netbackup/bin/bpclntcmd -clear_host_cache

Windows:
cd [install path]\NetBackup\bin
bpclntcmd -clear_host_cache

Test pinging the hostname again after making any changes, to verify it is working.

7

After verifying ping or if ICMP is intentionally disabled:

OS utilities can be used to test TCP connectivity over port 1556 (PBX), the primary port NetBackup uses. This must be open bi-directionally, and both sides must be listening.

Linux:

A. curl
curl -kv telnet://nbmaster:1556

Example:
curl -kv telnet://nbmaster2:1556
* Rebuilt URL to: telnet://nbmaster2:1556/
*   Trying 192.168.2.48...
* TCP_NODELAY set
* Connected to nbmaster2 (192.168.2.48) port 1556 (#0)

* Closing connection 0

B, nc (netcat)
nc -v nbmaster 1556

Example:
nc -v nbmaster2 1556
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.2.48:1556.

C. telnet
telnet nbmaster 1556

Example:
telnet nbmaster2 1556
Trying 192.168.2.48...
Connected to nbmaster2.
Escape character is '^]'.

Note: To exit, hit ctrl+] to escape the telnet session, then type "quit" and press enter, or press the enter key, and the remote NetBackup host will close the connection.

Windows:

A. Test-NetConnection (powershell cmdlet):
Test-NetConnection -ComputerName nbmaster -Port 1556

Example:
Test-NetConnection -ComputerName nbmaster2 -Port 1556

ComputerName     : nbmaster2
RemoteAddress    : 192.168.2.48 
RemotePort       : 1556
InterfaceAlias   : LAN
SourceAddress    : 192.168.2.45
TcpTestSucceeded : True

B. telnet:
telnet nbmaster 1556

Example:
telnet nbmaster2 1556

If it connects, the screen goes blank and shows a cursor, meaning you are in a telnet session.
Note: to exit, hit ctrl+] to escape the telnet session, then type "quit" and press enter, or hit the enter key, and the remote NetBackup host will close the connection.

If connection succeeds, skip to step 9

8

If the connection attempt was not successful, then something may be blocking the TCP connection.

There are three common ways that TCP connections can be blocked:

1. By the OS firewall of the NetBackup Primary Server.
    See above section for checking the OS firewall.

2. By a network firewall or layer-3 switch acting as a firewall.

If it is verified that both sides are listening with netstat, but a telnet/curl/Test-NetConnection to make a TCP connection over port 1556 cannot be made, please raise a request with the network or firewall team to determine why port 1556 is not open between these hosts.

3. By the OS firewall on the problem client.
To rule this out, verify if an OS firewall is running:

Linux:

A. iptables -L

Example:
iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

B. systemctl status firewalld

Example:
systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
    Docs: man:firewalld(1)

Windows:
netsh advfirewall show allprofiles state

Example:
netsh advfirewall show allprofiles state

Domain Profile Settings:
----------------------------------------------------------------------
State                                 OFF

Private Profile Settings:
----------------------------------------------------------------------
State                                 OFF

Public Profile Settings:
----------------------------------------------------------------------
State                                 OFF
Ok.

The above examples were of the firewall being off. If the output differs, ask the appropriate OS admin to confirm that the firewall is either off, or that it is configured to allow traffic through TCP port 1556.
Also confirm that the local system is listening on port 1556 (it should be as long as the PBX service is running).

Linux:

A. netstat:
netstat -lnopt | grep 1556

Example:
netstat -lnopt | grep 1556
tcp        0      0 0.0.0.0:1556            0.0.0.0:*               LISTEN      1455/pbx_exchange    off (0.00/0/0)
tcp6       0      0 :::1556                 :::*                    LISTEN      1455/pbx_exchange    off (0.00/0/0)

B. ss:
ss -lnopt | grep 1556

Example:
ss -lnopt | grep 1556
LISTEN 0      5                 0.0.0.0:1556       0.0.0.0:*    users:(("pbx_exchange",pid=1455,fd=10))
LISTEN 0      128                  [::]:1556          [::]:*    users:(("pbx_exchange",pid=1455,fd=9))

Solaris:
A. netstat:
netstat -a -P tcp | grep LISTEN | grep 1556

Example:
netstat -a -P tcp | grep LISTEN | grep 1556
    *.1556               *.*                  0      0  256000      0 LISTEN
    *.1556                            *.*                               0      0  256000      0 LISTEN

Windows:
A. netstat:
netstat -ano | findstr LISTEN | findstr 1556

Example:
netstat -ano | findstr LISTEN | findstr 1556
TCP    0.0.0.0:1556           0.0.0.0:0              LISTENING       3300

9

Additional steps to resolve the inability to run nbcertcmd -ping:

If it is verified that ping works, TCP over port 1556 works, and nbcertcmd -ping works on the Primary Server, but nbcertcmd -ping does NOT work, then continue with the following tests:

A. Check if the client has a proxy configured at the OS level. This breaks NetBackup's Secure Communications - https://www.veritas.com/support/en_US/article.100051053

B. Verify the top SERVER entry in the NetBackup configuration on the client is correct:

Linux:
/usr/openv/netbackup/bin/nbgetconfig SERVER

Example:
/usr/openv/netbackup/bin/nbgetconfig SERVER
SERVER = nbmaster2.nbulab
SERVER = nbmaster2

Windows:
cd [install path]\NetBackup\bin
nbgetconfig SERVER

Example:
nbgetconfig SERVER
SERVER = nbmaster1
SERVER = nbmedia1

Note: Only the top item in the list is important for nbcertcmd -ping to work under normal circumstances.

C. Check for tunnel configuration on client, if the websvctunnels.cache file exists

Linux:
ls -lha /usr/openv/netbackup/websvctunnels.cache

Windows:
cd [install path]\NetBackup
dir /a websvctunnels.cache

D. If a specific error is returned from nbcertcmd -ping other than 26/41/8500, look up the error code in the NetBackup Status Codes Reference Guide: https://www.veritas.com/content/support/en_US/doc/44037985-159180701-0/id-SF970184045-159180701

E. If status 26/8500 is being returned and previously it has been verified that TCP traffic on port 1556 works, there is still a chance that TLS traffic is being blocked on port 1556. This is possible with application-aware firewall models.

This filtering could be happening from:

• The Primary Server's OS firewall
• The problem client's OS firewall
• A network firewall
• A layer-3 switch acting as a firewall

There are not many good tests for this, as "nbcertcmd -ping" not working when raw TCP works over the same port can be an indicator of many possible issues.

10

After verifying nbcertcmd -ping works, pull a CA certificate:

Run on the client:
nbcertcmd -getcacertificate

Linux:
/usr/openv/netbackup/bin/nbcertcmd -getcacertificate

Example:
/usr/openv/netbackup/bin/nbcertcmd -getcacertificate
NetBackup CA certificate is successfully stored from the master server nbmaster2.nbulab.

Windows:
cd [install path]\NetBackup\bin
nbcertcmd -getcacertificate

Example:
nbcertcmd -getcacertificate
NetBackup CA certificate is successfully stored from the master server nbmaster1.

If pulling the CA certificate worked, skip to step 12

11

Common errors that are seen if pulling the CA certificate fails:

A. EXIT STATUS 26: client/server handshaking failed:

This error can be received when running almost any nbcertcmd command.
A. If this error is returned on the Primary Server, it usually indicates that nbwmc is not running or working properly. Please troubleshoot nbwmc working on the Primary, and then test re-running the command after. https://www.veritas.com/content/support/en_US/article.100073385
B. If this error is seen on a media server or client, the first thing to rule out is the same as Step A. Re-run the failing command on the Primary Server and ensure that it is successful. Once you've validated that the Primary server command is good and the nbwmc services on it are running, then this is most likely indicating a communication issue. This can be caused by a number of things.

• Check the OS firewall settings on both the Primary Server and the problem media/client to ensure they aren't blocking the traffic.
• Check other clients on the same network segment (subnet, VLAN, etc) as the problem host, to ensure they can communicate with the primary with the same nbcertcmd commands.
• Ensure that there isn't an MTU mismatch on the interfaces in-use.
• Ensure that ping works, and test TCP port 1556, both bi-directionally between the Primary and problem host.
• Check the problem host for an OS environmental variable establishing an HTTP or HTTPS proxy system-wide.

If all checks are validated, then the issue is most likely to be a network firewall or network switch acting as a firewall that is selectively-blocking TLS traffic on port 1556.

B. EXIT STATUS 41: network connection timed out

This error can be received when running almost any nbcertcmd command.

• If you receive this error on the Primary Server, it usually indicates a hostname resolution issue for either the primary's hostname or for the name "localhost".
• On Unix & Linux, check the hosts file and make sure the first entry is for "127.0.0.1 localhost". Often this error can occur when there is a typo or character deletion on that line. On windows, winsock handles localhost resolution, and such a hosts file entry is not needed.
• Check "nslookup localhost" on the Primary Server. If this returns any hostname other than localhost (localhost.company.com is fine) or if it returns any IP address other than 127.0.0.1 or ::1 (IPv6 localhost) then DNS must be fixed.
• If you are familiar with the tools, you can also use something like strace on nbwmc to see if it is trying to connect to an unexpected IP address.
• If you receive this error on a media server or client, as always, the first course of action is to re-run the same nbcertcmd command on the Primary and ensure that you don't get the same error there. Assuming that that is fine, it is most likely a communications issue. As with other issues, check connectivity, name resolution, ports, firewalls, etc.

C. EXIT STATUS 8500: Connection with the web service was not established.

See troubleshooting for status 26 above (Error A).

D. EXIT STATUS 5969 : Response from the NetBackup Web Management Console service could not be parsed.

See troubleshooting for status 26 above (Error A). 

This error almost always indicates that the issue is on the Primary Server.

• If OpsCenter is installed on the Primary Server this can be the cause (the two should not co-exist on one server).
• 3rd party software on the Primary could also be trying to use the same ports as NBWMC.

E. EXIT STATUS 13 / 23 / 25 / 61 / 5949 / 5978 / 5942 / 7624 / 7625 / 7627 / 7640 / 7660 / 9301

Some certificate files may be corrupt. See: https://www.veritas.com/support/en_US/article.100039941 

F. EXIT STATUS 14: file write failed

This can be caused by a corrupt certmapinfo.json in [install path]/var/vxss/.

• Rename certmapinfo.json and retry the command.
• Afterwards, check if the old certmapinfo.json had information on more than 1 Primary Server listed.
• Environments that use AIR or have multi-homed (backed up by more than one Primary Server) environments may need more than one CA and Certificate pulled after renaming this file to return it to full operational ability.

G. EXIT STATUS 5954: The host name could not be resolved to the requesting host's IP address.

This error is generally because of the following:

• When you make a cert request with nbcertcmd, the client's CLIENT_NAME is packaged in the request.
• This can be overridden by using the -host option with the nbcertcmd to use a different CLIENT_NAME value.
• When the master server receives the request, it takes note of the IP it received the request from.
• The master tries to resolve the incoming IP from the client to a hostname if it can.
• The master then compares the hostname (or IP if it couldn't resolve it) to the CLIENT_NAME or -host that was included in the cert request.
• The error means that they didn't match.
• For non-new clients, it will also approve the cert request if the name it resolved is a mapping of the CLIENT_NAME or -host name that it received

In short, the goal is to make the names match by making a hosts file entry on the primary or changing the CLIENT_NAME value as needed.

H. EXIT STATUS 8504: The web service certificate is issued by an unknown Certificate Authority.

Pull a new CA with "nbcertcmd -getcacertificate" and then retry previous command

I. Exit Status 5930: The request could not be authorized.

• If using "nbcertcmd -renewcertificate" ignore this error and run the command "nbcertcmd -getcertificate -force" instead, then troubleshoot the error from that command.
• If running a command which requires a WEB login such as "nbcertcmd -createtoken" or "nbcertcmd -getsecconfig -certDeployLevel" then make sure the account specified for bpnbat -login is a user that is either an OS admin (root / Administrator) or an RBAC admin in NetBackup (create one with bpnbaz -AddRBACPrinicpal if needed).

J. EXIT STATUS 8509: The specified server name was not found in the web service certificate.

• This error may indicate that the top SERVER list entry of the client (or other host the command is being run on on) is not a valid Subject Alternative name in the Primary Server's internal Tomcat Certificate.
• If the top SERVER list entry is invalid on the host, change it by editing the bp.conf or Windows registry on the problem host.
• If the name should be valid, but does not work on any host using that name as the first SERVER entry, add the name for the Primary to the SAN list in the Tomcat cert via this TN: https://www.veritas.com/support/en_US/article.100034092

Note: Running the nbcertconfig command requires the following:

• Core NetBackup services must be running on the primary (pbx, 5 vnetd's, dbsrv/postgresql [internal database], and nbatd)
• The nbwmc service must be stopped.
• This process is performed on the Primary Server and requires downtime (backups will fail if executed during the procedure).
• Ensure that when running "nbcertconfig -t -f -sub name1,name2,name3,name4" to include every shortname and FQDN that will possibly be used to contact the primary, including the ones which are already in-use.

K. EXIT STATUS 8506: The certificate has expired.

This means that the automatic renewal of the internal tomcat certificate on the Primary Server failed 6 months ago, and it has now expired.

To manually renew it, see: https://www.veritas.com/content/support/en_US/article.100073668

L. EXIT STATUS 7648: Connection cannot be established because the host validation failed

• This may occur when the media server or client has a CLIENT_NAME that is set to a hostname that has an unapproved mapping in "Mappings for approval".
• If so, find mapping and approve it in the NetBackup Administration Console:
  • Select Host Management, then select the "Mappings for Approval" tab.
  • Then right-click the affected host from the list and select Approve

If this status is intermittent between the Primary Server and some Media servers or other Primary Servers used for AIR, perform the following:

• First, verify it is possible to pull a CA (nbcertcmd -getcacertificate) and a cert (nbcertcmd -getcertificate -force) on the problem host.
• Next, get a change window for a very brief downtime on the Primary Server, and try clearing out an internal nbwmc cache folder inside the [install path]/wmc/webserver/temp/access-caches/ folder per https://www.veritas.com/content/support/en_US/article.100052045.html 

If none of the above resolves the inability to pull a CA certificate, skip to step 13

12

After CA Certificate is deployed, pull a hostid certificate:

Use nbcertcmd -getcerificate -force to pull a hostid certificate.  

Example:
nbcertcmd -getcertificate -force
Host certificate and certificate revocation list received successfully from server nbmaster2.nbulab.

13

Additional troubleshooting steps if certificate deployment still fails:

The following commands help test connectivity between NetBackup hosts.  Use any error messages received as a search criteria to determine possible resolutions.

From Primary to problem host:
bptestbpcd

Usage:

Linux (to media): /usr/openv/netbackup/bin/admincmd/bptestbpcd -host MEDIANAME -verbose -debug

Linux (to client): /usr/openv/netbackup/bin/admincmd/bptestbpcd -client CLIENTNAME -verbose -debug

Windows (to media): cd [install path]\NetBackup\bin\admincmd\
                                  bptestbpcd -host MEDIANAME -verbose -debug

Windows (to client): cd [install path]\NetBackup\bin\admincmd\
                                 bptestbpcd -client CLIENTNAME -verbose -debug
Note: Use of the -debug flag makes bptestbpcd write to the commandline instead of a log file. If you are trying to collect verbose logs, use instead bptestbpcd -host/-client -verbose, without the "-debug" option.

From problem host to primary:
bpclntcmd -pn

Usage:
Linux: /usr/openv/netbackup/bin/bpclntcmd -pn -verbose -debug

Windows: cd [install path]\NetBackup\bin
                 bpclntcmd -pn -verbose -debug

14

Manually create and deploy a Certificate:

The instructions below are for a Windows Client, and a Linux Primary Server.  If both sides are either Windows or Linux, the copy/paste steps will not be needed, instead, copy the request and certificate txt files between the systems.

1. Create a certificate request file on the client using nbcertcmd:
nbcertcmd -createCertRequest -requestFile C:\request.txt -server PRIMARYSERVER

2. Open C:\request.txt in notepad, and copy the contents.

3. On the master, let's make a new file:
vi /tmp/request.txt
i (enter insert mode)

4. Paste the contents from C:\request.txt on the client server into vi, and then esc, :wq to save it.

5. Create a reissue token w/ GUI (token management) or cmd on Master Server
[GUI]
OR
nbcertcmd -createtoken -name support -reissue -host CLIENTNAME

Note: If the host has never had a certificate, generate a normal install token.

6. On the master, run the following:
nbcertcmd -signCertificate -requestFile /tmp/request.txt -certificateFile /tmp/cert.txt -token ####
The token key will be required from the step before, enter when prompted.

7. Run "cat /tmp/cert.txt" and copy the output

8. Make a new text file called C:\cert.txt and open it in Notepad.

9. Paste the output from /tmp/cert.txt on the master and save the new C:\cert.txt on the client.

10. Run:
nbcertcmd -deployCertificate -certificateFile C:\cert.txt

11. Run:
nbcertcmd -getcrl