How to troubleshoot NetBackup Web Management Console (NBWMC) Failure 

Description

This article will provide an overview of how to troubleshoot issues with the NetBackup Web Management Console (NBWMC) not starting, or not responding correctly. 

Potential causes can include any of the prerequisite systems (primarily PBX, VNETD, NBDB, NBAZDB, NBATD, JRE, DNS/name resolution) not working correctly, permissions not being set correctly, firewall issues, or underlying broken configuration of the certificate system. 

Below are some of the symptoms that may be observed. Keep in mind that some of these symptoms can be the result of different issues and may not be covered in this document:

• Cannot access Java GUI 
• Primary server down 
• Backups are failing  
• NBWMC is not starting or NetBackup Web Services not starting 
• NetBackup Web Management Console not starting 

 

1. Verify that PBX is up and running.

For Linux: Run the bpps -x command, pbx usually appears at the bottom:

• root     1047100       1  0 11:53 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange 

For Windows: Open the tasklist and search for pbx_exchange.exe:

• pbx_exchange.exe             16732 Services                   0     10,380 K     

 

2. If it is not running, attempt to start PBX. If it is running, go to step 3. 

• For Linux: /opt/VRTSpbx/bin/vxpbx_exchanged start 
• For Windows: net start vrtspbx 
  • Or start graphically from services.msc > Veritas Private Branch Exchange >  Start 

 

3. Verify that VNETD is running properly: (there should be at least 5 on a Primary server, possibly more if jobs are active).

If it is running skip to step 5.

For Linux: /usr/openv/netbackup/bin/bpps -x | grep vnetd 

Example output: 

• root     1047122       1  0 11:53 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone 
• root     1047173       1  0 11:53 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -proxy inbound_proxy -number 0 
• root     1047179       1  0 11:53 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -proxy outbound_proxy -number 0 
• root     1047181       1  0 11:53 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -proxy http_pbx_tunnel -number 0 
• root     1047184       1  0 11:53 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -proxy http_api_tunnel -number 0

For Windows: 

cd [install path]\NetBackup\bin 

bpps | findstr vnetd 

• vnetd           15580    0.000%            1.296   14M   3/21/23 10:56:33.388 
• vnetd             452    0.000%            0.281   21M   3/21/23 10:56:33.486 
• vnetd            9044    0.000%            0.281   21M   3/21/23 10:56:33.501 
• vnetd            2364    0.000%            1.703   23M   3/21/23 10:56:33.513 
• vnetd           16484    0.000%            0.234   19M   3/21/23 10:56:33.528 

 

4.  Stop and re-start vnetd (or all of NetBackup): 

For Linux: 

• /usr/openv/netbackup/bin/vnetd -terminate 
• /usr/openv/netbackup/bin/vnetd -standalone 

Or: 

• /usr/openv/netbackup/bin/goodies/netbackup stop 
• /usr/openv/netbackup/bin/goodies/netbackup start 

For Windows: 

• cd [install path]\NetBackup\bin 
• vnetd –terminate 
• vnetd –standalone 

Or: 

• cd [install path]\NetBackup\bin 
• bpdown -v -f 
• bpup -v –f 

 

5. Verify that NBDB (8.x.x - 10.1.1) or PostgreSQL (10.2+) is running:

For Linux: 

• 10.2 and later: /usr/openv/netbackup/bin/bpps -x | grep postgres

• 10.1.1 and earlier: /usr/openv/netbackup/bin/bpps -x | grep dbsrv

Example output: 

root     1047292       1  1 11:53 ?        00:03:30 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 5 

For Windows: 

cd [install path]\NetBackup\bin 

• 10.2 and later: bpps | findstr postgres

• 10.1.1 and earlier: bpps | findstr dbsrv 

dbsrv17         11984    0.000%           19.218 1097M   3/21/23 10:56:26.941 

If NBDB is not running, troubleshoot the NBDB issues and once resolved, check NBWMC functionality after restarting NetBackup services. 

 

6. Verify that NBDB is responding:

For Linux:  /usr/openv/db/bin/nbdb_ping 

Database [NBDB] is alive and well on server [NB_nbmaster2] 

For Windows: 

cd [install path]\NetBackup\bin 

nbdb_ping 

Database [NBDB] is alive and well on server [NB_nbmaster1] 

If NBDB is not responding, troubleshoot NBDB issues and once resolved, check NBWMC functionality after restarting NetBackup services. 

 

7. Verify that NBATD is running: 

For Linux: /usr/openv/netbackup/bin/bpps -x | grep nbatd 

root     1047350       1  0 11:53 ?        00:00:01 /usr/openv/netbackup/bin/private/nbatd -c /usr/openv/var/global/vxss/eab/data 

For Windows: 

cd [install path]\NetBackup\bin 

bpps | findstr nbatd 

nbatd           12064    0.000%            2.625   16M   3/21/23 10:56:30.49

If NBATD is running, skip to step 9.

 

8. Stop & start NBATD (or all NetBackup services) and see if it starts:

For Linux: 

kill –9 1047350 

(using the PID found in the previous step) 

/usr/openv/netbackup/bin/nbatd 

Or: 

/usr/openv/netbackup/bin/goodies/netbackup stop 

/usr/openv/netbackup/bin/goodies/netbackup start 

For Windows: 

Taskkill /F /PID 12064 

(using the PID found in the previous step) 

cd [install path]\NetBackup\bin 

nbatd 

Or: 

cd [install path]\NetBackup\bin 

bpdown -v -f 

bpup -v -f 

If it will not start, attempt to follow: Netbackup Authentication Service (nbatd) is not starting and is not generating any logs

After troubleshooting and resolving NBATD issues, check NBWMC functionality after restarting NetBackup services. 

 

9. Verify that NBATD is working/responding correctly (Optional: There is no need to check this right away if it's running) 

For Linux: 

Option 1 (Note, you may need to let all services try to start for this to work): 

/usr/openv/netbackup/bin/bpnbat -login -logintype AT 

Option 2 (Note, this issues a renewed AT certificate, and requires that you stop & restart NetBackup services after running it): 

/usr/openv/netbackup/bin/admincmd/bpnbaz -provisioncert mastername 

Option 3 (Note, this will reconfigure the AT broker certificates for nbatd on the master, this will require a stop & restart of NetBackup services after running): 

/usr/openv/netbackup/bin/admincmd/bpnbaz -configureauth -force 

For Windows:

Option 1: 

cd [install path]\NetBackup\bin 

bpnbat -login -logintype AT 

Option 2 (Note, this issues a renew AT certificate, and requires that you stop & restart NetBackup services after running it): 

cd [install path]\NetBackup\bin\admincmd 

bpnbaz -provisioncert mastername 

Option 3 (Note, this will reconfigure the AT broker certificates for nbatd on the master, this will require a stop & restart of NetBackup services after running): 

cd [install path]\NetBackup\bin\admincmd 

bpnbaz -configureauth -force 

 

10. Verify that NBAZDB is alive and responding.

Linux: /usr/openv/db/bin/nbdb_ping -dbn NBAZDB 

Database [NBAZDB] is alive and well on server [NB_nbmaster2]. 

Windows: 

cd [install path]\NetBackup\bin 

nbdb_ping -dbn NBAZDB 

Database [NBAZDB] is alive and well on server [NB_nbmaster1]. 

 If NBAZDB is alive and responding, skip to step 13.

 

11. Verify that NBAZDB is set to start (valid for 8.x - 10.1.1 only, databases.conf is not used by PostgreSQL)

Check databases.conf and make sure it has a line for NBAZDB 

Linux: cat /usr/openv/var/global/databases.conf 

"/usr/openv/db/data/NBDB.db" -n NBDB 

“/usr/openv/db/data/NBAZDB.db" -n NBAZDB 

Windows: 

cd [install path]\NetBackupDB\CONF\ 

type databases.conf 

"C:\Program Files\Veritas\NetBackupDB\data\NBDB.db" -n NBDB 

"C:\Program Files\Veritas\NetBackupDB\data\NBAZDB.db" -n NBAZDB 

If the NBAZDB line is present, skip to step 12 

If the NBAZDB line is not present, add it with nbdb_admin -auto_start NBAZDB 

Linux: 

/usr/openv/db/bin/nbdb_admin -auto_start NBAZDB 

Successfully added NBAZDB to databases.conf. 

Windows:

cd [install path]\NetBackup\bin 

nbdb_admin -auto_start NBAZDB 

Successfully added NBAZDB to databases.conf. 

Stop and Restart NetBackup services, then try pinging with: 

Linux: /usr/openv/db/bin/nbdb_ping -dbn NBAZDB 

Database [NBAZDB] is alive and well on server [NB_nbmaster2]. 

Windows: 

cd [install path]\NetBackup\bin 

nbdb_ping -dbn NBAZDB 

Database [NBAZDB] is alive and well on server [NB_nbmaster1]. 

If NBAZDB is responding, skip to step 13

If NBAZDB is still not responding, continue to step 12

 

12. If NBAZDB is set to start, but still doesn’t start up, it can be recovered from staging or a catalog backup.

• Recover from staging: 

Linux: /usr/openv/db/bin/nbdb_restore -recover -staging -dbn NBAZDB 

Windows: 

cd [install path]\NetBackup\bin 

nbdb_restore -recover -staging -dbn NBAZDB 

If NBAZDB is not available in staging, use the BAR GUI in the Java Admin Console to restore NBAZDB.db from the staging folder in the catalog backup, and recover it to the staging folder. Then perform the above. 

If NBAZDB is not recoverable from either staging or catalog backup, a new DB can be created. See: During Catalog backup, staging for NBAZDB is failing (The cause may be different, but the solution is valid) 

 

13. Verify that NBWMC is starting

Run bpps and look for Dnop following the "java" command on Linux, or on Windows look for nbwmc.exe:

Linux: /usr/openv/netbackup/bin/bpps -x | grep Dnop 

nbwebsvc 1054407       1  9 19:25 ?        00:06:11 /usr/openv/java/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dnbwmc -d64 -Xms4096m -Xmx4096m -XX:MetaspaceSize=192m -XX:NewRatio=1 -XX:SurvivorRatio=4 -XX:-UseAdaptiveSizePolicy -Djava.library.path=/usr/openv/lib:/usr/openv/java -DNB_HOSTNAME=nbmaster2.nbulab -DNB_INSTALL_DIR=/usr/openv -DVAR_GLOBAL=/usr/openv/var/global -Djava.util.logging.config.file=/usr/openv/wmc/config/tomcat.logging.properties -Dlog.conf.path=/usr/openv/netbackup/nblog.conf -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dwebsvc.group=nbwebgrp -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dcom.safelogic.cryptocomply.fips.approved_only=false -verbose:jni -Djava.security.egd=file:/dev/./urandom -XX:ErrorFile=/usr/openv/wmc/webserver/logs/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/openv/wmc/webserver/logs -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/usr/openv/wmc/webserver/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/openv/wmc/webserver/bin/bootstrap.jar:/usr/openv/wmc/webserver/bin/tomcat-juli.jar -Dcatalina.base=/usr/openv/var/global/wsl/webserver -Dcatalina.home=/usr/openv/wmc/webserver -Djava.io.tmpdir=/usr/openv/wmc/webserver/temp org.apache.catalina.startup.Bootstrap start 

  Windows: 

cd [install path]\NetBackup\bin

bpps | findstr nbwmc 

nbwmc           10372    0.000%            0.203   13M   3/21/23 18:35:10.961 

If NBWMC is starting, but not working, skip to step 15

If NBWMC is not starting, continue to step 14.

 

14. Verify that Java/JRE is functioning correctly:

Run the keytool. When the command is successful there is help dialog, otherwise there will be java errors.

Linux: /usr/openv/java/jre/bin/keytool 

Windows: 

cd [install path]\NetBackup\jre\bin 

Keytool 

Example Output (note that the exact output isn’t important, only that it returns the help list rather than an error): 

• Key and Certificate Management Tool 
• Commands:  
• -certreq            Generates a certificate request 
• -changealias        Changes an entry's alias 
• -delete             Deletes an entry 
• -exportcert         Exports certificate 
• -genkeypair         Generates a key pair 
• -genseckey          Generates a secret key 
• -gencert            Generates certificate from a certificate request 
• -importcert         Imports a certificate or a certificate chain 
• -importpass         Imports a password 
• -importkeystore     Imports one or all entries from another keystore 
• -keypasswd          Changes the key password of an entry 
• -list               Lists entries in a keystore 
• -printcert          Prints the content of a certificate 
• -printcertreq       Prints the content of a certificate request 
• -printcrl           Prints the content of a CRL file 
• -storepasswd        Changes the store password of a keystore 
• Use "keytool -command_name -help" for usage of command_name 

If Java/JRE is not responding correctly on Linux and it has been recently upgraded, see After upgrading JRE to 8u333, 8u341 or 8u351 on a master server, web services do not start for instructions on how to revert JRE to the previous version.

If JRE is not responding correctly, this must be investigated and repaired. Once JRE is responding correctly, stop/restart NetBackup services and check NBWMC functionality again. 

 

15. Check the wmc/webserver/logs/catalina.yyyy-mm-dd.log

• In the below examples, YYYY-MM-DD is used instead of the current date. Please replace it with the actual date, for example 2025-01-25. 
• If catalina.yyyy-mm-dd.log either does not exist, or has not been written to for one or more startup attempts, skip to step 16.
• Also C:\Program Files\Veritas\ is used as the NetBackup install path, but adjust as needed since the installation path may vary. 
• Note that for both Windows and Linux, the commands below are each a single long command, not multiple lines. 

Option 1 – Command line: 

Linux method: 

awk '/Server version/{ chunk="" } /SEVERE/{ logme=(chunk=="") } logme{ chunk=chunk $0 RS } END{ printf "%s", chunk }' /usr/openv/wmc/webserver/logs/catalina.YYYY-MM-DD.log 

Windows Powershell Method:

$linesOfInterest= @();$serverLineFound = $false;$severeCount = 0;switch -Regex -CaseSensitive -File "C:\Program Files\Veritas\NetBackup\wmc\webserver\logs\catalina.YYYY-MM-DD.log" {'\bServer version\b' {;$serverLineFound = $true;$severeCount = 0;continue;};'\bSEVERE\b' {;if (-not $serverLineFound -or ++$severeCount -gt 1) { continue };$linesOfInterest = @($_);continue;};default {;if ($severeCount -eq 1) { $linesOfInterest += $_ };};};$linesOfInterest 

Option 2 – Notepad/less - For use if copying the awk or PowerShell commands isn’t working correctly: 

Linux: 

less catalina.YYYY-MM-DD.log

G 

?Server\ version 

/SEVERE 

Note: Read lines including the first match for SEVERE from this point, and stop reading when getting to the next instance of SEVERE.

Windows: 

From an admin cmd prompt: 

notepad "C:\Program Files\Veritas\NetBackup\wmc\webserver\logs\catalina.YYYY-MM-DD.log" 

then in notepad 

Ctrl + End (to go to the bottom) 

Ctrl + f (open search) 

search for  

Server version 

and change search direction to up, press find only once, we want the LAST startup 

leave find dialog open, and change search direction to Down, and search for 

SEVERE  

and press find only once 

Note: Read lines including the first match for SEVERE from this point, and stop reading when getting to the next instance of SEVERE.

Note: If no SEVERE error is found, but there is a INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [#####] milliseconds: 

This indicates that nbwmc is starting, so check hostname resolution and DNS, as this is not an nbwmc issue.

If SEVERE errors are seen, these errors should be researched and resolved to continue.

 

16. If catalina.yyyy-mm-dd.log either does not exist, or has not been written to for one or more startup attempts: 

This is most likely a permissions issue. Possible fixes include using setupWmc to set up ownership/permissions on wmc and var/global/wsl directories.

Run setupWmc to try to fix the permissions with the following command:

Linux: /usr/openv/wmc/bin/install/setupWmc 

Windows: 

cd [install path]\NetBackup\wmc\bin\install 

setupWmc 

 

17. Additional steps if NBWMC is starting, but not responding:

A. Run the nbcertcmd -ping command to find additional errors:

Linux:  /usr/openv/netbackup/bin/nbcertcmd -ping 

Windows:  

cd [install path]\NetBackup\bin 

nbcertcmd -ping 

Common nbcertcmd -ping errors 

• 26 - nbwmc is broken, not running or there is a firewall issue 
• 41 - Indicates there is a hostname resolution issue (41 is a network timeout).
• 4-digit error codes (8500, 5969, etc) indicate that nbwmc is partially working, check the knowledge base for the specific error.

B. Perform a "mini-reset" of the NetBackup certificates: How to perform automatic mini reset of NetBackup certificates using configureCerts script

If the issue continues, please take note of the results of any of the checks in this document that failed to assist technical support in resolving the problem.