The NetBackup Web Management Console will not start or stay running

Problem

After upgrading NetBackup from 7.x to 8.1.1 or 8.2, the NetBackup Web Management Console (nbwmc) service fails to start and stay running.

Error Message

The tomcat (nbwmc) log file shows the following:
Located here: /usr/openv/wmc/webserver/logs/catalina.<date>.log 

27-Aug-2018 00:03:18.442 INFO [localhost-startStop-2] com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource. Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> 2sjifg9xxq4p29xfs9t5|33eb4f1a, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.sybase.jdbc4.jdbc.SybDriver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> 2sjifg9xxq4p29xfs9t5|33eb4f1a, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:sybase:Tds:localhost:13785?ServiceName=NBDB, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 3600, maxIdleTimeExcessConnections -> 600, maxPoolSize -> 10, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 3, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ]
27-Aug-2018 00:03:48.579 WARNING [C3P0PooledConnectionPoolManager[identityToken->2sjifg9xxq4p29xfs9t5|33eb4f1a]-HelperThread-#0] com.mchange.v2.resourcepool.BasicResourcePool. com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@b575cef -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
 java.sql.SQLException: JZ00L: Login failed.  Examine the SQLWarnings chained to this exception for the reason(s).
    at com.sybase.jdbc4.jdbc.ErrorMessage.raiseError(ErrorMessage.java:768)
    at com.sybase.jdbc4.tds.Tds.processLoginAckToken(Tds.java:5356)
    at com.sybase.jdbc4.tds.Tds.doLogin(Tds.java:725)
    at com.sybase.jdbc4.tds.Tds.login(Tds.java:579)
    at com.sybase.jdbc4.jdbc.SybConnection.tryLogin(SybConnection.java:422)
    at com.sybase.jdbc4.jdbc.SybConnection.handleHAFailover(SybConnection.java:3335)
    at com.sybase.jdbc4.jdbc.SybConnection.<init>(SybConnection.java:348)
    at com.sybase.jdbc4.jdbc.SybConnection.<init>(SybConnection.java:253)
    at com.sybase.jdbc4.jdbc.SybDriver.connect(SybDriver.java:232)
    at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
    at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
    at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
    at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
    at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
    at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
    at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
    at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
    at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

Cause

This condition has been known to occur when the password which Tomcat caches and subsequently uses to connect to the NetBackup ASA database is incorrect.

Solution

The solution is to update the password cache which Tomcat uses to connect to the NetBackup database.

To update the cached password, run the following:

Unix:  /usr/openv/db/bin/nbdb_admin -dba <newPassword>
Windows: <install_path>\NetBackup\bin\nbdb_admin -dba <newPassword>

The above command will allow a NetBackup Administrator to set the password to the NBDB database.  Upon doing this, the password cache for Tomcat is refreshed with the new password.  You will be prompted to restart nbwmc upon the success of the command.

Example:

[root@MyMaster~]# nbdb_admin -dba MySuperSecretPassword12

WARNING: After changing the DBA password, Veritas may not be able to recover
any information saved within the EMM database if the password is unavailable.
A secondary emergency backup copy of this password should be maintained.
If you do not maintain a corporate mechanism to escrow passwords,
Veritas recommends making a backup copy of the password on a local hard drive.
This file will contain the password in clear text and access to the file
will be controlled by the file system permissions.

Do you wish to create a backup file of your new DBA password? (y/n) y
Please specify the directory for the backup file (/usr/openv/db/data):

The file /usr/openv/db/data/nbdbinfo.dat has been created.
This will require restart of NetBackup Web Management Console service.
Change of dba password for NBDB successful.