Warning: NetBackup web services could not start on this host appears during the upgrade to NetBackup 8.0.

Problem

This condition has many symptoms.

1. The upgrade to NetBackup 8.0 is successful and the NetBackup Web Management Console services start but later may stop.
2. The Administrator is unable to open the NetBackup Administration Console because the certificates didn't generate.
3. Attempting to generate certificates will fail.
 

Error Message

During the upgrade to NetBackup 8.0 the pop-up message, WARNING: NetBackup web services could not start on this host, is presented. Additionally the error, The host MASTER does not have NetBackup Host ID-based security certificate installed, is displayed when opening the NetBackup 8.0 Administration Console. Last, generating the Host ID-based certificate will fail.

1. During the upgrade the error, NetBackup web services could not start on this host is displayed.

2. Opening the NetBackup Administration Console error:



3. Generating the Host ID-Based certificate error:

<install_path>\Veritas\NetBackup\bin>nbcertcmd -getCACertificate

/usr/openv/netbackup/bin/nbcertcmd -getCACertificate

nbcertcmd: The -getCACertificate operation failed for server <hostname>.
EXIT STATUS 41: network connection timed out



The integrated Tomcat server wrapped within the NetBackup Web Management Console service produces the following error upon startup:
NOTE: Log located here:

<install_path>\Veritas\NetBackup\wmc\webserver\logs\catalina.yyyy-mm-dd.log

/usr/openv/wmc/webserver/logs/Catalina.yyyy-mm-dd.log

17-Feb-2017 13:30:11.390 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
17-Feb-2017 13:32:10.789 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class com.netbackup.common.web.config.OrderedServletContextListener
 java.lang.RuntimeException: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection
    at com.netbackup.webapp.gateway.rest.config.IntraWebappJwtLoader.contextInitialized(IntraWebappJwtLoader.java:56)
    at com.netbackup.common.web.config.OrderedServletContextListener.contextInitialized(OrderedServletContextListener.java:21)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4810)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5255)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection
    at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:544)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
    at com.sun.proxy.$Proxy156.createIntraWebappToken(Unknown Source)
    at com.netbackup.webapp.gateway.rest.config.IntraWebappJwtLoader.contextInitialized(IntraWebappJwtLoader.java:53)
    ... 10 more
Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)
    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)
    at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:550)
    at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:450)
    ... 18 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
 

Cause

Prior to the upgrade the server.conf file was missing the port information, e.g. -x tcpip(LocalOnly=YES;ServerPort=13785).
Without establishing the ServerPort=13785, the NetBackup Web Management Console service (nbwmc) cannot connect to the NetBackup Database (NBDB).


Windows Examples:
This is an example of a server.conf file that's missing the port information:
<install_path>\Veritas\NetBackupDB\CONF\server.conf

-c 200M -ch 2730M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_MASTER
-o "C:\Program Files\Veritas\NetBackupDB\log\server.log" -os 1M -ti 0 -wc-

This is an example of a server.conf file that's healthy:
-c 200M -ch 2730M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_MASTER
-o "C:\Program Files\Veritas\NetBackupDB\log\server.log" -os 1M -ti 0 -wc- -x tcpip(LocalOnly=YES;ServerPort=13785)


UNIX Examples:
This is an example of a server.conf file that's missing the port information:
/usr/openv/var/global/server.conf

-c 200M -ch 1277M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_nbmaster2
-o /usr/openv/db//log/server.log -os 1M -ti 0 -ua -ud -wc-

This is an example of a server.conf file that's healthy:
-c 200M -ch 1277M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_nbmaster2
-o /usr/openv/db//log/server.log -os 1M -ti 0 -ua -ud -wc- -x tcpip(LocalOnly=YES;ServerPort=13785)




 

Solution

1. Stop all of the NetBackup services.

2. Add the port information to server.conf.

-x tcpip(LocalOnly=YES;ServerPort=13785)

Example server.conf file with port information:
Windows:   <install_path>\Veritas\NetBackupDB\CONF\server.conf

-c 200M -ch 2730M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_MASTER
-o "C:\Program Files\Veritas\NetBackupDB\log\server.log" -os 1M -ti 0 -wc- -x tcpip(LocalOnly=YES;ServerPort=13785)

Unix:   /usr/openv/var/global/server.conf
-c 200M -ch 1277M -cl 200M -gd DBA -gk DBA -gl DBA -gna 1 -gp 4096 -k -m -n NB_nbmaster2
-o /usr/openv/db//log/server.log -os 1M -ti 0 -ua -ud -wc- -x tcpip(LocalOnly=YES;ServerPort=13785)


3. Start all of the NetBackup Services.

4. Follow the procedure described in Security and Encryption guide: Deploying security certificates on a NetBackup host.
Go to section: To generate and deploy the host ID-based certificates on the host to which the Java GUI is connecting, when token is NOT required. 
Follow Steps 1, 2, and 3.

Note: The default Security Level is High and a token is required when the Security Level is Very High.

To check the security level, execute the following command;

Example:
Windows:  
<install_path>\Veritas\NetBackup\bin>nbcertcmd -getSecConfig -certDeployLevel -server master
Security for certificate deployment : High


Unix: 
/usr/openv/netbackup/bin/nbcertcmd -getSecConfig -certDeployLevel -server nbmaster2
Security for certificate deployment : High

When the security level is High a token is not required.

If the nbcertcmd fails to execute then follow the procedure in the Veritas NetBackup 8.0 Security and Encryption Guide - Web login requirements for nbcertcmd command options.
You'll need to log into the NetBackup Web Management Service (nbwmc) first via ( bpnbat) then execute ( nbcertcmd).

Example:

• To log in to the NetBackup Web Management Service, run the following command on the master server. (nbwmc runs only on the master server.):

Etrack : 3910511