The host does not have the NetBackup Host ID-based security certificate installed is displayed after an upgrade to NetBackup 8.0.
Problem
After an upgrade to NetBackup 8.0 the NetBackup Administration console produces a certificate error and the user is unable to login. Also noted is that the NetBackup Web Management Console service stops after a period of time.Error Message
The host WINREF does not have NetBackup Host ID-based security certificate installed. The certificate is mandatory to establish a secure connection.
Note: The host WINREF is not the name of the master server and is used for illustration.
The NetBackup Tomcat web service "NetBackup Web Management Console" (nbwmc) does not fully initialize.
The tomcat log
Windows:
<installdrivepath>\veritas\netbackup\wmc\webserver\logs\catalina.yyyy-mm-dd.log
Unix:
/usr/openv/wmc/webserver/logs/Catalina.yyyy-mm-dd.log
shows the following errors upon start up.
NOTE: Some of the log content has been truncated for clarity.
10-Jul-2017 10:18:57.988 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class com.netbackup.common.web.config.OrderedServletContextListener
java.lang.RuntimeException: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection
at com.netbackup.webapp.gateway.rest.config.IntraWebappJwtLoader.contextInitialized(IntraWebappJwtLoader.java:56)
at com.netbackup.common.web.config.OrderedServletContextListener.contextInitialized(OrderedServletContextListener.java:21)
...
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection
at org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:544)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
... 10 more
Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
... 18 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
... 21 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
... 26 more
10-Jul-2017 10:18:57.988 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
10-Jul-2017 10:18:57.988 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.startInternal Context [/netbackup] startup failed due to previous errors
10-Jul-2017 10:18:58.004 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
10-Jul-2017 10:19:01.035 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.netbackup.security.config.Initializer@7f37af9c]
10-Jul-2017 10:19:01.207 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext
10-Jul-2017 10:20:03.268 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'certificateServiceImpl': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.netbackup.security.property.service.PropertyService com.netbackup.security.certificate.service.CertificateServiceImpl.propertyService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'propertyServiceImplementation': Invocation of init method failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
Cause
The problem is noted when the NBDB is not the first database listed in the master server databases.conf file.Example Windows:
Path:
C:\Program Files\Veritas\NetBackupDB\CONF\databases.conf
File contents:
"C:\Program Files\VERITAS\NetBackupDB\data\BMRDB.db" -n BMRDB
"C:\Program Files\VERITAS\NetBackupDB\data\NBDB.db" -n NBDB
"C:\Program Files\VERITAS\NetBackupDB\data\NBAZDB.db" -n NBAZDB
Note: In this example the master server install is on the C:\ drive.
Example Unix/Linux:
Path:
/usr/openv/var/global/databases.conf
File contents:
"/usr/openv/db/data/BMRDB.db" -n BMRDB
"/usr/openv/db/data/NBDB.db" -n NBDB
"/usr/openv/db/data/NBAZDB.db" -n NBAZDB
Note: In this example NBDB is installed in the default path, /usr/openv/db/data
Solution
Modify the database order in the databases.conf file.1. Stop all NetBackup services.
Windows: C:\Program Files\Veritas\NetBackup\bin\bpdown -f -v
Unix/Linux: /usr/openv/netbackup/bin/goodies/netbackup stop
2. Using a text editor (notepad.exe for example) change the database order and move the NBDB record to the top of the databases.conf file.
Correct order from the previous example, Windows:
"C:\Program Files\VERITAS\NetBackupDB\data\NBDB.db" -n NBDB
"C:\Program Files\VERITAS\NetBackupDB\data\BMRDB.db" -n BMRDB
"C:\Program Files\VERITAS\NetBackupDB\data\NBAZDB.db" -n NBAZD B
Correct order from the previous example, Unix/Linux:
"/usr/openv/db/data/NBDB.db" -n NBDB
"/usr/openv/db/data/BMRDB.db" -n BMRDB
"/usr/openv/db/data/NBAZDB.db" -n NBAZDB
2a. Save the file but not as a .txt file.
Ensure the file name remains as databases.conf.
3. Start all NetBackup services.
Windows:
C:\Program Files\Veritas\NetBackup\bin\bpup -f -v
Unix/Linux:
/usr/openv/netbackup/bin/goodies/netbackup start
4. If the security certificate error is presented after successful NetBackup start, manually create the host ID-based certificate.
Step 1: Establish a trust with the master server by running:
nbcertcmd -getCACertificate
Windows:
C:\Program Files\Veritas\NetBackup\bin\
Unix/Linux:
/usr/openv/netbackup/bin
Step 2: Get the certificate by running the following from the same path:
nbcertcmd -getCertificate