Multiple ungraceful reboots fails to start EMM on Master Server with Errors "Database [NBDB] is not available"

Problem

Multiple ungraceful shutdown and reboots may cause EMM Database not to start on the Master Server.

Error Message

The /usr/openv/db/bin/nbdb_ping command shows error: Database [NBDB] is not available"

The /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts command hangs.

The /usr/openv/db/log/server.log shows no noticeable errors.

The /usr/openv/netbackup/logs/nbdb log shows the following errors:

20:08:11.619 [14387] <4> nbdb_admin: Entering.
20:08:11.621 [14387] <4> nbdb_admin: SQLANY = /usr/openv/db/
20:08:11.621 [14387] <4> nbdb_admin: SQLANY11 = /usr/openv/db/
20:08:11.621 [14387] <4> nbdb_admin: LD_LIBRARY_PATH = /usr/openv/db//lib:/usr/openv/db//bin:/usr/openv/lib:/usr/openv/db/bin
20:08:11.621 [14387] <4> nbdb_admin: SATMP = /usr/openv/tmp/sqlany
20:08:11.628 [14387] <4> create_util_db_file: Creating /usr/openv/db/bin/util_db.ini
20:08:11.914 [14387] <16> NBDBsystem: System call failed with status: 2304
20:08:11.915 [14387] <16> nbdb_admin: Error stopping ASA database server: NB_server1.
20:08:11.915 [14387] <4> nbdb_admin: Removing /usr/openv/db/bin/util_db.ini
20:08:11.915 [14387] <4> nbdb_admin: Exiting with rc = 15
20:09:07.012 [14514] <4> nbdb_ping: Entering.
20:09:07.012 [14514] <4> nbdb_ping: nbdb_ping
20:09:07.013 [14514] <4> nbdb_ping: SQLANY = /usr/openv/db/
20:09:07.014 [14514] <4> nbdb_ping: SQLANY11 = /usr/openv/db/
20:09:07.014 [14514] <4> nbdb_ping: LD_LIBRARY_PATH = /usr/openv/lib:/usr/openv/db/bin
20:09:07.014 [14514] <4> nbdb_ping: SATMP = /usr/openv/tmp/sqlany-------------------------------------------
...
20:09:07.018 [14514] <4> ping_database: Entering.
20:09:07.102 [14514] <4> ping_database: ODBC connection string=CS=utf8;UID=dba;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_server1;DBN=NBDB
20:09:07.102 [14514] <16> ping_database: ODBC connection failed.
20:09:07.102 [14514] <16> ping_database: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Connection error: Unable to initialize requested communication links, ErrCode -1, Sqlstate 08001
20:09:07.103 [14514] <4> ping_database: Exiting. rc = 25
20:09:07.103 [14514] <4> nbdb_ping: Database [NBDB] is not available.
20:09:07.103 [14514] <4> nbdb_ping: Exiting with rc = 25
20:15:21.666 [14712] <4> nbdb_admin: Entering.

Cause

This issue can be caused due to cached socket files under /usr/openv/tmp/sqlany/__SQLAny__, which is used to set up communication for Sybase. This was generated from a previous session that was not properly cleaned up due to abrupt failures that did not allow graceful Sybase shutdown.

Solution

Check and confirm the configuration in the following files:

cat /etc/hosts
cat /usr/openv/netbackup/bp.conf
cat /usr/openv/var/global/server.conf
cat /usr/openv/var/global/databases.conf
cat /usr/openv/db/bin/servername
cat /usr/openv/db/data/vxdbms.conf

Check and confirm the database server log does not show any noticeable errors.

cat /usr/openv/db/log/server.log

If all the above files look good, then follow the steps below:

1. Enable / Check nbdb log file (user.mmddyy_sssss.log) under /usr/openv/netbackup/logs/nbdb/ for errors, such as:
   • "NBDBsystem: System call failed with status: 2304"
     • Example:
       grep -i "System call failed" /usr/openv/netbackup/logs/nbdb/root.052815_00001.log​​​​​​​
   • "ping_database: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Connection error: Unable to initialize requested communication links, ErrCode -1, Sqlstate 08001"
     • Example:
       grep -i "ping_database" /usr/openv/netbackup/logs/nbdb/root.052815_00001.log
 
2. Stop NetBackup services first, then stop PBX service:
   /usr/openv/netbackup/bin/goodies/netbackup stop
/opt/VRTSpbx/bin/vxpbx_exchanged stop
 
3. Check the following directory to see if it contains large number of .tmp files:
   ls -lah /usr/openv/tmp/sqlany
 
4. If it does a large number of .tmp files...
   • Rename or remove the sqlany directory:
     mv sqlany sqlany.bak
   • Create a new sqlany directory:
     mkdir /usr/openv/tmp/sqlany
 
5. Start PBX service first, then start NetBackup services:
   /opt/VRTSpbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup start
 
6. Verify that the following commands return good results indicating that the EMM database is up:
   /usr/openv/db/bin/nbdb_ping
/usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts

Applies To

NetBackup 7.x, 8.x