Unable to start the NetBackup Enterprise Media Manager service. Server.log file reports an error "File is shorter than expected."

Problem

The NetBackup Enterprise Media Manager is not coming up.

Error Message

Error in the database server log (/usr/openv/db/log/server.log) shows:

11/01 03:48:45.     Transaction log: NBDB.log...
11/01 03:48:45. *** ERROR *** Assertion failed: 201129 (11.0.1.2222)
File is shorter than expected

Cause

The NetBackup Enterprise Media Manager (nbemm) service fails to start due to a corrupt NBDB transaction log file (NBDB.log) as seen in the server.log file.

Solution

Steps to remove the corrupt transaction log and recreate a new one.

Warning: There may be some transactions that could be lost.

1. Stop NetBackup services if it is not already stopped:
   /usr/openv/netbackup/bin/goodies/netbackup stop
   or
   /etc/init.d/netbackup stop
2. Remove NBDB from auto_start option:
   /usr/openv/db/bin/nbdb_admin -auto_start NONE
3. Start the ASA database engine:
   /usr/openv/db/bin/nbdbms_start_server
4. To add the necessary environment variables to your shell, run the command:
   /usr/openv/db/vxdbms_env.sh
   • Note: If using 'csh' for a shell, the command is 'source /usr/openv/db/vxdbms_env.csh' without the quotes.
5. Change directories to where the NBDB resides:
   cd /usr/openv/db/data
6. Rename the bad transaction log:
   mv /usr/openv/db/data/NBDB.log /usr/openv/db/data/NBDB.log_old
7. To force database recovery, while still in the /usr/openv/db/data directory, run the command:
   /usr/openv/db/bin/ dbsrv11 -f /usr/openv/db/data/NBDB.db
   • Note 1: This will force start that database and stop it right away.
   • Note 2: The NBDB.log will not be created until step 10.
   • Attention: You must be logged in as 'root' to perform library path settings options, as 'sudo' will not suffice.
   • If you receive an error (:: ld.so.1: dbsrv11: fatal: libdbserv11_r.so: open failed: No such file or directory Killed), you will need to follow the steps as below:
     • A. Verify that the library libdbserv11_r.so resides in /usr/openv/db/lib path. Verify the library links exist by running the command ldd.
       cd /usr/openv/db/bin
ldd dbeng11
       • If ldd returns:
         libdbserv11_r.so =>       (file not found)
libdbtasks11_r.so =>      (file not found)
       • Run:
         echo $LD_LIBRARY_PATH
LD_LIBRARY_PATH=/usr/openwin/lib
       • If the LD_LIBRARY_PATH does not include /usr/openv/db/lib, modify the LD_LIBRARY_PATH to include /usr/openv/db/lib/:
         LD_LIBRARY_PATH=/usr/openwin/lib:/usr/openv/db/lib/
       • Verify that the LD_LIBRRARY_PATH now includes /usr/openv/db/lib using the echo command:
         echo $LD_LIBRARY_PATH
     • B. Again, try to run:
       /usr/openv/db/bin/dbsrv11 -f /usr/openv/db/data/NBDB.db
       • If you still get the same error (:: ld.so.1: dbsrv11: fatal: libdbserv11_r.so: open failed: No such file or directory Killed), you need to export the path
     • C. To export the path, run:
       export LD_LIBRARY_PATH=/usr/openwin/lib:/usr/openv/db/lib/
cd /usr/openv/db/bin
ldd dbeng11
       • ldd should return an output similar to this:
         libdbserv11_r.so =>      /usr/openv/db/lib//libdbserv11_r.so
libdbtasks11_r.so =>     /usr/openv/db/lib//libdbtasks11_r.so
     • D. Try to run:
       /usr/openv/db/bin/dbsrv11 -f /usr/openv/db/data/NBDB.db
       or
       /usr/openv/db/bin/dbsrv11 -f NBDB
       • Example output should look like this:
         SQL Anywhere Personal Server Version 11.0.1.2632
OEM Authenticated Edition, licensed only for use with authenticated OEM applications.
Copyright (c) 2011, iAnywhere Solutions, Inc.
Portions copyright (c) 2011, Sybase, Inc. All rights reserved.
Use of this software is governed by the Sybase License Agreement. Refer to https://www.sybase.com/softwarelicenses
1 physical processor(s) detected.
Processor limit (Personal Server): 1
Maximum number of physical processors the server will use: 1
This server is licensed to:
NetBackup
Veritas Corporation
Running SunOS 5.10 Generic_142909-17 on SPARC
Server built for SPARC processor architecture
8192K of memory used for caching
Minimum cache size: 8192K, maximum cache size: 2355616K
Using a maximum page size of 2048 bytes
Cannot access "NBDB": file does not exist
Database server shutdown due to startup error
Database server stopped at Tue Nov 01 2011 18:07
   • Note: If the dbeng11 -f NBDB should fail with the error "A database server with that name has already started"...
     Please check /usr/openv/tmp and move the sqlany directory to different name.
     Also, move all the sqlany files to different names and restart the database server on step 3 above.
     Then the dbeng11 -f NBDB should work.
8. Stop the database by running the command:
   /usr/openv/db/bin/nbdbms_start_server -stop
9. Add NBDB to auto_start option:
   /usr/openv/db/bin/nbdb_admin -auto_start NBDB
10. To start the database, run the command:
    /usr/openv/db/bin/nbdbms_start_server
    • Note: If the command is stuck for a long time then interrupt the command with Ctrl+C and try running:
      /usr/openv/db/bin/dbsrv11 -f /usr/openv/db/data/NBDB.db
/usr/openv/db/bin/nbdbms_start_server
11. To test if the database is operational, run the command:
    /usr/openv/db/bin/nbdb_ping
    • Which should return output similar to the following:
      Database [NBDB] is alive and well on server [NB_master1].