Validation of NBDB fails, causing catalog backup status 2

Problem

Validation is failing intermittently. Running nbdb_admin -validate full is always successful, though running nbdb_backup -online /<staging path> may also fail.

Error Message

A similar message in the bpdbm log may be found:

01:06:20.514 [22046] <4> validate_database: Executing . /usr/openv/db/vxdbms_env.sh && /usr/openv/db/bin/dbvalid -o "/usr/openv/db/log/dbvalid.log" -q -fx -c
 "UID=dba;PWD=******;ENG=validation_eng;LINKS=shmem;START=/usr/openv/db/bin/dbeng11 -c 200M -ch 1024M -cl 200M -r \"/usr/openv/db/staging/NBDB.db\" -ds \"/us
r/openv/db/staging\";LOG=/usr/openv/db/log/dbvalid-connection.log"
01:06:20.761 [22046] <16> NBDBsystem: System call failed with status: 1792
01:06:20.761 [22046] <16> validate_database: Database validation failed for database NBDB
01:06:20.762 [22046] <4> db_error_add_to_file: Database validation failed for database NBDB.
01:06:20.762 [22046] <4> validate_database: Exiting. rc = 11
01:06:20.762 [22046] <16> do_online_nbdb_backup: error validating NBDB backup in /usr/openv/db/staging
 

From dbvalid-connection.log (must be enabled via touch file):

Fri Sep 06 2013 01:06:20
01:06:20 Attempting to connect using:
UID=dba;PWD=********;ENG=validation_eng;START='/usr/openv/db/bin/dbeng11 -c 200M -ch 1024M -cl 200M -r "/usr/openv/db/staging/NBDB.db" -ds "/usr/openv/db/sta
ging"';LOG=/usr/openv/db/log/dbvalid-connection.log;LINKS=shmem;CS=NO
01:06:20 Attempting to connect to a running server...
01:06:20 Trying to start SharedMemory link ...
01:06:20     SharedMemory link started successfully
01:06:20 Attempting SharedMemory connection (no sasrv.ini cached address)
01:06:20 Failed to connect over SharedMemory
01:06:20 No server found, attempting to run START line...
01:06:20 Failed to autostart server
01:06:20 Attempting SharedMemory connection (no sasrv.ini cached address)
01:06:20 Failed to connect over SharedMemory
01:06:20 Cannot connect to server

To enable dbvalid-connection.log create a touch file similar to below:
 
touch /usr/openv/db/VRTS_NBDB_DBVALID_LOGS

Log location: /usr/openv/db/log/dbvalid-connection.log

​Also, increase the verbosity of the log to capture more detailed logs.
In bp.conf file add the line below:

BPDBM_VERBOSE = 5

On a newly upgraded NBU 10.1 server running on Linux, you may also see the following in the /usr/openv/netbackup/logs/nbdb logs after running nbdb_backup -online:

• 14:07:00.267 [9626] <16> GetServerCacheSettings: Unable to locate '-gss' parameter
• 14:07:00.267 [9626] <2> GetServerCacheSettings: Exiting (initial=100M, min=100M, max=1024M, stack=) rc = 13

Cause

Lack of SharedMemory resources, or lack of stack size entry in the server.conf.forbackup (the configuration file used for the staging and DB validation portion of catalog backups).

Solution

1. If the primary server started receiving this error after an upgrade to 10.1, update the /usr/openv/var/global/server.conf.forbackup by appending a stack size setting of 8M (-gss 8M)to the end:
   • e.g. -c 100M -ch 1024M -cl 100M -gss 8M
2. If the above doesn't work, or if the server is below 10.1, increase SHMMAX to 3/4 of physical RAM

• echo Mem Size In Bytes > /proc/sys/kernel/shmmax
• echo Mem Size In Bytes > /proc/sys/kernel/shmall
• Run sync; sync; reboot to use the settings