Backups which contain a large number of files might intermittently fail with the Error db_FLISTsend failed: premature eof encountered (233).

Problem

Large backups, for example NDMP backups, might sometimes fail backing up a very large number of files.

Error Message

The job details reports "Error db_FLISTsend failed: premature eof encountered (233)".

Reviewing the bpbrm log from the media server performing the backup reports:

03:29:55.728 [311659.311659] <2> db_FLISTsend: db_begin() failed: network connection broken
03:29:55.728 [311659.311659] <16> non_mpx_backup_archive_verify_import: db_FLISTsend failed: network connection broken (40)

The associate bpdbm log from the master server reported:

03:29:56.162 [14254] <2> error_db: Q_ERRADD
03:29:56.162 [14254] <4> db_error_add_to_file: db_FLISTsend failed: network connection broken (40)

Cause

Due to the large number of updates occurring for the backup along with a master server that was also under heavy load resulted in the bpdbm process failing on the request to update the files list for the backup.  A closer look at the bpdbm log also showed errors such as:

03:26:52.422 [7486] <16> _fetch_cs_cache_info: Failed while preparing statement(SELECT * FROM hostid_to_alias WHERE name=? AND domain=? AND birth<= strftime('%s', 'now') AND ttl> strftime('%s', 'now');): 5(database is locked)

The "database is locked" messages are due to bpdbm (database manager) not being able to update the NBDB (Sybase database) because Sybase is already very busy handling other updates.

Solution

Ensure the master server is sized properly for the amount of work (for example the number of jobs) it needs to do.  Having sufficient memory and CPU to handle the processing should help to resolve the problem.

Try adding the MAX_ENTRIES_PER_ADD touch file on the media servers.  Insert a larger value to reduce the number of updates to the master server during backups.  The default value is 5000.  Try increasing to 25000 to see if that might resolve the issue (the maximum value is 100000).

UNIX: /usr/openv/netbackup/MAX_ENTRIES_PER_ADD
Windows: install_path\Veritas\NetBackup\MAX_ENTRIES_PER_ADD

To attempt to eliminate the "database is locked" messages add a "-gn 30" value to the server.conf file on Master server.  Allowing bpdbm to perform timely updates to the NBDB database should prevent the backup failures from occurring.

UNIX: /usr/openv/var/global/server.conf
Windows: install_path\Veritas\NetBackupDB\CONF\server.conf

You will need to restart services on the master server for the change to take affect.  The "-gn" setting defines the number of active tasks for the network database server.  The default value is 20, which should be sufficient for the majority of NetBackup environments.  The maximum value for -gn is 40.