ERROR CODE: error 26 bpbrm jobs failing with status 26.

Problem

ISSUE: Error Code 26 in bpbrm, jobs failing with Status 26 on very busy servers.

Error Message

<2> bpbrm Exit: ERROR 26 sent to parent process

Cause

DIAGNOSTIC STEPS:

Some jobs end with a Status 26 showing the following error in bpbrm log on the Media Server:

<2> bpbrm read_media_msg: media manager pid is 0, timed out waiting for msg after 30 minutes
<16> bpbrm multiplexed_backup: cannot get PID of media manager

Then the PID would end with the following message:

<2> bpbrm Exit: ERROR 26 sent to parent process

Review of the related bptm log file may show the bptm pid returning the following messages:

<16> emmlib_initializeEx: (-) Exception! CORBA::COMM_FAILURE
<16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'
<16> bptm: (-) Translating EMM_ERROR_CorbaCommFailure(3000000) to 25 in the NetBackup context
<16> bptm: error (25) initializing EMM interface.
<16> main: (-) Translating ERROR UNDEFINED(25) to 196 in the Media context
<2> bptm: error(196) initializing EMM interface.
<2> bptm: EXITING with status 47 <----------

If bptm logging levels are set to VERBOSE 5, the following message will also be seen:

<2> Orb::connectToObjectOnce: connection attempt failed: system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'

When this occurs, other commands that require EMM queries, such as vmquery , vmoprcmd and tpconfig may also return a Status 39.

In larger environments, when the system is heavily loaded with backup and duplication jobs, these jobs may show the above errors if the system does not have enough file descriptors defined on the operating system.  
The number of file descriptors can be viewed by checking the file descriptor limit in the '/usr/sbin/sysdef -i' command output - such as:

* Process Resource Limit Tunables (Current:Maximum)
0x0000000000000100:0x0000000000000400 file descriptors

These hex values translate to the following:

256:1024

Solution

RESOLUTION:

The problem was resolved by setting the file descriptors to a higher value. NetBackup needs a minimum of 8000 file descriptors defined in order to operate correctly. Below is an example of the command that will increase the file descriptor value to 8192.

Example:
ulimit -n 8192

The ulimit command will set the value for the particular session that NetBackup uses when it starts up and is not permanent. Administrators may choose either to set the system wide limit higher or to modify the NetBackup start up scripts so that a higher ulimit is defined for the application each time it starts. That can be done by adding the following line to the top of the '/etc/init.d/netbackup' file.

/usr/bin/ulimit -n 8192

The above line should be put in before the line starting with " NBPATH= ".