All NetBackup jobs begin to fail with "RB internal error (901)"

Problem

This issue may be observed in a busy NetBackup environment which has existed for many years.

All NetBackup jobs will start to encounter resource allocation errors which result in "NBU status: 901" as observed in Activity Monitor.

The Job Details show the RB internal error:
16-Dec-2019 19:12:42 - Info nbjm (pid=3777) starting backup job (jobid=25036743) for client MyClient, policy BigPolicy, schedule DailySchedule
16-Dec-2019 19:12:42 - Info nbjm (pid=3777) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=25036743, request id:{09069D50-2038-11EA-83AD-9A02C97DA7B0})
16-Dec-2019 19:12:42 - requesting resource DiskStorage
16-Dec-2019 19:12:42 - requesting resource MyMediaServer.NBU_CLIENT.MAXJOBS.MyClient
16-Dec-2019 19:12:42 - requesting resource MyMediaServer.NBU_POLICY.MAXJOBS.BigPolicy
16-Dec-2019 19:12:43 - Error nbjm (pid=3777) NBU status: 901, EMM status: RB internal error
RB internal error  (901)
 

Error Message

The debug log for nbrb (OID 118), at DebugLevel=1, shows that an allocation key above 2,147,483,647 cannot be found and is unknown:
[Debug] NB 51216 nbrb 118 [StopScanCallbackQueue::evaluate] fatal error; failing requestId={A4C913A2-2052-11EA-9E7A-FFA21F369A8E}
[Diagnostic] NB 51216 nbrb 118 [CallbackQueue::notifyClientFail] queueing notification of request failure for request ID {A4C913A2-2052-11EA-9E7A-FFA21F369A8E} with status 901
[Diagnostic] NB 51216 nbrb 118 [CorbaCall_requestFailed::execute] sending failure of request to nbjm for request ID {A4C913A2-2052-11EA-9E7A-FFA21F369A8E}, error code 901, reason request failed
[Debug] NB 51216 nbrb 118 [GroupProvider::isMountableDiskReadAllocation] unable to fetch the MDS allocation info from the database for allocationKey=2147866226
[Debug] NB 51216 nbrb 118 [EMMProvider::logdifference] mds has allocation nbrb database does not: 2147866226
[Debug] NB 51216 nbrb 118 [EMMProvider::deallocateAllocation] deallocating unknown allocation, allocationKey=2147866226
[Diagnostic] NB 51216 nbrb 118 [EMMProvider::deallocateAllocation] unable to find allocation key 2147866226 in database table RB_MDS_Allocation_Record
[Diagnostic] NB 51216 nbrb 118 [EMMProvider::doDeAllocate] unable to find allocation key 2147866226 in database table RB_MDS_Allocation_Record
[Application] NB 51216 nbrb 118 [No context] [Error] V-118-184 failed to execute SQL statement DELETE RB_MDS_TransportAllocation: error message , error code 100, native error code 0, SQL return code 100
[Debug] NB 51216 nbrb 118 [No context] 1 [RBDatabase::executeSQLStatement] failed to execute: DELETE RB_MDS_TransportAllocation rc = 100
 

Cause

When a NetBackup job starts, resources become allocated for the job.  As the job completes, NetBackup tries to deallocate those resources so future jobs can use them. 

When the allocation key is greater than 2,147,483,647, the deallocation fails, and resources do not become available to be reallocated.

Consequently, new jobs cannot run due to lack of resources to allocate.

Solution

This issue affects all NetBackup versions 8.2 and older.

There is no known workaround for this issue.

Contact Veritas technical support and reference Etrack 3993459 if you are experiencing the issue addressed by this article.

This issue is currently being considered by Veritas Technologies LLC to be addressed in the next major revision of the product.  Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests or introduces new risks to overall code stability.  Veritas’ plans are subject to change, and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.