Image cleanup job failure error 2067

Problem

After expiring images and removing disk volume manually, image cleanup job fails with error 2067.

Error Message

Nov 15, 2010 6:52:45 PM - Info nbdelete (pid=1061088) deleting expired images. Media Server: backup2 Media: @aaab4
Nov 15, 2010 6:52:45 PM - Error nbdelete (pid=1061088) Cannot obtain resources for this job : error [2067]
Nov 15, 2010 6:52:45 PM - Info nbdelete (pid=1061088) deleting expired images. Media Server: backup2 Media: @aaab6
Nov 15, 2010 6:52:46 PM - Error nbdelete (pid=1061088) Cannot obtain resources for this job : error [2067]
Nov 15, 2010 6:52:45 PM - requesting resource @aaab4
Nov 15, 2010 6:52:45 PM - Error nbjm (pid=483416) NBU status: 2067, EMM status: Disk volume not found
Nov 15, 2010 6:52:45 PM - Error nbjm (pid=483416) NBU status: 2067, EMM status: Disk volume not found
Nov 15, 2010 6:52:45 PM - requesting resource @aaab6
Nov 15, 2010 6:52:46 PM - Error nbjm (pid=483416) NBU status: 2067, EMM status: Disk volume not found
Nov 15, 2010 6:52:46 PM - Error nbjm (pid=483416) NBU status: 2067, EMM status: Disk volume not found
media manager - system error occurred  (174)

Cause

Disk/media ID related lock files still exist for removed disk storage and the processes are trying to access the "removed" resource(s) to cleanup the images, but cannot.

Information to collect and locate the issue, or to be reviewed by Support:

• nbsu output file - /usr/openv/netbackup/bin/support/nbsu
• nbdb_unload output file - /usr/openv/db/bin/nbdb_unload -dbn NBDB /tmp/nbdb_dump
• bpdbm log(s) - /usr/openv/netbackup/logs/bpdbm/
• admin log(s) - /usr/openv/netbackup/logs/admin/

Example:

From nbsu output, verify resources @aaab4 and @aaab6 do not appear in disk volume list anymore.
From bpdbm log & nbdb_unload output, verify resources @aaab4 and @aaab6 are not being referenced anymore.
The above 2 verification confirms the disk volume was removed properly and no image/fragment(s) are left in EMM.

Note: If any of the resource is still in EMM, advancement to Engineering may be required to have SQL script removing the leftover entries in EMM.

Check admin log:
18:52:45.026 [1061088] <2> nbdelete: set_required_media_server: no diskMediaID <-> mediaServer mapping : </usr/openv/netbackup/db/config/CLEANUP_MEDIASERVER_MAP>
18:52:45.026 [1061088] <4> nbdelete: allocate_media: Requesting resources for: @aaab4
...
18:52:45.527 [1061088] <2> requestFailed: got gotCallback, jmJobStatus = [2067], emmStatus = [2005067], mapped failure status to= [2067]
18:52:45.529 [1061088] <2> packageInitialResourceRequestResult: totalNumberOfAllocations == 1
18:52:45.529 [1061088] <2> packageInitialResourceRequestResult: retVal = 2067
18:52:45.529 [1061088] <16> RequestInitialResources: MultiResReq.cpp:2330 resource request failed [2067]
18:52:45.553 [1061088] <2> RequestInitialResources: retVal = 2067 emmStatus = 2005067
18:52:45.553 [1061088] <2> RequestInitialResources: returning
18:52:45.553 [1061088] <4> nbdelete: allocate_media: jmcomm_RequestInitialResources returned: 2067
18:52:45.553 [1061088] <16> nbdelete: RemoveFragments: Cannot obtain resources for this job : error [2067]
18:52:45.553 [1061088] <2> set_job_details: Tfile (730253): LOG 1289865165 16 nbdelete 1061088 Cannot obtain resources for this job : error [2067]

04:53:33.898 [2158714] <4> nbdelete: lockOperation: Attempting lock on: /usr/openv/var/nbdelete/backup2_aaab4.lock
04:53:33.898 [2158714] <4> nbdelete: lockOperation: obtained lock /usr/openv/var/nbdelete/backup2_aaab4.lock
04:53:33.898 [2158714] <4> nbdelete: allocate_media: Requesting resources for: @aaab4

04:53:35.353 [2158714] <4> nbdelete: lockOperation: Attempting lock on: /usr/openv/var/nbdelete/backup2_aaab6.lock
04:53:35.353 [2158714] <4> nbdelete: lockOperation: obtained lock /usr/openv/var/nbdelete/backup2_aaab6.lock
04:53:35.353 [2158714] <4> nbdelete: allocate_media: Requesting resources for: @aaab6

This means that NetBackup still thinks the resources are available, because it tries to read the lock file.

Check if those files exist in nbsu (NBU_dirs.txt):

NBU_dirs.txt:-rw-------   1 root     system            0 Oct  1 17:34 backup2_aaab4.lock
NBU_dirs.txt:-rw-------   1 root     system            0 Oct  1 17:34 backup2_aaab6.lock

After confirming the files exist for the removed storage, these lock files may be manually removed.

Solution

For the disk storage that no longer exists, manually remove the related lock files that disk/media ID, which should resolve this issue.

Applies To

When running "nbdelete -allvolumes -force", the same 2067 error appears in job details.