Diskpool removal fails with "DSM has encountered the following busy resource"

Problem

Diskpool removal fails with "DSM has encountered the following busy resource", which is a NetBackup status code 134 error (unable to process request because the server resources are busy).

Error Message

DSM has encountered the following busy resource: <Diskpool_Name>, mount point = PureDiskVolume
failed to change the state of disk pool, invalid command parameter

...OR...

DSM has encountered the following busy resource: <Diskpool_Name>, mount point = <OST_DiskVolume>
failed to delete disk pool, unable to process request because the server resources are busy

...OR...

NetBackup Administration Console > Media and Device Management may show:
/!\ Failed to delete disk pool <Diskpool_Name>: unable to process request because the server resources are busy(134)
DSM has encountered the following busy resource: <Diskpool_Name> mount point = <OST_DiskVolume>
(MM Status 134)

Cause

This error happens when the PureDisk Diskpool is removed after the PureDisk SPA was removed from the environment. The proper procedure requires that information is removed from NetBackup before the appliance is removed from the environment.

Troubleshooting:

Output from nbdevconfig command:

nbdevconfig -changestate -dp <Diskpool_Name> -stype PureDisk -state DOWN
DSM has encountered the following busy resource: <Diskpool_Name>, mount point = PureDiskVolume
failed to change the state of disk pool, invalid command parameter

Output from nbdb_unload:

EMM_DiskVolumeMountPoint:
'71','131','1000049','0','3','PureDiskVolume', '2','2009-02-0523:23:19.899587','2010-05-04 08:34:54.093981'

The 7th Column is NumberOfUsers and it should be zero but they are two.

Note: The directory path to the nbdb_unload command is...

Unix/Linux: /usr/openv/db/bin/
Windows: <install_path>\NetBackup\bin\

See a list of the columns below in the EMM_DiskVolumeMountPoint table:

CREATE TABLE"EMM_MAIN"."EMM_DiskVolumeMountPoint"
(
"DiskVolumeMountPointKey"unsigned int NOT NULL DEFAULT autoincrement ,
"DiskVolumeKey"unsigned int NOT NULL ,
"MachineKey"unsigned int NOT NULL ,
"State"integer NOT NULL DEFAULT 0 ,
"Flags"integer NOT NULL DEFAULT 0 ,
"MountPoint"varchar(4096) NOT NULL ,
"NumberOfUsers"integer NOT NULL DEFAULT 0 ,
"CreatedDateTime"timestamp NOT NULL DEFAULT current utc timestamp ,
"LastModifiedDateTime"timestamp NOT NULL DEFAULT utc timestamp ,
CONSTRAINT"PK_EMM_DISKVOLUMEMOUNTPOINT" PRIMARY KEY ("DiskVolumeMountPointKey"),
) IN"EMM_DATA"

Solution

This issue can occur when the volume has an open handle with nbrb. Please stop all running jobs before running the below command to clear nbrb holds on the volumes. This should reset the number of users back to zero.

Unix/Linux: /usr/openv/netbackup/bin/admincmd/nbrbutil -resetall
Windows: <install_path>\NetBackup\bin\admincmd\nbrbutil -resetall

Applies to:

Environments with MSDP/PureDisk storage servers.

Note: Similar scenarios may also happen with OST STS (OpenStorage, Storage Servers).