Verify that the disk is ready to decommission; avoiding RDSM errors when removing disk pools

Problem

Verify that the disk is ready to decommission prior to removing the configuration from the environment.

Error Message

RDSM errors when removing disk pools

Cause

• Backup policies are no longer using the Disk Storage Unit, including Storage Unit Groups and Storage Lifecycle Policies.
• Disk Storage Unit still exists within the environment, but is empty or has images that can be expired. Do not remove the disk storage unit until the entire process is complete.
• Inspection of the disk storage unit path ("ls -l" or "dir") shows the same information as what is currently contained in the catalog or Images On Media report.
• For disk pools, make sure that all volumes within the pool have been checked in the reports and catalog to ensure they are empty or to identify images ready to expire

Solution

Note: Unless otherwise noted, the directory path to these commands is as follows...
•  UNIX/Linux: /usr/openv/netbackup/bin/admincmd/
•  Windows: install_path\NetBackup\bin\admincmd\

First, to check for any incomplete SLP backup, duplications, or imports - which have yet to be written to the soon to be deleted disk pool:

nbstlutil report

This output will highlight the Storage Lifecycle Policy with incomplete images. If zero images are returned from the above command you may proceed with the step 'Finding the Images on Disk' below, Use the command below to show the SLP configurations to determine which storage unit is using the disk pool that needs to be removed from the configuration.

nbstl -L

To cancel images from completing their SLP duplications (only if the customer specifies the image copies are not needed), and if the SLP has an destination/operation step configured to use the diskpool:

nbstlutil cancel -lifecycle <lifecycle_name>

Find the Images On Disk report and select the disk type and/or path to get a listing of images associated with that path. Alternatively, you can use the command line to pull a more specific report for a particular storage unit:

bpimmedia -stype <Storage Server Type> -dp <disk pool name>

Verify that this report is empty (no entity found). If images still appear in this report, these copies must be expired from the disk. If duplicating these images as part of the steps to remove the disk from the environment, be sure to expire the copy number associated with the image on the disk. For information on manual duplications, consult the Commands Guide, or use the Catalog menu option in the Administration Console with the Duplicate action selected and search for the images needing manual duplication.

To expire a specific copy number (copy 1 in this example):

bpexpdate -backupid client_1234567890 -copy 1 -d 0

Alternatively, you can search the Catalog for the specific copy and select and right-click to Expire the copy of the image(s) in question.

Alternatively, you may need to expire all images from a specific Disk Pool.

bpexpdate -stype %servertype% -dp %diskpoolname%

Once the image has been expired using one of these two methods, we must ensure that an Image Cleanup runs successfully with an exit status of 0. You can manually start an image cleanup using the following syntax:

bpimage -cleanup -allclients

If nbdelete is failing due to images not found on the disk (status code 83), you can follow with the below command:

nbdelete -allvolumes -force

Be sure when running nbdelete with the -force option that all disks and media servers that are valid are up and running. If nbdelete is unable to contact the media server, or if the media server was contacted but the disk path was not available, the -force option will proceed with removing images in a "to be deleted" state from the EMM tables. After the images are removed from the EMM tables, there will be no further attempts to remove the images from disk. If a disk or media server was temporarily offline while this command was run, it could leave orphaned images on the disk.

It is possible for the catalog record to be deleted before the nbdelete command has cleared the EMM Images, ImageFragment and ImageCopy tables in the database. Re-check the reports (Disk Reports > Images On Disk) to ensure they now return no data for that Disk Pool. Failure to successfully decommission a disk storage unit can lead to Image Cleanup errors.

Once the disk is empty of NetBackup image data, and the image cleanup for all clients has run successfully since expiring the images on the disk, we can move forward with removing the configuration for the disk storage units.

Remove the storage unit from any existing Storage Lifecycle Policies.

nbstl <Lifecycle Name> -modify_current -residence <stu1>,<stu2>,[<stu#>]*

Note: See Related Articles (100011045) below for more details; the values in the above example are for example purposes only.

Remove the disk storage unit(s).

bpstudel -label <storage_unit_name>

If applicable, remove the disk pool(s).*

nbdevconfig -deletedp -dp <disk-pool-name> -stype <server-type>

If applicable, remove credentials for the Storage Server from the media server(s).*

*Note: For OpenStorageTechnology disk types (OST)

Note: For Advanced Disk (AdvancedDisk) there are no credentials, so the tpconfig step below is not needed, but you must remove the storage server, storage units, and diskpool.

From the media server:

tpconfig -delete -storage_server -stype <server-type> -sts_user_id <user id>

Note: The directory path to this command is as follows...
•  UNIX/Linux: /usr/openv/volmgr/bin/
•  Windows: install_path\Volmgr\bin\

*Note: If -sts_user_id is not immediately known, check "tpconfig -dsh" for this value for the correct Storage Server. This step must be performed prior to the media server being decommissioned.

If applicable, remove the storage server.*

nbdevconfig -deletests -storage_server <storage-server-name> -stype <server-type>

*Note: For basic disks and basic disk staging storage units, these will not apply. This step is only required if the Storage Server itself is being decommissioned.