All backups to an OpenStorage (DataDomain) storage device (after plug-in upgrade and password change) fail with error "Disk storage is down (2106)"

Problem

After updating DataDomain (DD) plug-in and running the tpconfig command to change the password on the NetBackup media server configured for OpenStorage (OST), all subsequent backups written to the DD storage have reportedly failed with the error Disk storage is down (2106).

Error Message

Error nbjm (pid=xxxx) NetBackup status 2106, EMM status Storage server is Down or unavailable.
Disk storage is down (2106)

Cause

Both the primary/master server and the media server were specified as "media servers" under Credentials > Storage Servers. Additionally, the OST(DD) plug-in and the NetBackup Credentials for the Storage Server had only been updated on the media server, but not the master server, and the mis-match in the "plug-in" and the "Storage Server" credentials between the two "media servers" configured for the storage servers cause the OST(DD) Disk Pool and Storage Servers to go to a DOWN state.

Troubleshooting:

Check the OST(DD) configuration on the upgraded Media server:

Windows:
In an Administrator Command Prompt, change directories to: install_path\NetBackup\bin\admincmd

UNIX/Linux:
In a terminal/SSH session, change to directories to: /usr/openv/netbackup/bin/admincmd

Then run the following commands and check for entries similar to the example output below:

• nbdevquery -liststs -stype DataDomain -U
  • State: DOWN
• nbdevquery -listdp -stype DataDomain -U
  • Disk_Pool: OST_DD_Pool01 = UP
  • Storage Server: OST_DD_Storage01 = DOWN
• nbdevquery -listdv -stype DataDomain -U
  • OST_DD_Storage01 = DOWN | Disk Volume name: ddboost
  • Flag: InternalDown
• bpstsinfo -pi -stype DataDomain
  • DataDomain Plug-in information displays properly.

Syntax to change the state of an OST(DD) disk volume:

• nbdevconfig -changestate -stype <server_type> -dp <disk_pool_name> - dv <vol_name> -state <state>

Example commands attempting to change the state of an OST(DD) disk volume with output below:

• nbdevconfig -changestate -stype DataDomain -dp OST_DD_Pool01 -dv ddboost -state UP
  • Command completed successfully
• nbdevquery -liststs -stype DataDomain -U
  • State: DOWN
• nbdevquery -listdp -stype DataDomain -U
  • Disk_Pool: OST_DD_Pool01 = UP
  • Storage Server: OST_DD_Storage01 = DOWN
• nbdevquery -listdv -stype DataDomain -U
  • OST_DD_Pool01 = DOWN | Disk Volume name: ddboost
  • Flag: InternalDown

No Changes?

• Check the DataDomain storage server information is displayed properly.
  • bpstsinfo -si -stype DataDomain -sn <DD_STS_name>
• Open the NetBackup Administration Console > Credentials > Storage Servers.
  • Highlight the Storage Server (OST_DD_Storage01) in the top pane
  • In the bottom pane BOTH the master and media servers are listed as media servers for the Storage Server

Solution

Copy the OST(DD) plug-in installation files to the NetBackup primary/master server, and performed the following actions:

1. Upgrade the OST(DD) plugin on the master, to match the version on the media server, and restart the NetBackup services.
   • Restart NetBackup services on a Windows Server with commands:
     install_path\NetBackup\bin\bpdown -f -v
install_path\NetBackup\bin\bpup -f -v
   • Restart NetBackup services on a Unix/Linux Server with commands:
     # /usr/openv/netbackup/bin/goodies/netbackup stop
# /usr/openv/netbackup/bin/goodies/netbackup start
2. Run the tpconfig command to change (-update) the OST(DD) credentials for the master server:
   • Syntax:
     tpconfig -update -storage_server <storage_server_name> -stype DataDomain -sts_user_id <user_name> -password <new_password>
     • Windows location: install_path\Volmgr\bin
     • UNIX/Linux location: /usr/openv/volmgr/bin
3. Run the bpstsinfo command to confirm the password change on the primary/master server allows the primary/master server to access the OST(DD) storage server:
   • bpstsinfo -si
     • Windows location: install_path\NetBackup\bin\admincmd
     • UNIX/Linux location: /usr/openv/netbackup/bin/admincmd
   • Note: The bpstsinfo -si command (run from the master server) should display a similar output to what was displayed when this command was run on the media server.
   • Wait a few minutes (10 to 20 minutes) before proceeding.
4. On the master server, run the following commands to check the current state of the OST(DD) storage servers, disk volumes, and disk pools; with states that should be returned in the output below.
   • nbdevquery -liststs -stype DataDomain -U
     • State: UP
   • nbdevquery -listdp -stype DataDomain -U
     • Disk_Pool: OST_DD_Pool01 = UP
     • Storage Server: OST_DD_Storage01 = UP
   • nbdevquery -listdv -stype DataDomain -U
     • OST_DD_Pool01 = UP | Disk Volume name: ddboost
     • Flag: InternalUP

Applies To

NetBackup OpenStorage (DataDomain) Environment:

• Master Server: NBU_Master
• Media Server: OST_DD_Media
• Storage Server: OST_DD_Storage01
• Disk Storage Unit: DataDomain_SU01
• Disk Type: OST(DD)
• Disk Pool: OST_DD_Pool01

Note: All server names listed above are generic names used only for the examples in this document