20 steps to analyze, correct, and troubleshoot device issues on Window Platforms

Problem

The following 20 steps will help analyze, correct, and troubleshoot many different device issues.

Error Message

A few examples (but not limited to):

• Status codes : 84 , 96 , 2009
• Drives keep going down or randomly go down.
• No drives available

Cause

Various causes. Sometimes this is environmental (OS, hardware, network, etc), other times it's bad configuration which may have changed, or gone into effect after a reboot, and periodically it's hard to determine if it's related to NetBackup software. Hence some verification and troubleshooting steps are required.

Solution

VERIFY

1. Does the OS see the devices ?

• Check the OS Device Monitor and make sure they are listed and do not have any yellow exclamation marks.
• From a command line :  install_path\Volmgr\bin\scan
  • This tells us what the OS sees
  • IF the OS does not see the devices, NetBackup will not see them.  Please contact the hardware manufacturer

2. Are there any Phantom Devices listed that need to be removed?

• From command prompt run:
  set devmgr_show_nonpresent_devices=1
• start devmgmt.msc
• Remove any tape drives, library or robots that are “grayed out”. Please review the following documentation:
  100023371 - How to delete "ghost" or "phantom" devices from the Windows Device Manager when using NetBackup

3. Does NetBackup see the devices configured ?

• From a command line :  install_path\Volmgr\bin\tpconfig -d
  • This tells us what NetBackup sees as being configured

PRELIMINARY TROUBLESHOOTING:

4. Power cycle everything.

• Make sure library is down for at least 5 min.

5. Update ALL drivers (tape drives, robot , tape library and HBA)

• Nothing should be older than 2010.
• ALWAYS obtain drivers from the manufacturer. Veritas has not updated their drivers since 2007

6. Install the latest device mappings, see Veritas Downloads.

• To confirm what version you are on :
  C:\Program Files\Veritas\Volmgr\bin\tpext -get_dev_mappings_ver

7. Check to see if the drives need cleaning.

• Even if the drives are clean by looking at the library’s GUI, they must been in a clean state on the NetBackup side.
• Run install_path\Volmgr\bin\tpclean -L
• If the output contains the message, NEEDS CLEANING you must clean the drive.
• You can clean the drive in the admin console or from a command line. Please review the following documentation:
      NetBackup Guides
      100006445 - How to use NetBackup's drive cleaning utility, tpclean, with a Library that contains a cleaning tape slot.
      100016842 - How to set up automatic tape cleaning in NetBackup Enterprise Server

8. If using McAfee antivirus, it will need to be disabled or configured to make exceptions for NetBackup  processes.  If this is not done, the most common error you will see is a status 84.
9. For servers that are still running Windows 2003 or earlier, be sure that the ‘Removable storage service’ is stopped and disabled.

TRY RECONFIGURATION

If you have had an issue with drives inconsistently going down, it is sometimes best to delete all devices and reconfigure

Note: This works best if it is Windows and SSO is NOT being used.

10. CONFIRM NO BACKUPS ARE RUNNING
11. Release all allocations to devices:

    From the command line run: install_path\Veritas\NetBackup\bin\admincmd>nbrbutil -resetAll
        (If backups are running, this will cause them to fail. This is why no backups should be running )

12. Delete all devices:

    OPTION 1 – Delete ALL devices.

    From the command line: install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices

    OPTION 2 – Delete all devices on a specified media server.

    From the command line: install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices -machinename -machinetype media

13. Use the device configuration wizard to configure your devices:

    Be sure there are no errors during the wizard or any "unused element" . If so, drag the tape drive to the robot in the wizard.

14. Inventory the robot(s).

ADDITIONAL TROUBLESHOOTING

15. Look for link state errors on HBA and tape devices
16. Look in the System Event viewer for any hardware related errors including tape alerts.
17. Confirm the following folders have been created in the NetBackup logs directory on the server that contains the devices.

    install_path\NetBackup\logs\bptm
    install_path\NetBackup\logs\bpbrm
    install_path\volmgr\debug\daemon
    install_path\volmgr\debug\ltid
    install_path\volmgr\debug\reqlib
    install_path\volmgr\debug\robots

18. To increase logging :

    Increase verbosity to 5 in the server host properties -> logging.
    Add the line 'VERBOSE' to install_path\volmgr\vm.conf
    create the following touch files:
    install_path\volmgr\ROBOT_DEBUG
    install_path\volmgr\AVRD_DEBUG
    install_path\volmgr\DRIVE_DEBUG
    install_path\volmgr\SSO_DEBUG

19. Run through issue and review logs.  Be sure to change logging back to 0 and remove VERBOSE from vm.conf

20. Information to gather for technical support from the vendor or NetBackup support:

• Environment Information (use the server where the devices are located at)
• Server Name:
• OS:
• NBU Version:
• Details for the failed job:
• All logs in #17.
• NBSU output...
  • NB 8.1 & lower: install_path\NetBackup\bin\support\nbsu -c -t
  • NB 8.1.1 & higher: install_path\NetBackup\bin\support\nbsu
• Mention any changes that happened around the time an issue started occurring, such as a recent product upgrade, partial hardware replacement, OS patch, or addition of third party software.

Access the Related Articles for more information.