How to troubleshoot NDMP Backups failures when status code 99 (or other NDMP backup failure) is reported. Includes logging instructions.

Problem

How to troubleshoot NetBackup for NDMP Backup failures when status code: 99 (NDMP backup failure) is reported.  Includes logging instructions.
This article can be used for most NDMP status codes.

Error Message

STATUS CODE: 99

Cause

Status 99 in NDMP backups is generic. Further troubleshooting is required.

Solution

During regular (standard) NDMP backups, avoid potential NDMP communications failures between NetBackup media servers and the Network Attached Storage (NAS) host.
The following troubleshooting steps may help isolate the root cause of NDMP backup issues. If they do not, proceed with logging instructions in Step 9.

Note:  Terminology for the types of NDMP backups can be confusing:

• Remote NDMP:  The backup is performed on the media server, using the device paths defined to the media server.
• Local:  The backup is performed using the device paths attached/zoned to the NAS host.
• 3-Way:  The backup of a NAS host is performed using the device paths attached/zoned to another NAS host

To isolate the root cause of a NDMP backup failure:

1. On the media server, test the connection to the NAS device on the NDMP port (10000) by typing the telnet command.  Try both the hostname and its IP address.  For example: telnet ndmp_host 10000

2. To test connectivity and credentials of the NDMP host and its devices:

Windows: C:\Program Files\Veritas\Volmgr\bin\
UNIX: /usr/openv/volmgr/bin/ 

•  Type tpautoconf -verify [NAS hostname]
•  Type tpautoconf -probe [NAS hostname] to  probe the scsi bus for devices (local or 3-Way only).
•  On another master or media server, type the above two commands to check for another route or network path.
• Compare the probe results with the native NAS commands:

          NetApp:  sysconfig -t
          EMC Celerra:  server_devconfig [datamover name] -l -s -n   (list, scsi, non-disk)
         Other:  See system documentation

3. On the NAS, and type the command ndmpd status to verify that the NDMP daemon is running.  If it is not, enable the ndmp daemon and verify again by typing the command ndmpd status.  For information on starting the ndmp daemon, see the NAS system documentation.

4. For EMC Celerra devices, ensure that the SnapSure feature is enabled.  See 000013977 for information on SnapSure and other considerations.

5. On the media or master server, type the telnet and tpautoconf command tests again from Step 1 to test connectivity, credentials, and devices.

6.  If Local or 3-Way backup, do the following to verify that the volume can be accessed and backed up:

• First, mount a scratch tape in the drive using robtest or some other means and perform a backup.
• On the NAS device, type the appropriate native command:

Network Appliance (NetApp): dump 0uf nrst0a /<file system> (Substitute the proper non-rewind device path if it is not nrst0a as in this example).
Note:  For Cluster-mode NetApp, refer to the NetApp Document 'Clustered Data ONTAP 8.2 Command Map for 7-Mode Administrators' as the dump command does not work for C-Mode volumes.
Excerpt:

Understanding the different clustered Data ONTAP shells for CLI commands
A cluster has three different shells for CLI commands:

• The clustershell is the native shell, started automatically when you log in to the cluster.
It provides all the commands you need to configure and manage the cluster.
• The nodeshell is a special shell that lets you run a subset of 7-Mode commands.
These commands take effect only at the node level. You can switch from the clustershell to a nodeshell session to run nodeshell commands interactively, or you can run a single nodeshell command from the clustershell. You will know a command is a nodeshell command if it has the (long) form:

system node run -node {nodename|local} commandname

This command will write data from the specified file system on the Network Appliance server directly to tape. If this dump does not complete successfully, then the problem is most likely a NetWork Appliance problem. To resolve the issue, open a call with Network Appliance technical support. If the dump does complete successfully, open a call with NetBackup support.

For EMC Celerra, refer to the server_archive command in the system documentation.

Note:  The Backup Path specified in the NetBackup policy must be the exact case-sensitive path as the filer sees it.

7.  For performance issues, try backing up the path in the policy to a null device and note the timing:

NetApp:  dump 0f null /vol/volname
EMC Celerra:  server_archive <datamover name> -w -f /dev/null/ -J <file path>

8. Based on your findings, proceed with any required configuration changes or, if there are no changes, begin backups or restores.

9.  If none of the above troublesooting steps resolve the issue, enable debug logging on the NetBackup media server and on the NAS device.

To enable debug logging:

The following will work on a Windows or Unix master or media server:

• On the NetBackup console, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties dialog box.
• Go to the Logging section and change the global logging level to 5 (maximum). Please make a note of the current logging level before changing it in order to reset it after debugging is done.
• Click OK to exit.

1. Make sure that the following legacy log directories exist:

For Windows media server:
\NetBackup\logs\bpbrm
\NetBackup\logs\bptm
\NetBackup\logs\ndmpagent

For Unix media server:
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/logs/ndmpagent

2.  Enable legacy logging on the media server

• On the NetBackup consoled, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties diaglog box
• Go to the Logging section and change the global logging level to 5 (maximum).  Please make a note of the current logging level before changing it in order to reset it after debugging is done.
• Click OK to exit

3. Enable unified logging on the media server by running:

For Windows media server:
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6

For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6  -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6  -s DiagnosticLevel=6

4. Enable ndmpd debug logging on the NAS device:

• For Network Appliance NAS, type the following to enable ndmpd debug logging:  ndmpd debug 70
  Note:  For OnTap 8.1 and above, the commands are different. Refer to 000015841 for 7-Mode or 000022198 for Cluster Mode (C-Mode).
  OS version of any filer can be determined by viewing the output of tpautoconf -verify [NAS host name]
  If necessary, further ndmpd event logging can be enabled.  Refer to the ndmpd.backup.log.enable command in this Network Appliance Support article.
• EMC Celerra:  Enable debug logging on the datamover as described in Article 000011595
• For other NAS devices, see the vendor documentation.

5. Reproduce the error.

6.  Disable ndmpd debug logging on the NAS device:

• For Network Appliance NAS, type the following to disable ndmpd debug logging:  ndmpd debug 0
  Refer to the NetApp articles listed in Step 4 for OnTap 8.1 and above.
• EMC Celerra, see the disable commands in 000011595.
• For other NAS devices, see the vendor documentation.

7. Disable unified logging on the media server by running. 

For Windows media server:
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 134 -s DebugLevel=0  -s DiagnosticLevel=0
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 151 -s DebugLevel=0  -s DiagnosticLevel=0

For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 134 -s DebugLevel=0  -s DiagnosticLevel=0
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 151 -s DebugLevel=0  -s DiagnosticLevel=0

8.  Disable legacy logging:

• On the NetBackup console, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties dialog box.
• Go to the Logging section and change the global logging level to its pervious level, as noted above.
• Click OK to exit

9. Do one of the following to retrieve logs from the NAS device:

• Pull the datamover server_log from the EMC Celerra:  server_log server_2 -a -s > /output.file  (assumes ‘server_2’ is the datamover name)
• Pull the system log, backup log, and the ndmpdlog from the Network Appliance NAS:
  • /vol/etc/messages
  • /vol/etc/logs/backup
  • <volume>/etc/log/ndmpdlog (/vol/etc/log/mlog in DataOnTap 8.x)
  • For remote access to the files, try windows browser  (\\Filer_Name\C$\etc\File_Name)

10. Gather unified logs:

• Create a temp directory.  Example:  C:\temp\vxul or /tmp/vxul
• Run the following command to retrieve all unified logs for today's date:

11.  Retrieve the bpbrm, bptm, ndmpagent logs from the NetBackup media server for today's date.

12.  Cut and paste the Job Details for the job in the Activity Monitor during the backup attempt.

13. Compress the temp directory (Zip for Windows, tar for Unix) and send to Veritas NetBackup Support along with the legacy logs, unified logs, ndmp logs and Job Details. 
 

Error when adding credentials for NDMP host:

Unable to validate the filer wide credentials. NDMP failed to verify host (58).

Ensure the ndmp userid is created with MD5 encryption.  For Network Appliance, it can be switched between plaintext and MD5.  Refer to this article.
For EMC, use MD5.  Search for the document 'Adding an NDMP user in the NetBackup Admin Console' on the EMC Support website (requires an account).