on 08-12-2013 12:40 PM
During regular (standard) NDMP backups, avoid potential NDMP communications failures between NetBackup media servers and the Network Attached Storage (NAS) host.
The following troubleshooting steps may help isolate the root cause of NDMP backup issues. If they do not, proceed with logging instructions in Step 8.
NOTE: Terminology for the types of NDMP backups can be confusing:
To isolate the root cause of a NDMP backup failure:
1. On the media server, test the connection to the NAS device on the NDMP port (10000) by typing the telnet command. Try both the hostname and its IP address. For example: telnet ndmp_host 10000
2. To test connectivity and credentials of the NDMP host and its devices:
NetApp: sysconfig -t
EMC Celerra: server_devconfig [datamover name] -l -s -n (list, scsi, non-disk)
Other: See system documentation
3. On the NAS, and type the command ndmpd status to verify that the NDMP daemon is running. If it is not, enable the ndmp daemon and verify again by typing the command ndmpd status. For information on starting the ndmp daemon, see the NAS system documentation.
4. For EMC Celerra devices, ensure that the SnapSure feature is enabled. See TECH170310 for information on SnapSure and other considerations.
5. On the media or master server, type the telnet and tpautoconf command tests again from Step 1 to test connectivity, credentials, and devices.
6. If Local or 3-Way backup, do the following to verify that the volume can be accessed and backed up:
Network Appliance (NetApp): dump 0uf nrst0a /<file system> (Substitute the proper non-rewind device path if it is not nrst0a as in this example).
This command will write data from the specified file system on the Network Appliance server directly to tape. If this dump does not complete successfully, then the problem is most likely a NetWork Appliance problem. To resolve the issue, open a call with Network Appliance technical support. If the dump does complete successfully, open a call with NetBackup support.
For EMC Celerra, refer to the server_archive command in the system documentation.
NOTE: The Backup Path specified in the NetBackup policy must be the exact case-sensitive path as the filer sees it.
7. For performance issues, try backing up the path in the policy to a null device and note the timing:
NetApp: dump 0f null /vol/volname
EMC Celerra: server_archive <datamover name> -w -f /dev/null/ -J <file path>
8. Based on your findings, proceed with any required configuration changes or, if there are no changes, begin backups or restores.
9. If none of the above troublesooting steps resolve the issue, enable debug logging on the NetBackup media server and on the NAS device.
To enable debug logging:
1. Depending on your media server platform, do the following:
For Windows media server:
For UNIX media server:
2. Make sure that the following legacy log directories exist on the media server that services the backup:
For Windows media server:
\NetBackup\logs\bpbrm
\NetBackup\logs\bptm
\NetBackup\logs\ndmpagent
For Unix media server:
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/ndmpagent
3. Run the following commands on the media server that services this backup to increase the unified logging levels:
For Windows media server:
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6
For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6
4. Enable ndmpd debug logging on the NAS device:
5. Reproduce the error.
6. Disable logging:
7. Retrieve the bpbrm, bptm, ndmpagent logs from the NetBackup media server for today's date.
8. Do one of the following to retrieve logs from the NAS device:
9. Type the commands on the media server to disable logging:
For Windows media server:
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0
For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0
10. Create a temp directory.
For example: C:\temp\vxul or /tmp/vxul
11. Run the following command to retrieve all unified logs for today's date:
For Windows media server: <install path>\NetBackup\bin\vxlogmgr -c -t 1 -f c:\temp\vxul
For Unix meda server: /usr/openv/netbackup/bin/vxlogmgr -c -t 1 -f /tmp/vxul
Where the '-t 1' value will retrieve the unified logs from the last hour. Adjust the value as appropriate to go back past the beginning of the job.
12. Compress the temp directory (Zip for Windows, tar for Unix) and send to Symantec NetBackup Support along with the legacy logs and the NAS logs.
13. To create a long listing of the policy, run the following comand on the NetBackup master server:
Windows master server: <install path>\NetBackup\bin\admincmd\bppllist <name of policy> -L
Unix master server: /usr/openv/netbackup/bin/admincmd/bppllist <name of policy> -L
14. On the media server, on the command line, run the nbsu tool as follows to create the output file:
Windows media server:
Unix media server:
The script will create an output file and tell you where it was stored.
15. Cut and paste the Job Details for the job in the Activity Monitor during the backup attempt. Send that to Symantec Support along with the nbsu output file and the bppllist output.
Related Articles:
Quite nice!
Please add logging of Oracle ZFS Appliance if you have any information.
Nice to have all NDMP related information at one spot .
good information
Please always check the information on the permanent link as the technote may change (http://www.symantec.com/docs/TECH56492 ). An example is that the technote has recently been updated to include better disabling of logging by using the vxlogcfg command with the "-r" flag to remove the lines added when using "vxlogcfg -a"
i.e.
For Windows media server:
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6
For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6
*** PS. By default DiagnosticLevel=6 is set. Unfortunately, sometimes the default levels don't get reset properly. Adding the diagnostic level in the command ensures that regardless of the default settings, the appropriate process logging will be enabled.
Can someone pl advise for below error:
media_server>tpautoconf -verify netapp_filer
Connecting to host "netapp_filer" as user "nbackup"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
ndmp_connect_client_auth failed
: host "netapp_filer" failed
NDMP failed to verify host
its the authentication error. Check for the correct credentials entry of the NDMP user in the NBU java console under the Media and device management>>credentials.
Thank you so much for this post. I've been having intermittent issues with this and this troubleshooting tip should come in handy.