When a NetBackup NetApp clustered Data ONTAP (cDOT) Data Storage Virtual Machine (SVM) backup or restore fails with the error 2813

Problem

When a NetBackup NetApp clustered Data ONTAP (cDOT) Data Storage Virtual Machine (SVM) backup or restore fails with the error 2813

 

Error Message

Using a restore as the example, but the same errors occur for a backup. 

When a failure occurs the NetBackup detailed job status for the restore will indicate the following :
Dec 21, 2016 11:39:22 AM - Info ndmpagent (pid=23902) INF - Restoring NDMP files from /prod-nfs/nfs_vol/test-directory/ to [See line below]
Dec 21, 2016 11:39:22 AM - Info ndmpagent (pid=23902) INF - Restoring NDMP files from [See line above] to /prod-nfs/restore_test/test-directory
Dec 21, 2016 11:39:22 AM - Info ndmpagent (pid=23902) NDMP Remote disk  
Dec 21, 2016 11:39:28 AM - Info ndmpagent (pid=23902) This is CDOT restore
Dec 21, 2016 11:39:28 AM - Info ndmpagent (pid=23902) DAR enabled
Dec 21, 2016 11:39:29 AM - Error ndmpagent (pid=23902) ndmp_data_connect_v3 failed, status = -1 (-1)
Dec 21, 2016 11:39:29 AM - Error ndmpagent (pid=23902) NDMP restore failed from path /prod-nfs/nfs_vol/test-directory
Dec 21, 2016 11:39:29 AM - Info ndmpagent (pid=23902) done. status: 25
         :                  :
NDMP policy restore error  (2813)


The ndmpagent (OID 134) on the NetBackup Media Server NDMP restore host will mirror the error in the detailed job status :  
21/12/16 11:39:29.371 [Debug] NB 51216 ndmpagent 134 PID:23902 TID:140143889221504 File ID:134 [No context] 1 [NdmpDataSideNas::DataConnPrepare] ndmp_cab_data_conn_prepare returned 0 (NDMP_NO_ERR)
21/12/16 11:39:29.425 [Debug] NB 51216 ndmpagent 134 PID:23902 TID:140143889221504 File ID:134 [No context] 1 [NdmpDataSideNas::DataConnect] ndmp_data_connect failed, status = 23 (NDMP_CONNECT_ERR)
21/12/16 11:39:29.425 [Application] NB 51216 ndmpagent 134 PID:23902 TID:140143889221504 File ID:134 [No context] [Error] V-134-91 ndmp_data_connect_v3 failed, status = -1 (-1)


The ndmpd.log on the cDOT node which hosts the NDMP restore control Logical Interface (LIF) with debug enabled and filter set to all will report "couldn't find any appropriate preferred local src IP address" 
00000004.000056f6 00ec5dff Wed Dec 21 2016 22:37:15 +11:00 [kern_ndmpd:info:3952] [69397]  ERROR: DATA cb_data_connect_failed (2): 33 (-10:57:46)
00000004.000056f7 00ec5dff Wed Dec 21 2016 22:37:15 +11:00 [kern_ndmpd:info:3952] [69397]   INFO: DATA: sending listen/connect failed ABORT (-10:57:46)
00000004.000056f8 00ec5dff Wed Dec 21 2016 22:37:15 +11:00 [kern_ndmpd:info:3952] [69397]  ERROR: NDMP Vserver Connect: couldn't find any appropriate preferred local src IP address (-10:57:46)
00000004.000056f9 00ec5dff Wed Dec 21 2016 22:37:15 +11:00 [kern_ndmpd:info:3952] [69397]  ERROR: NDMP Connect: Unable to Connect to [192.168.5.141]:4410 due to Can't assign requested address (-10:57:46)
 

Cause

An NDMP backup/restore connection is made up of a control and data path. As per the NDMP service Preferred Interface Role definition for a Data SVM. Either an Intercluster or Data (LIF) is required for the NDMP data path. This MUST be local to the node hosting the volume and be on the same subnet as the NDMP control path

e.g. From the error section above the Data SVM is prod-nfs and target volume is restore_test.

The NDMP service for the Data SVM prod-nfs shows the Prefered Interface Role for the ndmp data path: 
cm-netapp::*> service ndmp show -vserver prod-nfs
  (services)

                             Vserver: prod-nfs
                        NDMP Version: 4
                     :              :
 Enable Logging of VM Stats for Dump: false
              Enable NDMP on Vserver: true
            Preferred Interface Role: intercluster, data
              Secondary Debug Filter: -
Is Secure Control Connection Enabled: false

The current node for the volume restore_test is cm-netapp-02:
cm-netapp::*> vol show -vserver prod-nfs -fields node,volume
vserver  volume  node
-------- ------- ------------
prod-nfs nfs_vol cm-netapp-01
prod-nfs prodnfs_root
                 cm-netapp-01
prod-nfs restore_test
                 cm-netapp-02
3 entries were displayed.

The Data SVM LIF is hosted by node cm-netapp-01:
cm-netapp::*> net int show -vserver prod-nfs
  (network interface show)
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
prod-nfs
            prod-nfs_nfs_lif1
                         up/up    192.168.5.147/24   cm-netapp-01  e0c     true
1 entries were displayed.

There is no Intercluster LIF:
cm-netapp::*> net interface show -role intercluster
  (network interface show)
There are no entries matching your query.

 

Solution

There are a range of methods that can established a successful restore via the Data SVM, such as the following:

1. Migrate the Data LIF used for the NDMP control connection to the same node as the volume.  
2. Migrate the volume to the same node as the Data LIF used for the NDMP control connection.
3. Create an intercluster LIF, which has connectivity to the NetBackup media server performing the restore, which resides on the same node as the volume and is on the same subnet as the Data SVM used for the NDMP control connection. 

Note : The restore can also be performed via the Admin SVM using various LIF types and scenarios.