Error Message
In the NetBackup Activity Monitor Job details :
23/11/2015 12:36:46 AM - started process bpbrm (9652)
23/11/2015 12:36:46 AM - connecting
23/11/2015 12:36:46 AM - connected; connect time: 0:00:00
23/11/2015 12:36:48 AM - Info bptm(pid=6500) start backup
23/11/2015 12:36:49 AM - begin writing
23/11/2015 6:13:03 AM - Error bpbrm(pid=9652) db_FLISTsend failed: file read failed (13)
23/11/2015 6:13:03 AM - Error ndmpagent(pid=7104) terminated by parent process
23/11/2015 6:13:03 AM - Info ndmpagent(pid=0) done
23/11/2015 6:13:03 AM - Error ndmpagent(pid=7104) NDMP backup failed, path = /root_vdm_1/fs_bkapp01/sh_bkapp01
23/11/2015 6:13:04 AM - Warning ndmpagent(pid=7104) EMCVNX01.test.net: Backup was aborted
23/11/2015 6:13:04 AM - Info ndmpagent(pid=7104) EMCVNX01.test.net: server_archive: emctar vol 1, 2038329 files, 0 bytes read, 2568769830912 bytes written
23/11/2015 6:13:04 AM - Error ndmpagent(pid=7104) EMCVNX01.test.net: Backup is aborted.
23/11/2015 6:13:08 AM - Info ndmpagent(pid=0) done. status: 13: file read failed
23/11/2015 6:13:08 AM - end writing; write time: 5:36:19
file read failed(13)
In the bpdbm log on the master server :
06:13:03.410 [4540.7300] <2> image_db: Q_IMAGE_ADD_FILES (locking)
06:13:03.441 [4540.7300] <2> DbmOdbcConnect::set_connection_id: -1 -> 3379503
06:13:03.488 [4540.7300] <8> NBE_CatImage::ndmp_addNode: duplicate NODE for inode 3071269
06:13:03.488 [4540.7300] <16> NBE_CatImage::DataMatches: mtime changed for inode 3071269 (old: 1448199332, new 1448213185)
06:13:03.488 [4540.7300] <32> Function: NBE_CatImage::ndmp_addNode: File: ../nbe_cat_image.cpp Line: 1770: invalid input data,
06:13:03.488 [4540.7300] <16> add_files_alt: Error adding file rec 13
06:13:03.503 [4540.7300] <2> process_request: request complete: exit status 13 file read failed; query type: 78
Cause
The default NDMP backup of EMC VNX filers do not use snapshots. Therefore if files are deleted and created during the active backup the reuse of an inode can occur.The NetBackup bpdbm process on the Master Server during a file add ( Q_IMAGE_ADD_FILES) to the backup image checks for metadata consistency. If there is an inode conflict the backup will be terminated due to potential issues during the restore.
Solution
On the EMC VNX ensure that the snapsure option is licensed and that the NDMP protocol is enabled to use it.
To check the snapsure license on the filer use the following command :
[nasadmin@EMCVNX01 ~]$ nas_license -list
key status value
site_key online 55 a9 e6 78
nfs online
cifs online
snapsure online
To check that snapsure is enabled for the NDMP protocol use the command :
[nasadmin@EMCVNX01 ~]$ server_param server_2 -facility NDMP -list
server_2 :
param_name facility default current configured
maxProtocolVersion NDMP 4 4
scsiReserve NDMP 0 0
CDBFsinfoBufSizeInKB NDMP 1024 1024
bufsz NDMP 128 128
: :
snapTimeout NDMP 5 5
: :
snapsure NDMP 0 0
v4OldTapeCompatible NDMP 1 1
If snapsure is not enabled use the following command to enable it for NDMP :
[nasadmin@EMCVNX01 ~]$ $ server_param server_2 -facility NDMP -modify snapsure -value 1
server_2 : done
Verify that snapsure is enabled :
[nasadmin@EMCVNX01 ~]$ server_param server_2 -facility NDMP -list
server_2 :
param_name facility default current configured
maxProtocolVersion NDMP 4 4
scsiReserve NDMP 0 0
CDBFsinfoBufSizeInKB NDMP 1024 1024
bufsz NDMP 128 128
: :
snapTimeout NDMP 5 5
: :
snapsure NDMP 0 1 1
v4OldTapeCompatible NDMP 1 1
In NetBackup set the SNAPSURE environment variable in the Backup Selection List for the NDMP backup policy above the path to be backed up :
set SNAPSURE=y
/root_vdm_1/fs_bkapp01/sh_bkapp01