EMC VNX/Celerra backup fails with db_FLISTsend failed: file read failed (13)

Article: 100024883
Last Published: 2021-02-24
Ratings: 0 0
Product(s): NetBackup

Error Message

In the NetBackup Activity Monitor Job details : 
23/11/2015 12:36:46 AM - started process bpbrm (9652) 
23/11/2015 12:36:46 AM - connecting 
23/11/2015 12:36:46 AM - connected; connect time: 0:00:00 
23/11/2015 12:36:48 AM - Info bptm(pid=6500) start backup 
23/11/2015 12:36:49 AM - begin writing 
23/11/2015 6:13:03 AM - Error bpbrm(pid=9652) db_FLISTsend failed: file read failed (13) 
23/11/2015 6:13:03 AM - Error ndmpagent(pid=7104) terminated by parent process 
23/11/2015 6:13:03 AM - Info ndmpagent(pid=0) done 
23/11/2015 6:13:03 AM - Error ndmpagent(pid=7104) NDMP backup failed, path = /root_vdm_1/fs_bkapp01/sh_bkapp01 
23/11/2015 6:13:04 AM - Warning ndmpagent(pid=7104) EMCVNX01.test.net: Backup was aborted 
23/11/2015 6:13:04 AM - Info ndmpagent(pid=7104) EMCVNX01.test.net: server_archive: emctar vol 1, 2038329 files, 0 bytes read, 2568769830912 bytes written 
23/11/2015 6:13:04 AM - Error ndmpagent(pid=7104) EMCVNX01.test.net: Backup is aborted. 
23/11/2015 6:13:08 AM - Info ndmpagent(pid=0) done. status: 13: file read failed 
23/11/2015 6:13:08 AM - end writing; write time: 5:36:19 
file read failed(13)


In the bpdbm log on the master server :
06:13:03.410 [4540.7300] <2> image_db: Q_IMAGE_ADD_FILES (locking)
06:13:03.441 [4540.7300] <2> DbmOdbcConnect::set_connection_id: -1 -> 3379503
06:13:03.488 [4540.7300] <8> NBE_CatImage::ndmp_addNode: duplicate NODE for inode 3071269
06:13:03.488 [4540.7300] <16> NBE_CatImage::DataMatches: mtime changed for inode 3071269 (old: 1448199332, new 1448213185)
06:13:03.488 [4540.7300] <32> Function: NBE_CatImage::ndmp_addNode: File: ../nbe_cat_image.cpp Line: 1770: invalid input data,
06:13:03.488 [4540.7300] <16> add_files_alt: Error adding file rec 13

06:13:03.503 [4540.7300] <2> process_request: request complete: exit status 13 file read failed; query type: 78

Cause

The default NDMP backup of EMC VNX filers do not use snapshots. Therefore if files are deleted and created during the active backup the reuse of an inode can occur. 

The NetBackup bpdbm process on the Master Server during a file add ( Q_IMAGE_ADD_FILES)  to the backup image checks for metadata consistency. If there is an inode conflict the backup will be terminated due to potential issues during the restore. 

Solution

On the EMC VNX ensure that the snapsure option is licensed and that the NDMP protocol is enabled to use it. 

To check the snapsure license on the filer use the following command :
[nasadmin@EMCVNX01 ~]$ nas_license -list
key                 status    value
site_key            online    55 a9 e6 78
nfs                 online
cifs                online
snapsure            online

To check that snapsure is enabled for the NDMP protocol use the command :
[nasadmin@EMCVNX01 ~]$ server_param server_2 -facility NDMP -list
server_2 :
param_name                       facility  default     current   configured
maxProtocolVersion                  NDMP          4          4
scsiReserve                         NDMP          0          0
CDBFsinfoBufSizeInKB                NDMP       1024       1024
bufsz                               NDMP        128        128
                     :                         :
snapTimeout                         NDMP          5          5
                     :                         :
snapsure                            NDMP          0          0
v4OldTapeCompatible                 NDMP          1          1

If snapsure is not enabled use the following command to enable it for NDMP :
[nasadmin@EMCVNX01 ~]$ $ server_param server_2 -facility NDMP -modify snapsure -value 1
server_2 : done

Verify that snapsure is enabled :
[nasadmin@EMCVNX01 ~]$ server_param server_2 -facility NDMP -list
server_2 :
param_name                       facility  default     current   configured
maxProtocolVersion                  NDMP          4          4
scsiReserve                         NDMP          0          0
CDBFsinfoBufSizeInKB                NDMP       1024       1024
bufsz                               NDMP        128        128
                     :                         :
snapTimeout                         NDMP          5          5
                     :                         :
snapsure                            NDMP          0          1          1
v4OldTapeCompatible                 NDMP          1          1

In NetBackup set the SNAPSURE environment variable in the Backup Selection List for the NDMP backup policy above the path to be backed up :
set SNAPSURE=y
/root_vdm_1/fs_bkapp01/sh_bkapp01
 

Was this content helpful?