Backup job fails with error: Cannot lstat/lstat64 . Errno = 5: I/O error.

Problem

Backup job fails with error: Cannot lstat/lstat64 filename;. Errno = 5: I/O error.

Error Message

The bpbkar log generated on client, will show below errors:

<master> <client> Error 4832 
Backup from client <client>: ERR - Cannot lstat <filename>. Errno = 5: I/O error 

or

<master> <client> Error 4832 
Backup from client <client>: ERR - Cannot lstat64 <filename>. Errno = 5: I/O error 

Cause

1. When the bpbkar daemon on the system is unable to read a file on the file system, it can fail with lstat or lstat64 errors.
2. This can also occur during catalog backups on the master server, which will fail with a status 124: NetBackup database backup failed, a path was not found or is inaccessible.
3. A file system or hardware error.
4. A file is removed during the backup.

Solution

1. What file system are the files on?

• Run ls -l from within the directory where the filename resides.
• Run ls -l <filename> on the filename itself.
• This will perform a simple stat on the file and verify that the filename can be seen in the directory.

2. What type of files do they appear to be?

• If this is a text or application file, try editing the filename or opening it in an application to verify the file can be read properly.
• Try copying the file to another file system to verify the operating system can read the file properly.  

3. Review the system logs ( syslog, /var/log/messages) for SCSI errors or anything related to the disk where the files are reported to exist. Use the timeframe reported in the Problems report or bpbkar log to check for corresponding errors in the system logs.

4. Unmount the file system where the files are reported and run a file system check to uncover any underlying file system issues. Contact the operating system vendor for instructions on how to unmount and check the file system.

5. Set up logging and run a bpbkar interactive test on the directory where the files are purported to exist:

• Add VERBOSE = 5 to the /usr/openv/netbackup/bp.conf configuration file on the client.
• touch /usr/openv/netbackup/bpbkar_path_tr
• mkdir /usr/openv/netbackup/logs/bpbkar directory if it doesn't already exist.

Run /usr/openv/netbackup/bin/bpbkar -nocont <directory> > /dev/null against the directory containing the file.

The above command will cause bpbkar to read the directory and write the output to /dev/null instead of disk or tape.
Running bpbkar manually is a good method to verify if files can be read without doing an actual backup of the client. Any errors will be written to the /usr/openv/netbackup/logs/bpbkar/log.<date> log.