Sign In
Problem
GENERAL ERROR: Getting "Cannot lstat/lstat64 . Errno = 5: I/O error" during backups.
Error Message
Backup from client : ERR - Cannot lstat lost+found/. Errno = 5: I/O error
Backup from client : ERR - Cannot lstat64 lost+found/. Errno = 5: I/O error
Solution
Error Codes/Messages:
The Problems report on the master server will list the following errors.
3/29/2005 12:49:11 AM <master> <client> Error 4832
Backup from client <client>: ERR - Cannot lstat <filename>. Errno = 5: I/O error
or
3/29/2005 12:49:11 AM <master> <client> Error 4832
Backup from client <client>: ERR - Cannot lstat64 <filename>. Errno = 5: I/O error
The /usr/openv/netbackup/logs/bpbkar/log.<date> file on the client will list the filename that generated the error.
03:15:29 [17734] <16> bpbkar: ERR - Cannot lstat64 <filename>. Errno = 5: I/O error
Overview:
When the bpbkar daemon on the system is unable to read a file on the file system, it can fail with lstat or lstat64 errors. These errors will occur in the Problems report. The client backup will fail with a status 1 (the requested operation was partially successful). This can also occur during catalog backups on the master server, which will fail with a status 124 (NetBackup database backup failed, a path was not found or is inaccessible).
The operating system returns an "Errno = 5: I/O error" from the lstat or lstat64 system call. Whether the error comes from lstat or lstat64 will depend on whether the lstat system call is used or whether the 64-bit lstat64 is used for files larger than 2GB in size.
Troubleshooting:
Check the NetBackup (tm) Problems report for errors, or enable bpbkar logging on the client. These will list the specific file(s) that are triggering the error message.
To get the error report for the specific job ID that failed with the status 1, run the following command on the master:
# /usr/openv/netbackup/bin/admincmd/bperror -jobid <jobid>
This will show the specifics for the individual job ID as opposed to searching the whole Problems report.
Resolution:
Typically, the lstat or lstat64 message followed by an "Errno = 5: I/O error" indicates a file system or hardware error. Another potential cause is when a file is removed during the backup.
Check the log for the file(s) causing the error. Then check the following to see why lstat errors are generated when reading the file:
I. What file system are the files on?
The Problems report on the master server will list the following errors.
3/29/2005 12:49:11 AM <master> <client> Error 4832
Backup from client <client>: ERR - Cannot lstat <filename>. Errno = 5: I/O error
or
3/29/2005 12:49:11 AM <master> <client> Error 4832
Backup from client <client>: ERR - Cannot lstat64 <filename>. Errno = 5: I/O error
The /usr/openv/netbackup/logs/bpbkar/log.<date> file on the client will list the filename that generated the error.
03:15:29 [17734] <16> bpbkar: ERR - Cannot lstat64 <filename>. Errno = 5: I/O error
Overview:
When the bpbkar daemon on the system is unable to read a file on the file system, it can fail with lstat or lstat64 errors. These errors will occur in the Problems report. The client backup will fail with a status 1 (the requested operation was partially successful). This can also occur during catalog backups on the master server, which will fail with a status 124 (NetBackup database backup failed, a path was not found or is inaccessible).
The operating system returns an "Errno = 5: I/O error" from the lstat or lstat64 system call. Whether the error comes from lstat or lstat64 will depend on whether the lstat system call is used or whether the 64-bit lstat64 is used for files larger than 2GB in size.
Troubleshooting:
Check the NetBackup (tm) Problems report for errors, or enable bpbkar logging on the client. These will list the specific file(s) that are triggering the error message.
To get the error report for the specific job ID that failed with the status 1, run the following command on the master:
# /usr/openv/netbackup/bin/admincmd/bperror -jobid <jobid>
This will show the specifics for the individual job ID as opposed to searching the whole Problems report.
Resolution:
Typically, the lstat or lstat64 message followed by an "Errno = 5: I/O error" indicates a file system or hardware error. Another potential cause is when a file is removed during the backup.
Check the log for the file(s) causing the error. Then check the following to see why lstat errors are generated when reading the file:
I. What file system are the files on?
Run ls -l from within the directory where the filename resides.
Run ls -l <filename> on the filename itself.
This will perform a simple stat on the file and verify that the filename can be seen in the directory.
II. What type of files do they appear to be?
If this is a text or application file, try editing the filename or opening it in an application to verify the file can be read properly.
Try copying the file to another file system to verify the operating system can read the file properly.
III. Review the system logs (syslog, messages, etc) for SCSI errors or anything related to the disk where the files are reported to exist.
Use the timeframe reported in the Problems report or bpbkar log to check for corresponding errors in the system logs.
IV. Unmount the file system where the files are reported and run a file system check to uncover any underlying file system issues.
Contact the operating system vendor for instructions on how to unmount and check the file system.
V. Set up logging and run a bpbkar interactive test on the directory where the files are purported to exist:
1. Add VERBOSE = 5 to the /usr/openv/netbackup/bp.conf configuration file on the client.
2. touch /usr/openv/netbackup/bpbkar_path_tr
3. mkdir /usr/openv/netbackup/logs/bpbkar directory if it doesn't already exist.
4. Run /usr/openv/netbackup/bin/bpbkar -nocont <directory> > /dev/null against the directory containing the file.
The above command will cause bpbkar to read the directory and write the output to /dev/null instead of disk or tape.
Running bpbkar manually is a good method to verify if files can be read without doing an actual backup of the client. Any errors will be written to the /usr/openv/netbackup/logs/bpbkar/log.<date> log.
Terms of use for this information are found in Legal Notices.
