How to manually identify/remove zero-byte or corrupt image header files which are preventing empty tapes from being returned to scratch and are preventing the "bpexpdate -deassignempty" command from completing…

The NetBackup scheduler automatically checks for tapes which are allocated to a media server, but hold no valid images (i.e. tapes that are "empty").  This automatic check is performed approximately every 12 hours.  If the check identifies an "empty" tape which is allocated to a media server and is not currently being used by a running backup, the tape is expired and returned to the scratch pool.

This check is necessary because if a backup has failed, a tape which was allocated for the failed backup may contain no valid images, but remain allocated to a media server and unused until it expires - which may be for days, weeks, months, or in some cases indefinitely, depending on the retention period of the schedule which execute the failed backup.

Another situation which may also result in a tape that is allocated to a media server becoming "empty" is when all of the images on a tape have been expired prior to the tape's expiration date - for example, by a Storage Lifecycle Policy.  The empty tape will remain allocated to a media server until its media expiration date is reached (possibly "forever," if the retention period is infinite), or until it is identified as an empty tape.  In this particular situation, failure to expire empty yet allocated tapes may ultimately result in NetBackup having no tapes available for backups or duplications.

In addition to the check for empty tapes that is automatically performed by the scheduler, the NetBackup command bpexpdate -deassignempty can be used to manually check for empty tapes.  This command can be executed by the NetBackup administrator at any time.

The check for empty tapes, which is performed by both the scheduler and the bpexpdate -deassignempty command, normally completes its operation successfully, but if there are any zero-byte or corrupt image header files in the NetBackup image database, the operation will fail.  If the command finds a bad image header, it will display this error message:

# bpexpdate -deassignempty
Search for empty media that meet the following criteria:
       Host: All
       Media id: All

Continue? y/n (n)y
Could not update media list, file read failed
file read failed
#

The error message file read failed indicates that there is at least one zero byte or corrupt backup image header somewhere in the NetBackup image database on the master server.  Empty (size zero bytes) image header files normally occur if the file system holding the image database has run out of space during a backup, causing the backup to abort, and leaving the unused image header file on the disk.  Corrupt image header files can occur if NetBackup was aborted unexpectedly (for instance, by a system crash) or if some other unexpected event has occurred.  In order for the bpexpdate -deassignempty command to be able to run to completion, all zero byte and corrupt backup image headers must be manually removed from the image database.

The following procedure can be used to identify and remove the bad image header file(s):

1. On the master server, enable bpdbm logging.

On a UNIX or Linux master server, make sure that the directory /usr/openv/netbackup/logs/bpdbm exists - mkdir this directory if it does not exist.

On a Windows master server, make sure that the directory <install_path>\VERITAS\NetBackup\logs\bpdbm exists - create this folder if it does not exist.

It is not necessary to restart NetBackup after creating the bpdbm log directory - a new bpdbm process will be started to check for "empty" tapes, and it will start writing into the bpdbm log file.


2. On the master server, execute the command bpexpdate -deassignempty, and observe the file read failed error message.


3. Search today's bpdbm log file on the master server for the path of the bad image header which caused the bpexpdate -deassignempty command to abort.

If there is an empty image header file in the image database, messages similar to the following may be seen near to the end of the bpdbm log - the last occurrence of the db_get_image_info line must be found with the warning that the image header file is empty, no data retrieved:

15:20:11.884 [18014] <16> db_get_image_info: /usr/openv/netbackup/db/images/nbclient/1212000000/test_policy_1212513625_FULL is empty, no data retrieved
15:20:11.885 [18014] <2> db_error_add_to_file: dberrorq.c:midnite = 1216328400
15:20:11.897 [18014] <16> list_client_images: Bad image header: test_policy_1212513625_FULL, error: file read failed (13)
15:20:11.897 [18014] <2> bpdbm: request complete: exit status 13 file read failed

In the example above, the zero byte backup image header file which caused the bpexpdate -deassignempty command to fail is /usr/openv/netbackup/db/images/nbclient/1212000000/test_policy_1212513625_FULL.

If there is a corrupt image header file in the image database, messages similar to the following may be seen near the end of the bpdbm log - again, find the last occurrence of the db_get_image_info line with the warning that the file may be corrupt:

19:12:56.791 [12920] <2> db_get_image_info: /usr/openv/netbackup/db/images/nbclient/1212000000/test_policy_1212513625_FULL does not contain valid image info, the file may be corrupt
19:12:56.792 [12920] <2> db_error_add_to_file: dberrorq.c:midnite = 1216328400
19:12:56.834 [12920] <16> list_client_images: Bad image header: test_policy_1212513625_FULL, error: events out of sequence - image inconsistency (229)
19:12:56.835 [12920] <2> bpdbm: request complete: exit status 13 file read failed

In the example above, the corrupt backup image header file which caused the bpexpdate -deassignempty command to fail is /usr/openv/netbackup/db/images/nbclient/1212000000/test_policy_1212513625_FULL


4. After the zero byte or corrupt backup image header file has been identified from the bpdbm log, it should be removed.  Also remove any other files associated with the same backup image.  The files associated with the image may include some files in the catstore subdirectory.

In the above example, the files associated with one of the backup images in the example above may include:

Great care must be taken to only remove files which start with the same name and timestamp as the bad image header file which was listed on the is empty, no data retrieved or the file may be corrupt error message in the bpdbm log file (see examples above).  Removing any other files will prevent NetBackup from restoring data from valid backup images.

Note:  Do not remove any other files from the catstore subdirectory, even if they are zero bytes.  It is normal for valid backup images to have zero byte files in the catstore subdirectory.  The failure of the bpexpdate -deassignempty command only relates to zero byte image header files (ending in FULL, INCR, UBAK or UARC) and not to any other zero byte files.  If files relating to other backup images are removed from the NetBackup image database, it will not be possible to restore from the images.


5. Execute the bpexpdate -deassignempty command again - if it still reports the file read failed error, return to step 3 and identify the next bad image header file.

If the bpexpdate -deassignempty command completes with no error message, all of the bad image headers have been removed.  If the command fails with an error other than file read failed, please consult the Veritas NetBackup knowledge base, or open a new support case to report the problem.


6. Finally, the bpdbm log directory created in step 1 can also be removed, if desired.


Alternative Method:
On a UNIX or Linux master server, an alternative method can be used to identify and remove all of the zero byte image header files in a single pass, but this alternative method must only be used when NetBackup is not running.  It is not safe to use this method while NetBackup is running, because it may incorrectly identify zero byte image header files which have been created during running backups.

When NetBackup is stopped, run these commands exactly as shown to search for zero byte image header files in the NetBackup image database on the master server:

# find /usr/openv/netbackup/db/images -name '*_FULL' -size 0
# find /usr/openv/netbackup/db/images -name '*_INCR' -size 0
# find /usr/openv/netbackup/db/images -name '*_UBAK' -size 0
# find /usr/openv/netbackup/db/images -name '*_UARC' -size 0

Each of the zero byte backup image header files identified by the find commands must be removed, and also any other files associated with the same backup image.  The files associated with the image may include some files in the catstore subdirectory.

For example, if an image header file with the path /usr/openv/netbackup/db/images/nbclient/1212000000/test_policy_1212513625_FULL is identified as being size zero bytes, the files associated with the invalid backup image may include:

Great care must be taken to only remove files which start with the same name and timestamp as the zero byte image header files identified by the find commands.  Removing any other files will prevent NetBackup from restoring data from valid backup images.

Note:  Do not remove any other files from the catstore subdirectory, even if they are zero bytes.  It is normal for valid backup images to have zero byte files in the catstore subdirectory.  The failure of the bpexpdate -deassignempty command only relates to zero byte image header files (ending in FULL, INCR, UBAK or UARC) and not to any other zero byte files.  If files relating to other backup images are removed from the NetBackup image database, it will not be possible to restore from the images.

When all of the zero byte image header files have been removed, restart NetBackup, and rerun the bpexpdate -deassignempty command to check that it completes successfully.  If it does not complete successfully, there may still remain corrupt image header files somewhere in the image database.  In that case, the instructions listed above (steps 1-6) will need to be followed as well.