NetBackup phase 2 import failed - no images were successfully processed

Problem

Phase 1 import is successful and phase 2 import failed with "the image was not created with TIR information" and "no images were successfully processed" error messages.

Error Message

16:53:28 INF - Created bpimport process, pid: 5692
Import phase 1 started 05/24/2016 16:53:29
16:53:29 INF - Create DB information for media id 001380.
16:53:30 INF - Initiation of bptm process to phase 1 import media id 001380 was successful.
16:53:33 INF - Waiting for mount of media id 001380 on server nbmedia1 for reading.
16:54:33 INF - Recreating image information for client nbclient1, policy Backup_nbclient1, and schedule Monthly_Full, done on 06/23/2015 19:39:28.
16:54:34 INF - Recreating copy 1, fragment 1 information for image.
16:55:45 INF - Found existing image information for client nbclient1, policy Backup_nbclient1, and schedule Monthly_Full, done on 06/23/2015 19:39:28.
16:55:45 INF - Recreating copy 1, fragment 2 information for image.
16:56:03 INF - Found existing image information for client nbclient1, policy Backup_nbclient1, and schedule Monthly_Full, done on 06/23/2015 19:39:28.
16:56:03 INF - Recreating copy 1, fragment 3 information for image.
...
16:57:05 INF - Creation of database information for import of media id 001380 complete.
...
10:21:40 INF - Created bpimport process, pid: 3788
Import phase 2 started 05/25/2016 10:21:40
10:21:43 INF - Importing policy Backup_nbclient1, schedule Monthly_Full (nbclient1_1435077568), media id 001380, created on 06/23/2015 19:39:28.
10:21:43 INF - Importing by reading INDEX file information only.
10:21:47 INF - Import of policy Backup_nbclient1, schedule Monthly_Full (nbclient1_1435077568) failed, the image was not created with TIR information.
...
10:21:47 INF - Status = no images were successfully processed.

Cause

The tape header may be corrupted. It's possible that tape drive hardware faults have prevented the Empty Header being written when they should have been, but this is very very rare.

How to see if there is an empty header on the tape:

1. Load the 'problem' tape into a drive with the following command, it will take a few moments for the command prompt to return.
   • Example (to load media ID E03002 into a drive):
     tpreq -m E03002 -f E03002
2. Show which drive the tape is loaded into:
   vmoprcmd
   • Example, command output:
     IBM.ULT3580-TD4.001 Yes Yes E03002 E03002 Yes hcart
  nbmaster1 /dev/nst4 SCAN-TLD
  nbmedia1 {4,0,1,3} ACTIVE
3. In the output from the example above the tape is loaded in drive with the path {4,0,1,3}.
4. Run scsi_command -map -d {4,0,1,3} command (substitute the correct drive path in example below) and send output to a file.
   • Example (Windows):
     scsi_command -map -d {4,0,1,3} > scsi_map_output.txt
   • Example (Unix/Linux):
     scsi_command -map -d /dev/nst7 > scsi_map_output.txt
5. Unload the tape media from the drive with the following command.
   • Example (unloading media ID E03002 from a drive):
     tpunmount -f E03002

Note: The path for the tpreq, vmoprcmd, scsi_command, & tpunmount commands are as follows...

Unix/Linux: /usr/openv/volmgr/bin/
Windows: install_path\Volmgr\bin\

Example (scsi_map command outputs):

• From a customer's server:

  • ...(snip)...
    32768006: file 4: record 1: size 1024: NBU BACKUP header    <<—— This is a valid, not empty header.
              backup_id nbclient1_1435077568: frag 3: file 3: copy 1
          expiration 1592757568: retention 8: block_size 65536
          flags 0x8: mpx_headers 0: resume_count 0: media 001380
32768007: file 4: record 2: size 65536
43046721: file 4: eof after 10278715 records: 673625801728 bytes
eot

• From a Veritas test server:

  • ...(snip)...
    00000006: file 3: record 2: size 4096
00000007: file 3: eof after 2 records: 5120 bytes
00000008: file 4: record 1: size 1024: NBU EMPTY header (file 3)    <<—— This shows an empty header.
    00000009: file 4: eof after 1 records: 1024 bytes
eot

Solution

1. If the tape is full, normally there is no empty header, this would suggest that the backup continued onto another tape (or in other words, when a backup job spans media there is no empty header on the tapes that became full). If this is the case, then there is another tape somewhere that needs importing.
2. If the tape is NOT full (from bpmedialist output), then there should be an Empty Header at the end. If the header is missing it will cause issues as NetBackup will think there are more tapes for that image (because a missing header signifies a tape that became full, and the backup then spanned across to another media).

Note: The path for the bpmedialist command is as follows...

Unix/Linux: /usr/openv/netbackup/bin/admincmd/
Windows: install_path\NetBackup\bin\admincmd\