How to expire images on VTL tape

Problem

How to expire images on VTL tape?

Solution

A Virtual Tape Library (VTL) is the same as a physical tape library to NetBackup (NBU). Virtual tapes are the same as physical tapes.  Although a backup image may expire, until all backup images on a tape expire, the tape cannot be reused, and on virtual tapes space doesn't become available.  When an available tape gets selected for a backup it gets labeled when it is first put into a drive.  This is the case for a new tape or a tape that has been written to in the past and for a tape that has been written until it is full.  Any tape in a library, Virtual or Physical, that was once full will stay that way until that tape is labeled.  In a VTL, and especially a VTL on top of a deduplication engine there can be a time lag from when a tape is labeled (which notifies the VTL to erase the data on the tape) until that data is erased and space is made available.  

Because backup images are stored on virtual tape cartridges, until all images on a virtual tape have expired, the "tape" cannot be reused and thus disk space is not freed up on the VTL. In fact, even when all images have expired, the media has been unassigned, and the virtual cartridge is returned to the NetBackup Scratch Pool or the volume pool to which it was originally assigned, the VTL is not aware of this and does not free up the space. Only when NetBackup writes from BOT (beginning of tape) does the VTL become aware the space can be freed up.

Most VTL vendors allow customers to configure tape cartridges of smaller capacity than the physical tape cartridge provides. This helps to increase the amount of free space because a smaller amount of capacity is tied up when all images on a piece of media have not expired. In addition, most VTL vendors do not allocate the entire capacity of the cartridge when it's configured (only a relatively small amount) and only allocate more space as needed to minimize the free space issue. This does not cause any problem because NetBackup is not aware of media capacity, but just keeps writing to the media until it reaches End-of-Tape (EOT).

Another way to free up space on a VTL more quickly is to configure virtual tapes with very small capacities. However, this results in more NetBackup overhead in managing a larger number of virtual media and for vendors that copy virtual to physical tape, this would result in a huge amount of wasted space on physical tape cartridges.

It's possible for a customer to force the VTL to free up space on unassigned media by re labeling the media. A customer could write a script that looked for unassigned media (or any media returned to the scratch pool), verifies it is media in a VTL, and then re-labels the media.  However, once media is relabeled the backup images can no longer be accessed and the media can no longer be imported. Re labeling unassigned media or media returned to the scratch pool is also a good way to make any backup images on the tape inaccessible, if company or industry regulations mandate doing so.

A method to use to avoid filling up a VTL is to leave a percentage of the space unused.  If every tape in a VTL were written to FULL the VTL will be full.  If a tape becomes available in NetBackup and is selected for a backup it will be labeled.  This kicks off the processes the underlying disk storage system uses to remove data and reclaim space.  It is not a simple 'rm' command and can take a significant amount of time before space becomes available.  Therefore a labeled tape will still return a write error if the disk subsystem has not freed up space.  However, if a percentage of the library's storage space is unused then upon labeling a virtual tape, the disk system will begin its clean up process and eventually return whatever space the tape was using to free space and while that is happening data can be written to the virtual tape but will go to other space on the disk.  NetBackup doesn't recommend a specific amount of space.  Five or ten percent might be acceptable depending on the length of time required for the disk system space reclamation processes to reclaim free space.  Label 5-10% of the tapes to make sure they are not hanging onto any space on the disk then move them to a pool that doesn't get used.