The NetBackup 8.2 Flex application plan and Flex instance rollback guidance and warnings

NetBackup 8.2 Flex application:

The NetBackup 8.2 Flex application intended for Flex 1.2 was posted briefly on our download site prior to discovering issues with the upgrade and because of a very serious data loss condition existing with the rollback of CloudCatalyst. NetBackup CloudCatalyst does not support rollback. 

If you are running NetBackup as a Flex instance, we recommend that you do not attempt the rollback capability, and that you upgrade to Flex version 1.3 and NetBackup version 8.2 when it is reposted in early December.  The new version Flex 1.3 for NetBackup 8.2 application has been corrected for the upgrade, will not allow the rollback of CloudCatalyst, and will provide a warning for master and media server rollbacks that customer support should be engaged to ensure a successful recovery.

Rollback capability for CloudCatalyst on all versions of NetBackup:

CloudCatalyst rollback should not be attempted with any version of NetBackup that offers CloudCatalyst support, and NetBackup Flex master or media server rollbacks should only be attempted as a last resort if there were serious problems with the upgrade.  Customer support should be engaged in master or media server rollback situations to ensure a good recovery.  If a rollback is performed, there is a risk of data loss and data leakage.  The longer the system was up and running prior to a rollback, the greater the chance of data loss and leakage. The data loss is not limited to losing backup data for jobs that run post upgrade and prior to rollback.  

Rollback issues with Flex, all NetBackup versions:

• CloudCatalyst should not be rolled back.  CloudCatalyst configuration information is persisted in the cloud/object storage. The capability to roll back the cloud/object storage does not exist. Rolling back CloudCatalyst after the upgrade completes will result in an unusable system.
• Incremental or transaction log-based database backups that ran post upgrade and prior to rollback. If transaction logs were truncated, a full database backup should be performed to ensure the database is protected.
• Incremental Windows filesystem backups.  If the archive bit is used for incremental backup, it will be reset upon completion of an incremental backup. If a rollback occurs and the incremental backup is lost, subsequent incremental backups will not detect that these files changed, and the files will not be backed up again until a full backup is performed.  A full backup should be performed to ensure that the Windows filesystems are protected. Files that were modified in the lost incremental and subsequently deleted prior to the next full will be lost.  
• Backup expiration catalog and storage inconsistency. If backup images expire and cleanup begins post upgrade and prior to a rollback, there is the chance that backup data will be removed from storage units external to the instance (e.g. a MSDP media server, Cloud or CloudCatalyst storage, OST storage, tape storage). When a rollback of the master server catalog occurs, the catalog will indicate that there is a valid backup even though the data may have been removed from storage. In addition to having backups that cannot be restored, duplicated, or replicated, this has the potential to affect scheduling of subsequent backups (delaying  or performing incrementals instead of fulls).
• Orphaned backups on storage. If backup images are created on external storage post upgrade and prior to rollback of the master server, backup images will exist on storage but not in the NetBackup catalog. This will result in situations where the backups on storage will never be removed (data leakage).  This may be addressed by importing the images from storage and/or leveraging the consistency check tools.

Media server / MSDP rollback (if Flex instance is a media server)

• The backups between upgrade and rollback will not be restorable (all lost) even though NetBackup has those in the catalog.
• Unfinished SLP jobs will fail (NetBackup master and storage inconsistency).
• Deleted backups during the time will come back and become storage leak.