After resizing the virtual disk, there is a potential for data loss when backing up VMware clients with NetBackup using Changed Block Tracking (CBT).

Article: 100014411
Last Published: 2014-12-08
Ratings: 0 0
Product(s): NetBackup

Problem

Document history:
November 7, 2014: Initial publication
November 13, 2014: Updated with additional information
December 8, 2014: Updated with additional information

A potential for data loss has been identified with VMware clients that are using Changed Block Tracking (CBT) after the virtual disk has been resized.  This issue is not caused by NetBackup; it is a third party issue identified by VMware which affects NetBackup VMware backups.

Error Message

There is potential for data loss when VMware virtual machines protected by NetBackup for VMware have the block level incremental backup (BLIB) feature enabled or if the VM has Changed Block Tracking (CBT) enabled.

What is affected:
VMware policies with Block Level Incremental Backup enabled OR any VMware policy backing up a virtual machine that has CBT enabled where the virtual disk (VMDK) has been resized to cross the following boundaries: 128GB, 256GB, 512GB, 1024GB.

Note: These are the boundaries Symantec Corporation has discovered in internal testing to be affected; there may be additional boundaries affected. More information on this issue is available from VMware:

 http://kb.vmware.com/kb/2090639

Cause

The root cause is that CBT information is corrupted when crossing boundaries such as those listed above.

This is silent data loss. A VM recovered from a backup which had corrupted CBT information may still boot and appear as if everything is in place, but a few blocks of data may be missing.  Without knowing specifically which VMDK blocks are missing and which files those VMDK blocks correspond to, this data loss is undetectable.

Solution

VMware has fixed this issue in ESXi 5.0 (Build Number: 2312428) and 5.1 (Build Number: 2323236) which can be downloaded from VMware at the following link:
 https://www.vmware.com/patchmgr/findPatch.portal.

VMware has also committed to fixing this issue in ESXi 5.5 but at this time a patch isn't available for that version.

Important: These patches only prevent CBT from becoming corrupted in the future. It is still necessary to reset CBT as described below to correct any already corrupted CBT.

To correct corrupted CBT information, CBT must be reset.  This will cause the next scheduled backup to include all in-use blocks regardless if it is a full or incremental schedule.  For instructions, please refer to the Related Article linked below.

Note: Please note that unchecking the Enable block-level incremental backup box in the VMware tab of a NetBackup for VMware Policy is not a valid workaround; CBT is used regardless of this setting.

Another workaround is to use this PowerCLI script:
Get the VMs with CBT enabled:
$vms=get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}
 
Create a VM Specification to apply with the desired setting:
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ChangeTrackingEnabled = $false
 
Apply the specification to each VM, then create and remove a snapshot:
foreach($vm in $vms){
$vm.ExtensionData.ReconfigVM($spec)
$snap=$vm | New-Snapshot -Name 'Disable CBT'
$snap | Remove-Snapshot -confirm:$false}
 
Check for success:
get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}

Note: Because this is not a NetBackup issue, there is no way to fully address and detect all affected VMs within the product. Veritas Corporation strongly recommend resetting CBT after any VMDK resize and on any VM where it is known that the disk has been resized in the past.  Veritas Corporation is currently investigating ways the impact of this issue can be mitigated with NetBackup and will update this document as new information becomes available.  Please use the Subscribe via email link to receive email notifications as this document is updated.

Best Practices:
Veritas strongly recommends the following best practices:
1. Always perform a full DR backup prior to making any changes to your environment.
2. Always make sure that your environment is running the latest version and patch level.
3. Perform periodic "test" restores.
 

Applies To

This third party issue currently affects all versions of NetBackup for VMware.

Was this content helpful?