Problem
When comparing the byte count and job duration of differential and incremental backup jobs of VMware virtual machines, the figures indicate that a full backup has been run.
A review within the vSphere Client of the summary information for the virtual machines does show that they are configured to use VM Hardware Version 7 and so meets the required level. However, if the Changed Block Tracking settings within each virtual machine are not set, then a full backup is the result.
Cause
Backup Exec has been designed to create or modify these settings without needing to shutdown your virtual machines and attempts to make the required changes at the start of the Full Backup job. The settings will remain in place once they are correctly set. In some environments the request by Backup Exec to modify the settings fails and causes a full backup to be run even when a differential or incremental backup is selected.
It is possible to review the ctkEnabled settings inside the VMX files of the virtual machines (without shutting them down). In order to review the Changed Block Tracking Settings inside the vSphere Client:
1) Make sure the policy based full backup job of the Virtual Machine has run at least once (there will correctly be no settings if this has not happened)
2) Shutdown the virtual machine
3) In the vSphere Client, edit the settings of the virtual machine
4) Click on the Options tab
5) Under Advanced section select General
6) Click the Configuration Parameters button - this will be grayed out if the virtual machine is running.
7) Confirm if ctkEnabled and one or more scsiN:N.ctkEnabled entries exist and are set to true, as can be seen in Figure 1 below.
NOTE: there should be one scsiN:N.ctkEnabled (where N:N represents the scsi device identifiers) per virtual Hard Disk
Fig 1. Correct Changed Block Tracking Settings
* If these settings are not created and set to true by the initial backup, then all jobs will run as a full backup.
** Investigation has shown that this can be caused by the credentials in the Backup Selection List being linked with an ESX security role that does not have enough permissions to modify the changed tracking settings. In particular the role called "VMware Consolidated Backup user (sample)" does not have enough permissions.
Solution
Configure the Windows based user (if using a Virtual Center) or the ESX/Linux user (if using direct to ESX Host) so it is linked to an ESX security role that has enough permissions. The specific requirements for AVVI Backups are discussed in the article provided at the bottom of this document.
To confirm which User or Group of users is assigned to a specific Virtual Machine and which role they are assigned, review the permissions screen/tab of the Virtual Machine itself inside the vSphere Client. Additionally, this screen also shows what level the permissions are defined as VMware security does have a concept of security inheritance (propagation).
To review the user and role assignments applied to each datastore (which can also affect the backup and restore processes), inside the vSphere Client, select View --> Inventory --> Datastores and then select the required datastore and click the Permissions tab.
To review the specific permissions applied to each role, in the vSphere Client select View --> Administration --> Roles from the menu. Then right click the role of interest and select "Edit Role..." Note: the built in roles of Read-only and Administrator cannot be edited, but can be cloned to either review the settings or base a custom role around the settings.
In practice, once the correct security is configured, most systems will then perform incremental and differential backups. However some virtual machines may require a one-time reboot due to the running state of the system still denying the ability to enable the Changed Tracking settings. Veritas cannot predict which virtual machines might need a restart.
Workaround:
Should the above solution fail to allow differential and incremental backups to run correctly, then follow How to reset Change Block Tracking (CBT) for VMware Backups.