Backup Exec Continuous Protection Server (CPS) Snapshots are being deleted by Windows before reaching the age at which CPS will automatically delete them.
Snapshot deleted successfully by an external source
The shadow copies of volume D: were aborted because of an IO failure on volume D
The oldest shadow copy of volume G: was deleted to keep disk space usage for shadow copies of volume G: below the user defined limit
A Backup Exec Continuous Protection Server (CPS) Snapshot is a Microsoft Shadow Copy of the volume on which the CPS Backup Destination resides. The loss of CPS Snapshots should be considered an event with the potential to result in the inability to restore data; this should be taken very seriously.
It must be understood that CPS cannot prevent the Microsoft Volume Shadowcopy Service from deleting CPS snapshots.
In order to communicate the severity of the loss of CPS snapshots it's necessary to highlight certain aspects of the protection provided by CPS.
CPS Business Server
Business Servers store data that is routinely saved by end-users during the course of their daily operations. Once data is saved to the Business Server, the data may be backed up to a Protection Server.
CPS Protection Server
The Protection Server is the destination of the CPS backup data. It is typically installed at a primary location or a central office. The Protection Server hosts the backup destinations where snapshots are taken.
CPS communicates with the Microsoft Volume Shadowcopy Service to accomplish the following:
1. To generate snapshots on a schedule defined in CPS.
2. To delete snapshots according to the aging policy as defined in CPS.
For more information on defining the CPS snapshot policy please see the CPS Administrators Guide; a link to the guide can be found in the Related Documents section below.
CPS Backup Destination
CPS Backup Destinations are folders that reside on Windows servers that CPS uses to store backup data so that CPS can take advantage of the Microsoft Windows Volume Shadow Copy feature. For more information on CPS Backup Destinations please see the CPS Administrators Guide; a link to the guide can be found in the Related Documents section below.
Why the loss of snapshots should be taken seriously
A CPS backup job will backup data from a CPS Business Server to a CPS Protection Server's Backup Destination. The data held in the Backup Destination are replicas of the original data. When a CPS backup job runs, it replaces the out-of-date data in the Backup Destination with up-to-date data.
In order to provide the ability to restore previous versions of CPS backup data, CPS uses the Microsoft Volume Shadowcopy Service to generate a snapshot of the volume where the CPS Backup Destination resides. These "volume level" snapshots contain the data that CPS must have available in order to restore previous versions of any and all data. If these snapshots are not available, if Windows has deleted these snapshots, then the ability to restore previous versions of any data is not possible. This is why the loss of snapshots should be considered a very serious event.
There are specific situations which can necessitate CPS restore data from the CPS Snapshots. For information on these specific examples please review TECH48459. It is imperative the technote be reviewed, failure to do so could result in the occurrence of a situation in which there is no CPS Backup Data available for restore.
Replacing a Business Server or a protected volume on a Business Server can result in the loss of backup data if the Continuous Backup Job protecting the volume is not canceled prior to the replacement.
Steps which can be taken to mitigate the severity of the loss of CPS snapshots
Adjust the locations of the CPS Backup Destination and Windows Shadow Copies
Use a dedicated volume for the CPS Backup Destination. Windows Shadow Copies are generated for the entire volume, not just a specific directory. Using a dedicated volume will keep the size of the shadow copies lower than when using a volume where additional data is stored.
Use a separate, dedicated hard drive (not just a logical volume) for Shadow Copies. The distinction between a dedicated volume for the Backup Destination and the dedicated hard drive for the snapshots is made because most events that result in the loss of snapshots is due to activity on the hard disk where the snapshots reside, even if that hard disk activity is taking place to satisfy the needs of a different volume. A sperate dedicated hard drive for snapshots will mean that only snapshot activity will result in hard disk activity. Configuring the location of Shadow Copies is done through the Shadow Copies tab in the properties of a drive, for more information see the section below titled Shadow Copy Settings. It will be necessary to delete any existing shadow copies prior to making this change. To save current CPS snapshot data use Backup Exec for Windows Servers as described below in the section titled Archive the CPS Snapshots to Tape.
An example of the strategy above:
1. Place the Backup Destination on the D: drive and dedicate the D: drive to only the Backup Destination.
2. Configure the Shadow Copies for the D: drive to be stored on the E: drive and dedicate the E: drive to only the Shadow Copies.
Archive the CPS Snapshots to Tape
To preserve the data in the current CPS snapshots, use Backup Exec for Windows Servers to backup the CPS shadow copy contents on a regular basis. For more information on how to backup CPS components with Backup Exec for Windows Servers please review the following:
http://www.symantec.com/docs/TECH47911: What are the Backup Exec Continuous Protection Server (CPS) components that are displayed in Backup Exec for Windows Servers?
Adjusting the CPS Snapshot Schedule
It may be necessary and prudent to adjust the schedule on which CPS generates snapshots. Don't forget the purpose of the snapshot feature is to provide multiple file versions for restore and the default snapshot schedule may not be the best strategy for every CPS backup schedule.
If CPS is backing up data on a specific schedule then there is no reason to have CPS generate more than one snapshot per Backup Destination per day. Suppose the CPS backup job starts at 6:00 PM, after everyone has left for the day and the job lasts approximately 1 hour; schedule CPS to generate a single snapshot at approximately 8:00 PM to allow sufficient time for the job to come to a completion.
If CPS is continuously backing up data from Hour A through Hour B then schedule CPS to generate snapshots at a reasonable frequency after the backup period has started and until shortly after the backup period has expired. Suppose the CPS job starts at 8:00 AM and expires at 6:00 PM. There is no reason for a snapshot to be generated at job start time, so set the first snapshot to be generated at 9:00 and schedule the snapshot frequency at a rate the disk subsystem can handle. A successful snapshot frequency may be every hour if the disk subsystem can maintain that pace or lower if not, there is no benchmark and performance will vary.
If CPS is continuously backing up data 24 hours per day then schedule the snapshots at a frequency the disk subsystem can maintain. Again, there is no benchmark and performance will vary.
For directions on settings a CPS snapshot schedule please refer to the CPS Administrators Guide.
The CPS Indexing service sole function is to index CPS snapshots and provide those indices to the user via the Search and Recent Activity tab views in the BERetrieve web console. If snapshots are deleted, the Indexing service will attempt to revise the current index by removing the deleted data and updating the indices. It is possible the loss of "some" snapshots could be to such an extent that all snapshots may need to be deleted in order for successful indexing operations to resume. If it does become necessary to delete all snapshots and start over, use Backup Exec for Windows Servers as described above in the section titled Archive the CPS Snapshots to Tape to save the current snapshot data.
Shadow Copy Settings
CPS does not control or allow access to Windows Shadow Copy settings. These settings are managed via the properties of a drive as seen below in figures 1 and 2.
The Shadow Copies tab, as seen below in figure 1, allows the selection of any drive and then selecting the Settings button to adjust Shadow Copy properties.
The Settings window displays the volume being managed, the volume where Shadow Copies are stored and the maximum size limit, if enabled. The location of shadow copies can be changed only when there are no shadow copies for the selected volume.
Windows also provides a command line interface called vssadmin as seen in figure 3. It is not possible to delete CPS shadow copies via vssadmin as CPS shadow copies are application shadow copies and are excluded from vssadmin delete operations. For additional information on vssadmin please refer to Microsoft documentation.
Viewing Shadow Copies
The shadow copy files, which are also referred to as snapshots, are located in the <drive>:\System Volume Information folder as seen below in figure 4. In order to view the contents of the folder it will be necessary to grant access to the logged in user.
Loss of CPS Shadow Copies Due to an External Source
Windows can delete CPS snapshots for a variety of reasons, which typically include but are not limited to:
Heavy Disk I/O
Low Disk Space
Total Number of Snapshots Reaching 64.
Defragmenting a Disk with a Small Cluster size.
When Windows deletes a CPS snapshot an alert will appear on the CPS Active Alerts tab with the following warning:
Snapshot deleted successfully by an external source
Make note of the time this alert was recorded in CPS Alerts as it will help locate a corresponding error in the Event Viewer logs.
Why Windows deletes CPS snapshots?
To determine why Windows has deleted a CPS snapshot examine the System log and look for an Error or Information event as seen in the examples below. Use the time of the CPS Alert to help narrow the search for corresponding events.
Example of Disk I/O causing a snapshot to be deleted
Event Type: Error
Event Source: VolSnap
Event ID: 14
Description: The shadow copies of volume D: were aborted because of an IO failure on volume D:.
Example of Low Disk Space causing a snapshot to be deleted
Event Type: Information
Event Source: VolSnap
Event ID: 33
Description: The oldest shadow copy of volume G: was deleted to keep disk space usage for shadow copies of volume G: below the user defined limit.
Using Performance Monitor to Validate the Events
If there is any doubt as to the validity of the events, then the Windows Performance Monitor can be employed to gather information about the disk to aid in understanding the conditions at the time of the snapshot deletion. The following steps can be used to configure Windows Performance Monitor to gather information relative to Disk I/O and Disk Space.
Open Windows Performance Monitor by clicking Start > Run and inputting perfmon as seen below in figure 5.
Windows Performance Monitor will open as seen below in figure 6.
To create a new log, right click Counters and select New Log Settings as seen below in figure 7.
Create a name for the new log as seen below in figure 8.
Click the Add Counters button as seen below in figure 9.
Select PhysicalDisk for the Performance Object and select the following counters for the disk where the snapshot of the Backup Destination resides:
· Avg. Disk Queue Length
· Avg. Disk Read Queue Length
· Avg. Disk Write Queue Length
· Avg. Disk sec/Read
· Avg. Disk sec/Transfer
· Disk Writes/sec
An example can be seen below in figure 10
Select LogicalDisk for the Performance Object and select the following counters for the disk where the snapshot of the Backup Destination resides:
· % Free Space
· Free Megabytes
An example can be seen below in figure 11
Below in figure 12 is an example of what the log Counters will display.
As seen below in figure 13, on the Log Files tab select the "Log file type" as Text File (Tab delimited), set the "End file names with" to yyyymmddhh.
On the Schedule tab set the schedule for starting and stopping to manual as seen below in figure 14.
Figure 15 shows the prompt to create the directory where the log files will be kept.
Figure 16 shows the newly created log object in Performance Monitor Counters.
To have Performance Monitor start capturing data based on the log, right click the log and select Start as seen below in figure 17.
To have Performance Monitor stop capturing data based on the log, right click the log and select Stop as seen below in figure 18.
Figure 19 shows the directory where the Performance Monitor log was saved.
This log can be opened in text format or imported into a spread sheet application that accepts tab delimited files.
Interpreting the data
Interpretation of the data should be done by a technician familiar with the disk subsystem and the Windows operating system. For this reason it is recommended the logs be reviewed by a support technician for the hardware vendor and/or a support technician for Microsoft.
Low Disk Space
The data relative to Low Disk Space should be self-explanatory. If low disk space data is recorded close to the time a snapshot is deleted then a corresponding Event in the System log that states a snapshot was deleted due to low space is expected.
Heavy Disk I/O
The data relative to Disk I/O may be more difficult to understand and may require contacting Microsoft to get a complete understanding. But the following information can be considered:
Counter: PhysicalDisk \ Avg. Disk Queue Length
Threshold: Should not be higher than the number of spindles plus two.
Significance: This counter indicates the average number of both read and writes requests that were queued for the selected disk during the sample interval.
Counter: PhysicalDisk \ Avg. Disk Read Queue Length
Threshold: Should be less than two.
Significance: This counter indicates the average number of read requests that were queued for the selected disk during the sample interval.
Counter: PhysicalDisk \ Avg. Disk Write Queue Length
Threshold: Should be less than two.
Significance: This counter indicates the average number of write requests that were queued for the selected disk during the sample interval.
Counter: PhysicalDisk \ Avg. Disk sec/Read
Threshold: No specific value.
Significance: This counter indicates the average time, in seconds, of a read of data from the disk.
Counter: PhysicalDisk \ Avg. Disk sec/Transfer
Threshold: Should not be more than 18 milliseconds.
Significance: This counter indicates the time, in seconds, of the average disk transfer. This may indicate a large amount of disk fragmentation, slow disks, or disk failures. Multiply the values of the Physical Disk\Avg. Disk sec/Transfer and Memory\Pages/sec counters. If the product of these counters exceeds 0.1, paging is taking more than 10 percent of disk access time, so more RAM is needed.
Counter: PhysicalDisk \ Disk Writes/sec
Threshold: Depends on manufacturer's specification.
Significance: This counter indicates the rate of write operations on the disk.
If, after a review of the log information gathered, there does not appear to be a satisfactory reason explaining the deletion of the snapshots, then please contact Microsoft to enlist their help in determining why the Microsoft Volume Shadowcopy Service has deleted the shadow copy snapshots generated by CPS.
For additional information on Microsoft documentation that may help to interpret the Performance Monitor log and Microsoft Shadow Copies see the following articles:
Measuring Application Performance
Examining and Tuning Disk Performance
Designing a Shadow Copy Strategy
How to enable the Volume Shadow Copy service's debug tracing features in Microsoft Windows Server 2003
Volume Shadow Copy Service Technical Reference