Problem
A NetBackup NDMP backup policy, which is protecting an EMC Network Attached Storage (NAS) device, can be configured to run "tar" format NDMP backups by including the directive “set type=tar” in the policy's backup selection.
For the reasons explained below, Veritas recommends against using the directive “set type=tar” in NDMP backup policies protecting EMC NAS devices.
If the directive “set type=tar” is present in a policy's backup selection, and the policy is used to perform an incremental NDMP backup of an EMC NAS device (client), it is possible that information about one or more of the parent directories in the path of a file saved by the incremental backup may not be present in the backup image. This can happen because, on an EMC NAS device, a “tar” format incremental NDMP backup image only contains data for files and directories that were created or modified since the last backup of the client at a lower dump level.
A "tar" format incremental NDMP backup image of an EMC NAS device only includes information about a directory if one of the following applies:
- The directory was created or renamed since the last backup
- The directory had a new file/sub-directory created in it since the last backup
- The directory had an existing file/sub-directory in it renamed or deleted since the last backup
Any existing directory that does not satisfy any of the above conditions does not have its information saved in a "tar" format incremental NDMP backup image. This is a limitation of the implementation of "tar" format NDMP backups on the EMC NAS device, and is outside of the control of NetBackup.
Note: A file that has been modified since the last backup is always saved by an incremental NDMP backup, even if information about its parent directory is not saved in the incremental backup image.
If information about some directories is missing from an incremental NDMP backup image, for the reason described above, this may cause difficulties for users or adminstrators who want to perform a partial restore of some of the files and/or directory trees from that incremental backup image.
Selecting a directory for restore in the NetBackup Backup, Archive, Restore (BAR) GUI is an easy way to tell NetBackup to restore the whole of the directory tree including all of the files and sub-directories beneath that directory, instead of having to select every one of them individually - but if information about a directory was not saved in an incremental NDMP backup image, then it will not be possible to select the directory when browsing the backup image in the BAR GUI, and this method cannot be used to select everything beneath that directory for restore - it would be necessary to individually select each of the files and/or directories beneath the missing directory. If the tree beneath a missing directory is very wide and deep, it may require a very large amount of time and effort to manually search for and select all of the files and sub-directories in the tree beneath the missing directory, which were saved by the incremental NDMP backup.
This problem does not affect a restore of all of the data in the backup image, because information about the top level directory (the path specified in the backup policy) is saved in all backup images, and so it is always available for restore selection.
There are also data security risks, that may arise because information about some directories has not been saved in a "tar" format incremental NDMP backup image. If files are being restored only from an incremental NDMP backup image, and NetBackup needs to re-create part of the directory structure above those files (because it has been deleted from the client, or because the files are being restored to an alternate location), then it will not be possible to restore the original security attributes of any directories that are not present in the incremental NDMP backup image. The access permissions on the directories that are created to hold the restored files may not be the same as the original directories. This problem does not arise if data are restored from the full backup image before restoring data from its associated incremental backup image(s).
Example:
A volume on an EMC NAS client contains two files, with full paths /fs6/projects/project1/data1 and /fs6/projects/project1/data2. An NDMP backup policy is configured to use the "
tar" format backup type to protect all data beneath the volume /fs6, with these directives in the policy's backup selection:
set type=”tar”
/fs6
If a full backup of /fs6 is run by the NDMP backup policy, the full backup image will contain data for all of the files and directories beneath /fs6:
/fs6/
/fs6/projects/
/fs6/projects/project1/
/fs6/projects/project1/data1
/fs6/projects/project1/data2
If the file “data2” is modified after the full backup is complete, and an incremental backup of /fs6 is run by the NDMP backup policy, the incremental backup image will only contain data for the modified file “data2”, and for the directory /fs6 at the top level of the backup path. Only the following paths will be included in the "
tar" format incremental backup image:
/fs6/
/fs6/projects/project1/data2
The incremental backup image will
not contain data for the parent directories which are in the path of the modified file, or for the unchanged file “data1”, because they have
not been modified since the full backup (updating the file "data2" does not modify the directory "/fs6/projects/project1/"). The following paths will
not be included in the "
tar" format incremental backup image:
/fs6/projects/
/fs6/projects/project1/
/fs6/projects/project1/data1
The reason why the parent directories /fs6/projects/ and /fs6/projects/project1/ are
not included in the incremental NDMP backup image is because the EMC NAS device's implementation of the NDMP “
tar” format incremental backups does
not save data for directories which have
not been modified since the last backup run at a lower dump level, even when they are in the path of a file that was saved by the incremental backup.
Data Restore Difficulties:
If a directory is not present in an incremental backup image, it is not possible to select the directory for restore when using the NetBackup
bprestore command, or when browsing the image in the NetBackup Backup, Archive, Restore (BAR) GUI. As a consequence, when browsing a "
tar" format incremental NDMP backup image of an EMC NAS client for restore in the NetBackup BAR GUI, the check boxes for some of the directories may be grayed out, and unavailable for selection, because they are not present in the incremental NDMP backup image:
This behavior is also seen when browsing for restore in a date range that includes both full and incremental NDMP backup images. If a directory was not saved by the
most recent incremental NDMP backup in the date range selected for restore, the directory will be shown as grayed out in the NetBackup BAR GUI.
In the above example, where both full and incremental NDMP backup images are in the selected date range, if a NetBackup user or administrator wants to restore everything in the directory structure beneath directory /fs6/projects/, this cannot be achieved by selecting the /fs6/projects/ directory in the GUI, because the directory is not present in the most recent incremental backup image, and so the check box is grayed out. It will be necessary to manually browse the directory structure beneath the directory, and select all the files and directories to be restored individually, as shown below:
In this example, the directory structure beneath /fs6/projects/ is very simple, and contains only two files, so it is very straightforward to select all of the files to be restored - but in a production environment the directory structure may include a very large number of sub-directories, with many levels of files and directories beneath each of them, and browsing the whole directory structure to make sure that all required files and directories are selected for restore may involve significant effort.
Data Security Risk:
If it is necessary to restore files to an alternate location, where the complete directory structure for the files does not already exist, NetBackup re-creates the directory structure by using the information (owner, group, permissions) of each directory path, which is stored in the backup image.
If some directories are missing from a “
tar” format incremental NDMP backup image written by an EMC NAS device, there may be an unwanted outcome when restoring files from an incremental NDMP backup image alone (i.e, when
not restoring from the full NDMP backup image of the client before restoring from the incremental backup images).
When files are restored only from a “
tar” format incremental NDMP backup image, any directories in the path of the restored files, that are not present in the incremental backup image, are created by NetBackup, and the directory ownership is set to "root", with default access permissions, which may allow world read and write access to the directory. This may have implications for data security on the client.
The only way to be sure that the correct ownership and permissions are set on all directories created during a restore from "
tar" format NDMP backups of an EMC NAS client is to select a date range for restore, that includes a full backup image and all of the incremental backup images following it, up to and including the desired restore date.
Notes:
A backup path that is specified in a policy's backup selection (/fs6 in the above example) is always present in all full and incremental backup images written by the policy. Selecting the top level of the backup path for restore results in the restore of the latest copy of all files and directories beneath the path, that were saved by the backups in the selected date range, even if they are shown as grayed out in the GUI - but bear in mind that, if restoring data only from incremental backup images, a directory that is not present in any of the backup images in the selected date range may not be restored with the correct security attributes.
For NDMP backups, dump level zero is used for a full backup, and contains all files and directories beneath the backup path; dump level 1 is a cumulative incremental backup, and contains files and directories that have been modified since the last full backup; dump levels 2 to 9 are used only for differential incremental backups, and contain files and directories that have been modified since the most recent backup that was run with a lower dump level.
Cause
Solution
For the reasons explained above, Veritas recommends against using the directive “set type=tar” in NDMP backup policies protecting EMC NAS devices.
It is preferable that the “dump” or “vbb” backup formats should be used in NDMP backup policies protecting EMC NAS devices.
When using the NDMP "dump" or "vbb" backup formats, it is strongly recommended that snapshots should be enabled, by use of the “set snapsure=y” directive in the backup selection of the NDMP backup policy, to avoid backup failures resulting from inode number re-use in “dump” or “vbb” format NDMP backups of large, busy volumes.
Examples:
NDMP backup format “
dump” (this is the default backup format, so it is not necessary to include a "set type=dump" directive in the policy's backup selection):
set snapsure=”y”
/fs6
NDMP backup type “
vbb”:
set type=”vbb”
set snapsure=”y”
/fs6
Notes:
When using the NDMP "vbb" backup format, only the top level directory of a volume can be included the backup selection in the policy.
A snapshot requires available free space on the volume - the amount of free space depends on how much of the data on the volume are changed during the backup - 20% free space is typically recommended.
Applies To
All EMC NAS devices that support NDMP with the “tar” backup format. This behavior has been verified on VNX and Celerra NAS devices.
This problem does not affect Hitachi NAS devices, which support NDMP with the “tar” backup format.
