If the NetBackup Catalog is stored on VxFS filesystem then catalog backup performance might be degraded at 8.1.2

Problem

After installing or upgrading to NetBackup 8.1.2, catalog backup performance may be degraded.  Other master server processing may also be impacted to a lesser extent.  But the impact to all types of processing will increase over time.

Observed on Linux-based master servers with the NetBackup catalog mounted on a VxFS filesystem.  But may potentially affect other non-Windows platforms and other file system types.

This includes NetBackup Appliance master servers which are running version 3.1.2.

Error Message

No errors may be present, but the catalog backup takes an extended length of time to complete the backup of the image catalog (/usr/openv/netbackup/db/images).

In addition, average read speeds of many tens, perhaps hundreds, of milliseconds may be observed for the file system on which the images database resides.  E.g.

$ vxstat -g nbuapp -r -i 1 -c 2

Cause

A change in the NetBackup code resulted in the situation where images which have a large filelist file (greater than 100mb, due to the backup of a million or more files) are written in a highly fragmented manner to the filesystem. Specifically these files are in the /usr/openv/netbackup/db/images/<client>/<ctime>/catstore directories.

If the total size of the filelist file is less than 100mb then they are not impacted as they are consolidated at the end of the backup into a single .f file in /usr/openv/netbackup/db/images/<client>/<ctime>/.

The fragmentation of a recently written large image file can be confirmed with appropriate file system commands.  In the examples below /usr/openv/netbackup/db is a symbolic link to a VxFS file system mounted on /cat/db.  Some file system commands must be executed against the mount path, instead of following symbolic links.  E.g.

/opt/VRTS/bin/fsadm -t vxfs -E -f /cat/db/images/.../large_file

Solution

There are potentially two parts to the solution.

A) Install the hotfix noted below to prevent new backup images, of large size, from being written in the highly fragmented manner.

B) The existing backup images will expire over time, causing the fragmented files to be deleted.  Optionally, the existing image files can be defragmented to improve performance immediately.  This example is for VxFS.

1) Locate the large image files that may be affected.  All such files will be located in a /catstore/ sub-directory.  Live backup files, which should not be defragmented because they are currently being written, will be in a /tmp/ sub-directory and must be excluded.

$ find -L /cat/db/images -type f -size +10000000c | grep '/catstore/' | grep -v '/tmp/' > /tmp/files_to_defrag.txt

If the master server was upgraded to NetBackup 8.1.2 (NetBackup Appliance 3.1.2) from an earlier version, backup images taken prior to the upgrade can be excluded.  This example excludes files older than 30 days.

$ find -L /cat/db/images -type f -size +10000000c -mtime +30 | grep '/catstore/' | grep -v '/tmp/' > /tmp/files_to_defrag.txt

If the binary was applied to the master server, but the defragmentation wasn't completed immediately after,  only the images between the time that server was upgraded to 3.1.2/8.1.2 and until the binary was implemented need to be considered for defragmentation.  

To search for affected files for a given date range:

find -L /cat/db/images -type f -size +10000000c -mtime +[EEB-days-ago] -mtime -[Upg-days-ago] | grep '/catstore/' | grep -v '/tmp/' > /tmp/files_to_defrag.txt

For: 

-mtime +[EEB-days-ago]  :   Use the number of days since installation of the binary.

-mtime -[Upg-days-ago]   :    Use the the number of days since the upgrade was performed.

2) The resulting file list can then be reviewed and defragmented, either all at once, or in smaller chunks.

$ cat /tmp/files_to_defrag.txt |  while read filename ; do /opt/VRTS/bin/fsadm -e -f $filename ; done

Alternatively, if NetBackup is confirmed down, the fragmentation of the entire volume can be reviewed and defragmented.  E.g.

$ netbackup stop
$ bpps -a
$ /opt/VRTS/bin/fsadm -t vxfs -E /cat  | tee /tmp/fragmentation.before
$ /opt/VRTS/bin/fsadm -t vxfs -de /cat
$ /opt/VRTS/bin/fsadm -t vxfs -E /cat | tee /tmp/fragmentation.after 

NOTE: If performing the above steps on a 3.1.2 NetBackup Appliance, it is necessary to specify the full path to the Veritas fsadm utility (/opt/VRTS/bin/fsadm) rather than calling the Linux version (/sbin/fsadm).   Please also double check the ' /sbin/fsadm ' usage, for other UNIX based operating systems when defining the 'vxfs' filesystem type  the ' -F ' switch might be required instead of ' -t '.

A supported hotfix has been made available for this issue. Please contact Veritas Technical Support to obtain this fix referencing this article. This hotfix has not yet gone through any extensive Q&A testing. Consequently, if you are not adversely affected by this problem and have a satisfactory temporary workaround in place, we recommend that you wait for the public release of this hotfix.

Veritas Technologies LLC currently plans to address this issue by way of a patch or hotfix to the current version of the software. Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.