DOCUMENTATION: Best Practice recommendations for enabling and gathering NetBackup (tm) logging
Veritas NetBackup (tm) 6.0 Troubleshooting Guide for UNIX and Windows, Chapter 3: Using Logs and Reports
Veritas NetBackup (tm) 6.5 Troubleshooting Guide for UNIX and Windows, Chapter 3: Using Logs and Reports
Veritas NetBackup (tm) 7.0 Troubleshooting Guide for UNIX and Windows, Chapter 3: Using Logs and Reports
Modification Type: Supplement
For every case that comes into Symantec Enterprise Technical Support, there is a significant percentage that require the collection of log files for troubleshooting. Starting with NetBackup 6.0 there are two styles of NetBackup logging. The previous logging method of creating a log directory for each daemon and enabling logging by changing the verbose setting is now called legacy logging. This method of logging is still used by some daemons within NetBackup.
NetBackup 6.0 also introduces a new style of logging, which is referred to as unified logging (VxUL). Unified logging is used by the Intelligent Resource Manager (IRM) and Enterprise Media Manager (EMM) services. This new logging method is enabled by default and can generate significantly larger log files, especially at higher debug levels. This logging method will keep logs based on the same setting used for legacy logging which is a default of 28 days.
Due to the changes in NetBackup 6.0 there are additional considerations for planning how to enable and gather logging. Enabling logging without the proper precautions can cause NetBackup to consume all available disk space on the system. This can cause problems for NetBackup and the operating system itself. One common problem that has been seen is logging consuming all the space and causing corruption of the EMM database.
The EMM server now uses a relational database to store media and device management information. If the system runs out of disk space this can potentially corrupt the EMM database and require that NetBackup be restored from the last good catalog backup. Depending on when this was last run, this could lead to the loss of valid backup images.
I. Setup of log directories
When setting up logging directories different approaches must be taken for legacy and unified log files. Symantec recommends that log files be moved to a dedicated file system. If this is not possible, then log files should be moved to a location other than the root file system or the file system on which NetBackup or other production data resides.
There are two considerations for selecting a partition to store log files: enough space to contain required logs, and not filling up a production file system. Since some of the logs are quite large, care should be taken to ensure that the file system is large enough to contain required logs. If the file system containing your image database or the EMM database is full, there is a possibility that you could lose data (as described in TechNote 282805, found below in the Related Documents section). It is therefore recommended that a separate file system, at least 64 GB in size be used, and it should be mounted or linked to the appropriate log directory. There are some log directories in that location, and those should be removed before mounting the new file system.
There are different methods to enable legacy and unified logs in NetBackup 6.0.
For legacy logging the verbose level is controlled by the VERBOSE setting in the /usr/openv/netbackup/bp.conf on UNIX systems or the Verbose setting in the Windows registry. Legacy logging also requires that log directories be created for each daemon or process individually. Unless directed by Symantec Technical Support do not use a higher debug level.
To lower the debug level for legacy logging do the following:
Set the Keep logs configuration setting to a lower value. The default is to keep 28 days of debug logging information. Setting this to a lower setting, such as 3 to 5 days, will result in fewer log files being kept on the server.
To change this setting via the NetBackup Java Administration Console:
It may be necessary to have an archive of older logs in order to troubleshoot long-running problems. A good practice would be to create a policy that will back up older logs before they are removed, and retain those logs for a time that is slightly greater than, or equal to, the time between your average full backups. For example, if you run a full backup once a month, then it would be a good idea to retain at least a month's worth of logs for reference.
Refer to the System Administrator's Guide for information on creating a policy. The policy will need to be active, and have a User Backup schedule. It would also be a good idea to keep the volumes used for these backups in a separate pool, so they don't get used for regular backups. Then a shell script could be run from cron to run a user backup everyday to back up older logs. The following script is provided merely as an example, and is not officially supported:
V. Summary of NetBackup logging
Newer features in the NetBackup 6.0 release require additional considerations when enabling and gathering logging. Features such as the EMM database can become corrupt if proper care isn't taken to prevent the file system from running out of disk space.