Problem
NetBackup Unified and VX Logs and Troubleshooting
Solution
This article is designed to assist in gathering information to allow troubleshooting of an issue with NetBackup.
Initial Information recommended for all issues:
- The exact error message (if an error is given)
- During which type of operation does the problem occur ? For example, Backup, Duplication, Vault, Restore etc.
- At what point in the job does the issue occur, for example, right at the beginning.
- Which system(s) are involved? For example, Master Server, Media Server - Please provide system names.
- Please submit the support report from the master and any relevant Media Servers in the environment.
- Is this a reoccurring issue ?
- What was the system doing at this time ?
- Please send in the contents of the details tab in the Activity Monitor for the job.
- What logs or screenshots are available?
- Has this worked previously? If so, when was the last time it DID work?
- Have there been any environmental; system; or configuration changes?
- When did the problem begin?
- How often does the problem occur?
- Is a similar configuration working properly?
/usr/openv/netbackup/bin/goodies/support/support >/tmp/support.txt
nbsu report:
/usr/openv/netbackup/bin/support/nbsu, The output is located in a subdirectory 'output'
For example, /usr/openv/netbackup/bin/support/output/nbsu
Logs:
Generally, for any reported problem the logs of the daemons/processes involved in that function will be requested. Note, that verbosity level at time of opening a case may not be sufficient and therefore, the issue will need to be captured with the level of logging increased and perhaps further set up/tuning of logs. In addition, these, initial investigations may lead down a sub-route that may require a repeat collection with additional logs enabled to continue the diagnostic process. Troubleshooting NetBackup can at times be complex, and it is not always possible to know whether a small set of logs would be required or a larger set. This then raises the question of gathering perhaps a small set of logs on the understanding that further more detailed logs may be required (thus saving the time and effort for the customer of gathering a large set that may not be needed).
Time scales:
Dependant on the issue, the logs covering just the time of the issue may be sufficient. For example, in the case of a Tape Drive going down it is likely that the logs just a few minutes either side of issue will suffice. In a more complex case, for example a resource or performance problem, then the logs several hours before the issue became apparent may be required.
System Design:
Consideration may be given to creation of separate filesystems for the main log areas, /usr/openv/netbackup/logs and /usr/openv/logs.
Types of logs:
NetBackup (version 6.0 and higher) uses two types of logging termed :
1. Legacy Logs : Used in all version of NetBackup, Legacy Logs are straight text files.
2. Unified or VX Logs: Used in NetBackup 6.0 and above. These logs need processing with the vxlogview command to make them readable.
Legacy logs:
These are located in /usr/openv/netbackup/logs, and /usr/openv/volmgr/debug, for example (bptm process)
/usr/openv/netbackup/logs/<process>/date.log
To enable a legacy log, the directory must be created in /usr/openv/netbackup/logs. Optionally, the VERBOSE entry can be set in either /usr/openv/netbackup/bp.conf or /usr/openv/volmgr/vm.conf. This is covered in more detail later.
Unified or VX Logs:
These are located in:
NetBackup - /usr/openv/logs/<process>
The log files following the naming convention :
<product id>-<originator id>-<host id>-<date>-<rotention>.log for example, 51216-111-2220372284-091020-0000000017.log
Unlike unified logs, VX logs require processing via the vxlogview command. The process that the log refers to is represented by the 'originator' id, that is the 2nd, 3 digit number. In the above example, 111 represents the NBEMM process.
To enable a VX log, the debug and diagnostic levels are set using the vxlogcfg command.
Originator ids:
The originator ids for a given process can be found in /usr/openv/netbackup/nblog.conf Although this file should not be edited manually, it is a useful reference and also shows the logging levels set, if other than default.
143.L10nResource=emm
143.OIDNames=mds
143.DebugLevel=6
143.DiagnosticLevel=6
The above example shows the originator id for the mds process, a sub process of nbemm.
Some processes do not create separate log files. For example, the mds process (143) will log into the main nbemm log.
Verbose settings for logs in /usr/openv/netbackup/logs:
By default, the unified verbose level is set to 0, this provides little information in the logs files. On occasion it may be sufficient, but in most cases there will be a requirement to increase this.
There are two ways this can be done :
To increase the verbose level of all logs (except vault)
Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf. <level> is a value between 0 and 5, with 5 being the highest.
To increase the verbose level of a specific process:
Add the entry <PROCESS>_VERBOSE = <level> into /usr/openv/netbackup/bp.conf, for example
BPTM_VERBOSE = 5
VAULT_VERBOSE = 5 (note, this must be set to increase the verbose level for /usr/openv/netbackup/logs/vault).
Note: Not all processes can be individually selected for a custom VERBOSE level in the bp.conf file.
Verbose settings for logs in /usr/openv/volmgr/debug:
For these logs, the only option is to add VERBOSE into /usr/openv/volmgr/vm.conf. No numerical value is accepted.
The vm.conf file contains configuration entries for media and device management. NetBackup can create this file, but if it does not exist, you must create it.
On Windows, the pathname is install_path\Volmgr\vm.conf.
On UNIX, the pathname is /usr/openv/volmgr/vm.conf.
Various NetBackup components read this configuration file on the host where the component runs. The NetBackup component is a command, daemon, process, or utility. The host can be a NetBackup administration client or a server where administration operations are requested
The logs available are as follows:
/usr/openv/volmgr/debug/acssi
/usr/openv/volmgr/debug/acsd
/usr/openv/volmgr/debug/robots
/usr/openv/volmgr/debug/daemon
/usr/openv/volmgr/debug/ltid
/usr/openv/volmgr/debug/oprd
/usr/openv/volmgr/debug/reqlib
/usr/openv/volmgr/debug/tpcommand
Further increased logging can be created by adding creating one or more of the following 'touch' files.
touch /usr/openv/volmgr/DEBUG
touch /usr/openv/volmgr/ROBOT_DEBUG
touch /usr/openv/volmgr/AVRD_DEBUG
touch /usr/openv/volmgr/SSO_DEBUG
touch /usr/openv/volmgr/database/DEBUG
touch /usr/openv/volmgr/VM_MQ_DEBUG
touch /usr/openv/volmgr/DRIVE_DEBUG
A process must restart to pickup changes in either the bp.conf or vm.conf files. Therefore processes such as bpbrm/ bptm will have an increased logging level as soon as a new backup starts. However, ltid or robots would not, and so the media manager processes would need to be restarted,
sg debugging (Solaris only)
It is possible to capture a debug log for the sg driver, this can be achieved one of two ways.
Add the following line to the sg configuration /kernel/drv/sg.conf file:
debug-level=1;
Reload the sg driver. This can be achieved either by rebooting the system or unloading and reloading the sg kernel module (see the Solaris man pages for modunload(1M) and modload(1M) for details).
Additionally it is possible to edit the /etc/system file. The sg driver has a verbosity flag called "sg_err_level" which defaults to 3. The sg driver can be made very verbose by adding the following to the /etc/system file, and then rebooting the machine, or, reloading the sg driver.
set sg:sg_err_level=4
The values between 3 and 9 may be used for different levels of
verbosity. Level 9 is the highest. Engineering advice is not to use
levels beyond level 4.
Java Logs:
On the Master server edit the file /usr/openv/java/Debug.properties
Add or uncomment the following lines :
printcmds=true
printCmdLines=true
debugMask=0x00040000
In legacy versions, the last setting should be:
debugMask=2
In /usr/openv/netbackup/logs create the following directories
bpjava-msvc
bpjava-susvc
Restart the GUI
When the issue is experienced in the GUI, run the following command :
/usr/openv/java/get_trace
A .log file will be created under /usr/openv/netbackup/logs/user_ops/nbjlogs for each instance of Java GUI you have open.
Further detail on VX logs:
The vx logs are created in the /usr/openv/logs directory by default, they have the following format, the first and second fields representing the -p and -o options used in the vx... commands.
51216-111-2201603439-080923-0000000000.log
To enable these logs, set the DebugLevel and DiagnosticLevel using the vxlogcfg command:
For example, to increase the 111 verbose level,
vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
The setting change can be confirmed with the following command:
vxlogcfg -l -p 51216 -o 111
The default logging levels for vx logs are as follows: DebugLevel = 1 DiagnosticLevel = 6
vx logs are created in a raw format, and are virtually unreadable until processed through the vxlogview command. This increases the size of the logs.
For this reason it may be advantageous to send the logs to Veritas in this format. Additionally, should a case require escalation to Engineering, very often the logs are requested in the raw format.
Collecting VX logs:
The vxlogview command has an option to process the logs for a given jobid. For this reason, even if vxlogview is run for a specific originator id, all the logs contained in /usr/openv/logs are searched. Processing individual originator ids can therefore be speeded up by copying the required logs to an alternative location and running the vxlogview command with the -G <location of logs> option.
This method can also be used in general if the raw logs are to be sent in, as it easily collects the required logs into one location.
Copying VX logs to an alternative location:
There are 3 ways to copy the logs ( the directory specified with -f must be on the same device as opt/openv/netbackup/logs ). Only one of the following examples needs to be used.
(1) Copy all the 111 (nbemm) logs for the past 'n' days into /tmp/vx .
vxlogmgr -o 111 -c -f /netbackup/tmp -n 1
(2) Copy all the 111 (nbemm) logs for the past 1 hour, relative to when the command was run: This is very simple, however please note that this command is relative to when it was run, for example if you run the command at 15.00, it will copy the logs between 14.00 - 15.00. Therefore it is vital to ensure that a sufficient time is specified to allow the collection of logs over a time period large enough to capture the issue.
vxlogmgr -o 111 -c -f /netbackup/tmp -t 01:00:00
(3) Copy all the 111 (nbemm) logs between the two times/dates ... This is perhaps the best and most accurate method. However, please see the 'Important note' below.
vxlogmgr -o 111 -c -f /netbackup/tmp -b '10/21/09 12:01:22 PM' -e '10/21/09 12:21:22 PM'Important note regarding format of <date/ time> for -b or -e options with vx commands
The format of the date used in example (3) may vary depending on the locale settings of the system. Running the vxlogview --help command and looking for the -b or -e options (begin/ end) will display a 'live' example of the date/time format for that machine.
For example, running vxlogview --help and looking at the entry for -e shows :
-e,--endate EndDate Display messages logged up to the given time.
Date time should be in the format
"7/8/10 7:38:45 AM"
The format required for the date/ time on this particular system, is therefore as shown, but using single quotes on Unix, or double quotes on Windows.
Detailed example for option (3)
To copy the emm log between 6th July 7.00 AM and 8th July 7.20 AM to a folder /netbackup/tmp in order to send to support the raw logs for only this time period (unix).
vxlogmgr -o 111 -c -f /netbackup/tmp -b '7/6/10 7:00:00 AM' -e '7/8/10 7:20:00 AM'
Following are the files that were found:
/usr/openv/logs/nbemm/51216-111-2220372284-100706-0000000000.log
/usr/openv/logs/nbemm/51216-111-2220372284-100707-0000000000.log
/usr/openv/logs/nbemm/51216-111-2220372284-100708-0000000000.log
Total 3 file(s)
Copying /usr/openv/logs/nbemm/51216-111-2220372284-100706-0000000000.log ...
Copying /usr/openv/logs/nbemm/51216-111-2220372284-100707-0000000000.log ...
Copying /usr/openv/logs/nbemm/51216-111-2220372284-100708-0000000000.log ...
If the -A <filename> option is used, a compressed file is created :
vxlogmgr -o 111 -c -f /netbackup/tmp -b '7/6/10 7:00:00 AM' -e '7/8/10 7:20:00 AM' -A vxlogs
File vxlogs.tar.gz is generated in path /netbackup/tmp
Reading vx logs:
vxlogview -p 51216 -o 111 -d all >/tmp/emm.txt
The command must be run with the -d all option, otherwise vital details will not be included.
Following on from the examples to move the log files, various variations can be run to convert the logs for a specific time period. Only one of the following examples needs to be used.
(1) View just the past 1 days of the nbemm log:
vxlogview -p 51216 -o 111 -d all -n 1 >/tmp/logs/nbemm.txt
(2) View the past 20mins of nbemm logs, relative to when the command was run:
vxlogview -p 51216 -o 111 -d all -t 00:20:00 >/tmp/nbemm_log.txt
(3) View the nbemm log between two specific times:
vxlogview -p 51216 -o 111 -d all -b '04/30/08 07:00:00 AM' -e '04/30/08 07:30:00 AM' >/tmp/nbemm_log.txt
(See the 'Important note regarding format of <date/ time> for -b or -e options with vx commands' section above)
Example:
For example, running vxlogview --help and looking at the entry for -e shows:
-e,--endate EndDate Display messages logged up to the given time.
Date time should be in the format
"7/8/10 7:38:45 AM"
The format required for the date/ time on this particular system, is therefore as shown, but using single quotes on Unix, or double quotes on Windows.
Detailed example for option (3)
The logs between 6.20 on the 6th July 2010 and 6.50 on the 7th July 2010 can be displayed like this (unix) :
vxlogview -p 51216 -o 111 -d all -b '7/6/10 6:20:00 AM' -e '7/7/10 6:50:00 AM' >/tmp/emm.txt
VX Logs and 'sub-components'
If the vx logs for a sub-component of a process are also enabled, this detail can be included in the logs by use of the -i option instead of the -o option.
For example, if the mds logging (143) is enabled, this detail will be included in the output of the emm log if the -i option is used as opposed to -o , as the mds does not create a separate log.
vxlogview -p 51216 -i 111 -d all -n 1 >/tmp/nbemm_log.txt
VX Logs and converting between Timezones
Running vxlogview will cause the logs times to be converted to the timezone of the server on which the commands are run, this can be inconvenient :
The use of the -z option can show the timezone of the server the logs were created on, eg.
/usr/openv/netbackup/bin/vxlogmgr -G . -s -z
(Note, when using the OpsCenter vxlogview command, -G . will not work, a path to the logs must be specified)
Following are the files that were found:
File ./51216-200-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.
File ./51216-231-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.
File ./51216-111-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.
Using the following vxlogview command, we can convert and read the logs with the times adjusted for (example) the timezone -8:00 hours behind GMT.
vxlogview -G . -p 51216 -o 111 -z GMT-8
'Tailing' the vx logs
Using the -S option allows the logs to be display in 'real time', similar to running tail -f /usr/openv/netbackup/<process>/<logfile> on a legacy log.
For example, the following command would 'tail' the nbjm log.
vxlogview -p 51216 -i 117 -S
Related Knowledge Base Articles
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.