Logging Assistant FAQ

Article: 100028985
Last Published: 2019-12-03
Ratings: 0 0
Product(s): NetBackup & Alta Data Protection

Problem

This document collects information about Logging Assistant.

How to debug VxPBX?

Solution

What purpose is Logging Assistant expected to serve?

Logging Assistant simplifies the procedure of providing evidence for support cases. This is primarily achieved by facilitating these steps in the NetBackup Administrator Console that users with root/admin privileges have had to perform manually:

  • setting up debug logging on multiple NetBackup hosts, creating log directories, changing logging levels in varied configuration files
  • collecting accumulated debug logs
  • reducing debug log levels pertaining to a specific support case after it has been resolved

By simplifying these steps, it is expected, Logging Assistant will help save customers' time collecting and providing evidence to Veritas technical support.

After a support case is created, typical use of Logging Assistant would entail the following steps:

  1. Create a Logging Assistant record (the record ID could be the support case ID)
  2. Set up requisite debug logging using the Setup Debug Logging wizard. Use Job Analysis for logs' suggestions.  Or consult support
  3. Reproduce the unexpected behaviour for which the support case was created
  4. Collect debug logs using the Collect Debug Logs Wizard. Using this wizard, nbsu output can also be optionally be included
  5. Work with technical support during analysis of uploaded logs/evidence 
  6. Repeat steps 2 through 5 if additional logs are required
  7. Take corrective actions prescribed by support
  8. Disable or Set Minimum debug logging for the Logging Assistant record
  9. Delete the Logging Assistant record

Note: The Setup Debug Logging wizard can be used to change existing debug logging settings for a Logging Assistant record.

How to upload collected logs to the Veritas Evidence Server?
If you have a case ID provided by Technical Support of the form ########, rename the log files with the case ID number. Then manually upload the files to the Veritas Evidence Server. Refer to the following article:
NBFDRV64 segfault doing restore.

Does Logging Assistant help avoid having to reproduce the problem?

While the problem will still have to be reproduced for evidence to analyze, Logging Assistant simplifies the process by centralizing and automating associated management operations.  Overall, Logging Assistant will help prevent human error and minimize user time commitment during the process of reproducing a problem and providing the evidence. 

Logging Assistant helps you set up the correct logging levels on the right machines to capture detailed logging of a problem the first time.  Once you've used Logging Assistant to easily and correctly set the logging levels, reproduce the issue once and use Logging Assistant to collect and ship the logs to Veritas.  Read on for a longer explanation.

Today, not all problems encountered by NetBackup are recorded in the Activity Monitor or the NetBackup Problem Report (i.e. errordb); many are just written to debug logs.  However, customers rarely have debug logging enabled.

At default levels (DiagnosticLevel=6 & DebugLevel=1), VxUL logs may contain clues to help explain unexpected behaviour. However, the sheer number of VxUL log files (~100 OIDs) and huge sizes of log files even at default levels in busy environments means it is non-trivial task to pore through them.  In addition, there are instances when, even at default levels, debug logs may not contain clues to identify the root cause.  As a result, Veritas technical support typically asks for high verbosity debug logs upfront, so they don't have to request, for example, that the user re-crash their system multiple times.

End users are not expected to have to look through NetBackup debug logs for explanations of unexpected behaviour. Debug logs, even at default levels, are for Veritas technical support and Engineering to analyze. Issues requiring user attention should be reported in the Activity Monitor and/or NetBackup Problem Reports and new releases add useful details in this area.

To improve overall NetBackup supportability, the objective of Logging Assistant is to reduce time taken to provide evidence for a support case. More often than not, issues in NetBackup are predictably reproducible. When problem recreation is required, Logging Assistant can simplify the pre- and post- steps.

Setting up debug logging using Logging Assistant is good, but services have to be restarted for logging level changes to take effect, right?

In 7.6, most master server daemons that do legacy logging have been modified to respond to debug logging level changes in bp.conf without requiring a restart. The modified daemons are: bprd, bpdbm (since 7.5), bpjobd (since 7.5), bpcd, vnetd, bpcompatd, nbproxy, bpinetd.

On the master server, nbazd requires a manual restart for debug logging level changes in the configuration file to take effect.

Media Manager daemons (vmd, ltid, avrd, tldcd, tldd, etc.) continue to require a restart for debug logging level changes to take effect. Manual restart of these daemons is required after using Logging Assistant to setup or disable debug logging.

The nbproxy process does not require a manual restart, but it checks for debug log level changes every 10 minutes. Processes like nbfsd and bpspsserver remain up for a short period after a job completes. If a new job is started immediately, modified debug logging levels for nbfsd and bpresolver may not take effect.

What does Logging Assistant do when setting up, setting minimum, and disabling debug logging? What logging levels does Logging Assistant set?

 

  Configuration File or Location Setup Debug Logging Set Minimum Debug Logging Disable Debug Logging
Legacy Logging /usr/openv/netbackup/bp.conf (registry on Windows) <PROCESSNAME>_VERBOSE = 5, Create debug log dir <PROCESSNAME>_VERBOSE = 0 <PROCESSNAME>_VERBOSE = -2
NetBackup VxUL Logging /usr/openv/netbackup/nblog.conf DiagnosticLevel = 6, DebugLevel = 6 DebugLevel = 1 DebugLevel = 0
Media Manager processes logging /usr/openv/volmgr/vm.conf Add VERBOSE line. Create debug log dir Remove VERBOSE Remove VERBOSE
PureDisk Plugin /usr/openv/lib/ost-plugins/pd.conf LOGLEVEL = 10 LOGLEVEL = 0 LOGLEVEL = 0
Deduplication processes: spad, spoold /usr/openv/lib/ost-plugins/spa.cfg Logging = full,thread Logging = trace,thread Logging = trace,thread
Deduplication process: mtstrmd /usr/openv/lib/ost-plugins/mtstrm.conf Logging = full,thread Logging = short,thread Logging = short,thread
PBX /etc/vx/VxICS/VxPBX.cfg DebugLevel = 10 DebugLevel = 10 DebugLevel = 0
Cloud process: nbcssc /usr/openv/lib/ost-plugins/cloudstore.conf CSSC_LOG_LEVEL = 5 CSSC_LOG_LEVEL = 0 CSSC_LOG_LEVEL = 0
Java GUI /usr/openv/java/Debug.properties printcmds = true, debugMask = 0x00040000, enableATLogs = true comment printcmds & debugMask lines comment printcmds & debugMask lines, enableATLogs = false
OpsCenter VxUL logging log.conf DiagnosticLevel = 6, DebugLevel = 6 DebugLevel = 1 DebugLevel = 0
NetBackup Search VxUL logging searchlog.conf DiagnosticLevel = 6, DebugLevel = 6 DebugLevel = 1 DebugLevel = 0
nbazd /usr/openv/netbackup/sec/az/bin/VRTSaz.conf DebugLevel = 6 DebugLevel = 0 DebugLevel = 0

Starting NetBackup 7.6, <PROCESSNAME>_VERBOSE value is available for processes logging in NetBackup legacy log directories (i.e. in /usr/openv/netbackup/logs). For instance, to enable bpduplicate logging, set BPDUPLICATE_VERBOSE = 5 and create the admin log directory. Other processes logging in the admin directory will continue to log at default levels. (Please see an exception for Database Agent processes below.)

For example, for vltrun logging, the Setup Debug Logging wizard will set VLTRUN_VERBOSE to 5 and create the corresponding log directory. The Disable Debug Logging wizard will set VLTRUN_VERBOSE to -2. The Set Minimum Debug Logging wizard will set VLTRUN_VERBOSE to 0. For the Disable and Set Minimum wizards, the log directory is not removed.

For Database Agent processes, Logging Assistant sets Debug_Database = 5, Debug_Database = 0, Debug_Database = 0 in Setup, Set Minimum, and Disable wizards respectively.

With these settings, use of the global VERBOSE setting in bp.conf should rarely be required.

Can Logging Assistant help with identifying logging to setup?

The Setup Debug Logging wizard includes a Job Analysis step.  In the Job Analysis step, a user-specified job ID is analyzed to suggest logging to setup in preparation for a re-crash.

How does the Logging Assistant Job Analysis work?

With the submitted job ID, the Job Analysis function of Logging Assistant fetches job information using bpdbjobs. If job information includes an associated policy name, policy information is fetched using bppllist. From the job information, the master server, media server and client names are extracted. Based on policy type, job status code, and selected policy attributes, Job Analysis pre-selects suggested debug logging.

Is there a way to setup debug logging of processes not listed in the Problem Categories table?

There may be instances where debug logging of components or processes not covered in the Problem Categories selection table need to be setup. This can be achieved by specifying names of such components (or processes) in the Additional components textbox that is available under the Problem Categories selection table. See the next answer for a comprehensive list of keywords accepted by Logging Assistant in the Additional components textbox.

What processes or components does Logging Assistant allow setting up debug logging of? In other words, provide a comprehensive list of components that can be specified in the "additional components" textbox.

Logging Assistant eases the process of setting up debug logging for a varied set of NetBackup processes. Here's a comprehensive list of "keywords" valid to specify in the "additional components" textbox to setup debug logging

These OIDs or the corresponding names in nblog.conf can be used as keywords in the Setup Debug Logging wizard:

OID range OID name (OID)
1-150
nbatd (18)
nbemm (111)
nbpem (116)
nbjm (117)
nbrb (118)
bmrd (119)
bmrsavecfg (121)
bmrc (122)
bmrs (123)
bmrcreatefloppy (124)
bmrsrt (125)
bmrprep (126)
bmrsetup (127)
bmrcommon (128)
bmrconfig (129)
bmrcreatepkg(130)
nbsl (132)
ndmpagent (134)
libutil, libmessaging (137)
mmui (140)
bmrepadm (142)
mds (143)
da (144)
151-200
ndmp (151)
bmrovradm (154)
ace (156)
ncfrai (158)
ncftfi (159)
nbsvcmon (163)
nbvault (166)
dsm (178)
nbftsrvr (199)
nbftclnt (200)
201-250
fsm (201)
stssvc (202)
ncfive (210)
rsrcevtmgr (219)
dps (220)
mpms (221)
nbrmms (222)
nbstserv (226)
rdsm (230)
nbevtmgr (231)
bmrlaunch (248)
251-300
SPSV2RecoveryAsst (254)
aggs (261)
wingui (263)
winbargui (264)
expmgr (272)
nbkms (286)
nbaudit (293)
301-350
ncf (309)
ncfnbservercom (311)
ncfbedspi (317)
ncfwinpi (318)
dbaccess (321)
ncforaclepi (348)
351-400
ncflbc (351)
ncfgre (352)
ncftarpi (355)
ncfvxmspi (356)
ncfnbrestore (357)
ncfnbbrowse (359)
ncforautil (360)
ncfdb2pi (361)
nbars (362)
dars (363)
ncfnbcs (366)
impmgr (369)
nbim (371)
nbhsm (372)
nbism (373)
ncfnbusearchserverpi (375)
ncfnbdiscover (377)
ncfnbquiescence (380)
ncfnbdboffline (381)
nbci (385)
ncfvmwarepi (386)
nbrntd (387)
stsem (395)
nbutils (396)
nbevingest (398)
nbdisco (400)
401-450
ncfmssqlpi (401)
ncfexchangepi (402)
ncfsharepointpi (403)
ncffilesyspi (412)
flashbackup (428)
bmrb2v (433)
bmrb2vrst (434)
nbwmc (436)
nbwebservice (439)
ncws (443)
ncfvmsearchpi (444)
ncfnbhfr (445)
ncfnbsessionmgr (448)
nbostpxy (450)
451-500
security (466)
hosts (482)
nbconfigmgmt (483)
nbgateway (484)
nbwss (485)
nbcatalogws (487)
nbrbac (488)
nbadminws (489)
 

These are processes that log in legacy logs location (/usr/openv/netbackup/logs). These process names are accepted as keywords by Logging Assistant GUI (in Setup Debug Logging wizard).

  • Logging Assistant keywords for Vault processes:
    • vltadm
    • vlteject
    • vltinject
    • vltoffsitemedia
    • vltopmenu
    • vltrun
  • Logging Assistant keywords for processes that log in nbdb directory:
    • create_nbdb
    • dbadm
    • nbdb_admin
    • nbdb_backup
    • nbdb_move
    • nbdb_ping
    • nbdb_restore
    • nbdb_unload
    • nbdb_upgrade
  • Logging Assistant keywords for processes that log in admin directory:
    bpauthorize
    bpcatlist
    bpccname
    bpchangeprimary
    bpclient
    bpconfig
    bpcounts
    bpcoverage
    bpdir
    bpdrfiles
    bpduplicate
    bperror
    bpexpdate
    bpflist
    bpfrag
    bpgetconfig
    bpgethost
    bpimage
    bpimagelist
    bpimmedia
    bpimport
    bplicense
    bplocaladdrs
    bpmedia
    bpmedialist
    bpminlicense
    bpnbaz
    bppficorr
    bpplcatdrinfo
    bpplclients
    bppldelete
    bpplexclude
    bpplinfo
    bppllist
    bpplsched
    bpplschedrep
    bpplschedwin
    bpplvalid
    bppolicynew
    bppplinclude
    bprdreq
    bpretlevel
    bprsh
    bpschedreq
    bpsetconfig
    bpsnapconfig
    bpstuadd
    bpstudel
    bpstulist
    bpsturep
    bpsyncinfo
    bptestbpcd
    bpverify
    hosts
    nb_updatedssu
    nbadminws
    nbcatalogws
    nbcatsync
    nbcertupdater
    nbconfigmgmt
    nbdc
    nbdecommission
    nbdelete
    nbdevconfig
    nbdevquery
    nbdiag
    nbexecute
    nbfindfile
    nbftconfig
    nbgateway
    nbhba
    nbholdutil
    nbindexutil
    nbpemreq
    nbrbac
    nbreplicate
    nbsnapimport
    nbsnapreplicate
    nbstl
    nbsvgrp
    nbwss
    security



     
  • Logging Assistant keywords for NetBackup processes/components that log in their respective log directories:
    backint
    beds
    bmrrst
    bp
    bparchive
    bpbackup
    bpbkar
    bpbrm
    bpbrmds
    bpcd
    bpclimagelist
    bpclntcmd
    bpcompatd
    bpdb2
    bpdbjobs
    bpdbm
    bpdbsbdb2
    bpdbsbora
    bpdm
    bpdhcp
    bpfilter
    bpfis
    bpfsmap
    bphdb
    bpinetd
    bpinst
    bpjava-msvc
    bpjava-susvc
    bpjava-usvc
    bpjobd
    bpkeyutil
    bplist
    bpmount
    bpnbat
    bporaexp
    bporaexp64
    bporaimp
    bporaimp64
    bppfi
    bprd
    bpresolver
    bprestore
    bpstsinfo
    bpsynth
    bptm
    bpubsdb2
    bpubsora
    bpVMreq
    bpVMutil
    dbclient
    exten_client
    infxbsa
    liveupdate
    mtfrd
    nb_cat_helper
    nbconsole
    nbfsd
    nblogadm
    nblogadmagent
    nblogadmhelper
    nbliveup
    nbregopsc
    nbproxy
    nbwin
    patch
    sybackup
    symlogs
    tar
    user_ops
    vault
    vnetd
    vxms
    Note: nb_cat_helper logs in the bpdbm log directory.

Of other categories of Logging Assistant keywords:

  • Media Manager processes (Note: Services like ltid, vmd, avrd require manual restart after logging level changes)
    • Logging Assistant keywords:
      • avrd
      • daemon
      • ltid
      • reqlib
      • robots
      • tpcommand
  • Deduplication plugin (configuration file: pdplugin.conf)
    • Logging Assistant keyword: pdplugin
  • Deduplication services (mtstrm.conf or spa.cfg)
    • Logging Assistant keywords:
      • mtstrmd
      • spad
      • spoold
  • NetBackup Search processes (configuration file: searchlog.conf)
    • Logging Assistant keywords:
      • nbsearch
      • nbcij
  • OpsCenter (Note: NetBackup client needs to be installed and running on the OpsCenter server)
    • Logging Assistant keywords:
      • opscenteragent (146)
      • opscentergui (147)
      • opscenterserver (148)
      • opscenterapplication (423)
      • opscenterinfra (761)
  • Java GUI (configuration file: Debug.properties)
    • Logging Assistant keywords:
      • jnbSA
      • jbpSA
  • Cloud service
    • Logging Assistant keywords: nbcssc
  • NBAC services (Authentication, Authorization)
    • Logging Assistant keywords:
      • nbatd (18)
      • nbazd
  • Miscellaneous:
    • Logging Assistant keyword to collect tracklogs: acclmetadata

Does Logging Assistant help with VxMS debug logs?

Yes. The Logging Assistant keyword to setup VxMS logging is vxms.

Starting 7.6, VxMS debug logs go in /usr/openv/netbackup/logs/vxms log directory. To enable high verbosity VxMS logging manually, set VXMS_VERBOSE = 5 in bp.conf.

Tools like bpclntcmd, bpclimagelist don't seem to log in the "bplist" log directory anymore. Where do they log?

The bpclntcmd and bpclimagelist commands now log in their respective (bpclntcmd and bpclimagelist) log directories.

Does Logging Assistant allow setting custom debug log levels?

For simplicity and to keep the need to re-crashes at a minimum, Logging Assistant sets up high verbosity debug logging, without means of customizations to users.

For OID 199 (nbftsrvr) however, Logging Assistant sets DebugLevel to 4:
NBFDRV64 segfault doing restore.

For bpfis debug logging, BPFIS_VERBOSE is set to 9 (not 5), since snapshot provider debug log messages are available at 9.

Why does Logging Assistant collect all VxUL debug logs for OIDs 137 and 156?

NetBackup VxUL logging defines separate OIDs for executables and important C/C++ library components (which are linked into executables).  If Logging Assistant is used to set up VxUL logging of library component OIDs, during their log collection Logging Assistant collects VxUL logs of all OIDs.  As such, setting up library components OID logging causes extra VxUL logs to be collected, potentially slowing down the collection operation.  This issue affects all library components, not just 137 and 156.

ETrack 3114358 is open to have Logging Assistant handle VxUL library component OIDs more appropriately during log collection.

Does Logging Assistant maintain a history or trail of actions performed through the life-cycle of a Logging Assistant record?

No, not currently.  Information like the modification timestamp of debug log settings is available in the View Details dialog.

What's with these large values regedit displays for BPDBM_VERBOSE, BPRD_VERBOSE, BPBRM_VERBOSE, BPTM_VERBOSE, BPDM_VERBOSE, BPJOBD_VERBOSE, NBPROXY_VERBOSE?

These VERBOSE values are stored as numbers in the Windows Registry. The Windows registry stores numerical values as DWORDs, which are unsigned. However, NetBackup internally interprets these values as signed numbers. As such, regedit displays large values when Logging Assistant sets these configuration settings to -2. Please use bpgetconfig or nbgetconfig to view values of these configuration settings as interpreted by NetBackup.

The other <PROCESSNAME>_VERBOSE values are stored as strings; regedit displays these values as expected.

Does Logging Assistant collect core dump files or other evidence files?

Logging Assistant does not automatically collect core dump files. Core dump files that are generated  have to be manually uploaded to the Veritas Evidence Server.
Refer to the following article:
How to gather information when troubleshooting core dump files from NetBackup processes on UNIX or Linux

Why is Logging Assistant UI "modal?"  For instance, I cannot easily pick the Job ID from Activity Monitor to use with Logging Assistant Job Analysis? Is there a way around?

NetBackup GUI Wizards are "step-by-step" and it's not possible to switch on a different NetBackup view (like Activity Monitor) while using a Wizard. This behaviour is sometimes referred to as "modal."  Allowing for non-modal behaviour of Wizards involves a significant enhancement in the frameworks underlying the NetBackup Administration Console GUIs.

The recommended alternative is to use File > New Window from Here menu item to have independent views of NetBackup Administration Console.

If, say, nbjm debug logging is set up under 2 Logging Assistant records, what is the result of disabling (or setting minimum) debug logging for one of the records?

Logging Assistant uses a reference counting-like mechanism to ensure debug logging configuration changes to disable (or set to minimum) debug logging for a process/component are not actually performed if other records have defined a level for the same process/component.

Another way to say this is that when debug logging is disabled for the first of the Logging Assistant records, no debug level changes for nbjm will be performed. The Results page of the Disable Debug Logging wizard will display a warning to this effect.

How is required disk space for debug log collection calculated?

nbcplogs provides a --preview option to establish required disk space to collect specified debug logs. The required disk space information is indicative; it is calculated assuming a ~85% compression ratio of debug log files. NetBackup debug logs typically compress to sizes smaller than this; more often than not, required disk space information reported by Logging Assistant is an upper bound.  As well, since nbcplogs compresses individual debug log files before bundling them together, peak disk space usage during collection is also not high.

How do I set up debug logging for a clustered master server?

The master server name textbox in the Host Selection page of the Setup Debug Logging Wizard is editable. Specify nodenames of the clustered master server in comma separated form.

Specifying different names of *one* host in the Setup Debug Logging wizard's Host Selections page produces inconsistent results. Can this be fixed?

Consider this scenario: user has nbumaster1.customer.com in the master server name textbox and picks nbumaster1 as a media server (from the include-exclude combo) on the Host Selections page to setup debug logging on. Note: user cannot edit media server name.

Even if both names refer to the same host, Logging Assistant will treat these as different hosts and simultaneously execute two remote commands to set up debug logging using the two hostnames. If both names refer to the same host, this can cause changes made by one remote command execution to be overwritten by the other remote command, but Logging Assistant will report success.  Without further checks, Logging Assistant cannot assume the names refer to the same host.  ETrack 3133255 is open to have Logging Assistant handle multiple names of a host appropriately.

This problem can also result if different aliases of the same host are specified on the Host Selections page.

The current workaround is to ensure the same name of a host is used in Logging Assistant records, in the example above, edit master server name to match the name used to configure it as media server.

How does Logging Assistant get actions performed on media servers and clients from the master server?

From the master server, Logging Assistant communicates with vnetd on the remote host to have nblogadmagent spawned, which performs requested Logging Assistant actions on the host.

What if Logging Assistant is not functioning correctly?  What evidence will be required?

On the master server, Logging Assistant debug logs can be found in the nblogadm and nblogadmhelper log directories. On NetBackup media servers and clients, debug logs are written in the nblogadmagent directory.

More often than not, nblogadm and nblogadmhelper debug logs suffice for Logging Assistant problem analysis.

Why do I have to explicitly disable debug logging before deleting a Logging Assistant record? Could Logging Assistant internally disable debug logging before deleting the record?

If a Logging Assistant record is deleted without disabling debug logging, host and process names information in the record will be lost and those processes will continue to perform high verbosity debug logging.

For the second part of the question, the Delete Record flow will need to be enhanced to let user choose between Disable and Set Minimum debug logging and launch the corresponding wizard.

Does Veritas collect telemetry information for Logging Assistant?

Yes, information of Job Analysis performed using Logging Assistant is collected by the telemetry agent. Additionally, information like the following is collected:

  • number of Logging Assistant records
  • number of unique hosts on which debug logging is set up
  • support case IDs

References

Etrack : 3114358 Etrack : 3133255 Etrack : 3286146 Etrack : 3330805 Etrack : 3332808

Was this content helpful?