How to troubleshoot backups/restores on SLES/RHEL/Appliance VMware Backup Host

Description

New Logging information
FlashBackup Logging has been improved:
Set VERBOSE = 10 to log what entries are being cataloged.

FullVM restore logs will now be in Unified logging for ncfnbrestore

For all of the below logging, please enable the vmcloglevel at the bottom of the document. A log level of 3 is standard, however if you suspect a bug then please use 6.

Manual browsing for virtual machines are logged in: (Please set vmcloglevel to at least 3)

Master:

Add VERBOSE = 5 in bp.conf file 

To configure nbsl logging ,Run

vxlogcfg -a -p NB -o 132 -s DebugLevel=6

Log files :

# /usr/openv/netbackup/logs/nbproxy

# /usr/openv/logs/nbsl

VMware Backup Host:

Add VERBOSE = 5 in bp.conf file 
/usr/openv/netbackup/logs/bpVMutil

Query Base Browsing:

OID 386 ncfvmwarepi and ncfnbcs OID 366 and oid 

Log file :

/usr/openv/logs/ncfnbcs

To configure run:
vxlogcfg -a -p NB -o 386 -s DebugLevel=6
vxlogcfg -a -p NB -o 366 -s DebugLevel=6
vxlogcfg -a -p NB -o 480 -s DebugLevel = 6

Please also setup vmcloglevel to atleast 3 (6 is the highest), instructions can be found further down the document.

To view the log for the last 30 minutes run:
vxlogview -p NB -i ncfnbcs -d all -t 00:30:00 > nbcs.txt

For Snapshots gather: (Please set vmcloglevel to at least 3 and add VERBOSE = 5 in bp.conf)

Log file :

/usr/openv/netbackup/logs/bpfis
Also check Virtual Center task and events for any errors on the Snapshot.

For Backups gather (add VERBOSE = 5 in bp.conf)

log file :
/usr/openv/netbackup/logs/bpbkar
VxMS

To enable VxMS logging on a Linux backup host on 7.5.x/2.5.x versions:

1. Change to the /usr/openv/lib directory.
2. If the .vxms.rc file already exists, remove the # from the following line and save the file: #debug_level=TRC_TOP|PARAM_IN|PARAM_OUT|PARAM_FULL|INFO|DEBUG
3. If the .vxms.rc file does not exist, create it and add the following line:
4. debug_level=TRC_TOP|PARAM_IN|PARAM_OUT|PARAM_FULL|INFO|DEBUG

You can use the echo command to edit the .vxms.rc file (enter all on one line):
echo "debug_level=TRC_TOP|PARAM_IN|PARAM_OUT|PARAM_FULL|INFO|DEBUG" > .vxms.rc

For full VxMS logging:
echo "debug_level=TRC_TOP|PARAM_IN|PARAM_OUT|PARAM_FULL|INFO|DEBUG|TRACE1|TRACE2|TRACE3" > /usr/openv/lib/.vxms.rc

Also add the following for Transport and NCF information:

Pre NetBackup 7.5.0.7:
Create a file with name 'vixDiskLib.ini' in folder /usr/openv/lib/shared/vddk/

With NetBackup 7.5.0.7 and 7.6.x:
Create a file with name 'vixDiskLib.ini' in folder /usr/openv/lib/shared/vddk/lib64

Edit it and add these lines there (everything is case-sensitive, of course):
vixDiskLib.nfc.LogLevel = 4
vixDiskLib.transport.LogLevel = 6

The log files are generated in the /usr/openv/logs directory. For every backup, two log files are generated in the following format:
VxMS-<PID>log.<MMDDYY>_core
VxMS-<PID>log.<MMDDYY>_provider

where <PID> is the process id and <MMDDYY> is the month, day, and year.

To configure VxMS logging on a 7.6/2.6 and later Linux backup host:

1. Create the VxMS log directory:

/usr/openv/netbackup/logs/vxms
Note: For logging to occur, the VxMS directory must exist.And it should be in lower case

2. Add the following to the /usr/openv/netbackup/bp.conf file:

VXMS_VERBOSE=numeric value of 0 or greater
Set the VXMS_VERBOSE to 5 for most issues, 8 for file mapping related issues.

3. To change the log location, enter the following in the bp.conf file:

vxmslogdir=path to new log location

Note: If the VxMS log location is changed, the Logging Assistant does not collect the logs.

Restores

- Please set vmcloglevel to at least 3 and add VERBOSE = 5 in bp.conf
- master bprd log

Create VM calls during restores are logged in bpVMutil: 

/usr/openv/netbackup/logs/bpVMutil

Tar is now nbrestore for VMware Restores:
OID 357 - /usr/openv/logs/ncfnbrestore

To configure ncfnbrestore run:
vxlogcfg -a -p NB -o 356 -s DebugLevel=6
vxlogcfg -a -p NB -o 357 -s DebugLevel=6

To view nbrestore for the last 30 minutes run:
vxlogview -p NB -o 356 -i 357 -d all -t 00:30:00 > nbrestore.txt

Additional logging for Restore related issues
On the Master Server: bprd
On the VMware Backup Host: 
# /usr/openv/logs/ncfnbrestore
# /usr/openv/logs/VxMS-<pid>log.<date>_provider
# /usr/openv/logs/VxMS-<pid>log.<date>_core
# /usr/openv/netbackup/logs/bpVMutil

Instant Recovery Logging:

(Please set vmcloglevel to at least 3 and add VERBOSE = 5 in bp.conf)

On the Master Server: bprd

They are 3 main steps performed during Instant Recovery:

 - Export a backup image
 - Mount a backup image
 - Activate VM

If any of these steps failed “Activate Instant Recovery” job logs will have details about the failure and error code.
If Export a backup image fails look at /usr/openv/netbackup/logs/nbfsd on media server.
Mount backup image fails look at /usr/openv/netbackup/logs/nbfsd on VM Proxy host and also the Virtual Center Task and Events.
Activate VM fails look at /usr/openv/netbackup/logs/bpVMutil on VM Proxy host and also the Virtual Center Task and Events.

Binaries on Linux can be found in:

Pre NetBackup 7.5.0.7:
/usr/openv/lib/shared/vddk/
/usr/openv/lib/map/rvp/
/usr/openv/lib/map/libvix
/usr/openv/netbackup/bin/nbrestore

If NetBackup is version 7.5.0.7 or 7.6.x:
/usr/openv/lib/shared/vddk/lib64

To change vSphere timeouts and logging values on Linux

1. On the Linux backup host, create (or open) the following file:

/usr/openv/netbackup/virtualization.conf

2. To change a timeout value, enter a new dword line under [BACKUP], using the appropriate name (such as jobtimeout or poweroptimeout).  Include a value for the timeout. For example:

[BACKUP]
"jobtimeout"=dword:60
This example sets the job timeout to 60 seconds.
Note: If the file already contains a [BACKUP] line, do not add another [BACKUP] line.  Any other lines that already exist under [BACKUP] should remain as they are.

3. To change the level of vSphere API logging, enter a dword line for vmcloglevel with a logging value, under [BACKUP].  For example:

"vmcloglevel"=dword:6
The allowed values are 0 through 6, where 0 is no logging and 6 is the highest log level.
 

Editing SymVmTools entries
Create a file called /etc/SymVmTools/VmTools.cfg.
The format of this file will be key=value with one key value pair per line.  See an example below:

VmTools.cfg:
SymVmToolsDisableRefresh=1
SymVmToolsRefreshTimeout=4000

VMware Instant Access

Verify the Elastic Search Service and Storage Platform Web Service are running

• # systemctl status pdde es
• # systemctl status pdde spws

On the Appliances all Instant Access logs can be found in:

• /log/vpfs/

To easily gather the logs for an Instant Access operation you are about to perform run the below:

• # tail -f /log/vpfs/*

When the screen stops scrolling, hit Enter a few times to create a significant gap on the screen.

Then perform the Instant Access operation that is failing.  The log will be capture on the screen in the exact process flow so you can easily trace the issue.  Make sure your screen buffer is large enough to capture the information.

VMware Agentless Recovery