How to enable Backup Exec process debug and collect them using Veritas Quick Assist

Problem

How to debug Backup Exec processes to troubleshoot Backup and Restore Job failures. 
 

Solution

There are two ways to debug backup exec processes:

·        SGMON Utility

·        Registry Editor

If remote server backup or restore is involved, the remote agent process will need to be debugged along with the processes on BE server.

The above two methods enables debug logging for all Backup and Restore operation performed during the time of enabling debugging until debugging is stopped.

TIP: It is beneficial to have the failing job running on its own and if possible even reduce the data size to limit the size of the debug logs and the time it takes to run until failure.

NOTE: Disable Debug after creating the required debug logs to ensure that the drive does not fill up with unnecessarily debug logs.

By default, BE records errors on BE server in beremote (BEServer-beremote_AlwaysOnLog_Date_Time.log) and bengine log (BEServer-bengine_AlwaysOnLog_Date_Time.log) when the backup/restore fails. A RemoteServer-beremote_alwaysOnLog_Date_Time.log may also exist on remote server if the failure is with remote agent error. In some cases these logs may suffice. These logs get created by default with BE 15 FP1 and above.

DEBUG using SGMON Utility:

Launch SGMON from the Backup Exec console on BE server

1. Launch SGMON on BE server and Remote server from the Backup Exec and Remote Agent installation path respectively. Clear Log before selecting the processes that need to be debugged -

BE Server - Program Files\Veritas\Backup Exec\SGMON.exe
Remote Server – Program Files\Veritas\Backup Exec\RAWS\SGMON.exe

2. On BE Server select the following checkbox from Backup Exec Debug Monitor -> Debug Tab -
a. Job Engine, Agent for Windows, Agent Browser
b. Backup Exec Server
c. Device and Media



3. On remote server select the following checkbox from Backup Exec Debug Monitor -> Debug Tab -
a. Job Engine, Agent for Windows, Agent Browser

4. Run the Backup or Restore Job to reproduce the Job Failure. To create smaller size logs, run the backup or restore job where possible (i.e. when Issue is reproducible with smaller size data set) with appropriate size data set so that the error is reproduced in the quickest time possible.

5. Disable debugging via the “Deselect All” button

Debug capture sequence: Enable Debug -> Reproduce the error -> Disable Debug -> Close Debug Monitor to stop debug capture

  
DEBUG using Registry Editor:
 
 
Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

Enable debug on both the Backup Exec Server & Remote agent:

1. Open registry editor by typing Regedit in Run window (Press Win + R keys on your keyboard to launch Run window)

2. Locate and set CreateDebugLog to 1 to enable beremote and bengine debug on BE server. Enable the same Dword on remote server registry to enable remote agent i.e. beremote process debug.

CreateDebugLog Dword Path -> HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\Backup Exec for Windows\Backup Exec\Engine\Logging

3. Quit the registry editor

4. Disable debug on both the Backup Exec & Remote agent server by setting CreateDebugLog from 1 to 0 after the Job has been run and error reproduced.
                                                                                                                                                 

Log Collection -

In both the above methods, the debug logs get saved on Backup Exec Install Path\Log folder on BE server and Remote Agent Install path\Logs folder on remote server. Logs bundle can be collected with the help of Veritas Quick Assist tool. The instruction to use Veritas Quick Assist is attached to this article and can also be accessed from the link here. 

 

Accept EULA -> Choose “Collect data for support” -> select Backup Exec -> Select “All Data” -> Accept the default logs selected (Base Logs, Debug Logs, Job Log). After the Tool has completed save the collected logs and upload the saved logs from each server (BE Server and Remote servers involved) to the support case at https://my.veritas.com

Note: If login credentials for above site are not available, tech support can be requested to send instructions to upload the logs to the support case.
These are basic logs. Depending on the type of Issue specific instruction may be provided by tech support to gather verbose level debug.

Default Location of BE process debug logs:

BE Server - Program Files\Veritas\Backup Exec\Logs\BEServer-BengineXY.log
                    Program Files\Veritas\Backup Exec\Logs\BEServer-BeremoteXY.log

Remote servers - Program Files\Veritas\Backup Exec\RAWS\Logs\RemoteServerName-beremoteXY.log

BE Server SGMON Log Name - BEServerName-SGMon.log
Remote Server SGMON Log Name – RemoteServerName-SGMON.log


Scenarios/Errors where this basic debug is helpful -

1. Unable to connect to the remote agent
2. Local/Remote Server browse error
3. File or directory not found error
4. Communication failure occurred
5. Network connection to the remote agent has been lost
6. Unable to attach error during Backup or restore.
7. Access is denied error

**Select “Enable verbose logging” for device and media in Backup Exec Debug Monitor(SGMON) -> Settings ->Device and Media on top of what was already selected. These Logs may be required for any Backup/Restores failing when targeted or read from a storage (dedupe storage, Tape, Disk Storage).
Note: Deselect the Enable verbose logging checkbox after debug log is collected. Disable debugging via the “Deselect All” button in Backup Exec Debug monitor (SGMON.exe).

**Additional Debug for backup and restore failure of Database Agents http://www.veritas.com/docs/000087598

*******************************************************************************************************
The log location can be changed by editing the following registry string values, followed by Backup Exec service re-start. Ensure the new log path exists.
HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\Backup Exec For Windows\Backup Exec\Debug
Path
SGMon Path
Note: Disable Debug after creating the required debug logs to ensure that the drive does not fill up with unnecessarily debug logs.
*******************************************************************************************************