NetBackup gets unusable when no space is available for ‘/log’ partition on NetBackup appliance

Problem

On NetBackup appliance, there is a possibility that some NetBackup Appliance or NetBackup software is suddenly unavailable like, fail to login to NetBackup admin console, NetBackup commands return error or stuck.

When user encounters this situation, please confirm that whether '/log' partition is full.

Error Message

Log Partition Full on Netbackup Appliance

Cause

NetBackup software logs '/usr/openv/logs' and '/usr/openv/netbackup/logs' are redirected to '/log' partition. And other appliance related logs and core dump file(s) are also written to '/log' partition.

1. When core dump files are generated, as their size is large, it's possible to make '/log' partition full.

2. The settings of logging level for NetBackup could have a dramatic effect on disk usage. If the default logging level is changed, it's possible to make '/log' partition full unexpectedly.

Solution

For product logs, elevate to appliance Maintenance Mode, delete logs by command ‘vxlogmgr -d’ .

For core dump and other files, elevate to appliance Maintenance Mode, clean core dump files under /log/crash and other files properly. See Overriding the NetBackup appliance intrusion prevention system policy.


 

To prevent the NetBackup Appliance operating system partition (known as the 'System' partition) from reaching a 100% full state, configure the appliance for SMTP alerts so that the administrator will become aware of a potential space problem. (Appliances can also be configured for SNMP alerts, please refer to the 'About hardware monitoring and alerts' section in the Veritas NetBackup™ Appliance Administrator's Guide for details on Hardware and Software monitoring.)

   Example SMTP configuration steps using CLISH from Main_Menu:
Settings> Alerts> Email SMTP Add <Server> <Account> <Password>

Settings> Alerts> Email SenderID Set <Address>

Settings> Alerts> Email Hardware Add <Address>

Settings> Alerts> Email Software Add <Address>

NetBackup Appliances will alert with either a WARNING or ERROR when a disk partition reaches specific usage points. Appliances alert for partition space conditions in the following ways: 
     - 'WARNING' with UMI code V-475-103-1001 when the System partition (as well as /log partition) reaches a 80% but under 98% full state.
     - 'ERROR' with UMI code V-475-103-1000 (and also sends this UMI to callhome if callhome is configured) once the partition reaches 98% full state or higher.

Example WARNING alert when the OS or /log partition usage is greater than 80 but less than 98%:

The following list contains all of the currently unresolved hardware alerts for your appliance hostname_here (VTAS5555555): 
• The partition usage has exceeded warning threshold and will soon reach full capacity. Cleanup the partition and re-check status. If the issue is not resolved, contact Veritas Technical Support for assistance. 
o Time of event: 2017-08-20 15:54:17 (+08:00)
o UMI Event code: V-475-103-1001
o Component Type: Partition
o Component: Log
o Status: 81%
o State: WARNING
o Additional information about this error is available at following link:
V-475-103-1001

Example ERROR alert when the OS or /log partition usage is greater than 98%:

The following error(s) are detected on your appliance hostname_here (VTAS5555555): 
* The partition usage has exceeded critical threshold: full capacity imminent. Cleanup the partition and re-check status. If the issue is not resolved, contact Veritas Technical Support for assistance. 
o Time of event: 2018-12-05 18:04:22 (+08:00)
o UMI Event code: V-475-103-1000
o Component Type: Partition
o Component: Log
o Status: 99%
o State: ERROR
o Additional information about this error is available at following link:
V-475-103-1000 
If AutoSupport is enabled, this information is automatically transmitted to Veritas for further analysis. 

Producing a DataCollect package before you engage Technical Support may help to expedite the resolution process. For information on how to gather the logs that the DataCollect utility creates, refer to your appliance Administrator's Guide.