Running nbdeployutil from remote master servers may cause CPU to skyrocket and nbwmc to stop processing requests.

Problem

With NetBackup version 8.1.2/3.1.2 if nbdeployutil is initiated by a remote master server to busy master server, the nbwmc java process cpu utilization may skyrocket, and backup jobs will no longer be initiated.

Error Message

When this occurs, nbwmc may stop processes or shutdown completely. From the logs in /usr/openv/wmc/webserver/logs, the catalina.yyy-mm-dd.log may simply show a shutdown:

29-Oct-2019 23:34:25.654 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
29-Oct-2019 23:34:25.654 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["nbhttps-jsse-nio-3652"]
29-Oct-2019 23:34:28.349 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["nbhttps-jsse-nio-auto-1-41009"]

The tomcat.log may also show a termination:

10/29/19 23:34:21 - Stopping nbwmc. Using JRE_HOME /usr/openv/java/jre.Tomcat did not stop in time.To aid diagnostics a thread dump has been written to standard out.Killing Tomcat with the PID: 2573The Tomcat process has been killed

Cause

This issue can be caused by a master server running nbdeployutil to other busy master servers.

Solution

Administrators should check the /usr/openv/var/global/nbdeployutilconfig.txt file on the remote master server, and review the MASTER_SERVERS entries. Any master servers that are seeing problems should be removed from the list.

For example if the server linuxabcd is seeing hangs - that entity should be removed from the MASTER_SERVERS entries:

cat /usr/openv/var/global/nbdeployutilconfig.txt
[NBDEPLOYUTIL_INCREMENTAL]
MASTER_SERVERS=linux1234,linuxabcd,linux5678,linuxefgh
FREQUENCY_IN_DAYS=1
PURGE_INTERVAL=180