The NetBackup Messaging Broker Service does not start on a Windows Master Server

Problem

After upgrading to NetBackup 8.3 or higher, the NetBackup Messaging Broker Service should start along with other NetBackup services.  However, the service fails to start, citing the following message

Windows could not start the NetBackup Messaging Broker Service on Local Computer.  For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error 1066.

Error Message

There could be several different error messages which may surface depending on different NetBackup versions in use.

Example 1:

The startup error captured in the <InstallPath>\Veritas\NetBackup\mqbroker\logs\nbmqbroker.<date>.log file is as follows:

05/19/2021 20:40:36 [17492.18656] NBU Install Directory as read from registry :  D:\Program Files\Veritas\, bRetVal = 1
05/19/2021 20:40:36 [17492.18656] cmdstr: CMD /C "CALL "D:\Program Files\Veritas\netbackup\mqbroker\bin\setmqenv.bat" && CALL "D:\Program Files\Veritas\netbackup\mqbroker\rabbitmq-server\sbin\rabbitmqctl.bat" status"
05/19/2021 20:40:37 [17492.18656] Retry - 1Registering service NetBackup Messaging Broker Service 

Successfully registered service NetBackup Messaging Broker Service for monitoring.

[05/19/2021 20:40:40.14] Register the service with NetBackup Service Monitor done: 0

[05/19/2021 20:40:40.14] rabbitmq-server.bat

Error spawning D:\Program Files\Veritas\NetBackup\mqbroker\erlang\erts-10.6\bin\epmd -daemon (error 0)

05/19/2021 20:40:40 [17492.5920] Child exited
05/19/2021 20:40:40 [17492.5920] Finished reading from pipe
05/19/2021 20:41:01 [17492.18656] RunCommand status returned: 70
05/19/2021 20:41:01 [17492.18656] RunCommand:[CMD /C "CALL "D:\Program Files\Veritas\netbackup\mqbroker\bin\setmqenv.bat" && CALL "D:\Program Files\Veritas\netbackup\mqbroker\rabbitmq-server\sbin\rabbitmqctl.bat" status"] FAILED with error = 70
05/19/2021 20:41:01 [17492.18656] Updated the service status as SERVICE_START_PENDING
05/19/2021 20:41:01 [17492.18656] Service startup failed as child process has stopped.
05/19/2021 20:41:01 [17492.18656] Updated the service status as SERVICE_STOPPED
05/19/2021 20:41:01 [17492.18656] CProcessMgmt[Exit]::WaitForMQBROKERToRespond
05/19/2021 20:41:01 [17492.18656] ERROR: Process exited with error code 1

Example 2:

10:56:10.907 [error] Error in process <0.206.0> on node nbmqbroker_prelaunch_11972@localhost with exit value:
{eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2272}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]}

10:56:10.907 [error] Supervisor inet_gethost_native_sup had child at module inet_gethost_native at <0.206.0> exit with reason {eacces,[{erlang,open_port,[{spawn,"inet_gethost 4 "},[{packet,4},eof,binary]],[{file,"erlang.erl"},{line,2272}]},{inet_gethost_native,server_init,2,[{file,"inet_gethost_native.erl"},{line,184}]}]} in context child_terminated

Example 3: 

Also captured in the log: <InstallPath>\Veritas\NetBackup\mqbroker\logs\nbmqbroker.<date>.log

05/22/2023 14:49:03 [10936.12732] RunCommand:[CMD /C "CALL "D:\Program Files\Veritas\netbackup\mqbroker\bin\setmqenv.bat" && CALL "D:\Program Files\Veritas\netbackup\mqbroker\rabbitmq-server\sbin\rabbitmqctl.bat" status"] Successful
05/22/2023 14:49:03 [10936.12732] CProcessMgmt[Exit]::WaitForMQBROKERToRespond
05/22/2023 14:49:03 [10936.12732] Wait for service to respond is successful.
05/22/2023 14:49:03 [10936.12732] Set the service status to SERVICE_RUNNING
2023-05-22 14:49:42.836000+01:00 [erro] <0.129.0>
2023-05-22 14:49:42.836000+01:00 [erro] <0.129.0> BOOT FAILED
2023-05-22 14:49:42.836000+01:00 [erro] <0.129.0> ===========
2023-05-22 14:49:42.836000+01:00 [erro] <0.129.0> ERROR: epmd error for host My-Master: {could_not_start_server,inet_gethost_native} (unknown POSIX error)
2023-05-22 14:49:42.836000+01:00 [erro] <0.129.0>
BOOT FAILED
 

Cause

This issue can occur when a rogue file or directory named 'program' exists on the root of the volume where NetBackup is installed.  This file or directory may be hidden.

In the example above: D:\program

Solution

Identify if a file or directory named 'program' exists on the root of any volumes, and if so, delete or rename it.

1. Open a CMD prompt using 'Run as administrator'.

2. Navigate to the root of the volume.
Example: cd D:\

3. Run 'dir /a' to show objects, including hidden objects.

4. If 'program' is found, delete or rename the object.

5. Repeat step 1 - 4 for any other volumes or drive letters.

6. Attempt to start the NetBackup Messaging Broker Service.

7. Reboot the host at the next opportunity.

Additionally, a similar startup issue has been observed to occur when the Windows 8dot3 name convention has been disabled for the volume on which NetBackup is installed, and no 8dot3 name structure exists for 'Program Files' (PROGRA~1).
See article 100046867 for more details.