NetBackup database (NBDB) fails to start due to master server not recognized as a master

Problem

When attempting to open the Activity Monitor in Java Console the error message "Unable to connect to the Enterprise Media Manager server: The EMM server failed to process the request(78)" is displayed.  Also not able to present any policy from the policies tab or see any Media and Device Manager items.

Error Message

Not all NetBackup services are UP for a master server even when restarted.

1. ping the NetBackup database:
# ./nbdb_ping
Database [NBDB] is not available.

2. Start NBDB manually which was sucessful except NBDBis staying up manually.
# . /nbdb_admin -auto_start NBDB
Successfully added NBDB to databases.conf.


In the bpjobd log it was observed that pbx is failing to connect with bpdbm:

# less ../logs/bpjobd/log.061617
14:57:59.955 [37757] <8> do_pbx_service: [vnet_connect.c:2085] vnet_pbxConnect() failed, status=18, errno=104, use_vnetd=0, cr->vcr_service=bpdbm
14:57:59.955 [37757] <8> async_connect: [vnet_connect.c:1677] do_service failed 18 0x12
14:57:59.956 [37757] <8> vnet_vnetd_pbx_c_supported: [vnet_vnetd.c:3367] VN_REQUEST_PBX_C_SUPPORTED 13 0xd
14:58:00.036 [37757] <8> do_vnetd_service: [vnet_connect.c:2006] remote host supports PBX, but PBX is not running 0 0x0
14:58:00.036 [37757] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:1995] VN_REQUEST_SERVICE_SOCKET 6 0x6
14:58:00.036 [37757] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2009] service bpdbm
14:58:00.076 [37757] <8> vnet_pop_byte: [vnet.c:194] errno 2 0x2
14:58:00.076 [37757] <2> vnet_pop_byte: vnet.c.196: 139877089953568: Function failed: 9 0x00000009
14:58:00.076 [37757] <2> vnet_pop_string: vnet.c.276: 139877089953568: Function failed: 9 0x00000009
14:58:00.076 [37757] <2> vnet_pop_signed: vnet.c.319: 139877089953568: Function failed: 9 0x00000009
14:58:00.076 [37757] <2> vnet_pop_status: vnet.c.397: 139877089953568: Function failed: 9 0x00000009
14:58:00.076 [37757] <8> vnet_vnetd_service_socket: [vnet_vnetd.c:2025] status 9 0x9
14:58:00.076 [37757] <8> do_vnetd_service: [vnet_connect.c:2031] vnet_vnetd_service_socket failed 9 0x9
14:58:00.076 [37757] <8> async_connect: [vnet_connect.c:1677] do_service failed 9 0x9
14:58:00.076 [37757] <8> async_connect: [vnet_connect.c:1700] getsockopt SO_ERROR returned 111 0x6f
14:58:01.076 [37757] <8> async_connect: [vnet_connect.c:1700] getsockopt SO_ERROR returned 111 0x6f
14:58:03.077 [37757] <8> async_connect: [vnet_connect.c:1700] getsockopt SO_ERROR returned 111 0x6f
14:58:07.077 [37757] <8> async_connect: [vnet_connect.c:1700] getsockopt SO_ERROR returned 111 0x6f
14:58:15.077 [37757] <8> async_connect: [vnet_connect.c:1700] getsockopt SO_ERROR returned 111 0x6f
14:58:15.077 [37757] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED
        status: FAILED, (24) BAD_VERSION; system: (104) Connection reset by peer; FROM 0.0.0.0 TO bcback1.abc.com 172.xx.xxx.xx bpdbm VIA pbx
        status: FAILED, (9) READ_FAILED; system: (2) No such file or directory; FROM 0.0.0.0 TO bcback1.abc.com 172.xx.xxx.xx bpdbm VIA vnetd
        status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO bcback1.abc.com 172.xx.xxx.xx bpdbm

Output from  nbemmcmd -listhost -verbose command:

NBEMMCMD, Version: 7.6.1
The following hosts were found:
bcback1.abc.com
        MachineName = "bcback1.abc.com"
        FQName = "bcback1.abc.com"
        MachineDescription = ""
        MachineNbuType = server (6)
bcback1.abc.com
        ClusterName = ""
        MachineName = "bcback1.abc.com"
        FQName = "bcback1.abc.com"
        GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x77
        MachineNbuType = master (3)
        MachineState = not reachable by master (4)
        NetBackupVersion = 7.6.1.0 (761000)
        OperatingSystem = linux (16)
        ScanAbility = 5     

The following may be seen in the nbpem log:
06/16/2017 15:44:10.092 [Application] NB 51216 nbpem 116 PID:9099 TID:139698326468384 File ID:116 [No context] [Critical] V-116-128 nbpem will not start - ho
st bcback1.abc.com is not a Master Server

Checked bp.conf file and noticed that the master server name was listed as "MEDIA_SERVER= masterserver_hostname".
 

Cause

Master server name was listed as "MEDIA_SERVER= masterserver_hostname" in bp.conf file.  The nbemmcmd -listhost -verbose command shows " MachineState = not reachable by master (4)"

Solution

1. Remove MEDIA_SERVER= masterserver_hostname from /usr/openv/netbackup/bp.conf file
2. /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache
3. Restart NetBackup services.
All services should start successfully.