Status 7640 - [PROXY] Received status: 7640 with message Connection was closed by peer.

Problem

A client that had been backing up successfully started failing backups with Status 7640.

Error Message

The Job Details in the Activity Monitor show that bpbrm is able to connect to bpcd on the client host, but the pre-security JSON was not received from the remote host before the remote process closed the connection.  There looks to either be a problem between the processes bpcd and vnetd -proxy inbound on the client host or a problem receiving the pre-security JSON from the vnetd -proxy inbound across the inter-host connection.

Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] Received status: 7640 with message Connection was closed by peer.
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] Encountered error (CERT_PROTOCOL_READING_JSON_LENGTH) while processing(CertProtocol).
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) bpcd on clientname exited with status 7640: The peer closed the connection
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] Connecting host: mastername
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] pid: 14210
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] Received status: 7640 with message Connection was closed by peer.
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) [PROXY] Encountered error (CERT_PROTOCOL_READING_JSON_LENGTH) while processing(CertProtocol).
Jun 16, 2019 11:30:00 PM - Error bpbrm (pid=26295) cannot send mail because BPCD on clientname exited with status 61: the vnetd proxy encountered an error
Jun 16, 2019 11:30:00 PM - Info bpbkar (pid=0) done. status: 7640: The peer closed the connection
<cut>
Jun 16, 2019 11:30:00 PM - end writing
The peer closed the connection.  (7640)

From the NetBackup master server the command "bptestbpcd -client clientname -verbose" shows the same results, so both the master and media servers are encountering the same problem on the client host.

<16>bptestbpcd main: Function ConnectToBPCD(clientname) failed: 7640
<16>bptestbpcd main: The peer closed the connection
<16>bptestbpcd main: Connection was closed by peer.: 7640
        [PROXY] Encountered error (CERT_PROTOCOL_READING_JSON_LENGTH) while processing(CertProtocol).: 2
The peer closed the connection

From the NetBackup client, the command "bpclntcmd -pn -verbose" returns:

Can't connect to host mastername: cannot connect on socket (25)

On the NetBackup client, the command "bpclntcmd -pn -verbose -debug" provides additional details (the bpcd debug log on the client would also show similar details for the inbound connections above).  First, the inter-host connection from bpclntcmd to PBX/bprd was successful.  But then the next step, make an intra-host connection and transfer the inter-host connection to the vnetd -proxy outbound, did not find the expected process running on this host. Hence the reason the connection was closed by the client host and why the 'missing' process did not send the expected pre-security JSON.

<2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
<8> vnet_find_accepting_proxy: No proxies are currently accepting
<8> find_any_proxy: No proxies are currently running
<16> proxy_helper_get: No outbound_proxy proxies are currently running
<16> vnet_proxy_helper_connect_try: js_helper = NULL
<16> vnet_proxy_socket_swap: vnet_proxy_helper_connect() failed: 7652
<16> connect_to_service: vnet_proxy_socket_swap(NULL, 3, 310, 0, mastername, bprd, 1) failed: 7652
<16> connect_to_service: JSON data = {"allow_large_status": {"timestamp": 1560803634, "who": "vnet_tss_init", "line_number": 32, "comment": "allow vnet status > 255", "data": false}, "direct_connect": {"timestamp": 1560803634, "who": "connect_to_service", "line_number": 834, "comment": "connect parameters", "data": {"who": "vnet_connect_to_service", "host": "mastername", "service": "bprd", "override_required_interface": null, "extra_tries_on_connect": 0, "getsock_disable_to": 0, "overide_connect_timeout": 0, "connect_options": null}}, "status": {"timestamp": 1560803634, "who": "connect_to_service", "line_number": 981, "comment": "vnet status", "data": 7652}}
<8> vnet_connect_to_service: [vnet_connect.c:286] connect_to_service() failed 7652 0x1de4
<2> bprd_connect: vnet_connect_to_service(mastername) failed: 7652
<16> do_request: Can't connect to host mastername: cannot connect on socket (25)
Can't connect to host mastername: cannot connect on socket (25)

Cause

TCP connectivity is confirmed successful above and does not need to be checked.  Certificates and related configuration are not yet in use by the connections and do not need to be checked.  However, the reason that the vnetd -proxy inbound (affecting bpbrm and bptestbpcd from the servers) and outbound (affecting bpclntcmd -pn from the client) are not available on the client host needs to be identified and corrected.

A check of the NetBackup processes on the client confirms none of the vnetd processes are running.

# /usr/openv/netbackup/bin/bpps -a
NB Processes
------------
root 4653078   1   0   Jan 22   -  7:46 /usr/openv/netbackup/bin/nbdisco
root 3473708   1   0   Jan 22   -  1:47 /usr/openv/netbackup/bin/bpcd -standalone

windows:

C:\program files\veritas\netbackup\bin\bpps

Solution

Start the missing NetBackup client services on this client using the following commands:

Start the vnetd processes on the client host:

# /usr/openv/netbackup/bin/vnetd -standalone

Alternatively, stop and restart all of the NetBackup services:

# /usr/openv/netbackup/bin/goodies/netbackup stop
# /usr/openv/netbackup/bin/goodies/netbackup start

Windows:

c:\program files\veritas\netbackup\bin\bpdown -f -v

c:\program files\veritas\netbackup\bin\bpup -f -v

To check services are running:

# /usr/openv/netbackup/bin/bpps -a

If all services have started the output should look similar to:

NB Processes
------------
root 4063472   1   0 15:49:46   -  0:00 /usr/openv/netbackup/bin/vnetd -proxy outbound_proxy -number 0
root 5701798   1   0 15:49:47   -  0:00 /usr/openv/netbackup/bin/bpcd -standalone
root 9175222   1   0 15:49:46   -  0:00 /usr/openv/netbackup/bin/vnetd -proxy inbound_proxy -number 0
root 3146104   1   0 15:49:47   -  0:00 /usr/openv/netbackup/bin/vnetd -standalone
root 4587932   1   0 15:49:48   -  0:00 /usr/openv/netbackup/bin/nbdisco

Notice the three vnetd services that are now running that were not present in the earlier output from the bpps command.

Once these services are running on the client, then test the previously failing operation(s).

For root cause as to why the processes are not running, review the NetBackup vnetd debug log (if available) and the system messages files/logs.  It is likely someone intentionally stopped the processes but forgot to restart them.  But it is also possible that some problem may have caused them to terminate prematurely, and that problem may need to be identified and corrected to prevent a future failure.