STATUS CODE 54 or STATUS CODE 58: Some third-party software packages can link into the TCP/IP stack in Windows and cause a loss of connection between the Veritas NetBackup (tm) server and the bpcd process on the client.

Problem

Some third-party software packages can link into the TCP/IP stack in Windows and cause a loss of connection between the Veritas NetBackup (tm) server and the bpcd process on the client.

Error Message

snip from bpbrm log

11/10/2016 13:28:08 - Info bpbrm (pid=8344)     status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is currently executing. ; FROM 0.0.0.0 TO <ip address of client> <ip address of client> vnetd VIA pbx
11/10/2016 13:28:08 - Info bpbrm (pid=8344)     status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is currently executing. ; FROM 0.0.0.0 TO <ip address of client> <ip address of client> vnetd VIA vnetd
11/10/2016 13:28:08 - Error bpbrm (pid=8344) Cannot connect to <client name>

The below sequence was observed with a NBU 8.1 master server and NBU 8.1 client

snip from on-screen result of "bptestbpcd -client <client_name> -verbose -debug"

12:03:34.020 [11400.11280] <2> logconnections: BPCD CONNECT FROM <masterIP>.62154 TO <clientIP>.1556 fd = 620
12:03:34.030 [11400.11280] <2> vnet_connect_to_vnetd_bpcd: js_bpcd_info: 000000000262E250
12:03:34.037 [11400.11280] <2> vnet_pbxConnect_ex: pbxConnectExEx Succeeded
12:08:34.072 [11400.11280] <2> vnet_pop_byte: vnet.c.187: 0: Function failed: 11 0x0000000b    <------  5 minute delay
...
12:08:34.112 [11400.11280] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation  currently executing. ; FROM 0.0.0.0 TO MyClient <clientIP> vnetd VIA pbx status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is currently executing. ; FROM 0.0.0.0 TO MyClient <clientIP> vnetd VIA vnetd

Client PBX log - can see the inbound connection and the handoff to bpcd:

11/20/2017 12:03:33.854 [Application] VxICS 50936 103 PID:4700 TID:4984 File ID:103 [No context] [Info] PBX_Client_Proxy::parse_line, line = ack=25  From <masterIP>, Port 62154
11/20/2017 12:03:33.854 [Application] VxICS 50936 103 PID:4700 TID:4984 File ID:103 [No context] [Info] PBX_Client_Proxy::parse_line, line = extension=bpcd  From <masterIP>, Port 62154
11/20/2017 12:03:33.854 [Application] VxICS 50936 103 PID:4700 TID:4984 File ID:103 [No context] [Info] hand_off looking for proxy for = bpcd

Client bpcd log - can see the error:

12:03:33.854 [6656.6660] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

Cause

Some third-party software packages can link into the TCP/IP stack in Windows and cause a loss of connection between the NetBackup server and the bpcd process on the client.  A Winsock error code  10022, 10038 or 10036 can also be seen in the client's bpcd log.

Troubleshooting:

The following is a partial list of software that this issue has occurred with:

• AT&T Communications Manager
• Aventail VPN
• Citrix Extranet Client, Citrix Advanced Gateway Client
• Citrix Metaframe Client Remote Admin
• Columbitech WAP connector and Gateway
• DIAMONDCS Port Explorer (This application shows up in the services query as "Port Explorer")
• HTTPLook
• Juniper Networks Secure Application Manager
• Juniper Networks VPN Client
• Lavasoft WC Assistant (WCAssistantService)
• LavaSoft Web Companion
• LavasoftTcpService
• Microsoft Personal Firewall
• Microsoft Forefront
• Microsoft Proxy Server
• Neoteris Application Proxy
• NetSpective Terminal Server Agent.
• NOD32 Anti-virus Software  (See workaround for NOD32 in the Related Documents section below)
• Panda Antivirus
• Permeo Application Security suite (client module)
• PGP Desktop
• OpenText SOCKS Client
• SurfControl
• Kaseya Agent
• McAfee Solidifier
• Snow Inventory
• SogouImeBroker
• ComputerZService
• 鲁大师（Software which includes a network driver "netload.dll" from Qihoo 360)
• 堡垒机基础控件+堡垒机客户端（Which includes a network driver "bh_am_portal_interceptor64.dll")

Log Files:

The following install_path\Veritas\NetBackup\logs\bpcd\<date>.LOG extract shows what is observed on the client side bpcd log when this error occurs:

<2> bpcd main: offset to GMT 0
<2> bpcd main: Got socket for input -1
<2> logconnections: getsockname(-1) failed: 10038
<16> bpcd setup_sockopts: setsockopt 1 failed:
<2> bpcd main: setup_sockopts complete
<16> bpcd main: authentication failed: 17
<2> bpcd main: offset to GMT 0

or...

<2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.

Solution

The error can be avoided by completely uninstalling the software on the client that is relative to the failure.  However, it is recommended to contact the respective software manufacturer to investigate if other configuration options or workarounds are possible.

When the error occurs, NetBackup is trying to set SO_REUSEADDR (allow local address reuse) on the inbound socket connection so the port can be handed off from bpinetd.exe (the NetBackup Client Service) to bpcd.exe.  Some products, such as those listed above, may not allow this functionality due to various methods that can be used to violate system security.

Running the command "netsh winsock show catalog " reveals which software may be causing the issue