NetBackup for VMware backup fails with Status Code 6 and "Connection reset by peer" or "connection may have been broken" message

Problem

NetBackup for VMware backup fails with Status Code 6: the backup failed to back up the requested files

Error Message

Activity monitor details for a failed job:

Jul 25, 2011 11:32:20 AM - Error bpbrm (pid=22974) from client clnt: ERR - Error opening the snapshot disks using given transport mode: Status 23.
Jul 25, 2011 11:32:21 AM - Error bptm (pid=23018) system call failed - Connection reset by peer (at child.c.1296)
Jul 25, 2011 11:32:21 AM - Error bptm (pid=23018) unable to perform read from client socket, connection may have been broken
Jul 25, 2011 11:32:21 AM - Error bptm (pid=23013) media manager terminated by parent process
Jul 25, 2011 11:32:01 AM - estimated 88820 kbytes needed
Jul 25, 2011 11:32:02 AM - started process bpbrm (pid=22974)
Jul 25, 2011 11:32:05 AM - connecting
Jul 25, 2011 11:32:06 AM - connected; connect time: 0:00:00
Jul 25, 2011 11:32:21 AM - begin writing
Jul 25, 2011 11:35:16 AM - Error bpbrm (pid=22974) could not send server status message
Jul 25, 2011 11:35:18 AM - end writing; write time: 0:02:57
the backup failed to back up the requested files  (6)

Cause

The VMware backup host is unable to communicate properly to the ESX server. This can be due to various network related issues; e.g. forward/reverse name resolution, traffic on port 902 is being blocked by firewall or security device, hardware failure, etc.

In the example below, backups using NBD transport type are unable to communicate to the ESX server on port 902.

Troubleshooting:

Check for errors in bpbkar logs, which may be pointing towards VxMS logs:

11:32:09.431 AM: [19048.25760] <4> tar_base::V_vTarMsgW: INF - tar message received from vxms_error_handler
11:32:09.431 AM: [19048.25760] <2> tar_base::V_vTarMsgW: INF - VxMS Error message 1 = Error opening snapshot disk(s) using given transport mode(s)

11:32:09.843 AM: [19048.25760] <4> tar_base::V_vTarMsgW: INF - tar message received from stage_one_virtual
11:32:09.843 AM: [19048.25760] <2> tar_base::V_vTarMsgW: ERR - Error opening the snapshot disks using given transport mode: Status 23.

11:32:09.845 AM: [19048.25760] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 6: the backup failed to back up the requested files

Check for errors in VxMS logs, to determine the potential cause:

07/25/2011 16:32:08.0426 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : VixDiskLib: Trying to open disk [FC_PITEMC_MGMT] clnt_1/clnt.vmdk using mode nbd.
        07/25/2011 16:32:08.0426 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : NBD_ClientOpen: attempting to create connection to vpxa-nfc://[FC_PITEMC_MGMT] clnt_1/clnt.vmdk@<ESX-Server-On-Port>:902
        07/25/2011 16:32:08.0426 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : Started up WSA
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : Cannot connect to server <ESX-Server-On-Port>:902: No connection could be made because the target machine actively refused it
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : CnxAuthdConnect: Returning false because CnxAuthdConnectTCP failed
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : CnxConnectAuthd: Returning false because CnxAuthdConnect failed
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : Cnx_Connect: Returning false because CnxConnectAuthd failed
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : Cnx_Connect: Error message: Failed to connect to server <ESX-Server-On-Port>:902
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1693 <VFM_ESWARN> : [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server <ESX-Server-On-Port>:902
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : NBD_ClientOpen: Couldn't connect to <ESX-Server-On-Port>:902:902 Failed to connect to server <ESX-Server-On-Port>:902
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : DISKLIB-DSCPTR: : "vpxa-nfc://[FC_PITEMC_MGMT] clnt_1/clnt.vmdk@<ESX-Server-On-Port>:902" : Failed to open NBD extent.
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : DISKLIB-LINK  : "vpxa-nfc://[FC_PITEMC_MGMT] clnt_1/clnt.vmdk@<ESX-Server-On-Port>:902" : failed to open (NBD_ERR_NETWORK_CONNECT). 
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : DISKLIB-CHAIN : "vpxa-nfc://[FC_PITEMC_MGMT] clnt 1/clnt.vmdk@<ESX-Server-On-Port>:902" : failed to open (NBD_ERR_NETWORK_CONNECT).
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: g_vdInterfaceLogger:.\libvix.cpp:1689 <VFM_ESINFO> : DISKLIB-LIB   : Failed to open 'vpxa-nfc://[FC_PITEMC_MGMT] clnt_1/clnt.vmdk@<ESX-Server-On-Port>:902 with flags 0x1e NBD_ERR_NETWORK_CONNECT (2338).
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: VixInterface::vdOpen:.\VixInterface.cpp:198 <VFM_ESERROR> [Sys Error: No error] : VixDiskLib_Open() error. Exited
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: VixGuest::openLeafSnapshotDisks:.\VixGuest.cpp:437 <VFM_ESERROR> [Sys Error: No error] : vdOpen() error = 13. Calling closeLeafSnapshotDisks()
        07/25/2011 16:32:09.0431 [TID 0x000064a0]: VixGuest::openLeafSnapshotDisks:.\VixGuest.cpp:540 <VFM_ESERROR> [Sys Error: No error] : Exited with failure
        07/25/2011 16:32:09.0432 [TID 0x000064a0]: VixCoordinator::vixMapObjCtl:.\VixCoordinator.cpp:904 <VFM_ESERROR> [Sys Error: No error] : Exited with 23
        07/25/2011 16:32:09.0432 [TID 0x000064a0]: vix_map_objctl:.\libvix.cpp:1145 <VFM_ESERROR> [Sys Error: No error] : Error 23
End vfm_objctl: result(23)
Begin: vfm_close [07/25/11 11:32:09]

Confirm TCP port 902 is open, by running OS level commands.

Examples, where 'esxname' is the name of the ESX server:

• UNIX/Linux: telnet esxname 902
• Windows(PowerShell): New-Object System.Net.Sockets.TcpClient("esxname",902)

Solution

In the example above, the communication issues were rectified by:

• Consulting with the network team/system administrator, to ensure TCP port 902 was opened between the VMware Backup Host and all the ESX host, as it is required for HotAdd/NBD/NBDSSL transport for backups and restores, per the NetBackup Network Ports Reference Guide.

Applies To

NBU 8.x, 9.x and 10.x
ESX 4.x, 5.x, 6.x, 7.x, and 8.x