Vmware Backups Fail with Status 58 or connect failed STATUS (18) CONNECT_FAILED

Problem

Vmwar Backups Fail with Status 58  Or " connect failed STATUS (18) CONNECT_FAILED   " Or "status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it" Or "getsockopt SO_ERROR returned 10061 0x274d"

Error Message

Job Details shows error:

5/07/2012 11:20:04 a.m. - Info nbjm(pid=3716) starting backup job (jobid=17870) for client NBclient, policy VADP_7503_NBclient_Tape, schedule Full-Daily
5/07/2012 11:20:04 a.m. - estimated 0 Kbytes needed
5/07/2012 11:20:04 a.m. - Info nbjm(pid=3716) started backup (backupid=NBclient_1341444004) job for client NBclient, policy VADP_7503_NBclient_Tape, schedule Full-Daily on storage unit Lib1
5/07/2012 11:20:06 a.m. - started process bpbrm (5916)
5/07/2012 11:20:08 a.m. - Info bpbrm(pid=5916) starting bptm          
5/07/2012 11:20:08 a.m. - Info bpbrm(pid=5916) Started media manager using bpcd successfully      
5/07/2012 11:25:10 a.m. - Info bpbrm(pid=5916) connect failed STATUS (18) CONNECT_FAILED       
5/07/2012 11:25:10 a.m. - Info bpbrm(pid=5916)     status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd VIA pbx
5/07/2012 11:25:10 a.m. - Info bpbrm(pid=5916)     status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd VIA vnetd
5/07/2012 11:25:10 a.m. - Info bpbrm(pid=5916)     status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd
5/07/2012 11:25:10 a.m. - Error bpbrm(pid=5916) Cannot connect to NBclient        
5/07/2012 11:25:10 a.m. - end writing
can't connect to client(58)

bpfis log shows that the snapshot is successful

11:09:22.513 [6648.2696] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -id NBclient_1341443359 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 40 -vm_type 2 -vm_client NBclient -clnt CLNT1 -st FULL -rg other -fim VMware_v2:Virtual_machine_backup=2,nameuse=1,snapact=2,file_system_optimization=0,trantype=san:nbd,exclude_swap=1,disable_quiesce=0,drive_selection=0,skipnodisk=0,post_events=1,nameuse=1 -class VADP_7503_NBclient_Tape -sched Full-Daily -fso -S CLNT -jobid 17869
11:09:22.513 [6648.2696] <2> pfi_start_client: received bpcd success message
11:09:23.574 [6648.2696] <2> pfi_start_client: read start message from CLNT1, msg=<INF - BACKUP START 6140>
11:09:23.574 [6648.2696] <2> set_job_details: Tfile (17869): LOG 1341443363 4 bpfis 6140 Backup started

11:09:23.574 [6648.2696] <2> send_job_file: job ID 17869, ftype = 3 msg len = 43, msg = LOG 1341443363 4 bpfis 6140 Backup started

11:09:23.574 [6648.2696] <2> write_file_names: buffering file name 'NEW_STREAM' for output
11:09:23.574 [6648.2696] <2> write_file_names: buffering file name 'ALL_LOCAL_DRIVES' for output
11:09:23.574 [6648.2696] <2> write_file_names: successfully wrote buffer to COMM_SOCK
11:09:23.590 [6648.2696] <2> bpbrm write_filelist: wrote CONTINUE on COMM_SOCK
11:09:24.448 [6648.2696] <2> get_bpfis_msg: bpfis buff: INF - FIS_ID=NBclient_1341443359
11:09:24.448 [6648.2696] <2> get_bpfis_msg: bpfis buff: INF - ACT=SNAPSHOT_METHOD 0 NBclient VMware_v2
11:09:24.448 [6648.2696] <2> set_job_details: Tfile (17869): SNAPSHOT_METHOD 1341443364 NBclient VMware_v2
11:09:24.448 [6648.2696] <2> send_job_file: job ID 17869, ftype = 3 msg len = 50, msg = SNAPSHOT_METHOD 1341443364 NBclient VMware_v2
11:10:24.414 [6648.2696] <2> get_bpfis_msg: bpfis buff:
 

bpbrm Log shows error

11:20:09.320 [5916.3140] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=D:\Program Files\Veritas\NetBackup\var\host_cache\147\37073547+veritas_pbx,1,400,2,1,0+NBclient.txt
11:20:09.320 [5916.3140] <2> retry_getaddrinfo_for_real: [vnet_addrinfo.c:1061] getaddrinfo() failed RV=10109 NAME=NULL SVC=veritas_pbx
11:20:09.320 [5916.3140] <2> get_sig_pbx_port: [vnet_addrinfo.c:4416] RegOpenKeyEx failed STAT=2 REGISTRY=Software\VERITAS\VxPBX
11:20:09.320 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:884] using IANA SVC=veritas_pbx PORT=1556
11:20:09.320 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] adjusted service name 1556
11:20:09.320
[5916.3140] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1630] found via getaddrinfo NAME=NBclient SVC=veritas_pbx
11:20:09.320 [5916.3140] <2> vnet_sortaddrs: [vnet_addrinfo.c:3945] sorted addrs: 1 0x1
11:20:09.320 [5916.3140] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=D:\Program Files\Veritas\NetBackup\var\host_cache\05b\4320d05b+0,1,a,0,1,0+10.6.0.17.txt
11:20:09.320 [5916.3140] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1630] found via getaddrinfo NAME=10.6.0.17 SVC=NULL
11:20:09.320 [5916.3140] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4809] using interface  ANY
11:20:09.320 [5916.3140] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=D:\Program Files\Veritas\NetBackup\var\host_cache\147\37073547+vnetd,1,400,2,1,0+NBclient.txt
11:20:09.320 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:906] using SVC=vnetd PORT=13724
11:20:09.320 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] adjusted service name 13724
11:20:09.320 [5916.3140] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1630] found via getaddrinfo NAME=NBclient SVC=vnetd
11:20:09.320 [5916.3140] <2> vnet_sortaddrs: [vnet_addrinfo.c:3945] sorted addrs: 1 0x1
11:20:09.336 [5916.3140] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4809] using interface  ANY
11:20:09.336 [5916.3140] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=D:\Program Files\Veritas\NetBackup\var\host_cache\147\37073547+bpcd,1,400,2,1,0+NBclient.txt
11:20:09.336 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:906] using SVC=bpcd PORT=13782
11:20:09.336 [5916.3140] <2> retry_getaddrinfo: [vnet_addrinfo.c:910] adjusted service name 13782
11:20:09.336 [5916.3140] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1630] found via getaddrinfo NAME=NBclient SVC=bpcd
11:20:09.336 [5916.3140] <2> vnet_sortaddrs: [vnet_addrinfo.c:3945] sorted addrs: 1 0x1
11:20:09.336 [5916.3140] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4809] using interface  ANY
11:20:09.336 [5916.3140] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1
11:20:10.366 [5916.3140] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 10061 0x274d
11:20:10.366 [5916.3140] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1
11:20:11.364 [5916.3140] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 10061 0x274d
11:20:11.364 [5916.3140] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1
11:20:12.378 [5916.3140] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 10061 0x274d
11:20:13.392 [5916.3140] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1
11:20:14.406 [5916.3140] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 10061 0x274d
11:25:10.697 [5916.3140] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
11:25:10.697 [5916.3140] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
11:25:10.697 [5916.3140] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
11:25:10.697 [5916.3140] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED
 status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd VIA pbx
 status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd VIA vnetd
 status: FAILED, (44) CONNECT_TIMEOUT; system: (10061) No connection could be made because the target machine actively refused it. ; FROM 0.0.0.0 TO NBclient 10.6.0.17 bpcd
11:25:10.697 [5916.3140] <8> vnet_connect_to_bpcd: [vnet_connect.c:279] connect_to_service() failed 18 0x12
11:25:10.697 [5916.3140] <2> local_bpcr_connect: Can't connect to client NBclient
11:25:10.697 [5916.3140] <2> ConnectToBPCD: bpcd_connect_and_verify(NBclient, NBclient) failed: 25
11:25:10.697 [5916.3140] <2> set_job_details: Tfile (17870): LOG 1341444310 4 bpbrm 5916 connect failed STATUS (18) CONNECT_FAILED
11:25:10.697 [5916.3140] <16> bpbrm start_bpcd_stat: Cannot connect to NBclient
11:25:10.697 [5916.3140] <2> set_job_details: Tfile (17870): LOG 1341444310 16 bpbrm 5916 Cannot connect to NBclient

11:25:10.697 [5916.3140] <2> send_job_file: job ID 17870, ftype = 3 msg len = 60, msg = LOG 1341444310 16 bpbrm 5916 Cannot connect to NBclient

Solution

1> Check under Backup Policy > Vmware Tab > Backup Host >  Confirm  the Correct Backup Media Server is selected.
2> Check under Master Server Host properties > Vmware Access Hosts >  Correct Backup Media Server is selected.
3> Run bpclntcmd -clear_host_cache on Master / Media Server.
4> Retry Backup .