SAN client backup fails with status 83 after relocating the media instance to another node in the Flex HA cluster

Problem

SAN client backup fails after relocating the media instance to another node in the Flex HA cluster with pipe open failed (17 unknown failure) and media open error (83).

Error Message

The backup operation Detailed Status shows pipe open failed 17 and media open error 83:

Errors in nbftclient.log:

0,51216,200,200,1363,1747021834611,13018,140362676942592,0:,118:Send FAT_CLIENT_DEV_STATUS_CHNG_EVENT from <server> to 
<server>,48:FTClientMgr_i::reportDeviceStatusChangeException,1
0,51216,200,200,1364,1747021834628,13018,140362676942592,0:,118:Send FAT_CLIENT_DEV_STATUS_CHNG_EVENT from<server> to 
<server>,48:FTClientMgr_i::reportDeviceStatusChangeException,1
0,51216,137,200,18350,1747021834628,13018,140362676942592,0:,31:This is not the K8S environment,25:nbk8s_connection_strategy,1
0,51216,137,200,18351,1747021834628,13018,140362676942592,0:,31:This is not the K8S environment,25:nbk8s_connection_strategy,1
0,51216,200,200,1365,1747021835081,13018,140362679043840,0:,74:called from PipeInit EVPD Page 0x83 "" device name /dev/sg1 0 retries left,13:DeviceInquiry,1
0,51216,200,200,1366,1747021835081,13018,140362679043840,0:,143:DeviceInquiry failed 1376386 for device /dev/sg1 inquiry SYMANTECFATPIPE 0.0     
vflexa5340-1011v03.server with other references 4,8:PipeInit,1
0,51216,200,200,1367,1747021835081,13018,140362679043840,0:,73:Forcing validation of sequence via requested check condition on pipeId 28,8:PipeInit,1
...
0,51216,200,200,1377,1747021835082,13018,140362679043840,0:,67:DeviceIoControl SCMD_READ_BUFFER check condition 5/37/0 on /dev/sg1,15:GetBufferLength,1
0,51216,200,200,1378,1747021835082,13018,140362679043840,0:,183:PipeInit failed with 2097169: pPipe = 0x7fa8ac0072d0: pDevice = 0x7fa8a400f270 "/dev/sg1": 
pipeId 28: maxRetries 3: CorbanumBufSeg 16: CorbaBufSegSize 0: SingleThread 1: Synchronous 0,8:OpenPipe,1
0,51216,200,200,1379,1747021835082,13018,140362679043840,0:,24: backup cnt 0 Pipe ID 28,13:~FTClientPipe,1
0,51216,200,200,1380,1747021835082,13018,140362679043840,0:,140:Fatal error, failed to open the pipe for jobid=41, on device SYMANTECFATPIPE 0.0     
vflexa5340-1011v03.server, error = 2097169,23:FTClientMgr_i::openPipe,1
1,51216,137,200,18360,1747021838632,13018,140362676942592,0:,0:,27:Orb::connectToObjectRetries,2,(40|)
0,51216,200,200,1381,1747021838633,13018,140362681145088,0:,58: received semaphore indicated discovery task already 
done.,38:FATClientDevices::RefreshOSdevicePaths,1
0,51216,200,200,1382,1747021839633,13018,140362681145088,0:,61:Rescan of FT Devices not supported on this OS in this version,26:FTClientMgr_i::scanDevices,1
0,51216,137,200,18361,1747022025272,13018,140362681145088,0:,31:This is not the K8S environment,25:nbk8s_connection_strategy,1
0,51216,137,200,18362,1747022025274,13018,140362681145088,0:,24:pbxConnectExEx Succeeded,18:vnet_pbxConnect_ex,1
0,51216,137,200,18363,1747022025285,13018,140362681145088,0:,24:remote_proxy_version = 7,34:vnet_proxy_protocol_from_legacy_ex,1
...

Cause

During relocation, the sg devices are in an improper state on the SanClient machine.
 

Running a backup during this time will fail, since the media server is relocating and the devices are not in a proper state.

Allowing time for the relocating process to complete and device discovery to occur is needed before running backup operations, otherwise they will fail with the PipeInit Open operation failure as seen in the logs above.

Solution

In case of backup failure, execute the following command on the SanClient machine and trigger the backup again.

1. Stop the nbftclient service:

/usr/openv/netbackup/bin/nbftclnt -terminate

2. Check if the nbftclient process has exited:

/usr/openv/netbackup/bin/bpps -x | grep nbftclnt

3. Start the SAN Client service:

/usr/openv/netbackup/bin/bp.start_all

4. Check if the nbftclient process has started:

/usr/openv/netbackup/bin/bpps -x | grep nbftclnt