VMware backups over NBD/NBDSSL are failing with NFC related errors

Problem

When performing VMware using NBD/NBDSSL, some backups may fail with NFC error due to network configuration.

Error Message

Activity Monitor > Job Details:

Dec 8, 2020 6:28:45 AM - Info bpbkar (pid=23313) INF - Transport Type =  nbd 
Dec 8, 2020 7:07:46 AM - Critical bpbrm (pid=23260) from client VMCLIENT: FTL - cleanup() failed, status 11 
Dec 8, 2020 7:07:48 AM - Error bptm (pid=23328) media manager terminated by parent process 

Backup Host > VDDK logs:

DISKLIB-LIB   : RWv failed ioId: #49498 (2338) (34) .
VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
VixDiskLib: VixDiskLib_Read: Read 256 sectors at 2705536 failed. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 5235.
[NFC DEBUG] NfcFssrvr_IOEx: write = 0, compType = 0, offset = 1385234432, length = 131072 flags = 0x0 type = 2
[NFC ERROR] NfcNetTcpWrite: bWritten: -1
[NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
[NFC ERROR] NfcFssrvr_IOEx: failed to send io message
DISKLIB-LIB   : RWv failed ioId: #49499 (2338) (34) .
VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
VixDiskLib: VixDiskLib_Read: Read 256 sectors at 2705536 failed. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 5235.

Backup Host > VxMS_Provider logs:

   02/21/2021 11:24:25 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] DISKLIB-LIB   : RWv failed ioId: #49496 (2338) (34) .
   02/21/2021 11:24:25 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
   02/21/2021 11:24:25 : g_vixInterfaceLogger:libvix.cpp:1800 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_Read: Read 256 sectors at 2705536 failed. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 5235.
   02/21/2021 11:24:25 : vdRead:VixInterface.cpp:623 <ERROR> : Error 14009 in read with disk handle 22289888 startSector 2705536 numSectors 256
   02/21/2021 11:24:25 : read:VixFile.h:272 <ERROR> : Returned error 14009, offset 0x0000000052910000, length 0x0000000000020000   
   02/21/2021 11:24:25 : read:VixFile.h:273 <WARN> : Returning: 11

For details on how to collect VDDK and VxMS logs, see the Related Article (100015958) below.

Cause

The issue with VMware backups over NBD/NBDSSL failing with NFC errors could be due to an Intrusion Prevention System (IPS) or Intrusion Detection System (IDS) interfering with the NBD connections.

These systems are typically part of the network infrastructure and can inadvertently disrupt the communication between the Backup Host and ESX Host.

Solution

Recommend consulting with your network administrator to update the communication between the Backup Host and ESX Host to bypass the Intrusion Detection and Prevention System (IDPS) communications.

As this is a network-level change and not something that can be adjusted within NetBackup or VMware.