VMware HotAdd Backup intermittently fails with "Error 7 in read with disk handle"

Problem

VMware HotAdd Backup intermittently fails with "Error 7 in read with disk handle"

Error Message

Job Details:
Mar 1, 2022 11:43:35 PM - Critical bpbrm (pid=267280) from client vmclient: FTL - cleanup() failed, status 11
Mar 1, 2022 11:43:37 PM - Error bptm (pid=267419) media manager terminated by parent process
Mar 1, 2022 11:43:43 PM - Critical bpbrm (pid=267280) unexpected termination of client vmclient
Mar 1, 2022 11:43:43 PM - Info bpbkar (pid=0) done. status: 11: system call failed
 
VDDK/VxMS_Provider:
   03/01/2022 23:23:59 : vdRead:VixInterface.cpp:734 <ERROR> : Error 7 in read with disk handle 34836000 startSector 32944128 numSectors 256
   03/01/2022 23:23:59 : read:VixFile.h:272 <ERROR> : Returned error 7, offset 0x00000003ED600000, length 0x0000000000020000   03/01/2022 23:23:59 : read:VixFile.h:273 <WARN> : Returning: 11
   03/01/2022 23:23:59 : vixMapRead:VixCoordinator.cpp:1196 <WARN> : Returning: 11
   03/01/2022 23:23:59 : hv_vmware_map_read:libvix.cpp:1552 <WARN> : Returning: 11
   03/01/2022 23:24:00 : vdRead:VixInterface.cpp:734 <ERROR> : Error 7 in read with disk handle 34836000 startSector 32944128 numSectors 256
   03/01/2022 23:24:00 : read:VixFile.h:272 <ERROR> : Returned error 7, offset 0x00000003ED600000, length 0x0000000000020000   03/01/2022 23:24:00 : read:VixFile.h:273 <WARN> : Returning: 11
   03/01/2022 23:24:00 : vixMapRead:VixCoordinator.cpp:1196 <WARN> : Returning: 11
   03/01/2022 23:24:00 : hv_vmware_map_read:libvix.cpp:1552 <WARN> : Returning: 11
   03/01/2022 23:24:01 : vdRead:VixInterface.cpp:734 <ERROR> : Error 7 in read with disk handle 34836000 startSector 32944128 numSectors 256
   03/01/2022 23:24:01 : read:VixFile.h:272 <ERROR> : Returned error 7, offset 0x00000003ED600000, length 0x0000000000020000   03/01/2022 23:24:01 : read:VixFile.h:273 <WARN> : Returning: 11
   03/01/2022 23:24:01 : vixMapRead:VixCoordinator.cpp:1196 <WARN> : Returning: 11
   03/01/2022 23:24:01 : hv_vmware_map_read:libvix.cpp:1552 <WARN> : Returning: 11

DISKLIB-LIB   : RWv failed ioId: #476238 (327689) (9) .
VixDiskLib: Detected DiskLib error 327689 (Input/output error).
Vix_TranslateErrno: errno = 5
Foundry operation failed with system error: Input/output error (5), translated to 7
VixDiskLib: VixDiskLib_Read: Read 128 sectors at 0 failed. Error 7 (A file access error occurred on the host or guest operating system) (DiskLib error 327689: Input/output error) at 6151.
DISKLIB-LIB   : RWv failed ioId: #476239 (327689) (9) .
VixDiskLib: Detected DiskLib error 327689 (Input/output error).
Vix_TranslateErrno: errno = 5

Cause

This issue can occur when the SCSI controller are not in order.

Solution

This is a known VMware issue:
https://kb.vmware.com/s/article/67757

In case of multiple SCSI controllers, the best practice below must be completely followed to avoid issues above.

1. Make sure the OS disks are on the SCSI controller 0 with numeric orders (such as disk 0 on scsi0:0, disk 1 on scsi0:1, disk 2 on scsi0:2...).
2. Add SCSI controller 1/2/3 in numeric order.
3. Add SATA controller 0.
4. Make sure there is no CDROM device.
5. Make sure there is no device on IDE0/1.
6. Make sure there is no disk/device on SATA controller.
7. Reboot VM.