VMware backups, restores and Individual File Recovery operations may fail on Windows hosts with DLL search restrictions enabled

Problem

VMware backups, restores and Individual File Recovery operations may fail on Windows hosts with DLL search restrictions implemented

Error Message

bpbkar or tar log shows the following error message:

07:37:36.287 [21176.20652] <2> tar_base::V_vTarMsgW: FTL - VxMS initialization failed

Cause

NetBackup is unable to load VxMS Dynamic Link Libraries (DLLs) as they are installed in a separate directory from the binary files (bpbkar32.exe and tar32.exe) that call them

Solution

Check for the existense of the following registry value to confirm that the DLL loading restrictions are in place:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\CWDIllegalInDllSearch

Value of 0xFFFFFFFF (hexadecimal) will prevent bpbkar32.exe and tar32.exe from loading the VxMS DLLs.

To allow these processes to load VxMS DLLs, create per-process overrides by creating the following registry keys:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\bpbkar32.exe
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\tar32.exe

In each of these keys, create a DWORD value CWDIllegalInDllSearch and set it to 0.