An Active Directory GRT backup fails consistently with JET_ERR on Windows Server 2016 and 2019.

Problem

An Active Directory Granular Restore Technology (GRT) backup fails consistently with JET_ERR on Windows Server 2016 and 2019. 

Error Message

An example of the messages in the Detailed Status of the job in Activity Monitor may be like:

Jan 1, 2021 1:00:00 PM - Info bpbrm (pid=1234) from clientdomain_controller_client: TRV - Starting granular backup processing for(System State\Active Directory). This may take a while...
Jan 1, 20211:00:00 PM - Info bpbrm (pid=1234) from client domain_controller_client: TRV - Granular processing failed!
Jan 1, 2021 1:00:00 PM - Error bpbrm(pid=1234) from client domain_controller_client: ERR - Error encountered while attempting to get additional files for System State:\

Bpbkar log snippet may show:

12:08:37.168 [10524.9924] <2> ov_log::V_GlobalLog: INF - Open Z:\inprogress\<client_name>\allusers\full\1500004582\_vv_<client_name>\System State\Active Directory\ntds.dit!
12:08:37.871 [10524.9924] <2> ov_log::V_GlobalLog: INF - PDNT_index - +PDNT_col (0x9)
12:08:37.871 [10524.9924] <2> ov_log::V_GlobalLog: INF - PDNT_index - +ATTm589825 (0x9)
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF -     DSProv:Exception (-1090) opening [Z:\inprogress\<client_name>\allusers\full\1500004582\_vv_<client_name>\System State\Active Directory\ntds.dit] at B:248
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF - JET_ERR DbTable::Open -> -1090(0xfffffbbe) = -1090 This instance cannot be used because it encountered a fatal error. !               SES=67242272,DB=1 -Z:\inprogress\<client_name>\allusers\full\1500004582\_vv_<client_name>\System State\Active Directory\ntds.dit ()!
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF - Error on close -1090!
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF - Error on detach -1090!
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF -     DSProv:Error -1090 closing db at B:308
12:09:08.295 [10524.9924] <2> ov_log::V_GlobalLog: INF - Terminating instance Zinprogress<client_name>allusersfull1500004582_vv_<client_name>System StateActive Directory!.
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: INF -     DSProv:Failed to create the instance for [Z:\inprogress\<client_name>\allusers\full\1500004582\_vv_<client_name>\System State\Active Directory\ntds.dit] at P:93
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: INF - ADGran: Status SUCCESS (0x00000000) no provider enumerator in getADGInstanceName:775. Attach reason = 0x0001, Provider status = 0x000C
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: INF - ADGRAN : Attaching to ADGRAN with attach reason = 1, isLiveDB = 0, isADDomain = 0
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: INF - BEDS Resource ID - 0xF35
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: INF - ADGran: VFF error in ADGran_ChangeDir:104. Provider status = 0x000C
12:09:09.389 [10524.9924] <2> Root::start(): DBG - FS_ChangeDir() Failed! (0xE0008488:Access is denied.) (../Root.cpp:429)
12:09:09.389 [10524.9924] <2> ov_log::V_GlobalLog: ERR - raiPdiEnumerate():start() failed, error = 17

Cause

Unable to catalog to the AD structure across NFS.

Solution

If you are encountering this issue in NetBackup versions 7.7.3, 8.0, or 8.1, EEBs are available to help the client host get a successful granular backup of Active Directory.

• NetBackup 7.7.3: ET 3940293
• NetBackup 8.0: ET 3924973
• NetBackup 8.1: ET 3939266

However, no Emergency Engineering Binary (EEB) is needed starting in NetBackup version 8.1.1.

Please raise a support case and the representative will verify if the EEB is appropriate.

Note: The registry entry is still required for NetBackup 8.1.1 and above.

Either upgrading the NetBackup Client version to 8.1.1 or applying the associated available EEBs will allow NetBackup to perform an alternate method of the Active Directory Granular Backup. This alternative method, triggered by a manually created registry key 'ADLocation' will redirect an offline copy of the Active Directory database files from their default temporary location of an NFS mount, to a local temporary directory as defined in the ADLocation registry key. To ensure a successful backup, the client will need enough free disk space to hold a temporary copy of the Active Directory database files.

Consideration and configuration of the path to be specified in the ADLocation registry key

1. Determine how much space is needed by calculating the size of the Active Directory data set (default location within C:\Windows\ntds).
2. Identify a volume with enough space to hold a copy of the Active Directory data set.
3. On the identified volume, create a path to a directory for NetBackup to use during Active Directory Granular Backups. (Example: C:\NBU_AD_temp\tempad)

There are several factors to consider when performing steps 1, 2, 3 above:

1. The volume must be locally present on the client. Not CIFS or NFS or a URL.
2. The volume must have sufficient space to hold a persistent copy of Active Directory
3. The path must not be to the exclusive root of a volume or a Mount Point (i.e. C:\, E:\, or C:\<dir>\<mount-point>)
   • The Windows OS command 'mountvol' will reveal existing mount points to avoid.
4. The path's directory structure can be several levels deep, but should be at least two levels deep. The last directory in the path will be deleted and recreated on each backup.
   • Example: C:\NBU_AD_temp\tempad
5. The last directory in the specified path should initially be empty

Note: No Active Jobs should be running on the client at the time of this configuration

Creation and configuration of the ADLocation registry entry:

1. Open the registry editor on the client (regedit)
2. Navigate into: HKLM\Software\Veritas\NetBackup\CurrentVersion\Config
3. Inside Config, create a new REG_SZ (string) value called: ADlocation
4. Give ADlocation a value of the full directory path created in Step 3 above (Example: C:\NBU_AD_temp\tempad)
5. The next backup should use the directory specified in the ADlocation registry key, and the granular processing error should no longer occur.

Additional behavioral information about how the ADLocation feature works with Active Directory Granular Backups

After the Backup Job goes active, just prior to starting the Active Directory Granular processing, the last folder in the specified ADLocation directory path, is deleted. It is with this clean-slate, that Active Directory processing then begins. The folder is created newly, and is therefore empty. Then a copy of the Active Directory database files are placed in the newly created folder. The files are then processed by NetBackup. The files are left in the directory at the end of the job. When the next Active Directory Granular backup starts, the process repeats itself.

To be clear, this alternate method is triggered by the presence of the newly introduced registry key 'ADLocation'. If the key is present, then the alternate method will be used.

Do not use multiple data streams when using the ADLocation parameter until NetBackup 10.0 or later.

When multiple backup jobs (bpbkar32.exe processes) execute simultaneously, one job's cleanup may delete the ADLocation folder while another job is using it. This restriction is resolved in NetBackup 10.0. See ET 4031733.

NOTE:

The ADlocation key is provided to allow GRT processing on the client itself needed for the backup.

If the client operating system is upgraded to Windows Server 2022 then the registry key is no longer needed and may cause "database system error" message in browsing of the image.

If so, then the registry key should be removed.

If, after removal of the key, the following messages are seen during backup:

"WRN - Cleanup of AD Folder failed, this could happen when registry key ADLocation is not specified."

These can be ignored, so long as the backup status is successful.