Active Directory GRT restore failing with status code 2808 and 5

Problem

Active Directory GRT restore failing with status code 2808 and 5.

Error Message

Snippet of Activity Monitor > Job Details:

MS-Windows policy restore error(2808)
10/8/2013 7:40:05 PM - Info tar32(pid=0) done. status: 5: the restore failed to recover the requested files 
10/8/2013 7:40:05 PM - Error bpbrm(pid=4908) client restore EXIT STATUS 5: the restore failed to recover the requested files

Snippet of ncfgre log:

10/8/2013 19:39:58.920 [[fsys\adgran]       ] <FROM BEDS>ADGran : Restoring object DC=com,DC=testdomain/DC=com/OU=HQ_Users/CN=NBU_User, property whenCreated in ADGran_WriteObj:269 (../BEDSContext.cpp:124)

10/8/2013 19:39:58.920 [[fsys\adgran]       ] <FROM BEDS>ADGran : Restoring object of size 0x16 in ADGran_WriteObj:270 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.920 [RAIConsumer::_objectOpenAndWrite()] write() completed, status = 0 (../RAIConsumer.cpp:1000)
10/8/2013 19:39:58.920 [RAIConsumer::_objectOpenAndWrite()] id: System?State\Microsoft Active Directory\Active Directory\DC=com\OU=HQ_Users\CN=NBU_User, bytes written: 6023 (../RAIConsumer.cpp:1008)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:Failed to find the Deleted Object [CN=NBU_User] in [OU=HQ_Users,DC=com,DC=testdomain] at W:304 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:setting [cn] for add at L:598 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:skipping [CN=NBU_User][nTSecurityDescriptor] reason-[2] (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:skipping [CN=NBU_User][objectCategory] object is category 1 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:setting [sAMAccountName] for add at L:598 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:setting [objectClass] for add at L:598 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:setting [instanceType] for add at L:598 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:executing an add on CN=NBU_User,OU=HQ_Users,DC=com,DC=testdomain at L:236 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:error an object with the same name [CN=NBU_User,OU=HQ_Users,DC=com,DC=testdomain] already exists at L:371 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.967 [[fsys\adgran]       ] <FROM BEDS>    ADProv:Error 800700b7 committing mandatory properties during create [CN=NBU_User] W:215 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.983 [[fsys\adgran]       ] <FROM BEDS>ADGran: Status FS_EXTENDED_ERROR (0xE000FEA9) Could not restore or create object, DC=com,DC=testdomain/DC=com/OU=HQ_Users/CN=NBU_User, in ADGran_CloseObj:223 (../BEDSContext.cpp:124)
10/8/2013 19:39:58.983 [Object::close()] FS_CloseObj() Failure! Extended Error unknown (-536836984) (../Object.cpp:202)
10/8/2013 19:39:58.983 [RAIConsumer::writeToRai(CFEObj,Root)] create and write succeeded (../RAIConsumer.cpp:1193)
10/8/2013 19:39:58.983 [Info] V-158-14 \System State\Active Directory\Active Directory\DC=com\OU=HQ_Users\CN=NBU_User was restored

TIPS:
Before running the restore, run the following commands on the destination client to increase ncfgre, ncfrai, and ncf logging level, respectively:
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 352 –s DebugLevel=6
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 158 –s DebugLevel=6
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 309 –s DebugLevel=6

Run the restore again, and run this command to obtain the ncfgre log:
<install dir>\veritas\netbackup\bin\vxlogview -p 51216 -i 352 -t 00:30:00 > c:\ncfgre.log

Then return the logging level back to default (1):
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 352 -s DebugLevel=1
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 158 –s DebugLevel=1
<install dir>\veritas\netbackup\bin\vxlogcfg -a –p 51216 –o 309 –s DebugLevel=1

Cause

Consider the following scenario of event sequences:
Event Day 1:

Domain Admin created a new user called NBU_User inside HQ_Users organizational unit (OU). The distinguished name looked like this: CN=NBU_User,OU=HQ_Users,DC=com,DC=testdomain

Event Day 2:

A backup of AD GRT backup was created.

Event Day 3:

Domain Admin moved NBU_User from HQ_Users to HQ_Administrators OU.

Event Day 4:

User NBU_User was deleted from Active Directory by Domain Admin.

Event Day 5:

User NBU_User is now needed to be restored.

NetBackup Admin performed AD GRT Restore.  The user object of NBU_User was restored and was created again, but attributes of the user were not created.

Prior to the restore, the user object could be found in the Deleted Objects container (assuming it has not gone past the tombstone lifetime), which could be verified by using Windows built-in tool ldp.exe. This tool also showed some of the user's attributes, including lastKnownParent.  In this case, the lastKnownParent attribute is still pointed to: OU=HQ_Administrators,DC=com,DC=testdomain.

For more information on using ldp.exe to locate the Deleted Objects, see: http://technet.microsoft.com/en-us/magazine/2007.09.tombstones.aspx

Solution

Move the partially restored user object to the original OU (OU=HQ_Users,DC=com,DC=testdomain) and run the restore again.