Incremental Backing up System State on a Windows 2016 client can intermittently produce Exit Status 1.

Problem

Incremental Backing up System State on a Windows client can intermittently produce Exit Status 1. 

In a Full backup, the job may continue to complete with a Status 0. 

Error Message

Job Details shows "WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error)" followed by "status: 1: the requested operation was partially successful".

"Status 0x00000002" indicates  "File NOT Found"

08:01:39.200 [6668.6596] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\amd64_installed in brUtil::GenerateFileList
.
08:01:51.888 [6668.6596] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\servicing\version\10.0.14393.0\x86_installed in brUtil::GenerateFileList
.
08:01:54.966 [6668.6596] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 2 files missing on the snapshot
08:01:54.966 [6668.6596] <2> tar_base::V_vTarMsgW: ERR - Error encountered while attempting to get additional files for System State:\

See attached log snip from ...\netbackup\logs\bpbkar, and Verbose 5, General Log Level 2.

Cause

This issue is caused when the operating system is updated by Windows Update, which removes two files but does not properly un-register them from the VSS System Writer.

Microsoft has published the following article on this issue:
Third-party backup warnings after you install a servicing update in Windows Server 2016

Example: Windows Update changes the version of these two files:
C:\windows\servicing\version\10.0.14393.0\ amd64_installed
C:\windows\servicing\version\10.0.14393.0\ x86_installed

The way Microsoft updates the versions of these two files:  the file names remain the same, but the directory in which they reside is modified to reflect the new version number:

From:
C:\Windows\servicing\Version\ 10.0.14393.0

To:
C:\Windows\servicing\Version\ 10.0.14393.1051

NOTE:  The new version number folder may continue to change from the one shown above, with future updates.

It removes the two files from the original folder, C:\Windows\servicing\Version\ 10.0.14393.0, and places new versions into C:\Windows\servicing\Version\ 10.0.14393.1051.

Windows continues to register the old (now empty) path within VSS System Writer.

The consequence of the old path continuing to be registered to the VSS System Writer is that when NetBackup initiates a System State backup, the Microsoft System Writer tells NetBackup to back up those files in that path even though they are not there. In a Full backup, the job will continue to complete with a Status 0.

During an Incremental backup job, if those two files happen to be the only two files supposed to be in the snapshot, when those files are absent, the image will complete with Status 1 and post the following errors:

• "Component Set System Files has XX files missing on the snapshot"
• "E_SNAPSHOT_ALL_COMPONENT_FILES_MISSING_ON_SNAPSHOT"

You may find files other than the above two return Status 0x00000002. The file paths which return Status 0x00000002 can vary depending on the platform environment.

Solution

Verbose bpbkar log is needed to verify the error messages. If the error messages are found for any file, apply the Microsoft solution below. As mentioned above, this condition occurs with any file with an OS mismatch, and the same step needs to be performed for each file.

Creating verbose bpbkar log:

1. Go to the Client server and navigate to <Install_Path>\Veritas\Netbackup\logs\
2. Create a folder with the name "bpbkar"
3. In the Master GUI navigate to Host Properties -> Clients -> choose the client server name
            Logging -> Change Global Logging to 5
            Windows Client -> Client Settings -> Change General level to 2
4. Collect the backup job "Detailed Status" text from the master Activity monitor and then review the error messages in the client bpbkar log.

To disable logs select "Change to Minimum Logging" after the log review has been completed.

According to Microsoft, this issue can be resolved by creating new placeholder files with the same name for the files that are reported as not found.

1. Create the missing files (empty zero byte files) using 'echo' : 

                 echo > C:\Windows\servicing\Version\10.0.14393.0\amd64_installed
          echo > C:\Windows\servicing\Version\10.0.14393.0\x86_installed

NOTE: After multiple Windows Updates, there may be multiple empty folders that need correction.  Review the NetBackup bpbkar log for details.

OR

2. Copy the problem files amd64_installed and x86_installed from their new paths, into the problematic path(s): "C:\Windows\servicing\Version\10.0.14393.0".  Having a copy of these two files in both locations (the current path and the original path) will allow the VSS snapshot to find the files in the paths it expects to find them in.

Notes: 

• If you have permission errors updating the folders/files, it may be necessary to assume ownership of the subfolders within the folder into which the files need to be created/copied.
• Applying the latest Windows updates may also resolve the problem. If the system is fully patched, please follow one of the Microsoft workarounds above.