nbdeployutil - Overlap detection feature for NetBackup versions 7.7.2, 7.7.3, 8.0, 8.1, and 8.1.1

Description

The new version of nbdeployutil introduces the Overlap Detection feature. This feature is applicable only for Capacity Licensing. With this feature, nbdeployutil detects if the same data is backed up by multiple policies of the same policy type and deducts the charged size for overlapping backup selections. nbdeployutil also supports the obfuscation feature that provides a command line option to anonymize the master server name and the client name in the report.

This article has the following sections:

• Supported NetBackup versions and agents
• Version-specific support
• Installing the Emergency Engineering Binary
• How the Overlap Detection feature works
• How to use nbdeployutil
• Examples of the nbdeployutil report
• Additional options for nbdeployutil that are available with the Overlap Detection feature
• Known limitations and workarounds
• Updates to the nbdeployutil EEBs

Supported NetBackup versions and agents

Feature	Supported NetBackup versions	Supported NetBackup agents
Overlap Detection	7.7.2 7.7.3 8.0 8.1 8.1.1	File System (Standard and Microsoft Windows) Microsoft SQL Server NDMP VMware Microsoft Exchange Server

Version-specific support

NetBackup master server	NetBackup agents, policy types, or workloads
	Require EEBs to report accurate license usage	Report accurate license usage
7.7.2	NetBackup client version: 7.7.2 FS, SQL, Exchange Server, Vmware, NDMP	-
7.7.3	NetBackup client version: 7.7.2 & 7.7.3 FS, SQL, Exchange Server, Vmware, NDMP	-
8	NetBackup client version: 7.7.2 , 7.7.3, & 8.0 FS, SQL, Exchange Server, Vmware, NDMP	-
8.1	NetBackup client version: 7.7.2 , 7.7.3, & 8.0 FS, SQL, Exchange Server, Vmware, NDMP	NetBackup client version: 8.1 FS, SQL, Exchange Server, Oracle, Vmware, NDMP, BigData
8.1.1	NetBackup client version: 7.7.2 , 7.7.3, & 8.0 FS, SQL, Exchange Server, Vmware, NDMP	NetBackup client version: 8.1 & 8.1.1 FS, SQL, Exchange Server, Oracle, Vmware, NDMP, BigData

The Overlap Detection feature includes the Incremental Reporting feature of nbdeployutil. To know more about the Incremental Reporting feature, refer to the following article:  https://www.veritas.com/support/en_US/doc/24437881-129889739-0/v120885038-129889739.
 

Installing the Emergency Engineering Binary

EEB can be downloaded from Download center https://www.veritas.com/support/en_US/downloads . Download the nbdeployutil zip file based on the version of NetBackup that you have installed.
Use the following steps to get the latest version of nbdeployutil with the Overlap Detection feature:

1. From the command prompt, navigate to the location of the downloaded nbdeployutil zip file.
2. Run the following command:
   <eebinstaller> -create
   Where <eebinstaller> is the name of the extracted file.

How the Overlap Detection feature works

When you run nbdeployutil, either manually or using the Incremental Reporting feature, overlapping backup selection data is detected and automatically adjusted in the Charged Size column for the overlapping policies in the report. This feature ensures that customers are charged optimally and are not charged for duplicate backed up data for the File System (Standard and Microsoft Windows) and Microsoft SQL Server policy types.

How to use nbdeployutil

You can use the existing command line options while running nbdeployutil.
For more information, refer to the following article:
https://www.veritas.com/support/en_US/doc/123533878-127136857-0/v123549289-127136857

Examples

The following examples give an overview of the overlap cases and display how the overlap is reported.

Example 1: Policy Type - File System Standard and Microsoft Windows

Policy 1	Policy 2
Backup Selection: C:\Data1 [Size 4GB] C:\Data2 [Size 2GB]	Backup Selection: C:\Data2 [2GB] C:\Data3 [1GB]

In this example, the C:\Data2 backup selection is common in both Policy 1 and Policy 2. nbdeployutil detects the overlap and displays in the report. An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Overlap Size (GB)	Total (Readable) Size (GB)	Charged Size (GB)
Policy 1	Ok		0	6	6
Policy 2	Ok	Overlap detected for the policy and deducted from the Charged Size	2	3	1

For Policy 1 the customer is charged for C:\Data1 and C:\Data2. For Policy 2, the customer is charged only for C:\Data3.

Example 2: Policy Type - Microsoft SQL Server

Policy 1: DB_File_Overlap1	Policy 2: DB_File_Overlap2
Backup Selection:  Instance=INST_R  DB file=DB_01 [Size 1.4GB] Instance=INST_R  DB file=DB_02 [Size 1.4GB]	Backup Selection: Instance=INST_R  DB file=DB_02 [Size 1.4GB] Instance=INST_R  DB file=DB_03 [Size 1.4GB]

In this example, the DB file DB_02 is common in both policies DB_File_Overlap1 and DB_File_Overlap2. nbdeployutil detects the overlap and displays in the report. An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Overlap Size (GB)	Total (Readable) Size (GB)	Charged Size (GB)
DB_File_Overlap1	Database Estimation, Multistream Estimation, OK	DB size estimated via backup summation, Multi-stream backup estimation	0	2.8	2.8
DB_File_Overlap2	Database Estimation, Multistream Estimation, OK	DB size estimated via backup summation, Multi-stream backup estimation, Overlap detected for the policy and deducted from the Charged Size	1.4	2.8	1.4

For policy DB_File_Overlap2 the customer is charged for only DB file DB_03.

Example 3: Policy Type - VMware

Policy 1	Client: WIN2012-HN.RSV.VEN.veritas.com VM Primary Identifier: VM BIOS UUID Backup Selection: Exclude boot disk [Size 4GB]
Policy 2	Client: WIN2012-HN.RSV.VEN.veritas.com VM Primary Identifier: VM INSTANCE UUID Backup Selection: Include all disks [Size 12GB]
Policy 3	Client: WIN2012-HN.RSV.VEN.veritas.com VM Primary Identifier: VM display name Backup Selection: Exclude boot disk [Size 4GB]
Policy 4	Client: WIN2012-HN.RSV.VEN.veritas.com VM Primary Identifier: VM DNS name Backup Selection: Exclude data disk [Size 8GB]

In this example, all the four policies are for the same client, but are made by selecting different primary identifiers. Here, Policy 2 includes both the disks so it is consuming all the other policies. Therefore, only Policy 2 is charged.
An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Overlap Size (GB)	Total (Readable Size (GB)	Charged Size (GB)
Policy 1	OK	Modified size using file/policy data (removed overlapping files)	0	4	0
Policy 2	OK		0	12	12
Policy 3	OK	Modified size using file/policy data (removed overlapping files)	0	4	0
Policy 4	OK	Modified size using file/policy data (removed overlapping files)	0	8	0

Only Policy 2 is charged as it consumes the other three policies.

Example 4: Policy Type - NDMP

Policy 1	Policy 2
Backup Selection: /NDMP_Filer/Volume1 [Size 4GB] /NDMP_Filer/Volume2 [Size 2GB]	Backup Selection: /NDMP_Filer/Volume2 [2GB] /NDMP_Filer/Volume3 [1GB]

In this example, the /NDMP_Filer/Volume2 backup selection is common in both Policy 1 and Policy 2. nbdeployutil detects the overlap and displays in the report.
An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Overlap Size (GB)	Total (Readable Size (GB)	Charged Size (GB)
Policy 1	OK		0	6	6
Policy 2	OK	Overlap detected for the policy and deducted from the Charged Size	2	3	1

For Policy 1, the charges are applicable for /NDMP_Filer/Volume1 and /NDMP_Filer/Volume2. For Policy 2, the charges are applicable only for /NDMP_Filer/Volume3.

Example 5: Microsoft Exchange Server

Policy 1	Policy 2
Backup Selection: Microsoft Information Store:\DB1 Microsoft Information Store:\DB2	Backup Selection: Microsoft Information Store:\DB1 Microsoft Information Store:\TestDB

In this example, the backup selection Microsoft Information Store:\DB1 is common in both Policy 1 and Policy 2. nbdeployutil detects the overlap and displays in the report.
An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Overlap Size (KB)	Duplicate Selection	Total (KB)	Charged Size (KB)
Policy 1	Database Estimation, OK	DB size estimated via backup summation, Overlap detected for the policy and deducted from the Charged Size.	8260	/Microsoft Information Store/DB1/Database	39309	31049
Policy 2	Database Estimation, OK	DB size estimated via backup summation	0		41362	41362

For Policy 1, the charges are applicable only for Microsoft Information Store:\DB2. For Policy 2, the charges are applicable for both Microsoft Information Store:\DB1 and Microsoft Information Store:\TestDB.

Example 6: Microsoft Exchange Server (Consumption scenario)

Policy 1	Backup Selection: Microsoft Information Store:\DB1 Microsoft Information Store:\DB2
Policy 2	Backup Selection: Microsoft Information Store:\DB1 Microsoft Information Store:\TestDB
Policy 3	Backup Selection: Microsoft Information Store:\

In this example, the backup selection Microsoft Information Store:\ directive includes all the databases present in the Microsoft Exchange client. nbdeployutil detects the consumption and displays in the report.
An extract of the report specific to this example is as follows:

Policy Name	Accuracy	Accuracy Comment	Total (KB)	Charged Size (KB)
Policy 1	Database Estimation	DB size estimated via backup summation, Modified size using file/policy data (removed overlapping files)	39309	0
Policy 2	Database Estimation	DB size estimated via backup summation, Modified size using file/policy data (removed overlapping files)	41362	0
Policy 3	Database Estimation	DB size estimated via backup summation	109597	109597

For Policy1 and Policy2, the customer is not charged. For Policy3, the charges are applicable for Microsoft Information Store:\ that includes the DB1, DB2, TestDB and other available databases in that client.

Additional options for nbdeployutil that are available with the Overlap Detection feature

CLI option	Option for Incremental Reporting feature	Description
--overlap-details	In the nbdeployutilconfig.txt file, add SHOW_OVERLAP_DETAILS=1 Path of file :  Windows: Install_Dir\NetBackup\var\global\nbdeployutilconfig.txt UNIX: /usr/openv/var/global/nbdeployutilconfig.txt	Use this option while generating the capacity licensing report to display duplicate backup selections in the Duplicate Selections column of the report. Note: Use this option only for ASCII or English-only characters in the backup selection data. The following disclaimer is displayed when you run this option: The overlap-details flag creates the Duplicate Selection column with data, which could be potentially sensitive.
--disable-overlap	In the nbdeployutilconfig.txt file, add  DISABLE_OVERLAP_DETECTION=1  Path of file :  Windows: Install_Dir\NetBackup\var\global\nbdeployutilconfig.txt UNIX: /usr/openv/var/global/nbdeployutilconfig.txt	Use this option along with the existing gather and report options to disable the Overlap Detection feature.
--anonymize		Use this option to anonymize the master server name and client names in the report. By default, the master server names are replaced with MASTER_{X} and client names are replaced with HOST_{X}, where X is a numeric value.
--anonymize -anon-master=SOMENAME		Use this option to anonymize the master server name and client names in the report. The master server name is replaced with SOMENAME and the client name is replaced with HOST_{X}, where X is a numeric value.

Information that is displayed for the anonymize option in the report

For the following command:
nbdeployutil.exe --gather --capacity --report

The report is displayed as:

Master Server	Client Name	Policy Name	Policy Type
win-b2qj6keaalf	5035be4e-1ec8-f99c-34fd-e2c16110e96e	Policy 1	VMware

For the following command:
nbdeployutil.exe --gather --capacity --report --anonymize

The report is displayed as:

Master Server	Client Name	Policy Name	Policy Type
Master_0	Host_1	Policy 1	VMware

For the following command:
nbdeployutil.exe --gather --capacity --report --anonymize -anon-master=SOMENAME

The report is displayed as:

Master Server	Client Name	Policy Name	Policy Type
SOMENAME_0	Host_1	Policy 1	VMware

Information that is displayed for the duplicate Microsoft SQL backup selections in the report

In the generated report for File System and Microsoft SQL policies, OK is displayed in the Accuracy column when an overlap is detected and the charged size is deducted.
The duplicate backup selection for Microsoft SQL Server has the following format:
In case of Availability Group dbName is prefixed by agname in the following format:

Known limitations and workarounds

• The feature cannot deduct overlap size in the following scenarios:
  • Backup policy contains the following directives:
    • "System-State"
    • "Shadow-Copy components"
  • Compression is enabled for backups.
• 
  The overlap size is not deducted from the charge size and Accuracy is mentioned as Overlap in the report instead of OK.

• The Overlap Detection feature gathers more information for optimally calculating the charged size and might require more disk space than the earlier version of nbdeployutil. Use the --output <LOCATION> option along with the --gather option to redirect the gather output to a different location.
• If you have enabled catalog compression, nbdeployutil decompresses the catalog and the disk space usage increases. To run this utility effectively, ensure that there is sufficient disk space available. To frequently compress the catalog, set the Compress catalog interval option to a minimal value. For example, you can set it as one day. This setting ensures that the uncompressed files are not stored for a longer duration after you run nbdeployutil. For more information, refer to https://www.veritas.com/support/en_US/doc/123711489-126541862-0/v41242926-126541862.
• The Overlap Detection feature might take significantly more time to effectively gather, calculate, deduct, and then display the charged size for the overlapping backup policies. During this process, there is no other effect on your production environment and on the tasks running on the master server. However, you can also run the report option of nbdeployutil on a different computer.
• Overlap Detection feature is not supported from OpsCenter.

Updates to the nbdeployutil EEBs

1. Detect and report data overlap for File System, SQL, VMware, NDMP and MS Exchange workloads.
2. Incrementally gather and report capacity usage information.
3. Provide the ability to anonymize host names in the output report
4. Generate a UsageData_Sanitized.out file that provides the capacity (front-end data) usage information to the NetBackup telemetry.
   Note: The following fields are obfuscated in the file: policy name, client name, master server name