NetBackup™ Deduplication Guide
- Introducing the NetBackup media server deduplication option
- Quick start
- Planning your deployment
- Planning your MSDP deployment
- NetBackup naming conventions
- About MSDP deduplication nodes
- About the NetBackup deduplication destinations
- About MSDP storage capacity
- About MSDP storage and connectivity requirements
- About NetBackup media server deduplication
- About NetBackup Client Direct deduplication
- About MSDP remote office client deduplication
- About the NetBackup Deduplication Engine credentials
- About the network interface for MSDP
- About MSDP port usage
- About MSDP optimized synthetic backups
- About MSDP and SAN Client
- About MSDP optimized duplication and replication
- About MSDP performance
- About MSDP stream handlers
- MSDP deployment best practices
- Use fully qualified domain names
- About scaling MSDP
- Send initial full backups to the storage server
- Increase the number of MSDP jobs gradually
- Introduce MSDP load balancing servers gradually
- Implement MSDP client deduplication gradually
- Use MSDP compression and encryption
- About the optimal number of backup streams for MSDP
- About storage unit groups for MSDP
- About protecting the MSDP data
- Save the MSDP storage server configuration
- Plan for disk write caching
- Provisioning the storage
- Licensing deduplication
- Configuring deduplication
- Configuring MSDP server-side deduplication
- Configuring MSDP client-side deduplication
- About the MSDP Deduplication Multi-Threaded Agent
- Configuring the Deduplication Multi-Threaded Agent behavior
- Configuring deduplication plug-in interaction with the Multi-Threaded Agent
- About MSDP fingerprinting
- About the MSDP fingerprint cache
- Configuring the MSDP fingerprint cache behavior
- About seeding the MSDP fingerprint cache for remote client deduplication
- Configuring MSDP fingerprint cache seeding on the client
- Configuring MSDP fingerprint cache seeding on the storage server
- Enabling 400 TB support for MSDP
- About MSDP Encryption using NetBackup KMS service
- About MSDP Encryption using external KMS server
- Configuring a storage server for a Media Server Deduplication Pool
- About disk pools for NetBackup deduplication
- Configuring a disk pool for deduplication
- Creating the data directories for 400 TB MSDP support
- Adding volumes to a 400 TB Media Server Deduplication Pool
- Configuring a Media Server Deduplication Pool storage unit
- Configuring client attributes for MSDP client-side deduplication
- Disabling MSDP client-side deduplication for a client
- About MSDP compression
- About MSDP encryption
- MSDP compression and encryption settings matrix
- Configuring encryption for MSDP backups
- Configuring encryption for MSDP optimized duplication and replication
- About the rolling data conversion mechanism for MSDP
- Modes of rolling data conversion
- MSDP encryption behavior and compatibilities
- Configuring optimized synthetic backups for MSDP
- About a separate network path for MSDP duplication and replication
- Configuring a separate network path for MSDP duplication and replication
- About MSDP optimized duplication within the same domain
- Configuring MSDP optimized duplication within the same NetBackup domain
- About MSDP replication to a different domain
- Configuring MSDP replication to a different NetBackup domain
- About NetBackup Auto Image Replication
- About trusted primary servers for Auto Image Replication
- About the certificate to be used for adding a trusted master server
- Adding a trusted master server using a NetBackup CA-signed (host ID-based) certificate
- Adding a trusted primary server using external CA-signed certificate
- Removing a trusted primary server
- Enabling NetBackup clustered primary server inter-node authentication
- Configuring NetBackup CA and NetBackup host ID-based certificate for secure communication between the source and the target MSDP storage servers
- Configuring external CA for secure communication between the source MSDP storage server and the target MSDP storage server
- Configuring a target for MSDP replication to a remote domain
- About configuring MSDP optimized duplication and replication bandwidth
- About performance tuning of optimized duplication and replication for MSDP cloud
- About storage lifecycle policies
- About the storage lifecycle policies required for Auto Image Replication
- Creating a storage lifecycle policy
- About MSDP backup policy configuration
- Creating a backup policy
- Resilient Network properties
- Specifying resilient connections
- Adding an MSDP load balancing server
- About variable-length deduplication on NetBackup clients
- About the MSDP pd.conf configuration file
- Editing the MSDP pd.conf file
- About the MSDP contentrouter.cfg file
- About saving the MSDP storage server configuration
- Saving the MSDP storage server configuration
- Editing an MSDP storage server configuration file
- Setting the MSDP storage server configuration
- About the MSDP host configuration file
- Deleting an MSDP host configuration file
- Resetting the MSDP registry
- About protecting the MSDP catalog
- Changing the MSDP shadow catalog path
- Changing the MSDP shadow catalog schedule
- Changing the number of MSDP catalog shadow copies
- Configuring an MSDP catalog backup
- Updating an MSDP catalog backup policy
- About MSDP FIPS compliance
- Configuring the NetBackup client-side deduplication to support multiple interfaces of MSDP
- About MSDP multi-domain support
- About MSDP application user support
- About MSDP mutli-domain VLAN Support
- About NetBackup WORM storage support for immutable and indelible data
- MSDP cloud support
- About MSDP cloud support
- Create a Media Server Deduplication Pool (MSDP) storage server in the NetBackup web UI
- Creating a cloud storage unit
- Updating cloud credentials for a cloud LSU
- Updating encryption configurations for a cloud LSU
- Deleting a cloud LSU
- Backup data to cloud by using cloud LSU
- Duplicate data cloud by using cloud LSU
- Configuring AIR to use cloud LSU
- About backward compatibility support
- About the configuration items in cloud.json, contentrouter.cfg, and spa.cfg
- About the tool updates for cloud support
- About the disaster recovery for cloud LSU
- About Image Sharing using MSDP cloud
- About restore from a backup in Microsoft Azure Archive
- About MSDP cloud immutable (WORM) storage support
- Monitoring deduplication activity
- Monitoring the MSDP deduplication and compression rates
- Viewing MSDP job details
- About MSDP storage capacity and usage reporting
- About MSDP container files
- Viewing storage usage within MSDP container files
- Viewing MSDP disk reports
- About monitoring MSDP processes
- Reporting on Auto Image Replication jobs
- Managing deduplication
- Managing MSDP servers
- Viewing MSDP storage servers
- Determining the MSDP storage server state
- Viewing MSDP storage server attributes
- Setting MSDP storage server attributes
- Changing MSDP storage server properties
- Clearing MSDP storage server attributes
- About changing the MSDP storage server name or storage path
- Changing the MSDP storage server name or storage path
- Removing an MSDP load balancing server
- Deleting an MSDP storage server
- Deleting the MSDP storage server configuration
- Managing NetBackup Deduplication Engine credentials
- Managing Media Server Deduplication Pools
- Viewing Media Server Deduplication Pools
- Determining the Media Server Deduplication Pool state
- Changing OpenStorage disk pool state
- Viewing Media Server Deduplication Pool attributes
- Setting a Media Server Deduplication Pool attribute
- Changing a Media Server Deduplication Pool properties
- Clearing a Media Server Deduplication Pool attribute
- Determining the MSDP disk volume state
- Changing the MSDP disk volume state
- Inventorying a NetBackup disk pool
- Deleting a Media Server Deduplication Pool
- Deleting backup images
- About MSDP queue processing
- Processing the MSDP transaction queue manually
- About MSDP data integrity checking
- Configuring MSDP data integrity checking behavior
- About managing MSDP storage read performance
- About MSDP storage rebasing
- About the MSDP data removal process
- Resizing the MSDP storage partition
- How MSDP restores work
- Configuring MSDP restores directly to a client
- About restoring files at a remote site
- About restoring from a backup at a target master domain
- Specifying the restore server
- Managing MSDP servers
- Recovering MSDP
- Replacing MSDP hosts
- Uninstalling MSDP
- Deduplication architecture
- Configuring and using universal shares
- About Universal Shares
- Configuring and using an MSDP build-your-own (BYO) server for Universal Shares
- MSDP build-your-own (BYO) server prerequisites and hardware requirements to configure Universal Shares
- Configuring Universal Share user authentication
- Mounting a Universal Share created from the NetBackup web UI
- Creating a Protection Point for a Universal Share
- Using the ingest mode
- Changing the number of vpfsd instances
- Upgrading to NetBackup 10.0
- Troubleshooting
- About unified logging
- About legacy logging
- NetBackup MSDP log files
- Troubleshooting MSDP installation issues
- Troubleshooting MSDP configuration issues
- Troubleshooting MSDP operational issues
- Verify that the MSDP server has sufficient memory
- MSDP backup or duplication job fails
- MSDP client deduplication fails
- MSDP volume state changes to DOWN when volume is unmounted
- MSDP errors, delayed response, hangs
- Cannot delete an MSDP disk pool
- MSDP media open error (83)
- MSDP media write error (84)
- MSDP no images successfully processed (191)
- MSDP storage full conditions
- Troubleshooting MSDP catalog backup
- Storage Platform Web Service (spws) does not start
- Disk volume API or command line option does not work
- Viewing MSDP disk errors and events
- MSDP event codes and messages
- Unable to obtain the administrator password to use an AWS EC2 instance that has a Windows OS
- Trouble shooting multi-domain issues
- Appendix A. Migrating to MSDP storage
- Appendix B. Migrating from Cloud Catalyst to MSDP direct cloud tiering
- About migration from Cloud Catalyst to MSDP direct cloud tiering
- About Cloud Catalyst migration strategies
- About direct migration from Cloud Catalyst to MSDP direct cloud tiering
- About postmigration configuration and cleanup
- About the Cloud Catalyst migration -dryrun option
- About Cloud Catalyst migration cacontrol options
- Reverting back to Cloud Catalyst from a successful migration
- Reverting back to Cloud Catalyst from a failed migration
- Appendix C. Encryption Crawler
- Index
Running the migration to the new MSDP direct cloud tier server
Before you continue the process of installing and configuring the new MSDP direct cloud tier server, it is recommended that you set up logging. If any issues arise during installation, the logs help with diagnosing any potential errors during migration. The following items are recommended:
Ensure that the
/usr/openv/netbackup/logs/admindirectory exists before running the nbdecommission command.Set the log level to VERBOSE=5 in the
bp.conffile.Set loglevel=3 in
/etc/pdregistry.cfgforOpenCloudStorageDaemon.Set Logging=full in the
contentrouter.cfgfile.
To run the migration, go to the command prompt on the MSDP direct cloud tier server and run:
/usr/openv/netbackup/bin/admincmd/nbdecommission -migrate_cloudcatalystNote:
This utility needs to be run in a window that does not time out or close even if it runs for several hours or more. If the migration is performed on an appliance, you need to have access to the maintenance shell and it needs to remain unlocked while the migration runs. The maintenance shell must remain enabled even if it runs for several hours or more.
Select the Cloud Catalyst storage server to migrate and enter the information as prompted by the nbdecommission utility.
The following is an example of what you may see during the migration:
# /usr/openv/netbackup/bin/admincmd/nbdecommission -migrate_cloudcatalyst
MSDP storage server to use for migrated CloudCatalyst: myserver.test.com
Generating list of configured CloudCatalyst storage servers.
This may take a few minutes for some environments, please wait.
Cloud Storage Server Cloud Bucket CloudCatalyst Server Storage Server Type
1) amazon.com my-bucket myserver.test.com PureDisk_amazon_rawd
Enter line number of CloudCatalyst server to migrate: 1
MSDP KMS encryption is enabled for amazon.com.
Please confirm that CloudCatalyst was configured using
KMSKeyGroupName amazon.com:testkey
Continue? (y/n) [n]: y
Enter new disk volume name for migrated CloudCatalyst server: newdv
Enter new disk pool name for migrated CloudCatalyst server: newdp
Enter cloud account username or access key: AAAABBBBBCCCCCDDDDD
Enter cloud account password or
secret access key: aaaabbbbccccddddeeeeffffggg
You want to migrate amazon.com (bucket my-bucket) to
newmsdpserver.test.com (volume newdv, pool newdp).
Is that correct? (y/n) [n]: y
To fully decommission myserver.test.com after
CloudCatalyst migration is complete,run the
following command on the master server:
/usr/openv/netbackup/bin/admincmd/nbdecommission
-oldserver myserver.test.com
Administrative Pause set for machine myserver.test.com
Migrating CloudCatalyst will include moving the images to server
newmsdpserver.test.com deleting the old disk pool, storage unit, and
storage server, deactivating policies that reference the old storage
unit, and restarting MSDP on server newmsdpserver.test.com.
Before proceeding further, please make sure that no jobs are running on
media server myserver.test.com or media server newmsdpserver.test.com.
This command may not be able to migrate CloudCatalyst
with active jobs on either of those servers.
To avoid potential data loss caused by conflicts between the
old CloudCatalyst server and the migrated MSDP server, stop the
NetBackup services on myserver.test.com if they are running.
It is recommended to make one or both of the following changes
on myserver.test.com to prevent future data loss caused by
inadvertently starting NetBackup services.
1) Rename /usr/openv/esfs/bin/vxesfsd to /usr/openv/esfs/bin/vxesfsd.off
2) Change "ReadOnly" to "1" in the esfs.json configuration file
See the documentation for more information about esfs.json.
It is also recommended to perform a catalog cleanup and backup prior
to migration so that the catalog can be restored to its original
state in the event that migration is not completed.
Continue? (y/n) [n]: y
Successfully cloned storage server: amazon.com to:
newmsdpserver.test.com_newdv
Storage server newmsdpserver.test.com has been successfully updated
The next step is to list the objects in the cloud and migrate
the MSDP catalog. The duration of this step depends on how much data
was uploaded by CloudCatalyst.
It may take several hours or longer, so please be patient.
You may reduce the duration by not migrating the
CloudCatalyst image sharing information if you are certain that
you do not use the image sharing feature.
Do you wish to skip migrating CloudCatalyst image
sharing information? (y/n) [n]:
Jun 24 15:37:11 List CloudCatalyst objects in cloud
Jun 24 15:37:13 List CloudCatalyst objects in cloud
Jun 24 15:37:18 List CloudCatalyst objects in cloud
Jun 24 15:37:26 MSDP catalog migrated successfully from CloudCatalyst
Disk pool newdp has been successfully created with 1 volumes
Moved CloudCatalyst images from myserver.test.com to newmsdpserver.test.com
Disk pool awsdp (PureDisk_amazon_rawd) is referenced by the following
storage units:
awsdp-stu
Storage unit awsdp-stu: host myserver.test.com
Deactivating policies using storage unit awsdp-stu
Storage unit awsdp-stu is referenced by policy testaws
Deactivated policy testaws
Deleting storage unit awsdp-stu on host _STU_NO_DEV_HOST_
Deleted storage unit awsdp-stu
Deleted PureDisk_amazon_rawd disk pool awsdp
Deleted PureDisk_amazon_rawd storage server amazon.com
Stopping ocsd and spoold and spad
Checking for PureDisk ContentRouter
spoold (pid 55723) is running...
Checking for PDDE Mini SPA [ OK ]
spad (pid 55283) is running...
Checking for Open Cloud Storage Daemon [ OK ]
ocsd (pid 55150) is running...
Stopping PureDisk Services
ocsd is stopped
Run MSDP utility to prepare for online checking.
This may take some time, please wait.
Starting ocsd and spoold and spad
Checking for Open Cloud Storage Daemon
ocsd is stopped
Starting Open Cloud Storage Daemon: ocsd Checking for PDDE Mini SPA
spad is stopped
spad (pid 56856) is running... [ OK ]
Checking for PureDisk ContentRouter
spoold is stopped
spoold (pid 57013) is running...spoold [ OK ]
Starting PureDisk Services
spoold (pid 57013) is running...
Enabling data integrity check.
Starting data integrity check.
Waiting for data integrity check to finish.
Processing the queue.
CloudCatalyst server myserver.test.com has been successfully
migrated to newmsdpserver.test.com.
To avoid potential data loss caused by conflicts between the
old CloudCatalyst server and the
migrated MSDP server, stop the NetBackup daemons (or services)
on myserver.test.com if they are running.
Monitor the output of the nbdecommission command for errors. Other logs to monitor for activity and potential errors are in the storage_path/log/ directory. You should monitor the ocsd_storage log and monitor the spad and spoold logs for any cacontrol command issues.
If an error is encountered and you can correct the error, you can resume the migration from that point using the start_with option as noted in the output from the nbdecommission command. If you have any questions about the error, contact Veritas Support before you resume the migration.
During the migration, there are several prompts that are displayed when the migration is run. You can use command line options to supply answers to these prompts, if necessary. Veritas recommends that you use the interactive prompts because it makes the migration easier to use and less error prone than using the command line options. If you choose to use the command line, the options are documented in the NetBackup Commands Reference Guide.
During the migration process most of the prompts are self-explanatory and the number and type of prompts can change. The number and type of prompts depends on the following:
The version of Cloud Catalyst being used at time of the migration.
If the Cloud Catalyst server is running at the time of the migration.
If KMS is enabled on the Cloud Catalyst server.
Table: Migration prompts discusses additional information about a few of the prompts.
Table: Migration prompts
Prompts | Description |
|---|---|
No MSDP storage server found on myserver.test.com. Please create the MSDP storage server before running this utility. | This output is displayed when the nbdecommission -migrate_cloudcatalyst command is run on a media server that does not have an MSDP storage server configured. |
Disk pools exist for storage server PureDisk myserver.test.com. CloudCatalyst migration requires a new storage server with no configured disk pools. | The sample output is displayed when the nbdecommission -migrate_cloudcatalyst command is run on a media server that does have an MSDP storage server configured and does have existing disk pools configured. Cloud Catalyst migration can only be run on a new MSDP cloud tier server with no existing disk pools. |
Enter cloud bucket name: | If the Cloud Catalyst server is not running at the time of migration you need to manually enter the existing Cloud Catalyst bucket or container name. This information is used for migration. |
Enter CloudCatalyst server hostname: | If the Cloud Catalyst server is not running at the time of migration you need to manually enter the server hostname of the existing Cloud Catalyst server to be migrated. |
Is MSDP KMS encryption enabled for amazon.com? (y/n) [n]: | If the Cloud Catalyst server is not running at the time of migration you may need to manually enter the KMS configuration settings for the existing Cloud Catalyst server. |
Enter new disk volume name for migrated CloudCatalyst server: | Enter the name of the MSDP Cloud disk volume to be created on the new MSDP cloud tier server. This name is used for the migrated Cloud Catalyst data. |
Enter new disk pool name for migrated CloudCatalyst server: | Enter the name of the MSDP Cloud disk pool to be created on the new MSDP server and used for the migrated Cloud Catalyst data. |
Enter cloud account username or access key: Enter cloud account password or secret access key: | Enter the credentials for the cloud account that is used to access the Cloud Catalyst data to be migrated. If you use AWS IAM role to access the data, you should enter |