Auto Image Replication (AIR) will fail in targeted AIR mode if using load balancing media servers and those servers cannot resolve the hostname of the target storage server

Problem

When using Auto Image Replication (AIR) in targeted AIR mode to replicate data to another NetBackup domain, along with load balancing media servers will fail if those load balancing media servers do not have the ability to resolve the host name of the target storage server.
 

Example environment:

• Source NetBackup Domain
  Master Server: nbmaster01-source
  Media Server: nbmedia01-source
  MSDP Storage/Media Server: MSDPsts01-source
• Target NetBackup Domain
  Master Server: nbmaster02-target
  MSDP Storage/Media Server: MSDPsts02-target

Due to Network restrictions the following communication paths exist.
Both Master Servers can communicate with each other.
The Source Media Servers and MSDP Storage Server can communicate with each other.
Only the Source MSDP Storage Server is the only server with network access to the target MSDP Storage Server.

In this scenario, the following combinations of AIR will succeed or fail...

1. Backup and replication with targeted AIR OFF and load balancing ON: SUCCEEDS
2. Backup and replication with targeted AIR ON and load balancing ON: FAILS
3. Backup and replication with targeted AIR ON but load balancing OFF: SUCCEEDS

Error Message

The replication job will fail with internal job error code of 174 and status code 191, plus showing related interface to storage server invalid error messages.

15:41:46 - granted resource  MediaID=@aaaak;DiskVolume=PureDiskVolume;DiskPool=DP-MSDPsts01-source;Path=PureDiskVolume;StorageServer=MSDPsts01-source;MediaServer=nbmedia01-source
15:41:47 - Info nbmedia01-source (pid=138207) StorageServer=PureDisk:MSDPsts01-source; Report=PDDO Stats for (MSDPsts01-source): scanned: 4 KB, CR sent: 1 KB, CR sent over FC: 0 KB, dedup: 75.0%, cache disabled
15:41:47 - Info nbmedia01-source (pid=138207) Using OpenStorage to replicate backup id image_1493127303, media id @aaaak, storage server MSDPsts01-source, disk volume PureDiskVolume
15:41:47 - Info nbmedia01-source (pid=138207) Replicating images to target storage server MSDPsts02-target, disk volume PureDiskVolume
15:41:47 - Critical bpdm (pid=138207) sts_get_server_prop_byname failed: error 2060024 interface to storage server invalid
15:41:47 - Error nbreplicate (pid=138200) ReplicationJob::Replicate: Replication failed for backup id image_1493127303: media manager - system error occurred (174)
15:41:47 - Replicate failed for backup id image_1493127303 with status 174
no images were successfully processed  (191)

The bpdm log, with pdplugin logging turned up to level 10 will show these errors relating to not being able to resolve the target storage server:

[138207] <16> 386344:bptm:138207:nbmaster02-target: [ERROR] PDSTS: pd_test_for_get_server_prop_byname: Unable to resolve MSDPsts02-target : (2060024:interface to storage server invalid)
[138207] <16> 386344:bptm:138207:nbmaster02-target: [ERROR] PDSTS: impl_get_server_prop_byname: pd_test_for_get_server_prop_byname(PureDisk:MSDPsts02-target) failed (2060024:interface to storage server invalid)
[138207] <2> 386344:bptm:138207:nbmaster02-target: [DEBUG] PDSTS: impl_get_server_prop_byname: exit: (2060024:interface to storage server invalid)
[138207] <16> 386344:bptm:138207:nbmaster02-target: [ERROR] PDSTS: pi_get_server_prop_byname_v8: impl_get_server_prop_byname(PureDisk:MSDPsts02-target) failed (2060024:interface to storage server invalid)
[138207] <2> 386344:bptm:138207:nbmaster02-target: [DEBUG] PDSTS: pi_get_server_prop_byname_v8: exit: (2060024:interface to storage server invalid)
[138207] <2> set_job_details: Tfile (386344): LOG 1493127707 32 bpdm 138207 sts_get_server_prop_byname failed: error 2060024 interface to storage server invalid
[138207] <2> send_job_file: job ID 386344, ftype = 3 msg len = 115, msg = LOG 1493127707 32 bpdm 138207 sts_get_server_prop_byname failed: error 2060024 interface to storage server invalid
[138207] <32> bp_sts_open_proxy_target: sts_get_server_prop_byname failed: error 2060024
[138207] <16> rpl_build_target_svr_list: failure to open replication target server MSDPsts02-target: plug-in reports error 2060024

Cause

When using AIR replication the data transfer of the stored backup data is conducted between the Source and Target MSDP Storage Servers, therefore a communication path needs to exists between these servers.

AIR in non-targeted mode:

The Source Media Servers only need to communicate with the Source MSDP Storage Server, which in turn then communicates with the Target Storage Server.

AIR in targeted mode:

The Media Server being used for the replication will attempt to resolve the Target MSDP Storage Server name/IP address before continuing in the same data transfer path as non-targeted AIR. If the Media Server being used for replication fails when doing a host name lookup for the Target Storage Server the replication will fail. This is a limitation of the internal OST-plugin code that is used for communication between the NetBackup layer and the Storage Server at the present version.

This issue will only present itself in certain conditions where the customer is using a Load Balancing Media Server that has no host name resolution to the Target Storage Server AND using targeted AIR. If using non-targeted AIR or the Media Server can resolve the Target Storage Server, this issue will not be present.

Solution

This issue is recognized as a limitation and is being addressed by our Engineering and product management teams for this behavior to be changed in a future release.

Until this time the issue can be resolved by adding a "dummy" host name and IP address in the host file, or DNS server, so that the host name resolution phase passes and the replication job can continue.

This will work as we do not have to communicate with the Target Storage Server from a Load Balancing Media Server, we just need to be able to resolve the host name and IP address.