Synthetic backups fail with STATUS 609 and STATUS 610, when the reading media server is on the opposite side of a firewall from the writing media server

Overview:
Synthetic backups can fail with status609 and status 610 if a firewall prevents connections between the write and readmedia servers.

Both media servers can perform successful backup andrestore operations, because these operations do not involve connections betweenmedia servers or (if one is the master server) are using the vnetd port forconnections.   Synthetic backups require connections between the mediaservers and do not use the vnetd port.

Troubleshooting:

The job detail for the synthetic backup shows the connectionfailure.

2008-09-22 11:28:27 - beginreading
2008-09-22 11:51:22 - Errorbptm(pid=8965) synth writer cannot connect to synth job onhost=<myserver1> and port=62351, error=Connection timedout
2008-09-22 12:29:58 - endsynthetic reader 1; read time:01:01:34
2008-09-22 12:29:58 - ErrorReadExtents::giveEndpointStatus(pid=8427) received end point error = cannotconnect onsocket(25)    
2008-09-2212:29:58 - Error PlanAdmin::actionDone(pid=8427) read action failed, error=endpoint terminated with anerror(610)    
2008-09-2212:29:59 - Error ImageEndpoint::bpxmKill(pid=8427) could not send remote signalto PID:8963 on<myserver2>    
2008-09-2212:29:59 - Error SynthJob::handleTerminateSynth(pid=8427) termination requestedby reader manager, reason = readerfailed(609)    
2008-09-2212:30:01 - Error bptm(pid=8959) media manager terminated by parentprocess      
2008-09-22 12:31:04 - ErrorSynthJob::endJob_i(pid=8427) end bpsynth with status = reader failed(609)

Workarounds
Thereare two workarounds for NetBackup 5.0GA - 7.0GA that will allow for a successfulsynthetic backup.  

Option 1) Move the media to a media serverinside the firewall.

Make the media required for the synthetic backupavailable to a media server inside of the firewall.  Then update theNetBackup catalog to be aware of the new location for the media by using thiscommand.

# netbackup/bin/admincmd/bpmedia -movedb -m <mediaid>-newserver <hostname>

Option 2) Modify the firewallconfiguration

Change the firewall rules to allow the reading media serverto make connections to non-reserved ports on the writing mediaserver.  

If desired, the source ports used by the reading server tomake the outbound connection can be constrained, allowing for a smaller range ofports to be open on the firewall.  Change the following configurationsettings on the reading server:

CLIENT_PORT_WINDOW =<start_of_non-reserved_range><end_of_non-reserved_range>
CONNECT_OPTIONS =<writing_server_name> 1 1 1

If desired, the destination (listening)ports on the writing server awaiting the inbound connection can be constrained,allowing for a smaller range of ports to be open on the firewall.  Theconfiguration settings to change on the writing serverare:

SERVER_PORT_WINDOW = <start_of_non-reserved_range><end_of_non-reserved_range>

Narrowing a port range limits thenumber of source or destination ports available to all NetBackup processes andmay lead to other operational failures on busy servers, especially a masterserver.  The non-reserved port range starts at 1025 and the upper limit isoperating system dependent.

Solution

Apply NetBackup release update 7.0.1 to both the readingand writing media servers, which then use the PBX port for connection duringsynthetic backups.  If both servers are not at NB_7.0.1 or higher, thelegacy port range will continue to be used and the workarounds above arerequired.