How to follow best practices when using Preferred Network entries

Description

Note:  Preferred Network entries are not needed if NetBackup is configured with appropriate host names that the operating system (O/S) resolves to the correct IP addresses and then routes correctly.

When external constraints prevent the environment from being corrected, Preferred Network entries may be useful as follows:

• Can be used to prevent NetBackup from connecting to specific remote destination addresses.
• Can be used to cause NetBackup to connect only to specific remote destination addresses.
• Can be used to request a subset of local interfaces for source binding when making outbound connections.

Note: Preferred Network does not have any effect on local, intra-host, connections.

Caution: When used for source binding, the O/S may not honor the source binding list provided by NetBackup.  If the O/S implements the weakhost model, asymmetrical network routing may result.  If asymmetrical routing occurs, the remote host may reject the inbound connection if it implements the stronghost model.  Similarly, stateful network devices may also drop asymmetrical connections.   To ensure the use of specific outbound interfaces for specific remote hosts or networks, make sure O/S name resolution and routing configurations are correct; create static host routes if needed.  Also ensure that all network drivers are properly implementing the IP and TCP networking protocols.

Note: The local Preferred Network entries do not affect the forwarding profile that the local host returns to a remote host during initial CORBA connection setup; it will contain all the local plumbed interfaces.  However, the End Point Selection algorithm within the remote process will utilize it's local Preferred Network entries to evaluate the profile when selecting the destination for the subsequent CORBA connection.

Entry Syntax:

PREFERRED_NETWORK = <target[/subnet]>  <directive>  [<source[/subnet]>]

Multiple Preferred Network entries can be specified.   During evaluation, the entries are sorted by length of target subnet.   Entries with the largest (more precise) <target> subnet are compared before entries with a shorter (less precise) subnet.   If two entries have equal <target> subnet specification, they are compared in the order configured; top to bottom.

If a subnet is not specified, the default is /128 when the address is non-zero and /0 when the address is 0.  This applies to both <target> and <source> addresses.

A subnet of /0 cannot be used with a non-zero address because it effectively negates all of the bits in the address making the <target> or <source> match every address, i.e. 0/0. 

<target[/subnet]>

A hostname or range of addresses to be compared to the prospective source or destination addresses being evaluated.

A hostname:             myserver.domain
An IP address:          10.82.105.11
A network with subnet:  10.82.105.0/21
Any IPv4 address:       0.0.0.0
Any IPv6 address:       0::0
Any address:            0/0

Note: A host or network name that cannot be resolved will cause the <target> to be ignored, but any associated <source> will still be added to the source binding list.

<directive>

Determines how <target> is compared to the source and/or destination address being evaluated.

MATCH  

Applies to destination addresses.

If the address being evaluated matches <target>, then the address is immediately selected to be used and evaluation stops.   If <target> is not matched, evaluation continues with the next entry.  

 

ONLY

Applies to destination addresses.

If the address being evaluated does not match <target>, it  will not be used and evaluation stops for this address.  If this was the only potential destination, the connection will not be attempted.  If there is an additional potential destination, it will be evaluated starting over with the first entry.

PROHIBITED

The <target> applies to both source and destination addresses.  If a <source> is specified, it will be ignored and the <target> will still be prohibited.

If the matched address is a destination address, evaluation stops.  If this was the only potential destination, the connection will not be attempted.  If there are additional potential destinations, they will be evaluated starting over with the first entry. 

If the matched address is a source address, it will be removed from the binding list.  However, if <source> entries exist the shortened binding list may not be used; see next section.

Caution: On some platforms, prohibiting a local interface may cause unexpected results when connecting to remote hosts.  Prohibiting a local interface does not effect connections that are internal to the host.

<source[/subnet]>

Optional.   A hostname or IP address range that is requested to be used as the local interface for outbound connection to addresses in <target>.

Applicable to the directives ONLY and MATCH, but not to the directive PROHIBITED.

Note: The operating system may not honor the <source> request!

Note: A <source> which has already been PROHIBITED will not be requested by NetBackup.

Note: A host or network name that cannot be resolved, or that is not local, will be ignored.  But the evaluation status of the <target> is still used.

Source Binding Evaluation: 

The prospective source binding list is provided by the operating system and consists of all of the local plumbed interfaces that are not loopback (127.0.0.0/8, ::1) and not link-local (169.254.0.0/16, fe80::/64).

The Preferred Network entries are then evaluated in the sort order noted in the section above; longest subnet first, then position when subnets are equal.  Any local interfaces that match a <source> entry are moved to a second list if still present in the first list.  Any local interfaces that match a <target> PROHIBITED entry are removed from the first list if not already moved to the second list.

If local interfaces were moved to the second list, that list becomes the tentative binding list.  If the second list is empty, and interfaces were removed from the first list, then the shortened first list becomes the tentative binding list.

If a destination matches the <target> of an entry with a <source>, the tentative binding list is searched for the first match to an entry in <source>.  If a match is found, that interface will be the source requested by NetBackup for the outbound connection.

Otherwise, if the tentative binding list is the shortened first list, then it becomes the source binding list for the outbound connection.

Otherwise 'ANY' becomes the the source binding list for the outbound connection.

Note: If the source binding list is not 'ANY', most operating systems will arbitrarily use the first interface in the list provided by the application.  Because of this behavior, it is best to not use PROHIBITED entries for local interfaces and to minimize the use of <source> entries.

Recommendations and Guidelines:

Reminder: An O/S using the weakhost model will use the O/S routing table to determine the outbound interface regardless of the source binding list provided by NetBackup.  If that interface does not match the IP address provided by NetBackup, the outbound connection will be on a network that does not match the source IP and asymmetrical network routing of the response packet will occur.

Reminder: An O/S using the stronghost model may prevent or drop connections that use a source IP address that does not have a route to the destination.

Overall it is best to not use local interfaces with Preferred Network entries and instead configure the operating system appropriately.  If utilized, the following guidelines are recommended. 

• The best use of Preferred Network is to PROHIBIT NetBackup from connecting to specific remote addresses.  This can be reliably accomplished on all platforms because it is entirely under NetBackup control.
• It can also be useful to request or MATCH a specific local interface to a specific remote network address if the OS routing table contains multiple valid routes to that remote address and NetBackup should use only one of those routes.
• It is preferable to use MATCH over PROHIBITED for local soure interfaces.  The match attempts a connection using the specified source interface and will succeed if the routing tables and end-station model on the hosts and network devices in between will allow the connection.
• Avoid using PROHIBITED for local interfaces.  This causes outbound connections to be attempted to the destination address using, in turn, each of the remaining local interfaces.  Each local interface that is non-routable to the destination will result in a delay (10 seconds per non-routable source interface per connection).  The cumulative delays may cause other timers within NetBackup to be exceeded and various failure status codes may result.  See the related articles for details .

Always review the bplocaladdrs and bptestnetconn outputs closely and then sniff the network to confirm that the O/S is behaving as expected.

Note: See the Related Articles for source binding problems and solutions in NB 7.0.1 - 8.1, examples of using static routes (which always work), the weak-host model, and the strong-host model.
 

Troubleshooting :

Use this command to observe the local interfaces provided to NetBackup by the O/S and the order in which they are provided.

$ bplocaladdrs
10.82.105.11
10.82.105.8
10.82.10.10

Use this command to observe the order in which entries are evaluated and the evaluation results.   The 'TGT' or 'SRC' will indicate if the destination is permitted and which source binding list NetBackup will provide to the O/S.   A value of ‘ANY’ indicates that the outbound interface is not constrained by NetBackup.

$ bptestnetconn -asp -v6
…snip…
FL:    mymaster -> 10.82.105.14   :   5 ms FAST (< 5 sec) TGT PROHIBITED
FL:    mymedia -> 10.81.40.61     :   6 ms FAST (< 5 sec) SRC: 10.82.10.10
…snip…
PREFERRED_NETWORK rules are applied in this order:
[0] PREFERRED_NETWORK = 10.82.105.8 PROHIBITED
[1] PREFERRED_NETWORK = 10.81.40.0/24 MATCH 10.82.10.10

$ bptestnetconn -asp -v6 -H myclient
…snip…
FL:    myclient -> 10.81.40.127   :   6 ms FAST (< 5 sec) SRC: ANY
…snip…
PREFERRED_NETWORK rules are applied in this order:
[0] PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.0/24
[1] PREFERRED_NETWORK = 10.82.105.0/29 PROHIBITED
[2] PREFERRED_NETWORK = 10.82.104.0/24 MATCH 10.82.105.5 

Basic Examples :

PREFERRED_NETWORK = 12.34.0.0/16 MATCH

Allows connectivity to the hostnames that resolve to 12.34.0.0 to 12.34.255.255.  Does not affect outbound interface selection.

PREFERRED_NETWORK = 12.34.56.78 MATCH 98.76.54.32

Allows connectivity to the hostname that resolves to 12.34.56.78, and requests that the operating system use 98.76.54.32 as the outbound interface.

PREFERRED_NETWORK = 12.34.56.78/24 ONLY

Prevents NetBackup from connecting to any destination address outside the range of 12.34.56.0 to 12.34.56.255.   Does not affect outbound interface selection.

PREFERRED_NETWORK = 12.34.56.78/24 ONLY 98.76.54.32

Prevents NetBackup from connecting to any destination address outside the range of 12.34.56.0 to 12.34.56.255.   Requests that the operating system use 98.76.54.32 as the outbound interface.  

PREFERRED_NETWORK = 12.34.56.78/16 PROHIBITED

Prevents NetBackup from using any address between 12.34.0.0 and 12.34.255.255 as the source or destination for a connection.  If it matches a local interface, NetBackup will provide an ordered list of the remaining interfaces as the source binding list for the outbound interface when other entries do not specify a <source>.  The operating system may always use the first one in the list instead of utilizing the network routing tables to select the best one.  Using PROHIBITED with local interfaces is discouraged per the details in the prior section.

Complex Examples :  Consider a NetBackup server, named bob, with these network interfaces.  

eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.82.105.11 netmask fffff800 broadcast 10.82.111.255
eri0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.82.105.8 netmask fffff800 broadcast 10.255.255.255
eri1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 10.82.10.10 netmask fffff800 broadcast 10.82.15.255

Normal outbound connectivity to these four hosts is via the first interface.  Internal connections utilize the destination interface as the source interface.

$ bptestbpcd -host billcat
10.82.105.11 :54129 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy
10.82.105.11 :54152 -> 10.82.105.14:13724

$ bptestbpcd -host beetle
10.82.105.11 :54135 -> 10.82.104.249:13724

$ bptestbpcd -host lilo   
10.82.105.11 :54139 -> 10.82.56.79:1556

$ bptestbpcd -host 10.82.105.11
10.82.105.11:54144 -> 10.82.105.11:1556

$ bptestbpcd -host 10.82.105.8
10.82.105.8:52148 -> 10.82.105.8:1556

Using a local interface as the <target> for MATCH entries has no affect.  In this example the source interface is unaffected by the local MATCH entry.

PREFERRED_NETWORK = 10.82.105.8/32 MATCH  

$ bptestbpcd -host billcat
10.82.105.11 :54202 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy
10.82.105.11 :54206 -> 10.82.105.14:13724

$ bptestbpcd -host beetle
10.82.105.11 :54300 -> 10.82.104.249:13724

$ bptestbpcd -host lilo
10.82.105.11 :54302 -> 10.82.56.79:1556

$ bptestbpcd -host 10.82.105.11
10.82.105.11:54306 -> 10.82.105.11:1556

$ bptestbpcd -host 10.82.105.8
10.82.105.8:54309 -> 10.82.105.8:1556

Similarly, using a local interface as the <target> for ONLY entries has no affect on source binding.  It does however prevent connections to destination addresses (in the absence of other directives that more closely MATCH the destinations).  Connections internal to the host are not affected.

PREFERRED_NETWORK = 10.82.105.8/32 ONLY  

$ bptestbpcd -host billcat
<16> bptestbpcd main: ConnectToBPCD(billcat) failed: 25 cannot connect on socket

$ bptestbpcd -host muzzy
<16> bptestbpcd main: ConnectToBPCD(muzzy) failed: 25 cannot connect on socket

$ bptestbpcd -host beetle
<16> bptestbpcd main: ConnectToBPCD(beetle) failed: 25 cannot connect on socket

$ bptestbpcd -host lilo
<16> bptestbpcd main: ConnectToBPCD(lilo) failed: 25 cannot connect on socket

$ bptestbpcd -host 10.82.105.11
10.82.105.11:54306 -> 10.82.105.11:1556

$ bptestbpcd -host 10.82.105.8
10.82.105.8:54309 -> 10.82.105.8:1556

Using MATCH entries, the outbound connections to a specific host or network can be preferred over the defaults.   In this example, connections to a specific host and a separate network are requested to use the second outbound network interface.

PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.8
PREFERRED_NETWORK = 10.82.104.0/24 MATCH 10.82.105.8    

$ bptestbpcd -host billcat               (Preferred by the 1^st entry.)
10.82.105.8 :54192 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy                 (Implicitly permitted using defaults.)
10.82.105.11:54196 -> 10.82.105.14:13724

$ bptestbpcd -host beetle                (Preferred by the 2^nd entry.)
10.82.105.8 :54200 -> 10.82.104.249:13724

$ bptestbpcd -host lilo                  (Implicitly permitted using defaults.)
10.82.105.11:54202 -> 10.82.56.79:1556

Adding an ONLY entries prevents connections to any other hosts that are not on the specified network, or matched by prior entries.

PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.8
PREFERRED_NETWORK = 10.82.104.0/24 MATCH 10.82.105.8
PREFERRED_NETWORK = 10.82.56.0/24 ONLY

$ bptestbpcd -host billcat               (Preferred by 1^st entry.)
10.82.105.8 :54209 -> 10.82.105.15:13724

$ bptestbpcd -host 10.82.105.14          (Does not match 1^st or 2^nd, excluded by 3^rd.)
<16> bptestbpcd main: ConnectToBPCD(muzzy) failed: 25 cannot connect on socket

$ bptestbpcd -host beetle                (Preferred by 2^nd entry.)
10.82.105.8 :54214 -> 10.82.104.249:13724

$ bptestbpcd -host lilo                  (Required by 3^rd entry.)
10.82.105.11:54216 -> 10.82.56.79:1556

Changing the ONLY to PROHIBITED explicitly excludes connections with those destination hosts and implicitly allows connections to unspecified hosts.  The PROHIBITED network is non-local and does not affect source binding.

PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.8
PREFERRED_NETWORK = 10.82.104.249/32 MATCH 10.82.105.8
PREFERRED_NETWORK = 10.82.56.0/24 PROHIBITED

$ bptestbpcd -host billcat               (Preferred by the 1^st entry.)
10.82.105.8 :54224 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy                 (Implicitly permitted.)
10.82.105.11:54228 -> 10.82.105.14:13724

$ bptestbpcd -host beetle                (Preferred by the 2^nd entry.)
10.82.105.8 :54232 -> 10.82.104.249:13724

$ bptestbpcd -host 10.82.56.79           (Does not match 1^st or 2^nd, prohibited by 3^rd.)
<16> bptestbpcd main: ConnectToBPCD(lilo) failed: 25 cannot connect on socket

Conversely, moving the ONLY to the top of the list does not prevent the MATCH entries from being evaluated because the ONLY is for a less restrictive IP range than the MATCH entries so the latter are evaluated first for those hosts.

PREFERRED_NETWORK = 10.82.104.0/24 ONLY
PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.11
PREFERRED_NETWORK = 10.82.104.249/32 MATCH 10.82.105.8

$ bptestbpcd -host billcat               (Preferred by the 2^nd entry.)
10.82.105.11 :54392 -> 10.82.105.15:13724

$ bptestbpcd -host 10.82.105.14           (Does not match 2^nd or 3^rd, excluded by 1^st.)
<16> bptestbpcd main: ConnectToBPCD(muzzy) failed: 25 cannot connect on socket

$ bptestbpcd -host beetle                (Preferred by 3^rd before being required by 1^st.)
10.82.105.8 :54396 -> 10.82.104.249:13724

$ bptestbpcd -host 10.82.56.79           (Does not match 2^nd or 3^rd, excluded by 1^st.)
<16> bptestbpcd main: ConnectToBPCD(lilo) failed: 25 cannot connect on socket

The subnet on this ONLY entry matches both billcat and muzzy, but does not affect the outbound interface confirming that ONLY is used for destination address filtering and not source address filtering.  Otherwise all connections would fail as both local interfaces, 10.82.105.11 and 10.82.105.8, are not in that subnet.

PREFERRED_NETWORK = 10.82.105.14/31 ONLY
PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.8

$ bptestbpcd -host billcat               (Preferred by 2^nd entry.)
10.82.105.8 :54209 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy                (Preferred by 1^st entry.)
10.82.105.11:45662 -> 10.82.105.14:13724

$ bptestbpcd -host 10.82.104.249         (Excluded by 1^st entry.)
<16> bptestbpcd main: ConnectToBPCD(beetle) failed: 25 cannot connect on socket

In this example, all three remote hosts are reachable, but notice that the source interface is the one remaining after 10.82.105.11 is PROHIBITED.  This includes the apparent <target> MATCH for billcat, which actually failed to match because the <source> was previously PROHIBITED.  Notice that internal connections are not affected by PROHIBITED.

PREFERRED_NETWORK = 10.82.105.11/32 PROHIBITED
PREFERRED_NETWORK = 10.82.105.15/32 MATCH 10.82.105.11

$ bptestbpcd -host billcat               (Matched 2^nd, but 1^st prohibited that source.)
10.82.105.8 :54202 -> 10.82.105.15:13724

$ bptestbpcd -host muzzy                 (Implicit match & pruned source.)
10.82.105.8 :54206 -> 10.82.105.14:13724

$ bptestbpcd -host beetle                (Implicit match & pruned source.)
10.82.105.8 :54300 -> 10.82.104.249:13724

$ bptestbpcd -host 10.82.105.11         (Not affected by 1^st entry.) 
10.82.105.11:54306 -> 10.82.105.11:1556

$ bptestbpcd -host 10.82.105.8
10.82.105.8:54309 -> 10.82.105.8:1556

This example demonstrates two nuances of Source Binding Evaluation that result in the use of ANY interface instead of the non-prohibited interfaces.  The second entry removes the 10.82.10.10 local interface from the source binding list before the third entry is processed making that <source> unavailable.  The <source> on the first entry causes the shortened list created by the second entry to be ignored during all evaluations.

PREFERRED_NETWORK = 10.82.104.249 MATCH 10.82.105.0/24
PREFERRED_NETWORK = 10.82.10.10 PROHIBITED
PREFERRED_NETWORK = 10.82.56.0/24 MATCH 10.82.10.10

FL:   billcat -> 10.82.105.15   ... SRC: ANY (1^st source implicitly negates 2^nd target.)
FL:   muzzy  -> 10.82.105.14  ... SRC: ANY  ( same )
FL:   beetle -> 10.82.104.249 ... SRC: 10.82.105.11  (Matched 1^st, used 1^st in range.)
FL:   lilo -> 10.82.56.79   ... SRC: ANY (2^nd target explicitly negates 3^rd source.) 

Note: In the example above the <source> on the first entry matches two local interfaces.  The 10.82.105.11 interface was chosen over 10.82.105.8 as the source when connecting to beetle because that interface was returned first by the O/S as shown in the bplocaladdrs output in the section above.

This example shows how the binding list is shortened by prohibiting a local interface.  When ANY was the default source binding list, the outbound interface for these destinations was 10.82.105.11 (see first example above).  Prohibiting a different local interface causes NetBackup to provide a shortened list and the O/S selected 10.82.10.10 as the source IP.   Because this O/S uses the stronghost model, that interface is not valid for these destination IPs and the connection attempts fail.

PREFERRED_NETWORK = 10.82.105.8 PROHIBITED

FL:  billcat -> 10.82.105.15  ... SRC: 10.82.10.10,10.82.105.11
FL:     lilo -> 10.82.56.79   ... SRC: 10.82.10.10,10.82.105.11

$ bptestbpcd -host billcat
<16> bptestbpcd main: ConnectToBPCD(billcat) failed: 25 cannot connect on socket

$ bptestbpcd -host lilo
<16> bptestbpcd main: ConnectToBPCD(lilo) failed: 25 cannot connect on socket 

If the O/S is changed to the weakhost model, the TCP SYN for each connection is transmitted out the default interface (10.10.82.105.11) onto the 10.82.104.0 network, but with a source IP of 10.82.10.10.  If there is a network route from the 10.82.104.0 network to the destination hosts then the SYN will reach the destinations.  But the reply is only successful if there is an asymmetrical route back to the 10.82.8.0 network from the destination host.  Notice the spoofed source IP in the successful connection which does not reflect the network onto which the TCP SYN packet was actually sent.

$ bptestbpcd -host billcat
<16> bptestbpcd main: ConnectToBPCD(billcat) failed: 25 cannot connect on socket

$ bptestbpcd -host lilo
10.82.10.10:52842 -> 10.82.56.79:1556


Compatibility :  

Any legacy Required Interface or Required Network configuration is automatically converted to a Preferred Network representation internally.  

For the master server bob described above.  

REQUIRED_INTERFACE = bob

is equivalent to this for IPv4

PREFERRED_NETWORK = 0/0 MATCH 10.82.105.11

If IPv6 is enabled, via IP_ADDRESS_FAMILY=AF_UNSPEC, this would be the equivalent for IPv6

PREFERRED_NETWORK = 0/0 MATCH fc44:53f9:cb30:201:250:56ff:febc:e85f

Both bind the specified source interface for all outbound connections because 0/0 matches all destinations.  But notice the length of subnet (/0), any other directive with a source binding and a longer target subnet will supersede these entries.  Similarly, because both the IPv4 and IPv6 examples have the same subnet length, only the first of these two would be honored if both were configured.

Similarly, if a required network was configured

REQUIRED_NETWORK = 10.82.105/21

it translates to

PREFERRED_NETWORK = 10.82.105/21 ONLY

Which restricts destination addresses to the specified network without affecting source interface selection.

Note: If both REQUIRED_INTERFACE and PREFERRED_NETWORK are specified and conflict, REQUIRED_INTERFACE overrides.

Note: Unlike REQUIRED_INTERFACE, PREFERRED_NETWORK does not change the requesting_client or destination_client fields in user-directed requests to bprd for image list or restore.