How to use the 'Use specified network interface' client host property

Article: 100002210
Last Published: 2018-02-16
Ratings: 0 2
Product(s): NetBackup & Alta Data Protection

Description

The 'Use specified network interface' client host property, which corresponds to the REQUIRED_INTERFACE configuration setting, is a very misunderstood part of NetBackup.  

This setting requests that the operating system (OS) bind a specific source IP when creating outbound network connections.  It does not change the network interface that is used to transmit the packet.   The OS routing configurations determines the outbound network interface, and the resulting behavior can be very different from what the user is expecting.  See below for the effects on the networking layers and NetBackup; the changes can be subtle and may not be immediately noticed.

This setting was most useful in NetBackup 5.1 and prior versions.   It was retained in NetBackup 6.0 and later versions for backwards compatibility.

Starting with NetBackup 7.0.1, the PREFERRED_NETWORK setting should be used instead.   It has the same OS and network routing interactions, but it can be applied more selectively to specific outbound destinations instead of affecting all outbound destinations.

The Required Interface setting was initially implemented in NetBackup 3.x (the last century).  At the time, all supported Windows and UNIX operating system (OS) versions implemented the weak host end-station model and most backup client hosts had only a single network interface.

As networking capabilities evolved and customers began to add backup network interfaces to hosts, it was often convenient [but not recommended] to implement Required Interface on the media server hosts, e.g.

        REQUIRED_INTERFACE = mymm-bk.backupnet.mydomain

This allowed the media server (mm) bpbrm process to continue to connect to the client host interface on the public network [as configured in the backup policy] but request the OS to use the source binding for the mm interface on the backup network.  That way, when the client host created the [legacy] call-back connection to the bptm process on the media server it would use the media server interface on the backup network as the destination.  Consequently, the call-back would originate from the backup interface on the client.  The following network routing would then occur.

        bpbrm --> client        traffic on the public network
        bpbrm <-- client        traffic on the backup network

        client --> bptm         traffic on the backup network
        client <-- bptm         traffic on the backup network

Notice that the backup image, at either backup or restore time will flow across the backup network between the client and bptm.  The asymmetrical routing for the bpbrm connection was typically not noticed by customers, and was permitted by their firewalls.

This type of configuration allowed the new backup images, taken over the backup network, to be stored under the original public client name.  Thus older and newer backup images could be restored using a single client name.

If the client host was similarly configured with a Required Interface on the backup network, there was no effect on communication with the media server because the connection originating from the client to bptm was already using the backup network.

If a Required Interface was not specified on the media server, and the policy still referenced the client by its public hostname, then configuring the client with a Required Interface on the backup network resulted in the following.

        bpbrm --> client        traffic on the public network
        bpbrm <-- client        traffic on the public network

        client --> bptm         traffic on the public network
        client <-- bptm         traffic on the backup network

This placed the backup image, at restore time, on the backup network from bptm to the client.  However, during backup, the image was sent to bptm over the public network.  Probably not what was intended.

A better alternative was to change the policy client, from the hostname on the public network, to the hostname on the backup network.  Then link the backup hostname to the public hostname in the NetBackup image directory on the master server (see Best Practices for creating client alias relationships for image browse and restore).  In addition, it was useful to create a second storage unit for the media server using its hostname on the backup network.  The clients on the backup network could then be moved into policies that used the new storage unit.  This avoided all uses of Required Interface.

As network security evolved, customers required that all connections originate from the NetBackup servers.  Starting with NetBackup 6.0, the connection between bptm and the client also originate from the media server, by default.  This is transparent when using the appropriate policy client names on the intended target network (typically the backup network) and all traffic is on that network.

However, if the original public client names remain in the policy, then having the mm specify a required interface on the backup network causes the following:

        bpbrm --> client        traffic on the public network
        bpbrm <-- client        traffic on the backup network

        bptm --> client         traffic on the public network
        bptm <-- client         traffic on the backup network

Notice that the restore traffic from bptm to the client is also on the public network because the NetBackup servers know the client by its hostname on the public network!  Again this was often not noticed by customers until a large restore consumes enough bandwidth on the public network to impact the users. 

Note: If the Connect Options for the client are changed to 'x x 2' or firewalls do not permit connections from the media server to TCP ports 1556 and 13724 on the client, then legacy call-back will continue to be used as with NetBackup pre-6.0.  Note also that NetBackup 8.1 deprecates Connect Options and legacy call-back.

Using a Required Interface on a client host does not affect communication with the media server unless legacy call-back is used (because otherwise bpbrm and bptm originate both connections).   For client-initiated requests to the master server, it can result in asymmetrical routing between the client and bprd.

These unintended consequences of the OS weak host end-station model become more apparent when newer firewalls are installed or newer OS versions, that implement the strong host model (Windows) or reverse path filter (Linux), begin to detect the asymmetrical routes when communicating with older hosts. 

Solution

Syntax

The Required Interface setting, if used, should specify a hostname that resolves to the IP address of a networking interface or sub-interface that is local to the host.  That IP address should reverse resolve back to the same hostname.  E.g.

REQUIRED_INTERFACE = myhost.mydomain

With results forward and reverse name resolution.

$ bpclntcmd -hn myhost.mydomain
host myhost.mydomain: myhost.mydomain at 10.80.96.66
aliases:     myhost.mydomain     10.80.96.66

$ bpclntcmd -ip 10.80.96.66
host 10.80.96.66: myhost.mydomain at 10.80.96.66
aliases:     myhost.mydomain     10.80.96.66

Note: If the specified hostname does not resolve to an IP address, or resolves to an IP address that is not local to the host, the setting will be ignored.

Note: Do not specify an IP address, instead of a hostname.  This may negatively affect host certificates in NetBackup 7.7.2 and later versions.

Immediate/Local Operational Effects

This setting only affects outbound connections initiated from this host to other NetBackup hosts.   It does not affect inbound connections.   The effects are as follows.

1)      Causes the resolved IP address to be used as the source binding list when making outbound connections.

2)      This will cause the resolved IP address to be the source IP address for the outbound connection.  

Note: This does not mean that the OS will actually originate the connection from the networking interface to which the IP address is assigned.   See below. 

3)      If the connection is created for the purpose of a client-initiated list/browse, restore, or expiration request, then the 'requesting client' in the request sent to the master server will be set to the specified hostname.

Additional/Downstream Operational Effects

The items noted above may have the following additional effects, depending on the configuration of NetBackup on the originating and receiving hosts and the configuration of the networking layers in between.

4)      For source binding purposed, the Required Interface setting is equivalent to the following beginning in NetBackup 7.0.1.   Consequently, it takes precedence over any Preferred Network setting with respect to selecting the source binding list. 

PREFERRED_NETWORK = 0/0 MATCH thishost.mybackupnet.mydoamin

Note: See the related articles for best practices using Preferred Network settings. 

5)      The OS may cause the outbound connection to use a different network interface. 

The OS network routing table (netstat -n -r) controls the actual selection of the outbound interface.   If the Required Interface is not a normal outbound route to the destination IP address, the OS will send the TCP SYN to establish the connection from the normal interface, but with the source IP set to the Required Interface (resulting in source IP spoofing).   Since the transmitting interface is correct for the destination IP, the TCP SYN will typically reach the remote host.

Because the source and destination IP addresses are “as expected”, this behavior is unlikely to be noticed unless a packet trace (Wireshark/snoop/tcpdump) is taken on the client host.  The latter will confirm that the outbound packets are relative to the network routing table, which may be different from the Required Interface. 

6)      The connection reply may use a different, asymmetrical network route. 

The TCP reply packets will be returned to the source IP address.   Consequently, the reply packets may be transmitted from a different network interface on the remote host than the one on which they were received.   If a return route exists, they will be delivered to the Required Interface which may not be the interface from which the outbound packets originated.   This is called asymmetrical routing, and may cause the many large packets transporting a backup image to traverse one network path while the many large packets transporting a restore image will traverse the other network path. 

Because the source and destination IP addresses are “as expected”, this behavior is unlikely to be noticed unless a packet trace (Wireshark/snoop/tcpdump) is taken on the media server host.

Note: See the related articles for additional information about the weak host model. 

7)      The OS end-station model (strong host) on the source or destination host may cause the connection to fail. 

As noted above, the OS network routing table controls the actual selection of the outbound interface.   If the local OS is configured as a strong host or has reverse path filtering enabled, and the Required Interface is not one of the normal routing possibilities, then the TCP SYN will never be sent and the connection will fail. 

If the local OS is a weak host, and the TCP SYN is sent for a different interface with a spoofed source IP address and the destination is a strong host the connection may also fail.   Whether the remote host accepts the connection will depend on whether the receiving network interface is a normal route from which the specified source IP address is expected.   If not, the inbound connection may be dropped. 

Note: See the related articles for additional information about the strong host model. 

8)      Asymmetrical routing is not compatible with stateful network devices. 

Networking gateways and firewalls that provide certain types of routing, transformation, and security services need to receive, inspect and/or modify, and then forward packets traversing the network.   If they only see the packets travelling in one direction, they may drop the connection. 

9)      The master server may not recognize the Required Interface 

The NetBackup master server authenticates inbound connections by performing reverse hostname lookup on the source IP to get the peername [using getnameinfo()].   Because the Required Interface setting changes the source IP address, client-initiated operations may fail because the master server does not recognize the Required Interface as a valid NetBackup client. 

status 131: client is not validated to use the server

Be sure the master server can resolve the source IP to a hostname and that the resulting peername is configured a as a client, even if the policy it is in does not have an open window. 

10)   The Required Interface may be treated as a separate client. 

The NetBackup master server does additional authentication on requests to browse, restore, or expire backup images.   The intent is to ensure that any such requests originate either from the client that owns the image or from a valid NetBackup server.   

If the relationship between the peername [from the source IP] is not equivalent to with the client name and/or requesting client [from the request] then the bprd process may connect to the client to confirm it is the same host.   If host equivalence is still not achieved, the request will be rejected unless the peername is recognized as a valid NetBackup server. 

status 133: invalid request status 135: client is not validated to perform the requested operation

Note: These errors also occur if the peername does not have permission to access the images for the browse client.   Be sure to configure the altnames database appropriately.

Operating Systems Settings

As noted above, the end-station model implemented in the OS of the hosts can have useful or undesirable effects on the creation of connections when NetBackup requests a specific source binding instead of the default of ‘IPADDR_ANY’ source interface.

Review these OS settings to determine the weak host and strong host end-station model configuration.   See the OS documentation for specific values and defaults since they vary by version.

AIX:

no -a | grep rfc1122addrchk

HP-UX:

ndd -get /dev/ip ip_strong_es_model

Linux:

sysctl -a | grep 'net.*rp_filter'

Solaris:

ndd -get /dev/ip ip_strict_src_multihoming
ndd -get /dev/ip ip_strict_dst_multihoming
ndd -get /dev/ip ip6_strict_src_multihoming
ndd -get /dev/ip ip6_strict_dst_multihoming

Windows:

netsh interface ipv4 show addresses
netsh interface ipv4 show interface [NameOrIndex] | FINDSTR ”Weak host”
netsh interface ipv6 show addresses
netsh interface ipv6 show interface [NameOrIndex] | FINDSTR ”Weak host”

Other Considerations - List Restore client attribute

The List Restore client attribute is another part of request authorization processing that occurs after peername authentication and peername request authorization.   Check the bprd debug log to see if the client name was switched to the peername and then check the current setting in the client data base; correct if needed.

bpclient -client myclient.mydomain -L
bpclient -client peername-for-myclient.mydomain -L

Other Considerations – Altnames database

The altnames database is part of authorization processing which occurs after the List Restore attribute check noted above.   If the Required Interface resulted in the client name being changed to the peername (check the bprd debug log) then additional altnames configuration may be necessary or existing altnames configuration become obsolete depending on the situation.

cd netbackup
cd db
cd altnames
more mypeername

Note that altnames is useful to allow a peername to have access to backup images from the same host that are stored under a different client name.   Configure the peername as if it was a redirected client restore.

echo myclient.mydomain >> mypeername
echo myclient-bk.backupnet.mydomain >> mypeername

Other Considerations - Client Alias

This NetBackup 7.5 feature can perform additional processing after the altnames search.   It allows backups taken using several client names to be stored together and accessed via any of the names; i.e. all of those client names become aliases for each other.  

Note: See the related articles for best practices using client aliases.

Recommendations

The Required Interface setting is not needed in normal NetBackup configurations.  

Required Interface is only recommended to be used when the operating system has two equally valid IP addresses assigned to the same network interface (such as a node IP and a cluster/virtual/failover IP on a logical sub-interface) from which the local host will connect outbound to all other NetBackup hosts.  In that case, Required Interface can be used to have the source IP always be either the one associated with the node hostname or the one associated with the cluster hostname.  The better or more correct hostname will be a matter of implementation and/or preference.  

Any other situation, especially on hosts with multiple network interfaces on distinct IP subnets, other, better solutions should be used to prevent the creation of asymmetrical network routes.

The following guidelines cover most other situations.

Note: The OS exposes only the bound source IP address to the application, via getsockname(), not the actual outbound interface. Hence the source IP address in the NetBackup debug logs may not be accurate, similar to when static Network Address Translation (NAT) is used to obscure the true source of a connection.   Always use a network packet capture to confirm which network interface the inbound and outbound packets traverse.   It may be necessary to sniff each network interface in promiscuous mode to locate the bi-directional traffic.

A)      The best solution is to simply configure NetBackup with hostnames that resolve to the correct IP addresses that match the host network routing table, instead of using Required Interface. 

To connect to clients only on their backup interfaces, configure the policy and client name to be on the backup network, e.g. myclient-bk.backupnet.mydomain. 

Configure the storage unit residence with the media server hostname that is routable over the desired network from the master server, e.g. mymm-bk.backupnet.mydomain.   If some clients are only on the public network, they will have hostnames only on that network and the media server will use that network for only those clients.

Configure the server entries on the client to be the desired hostnames for outbound connections to the master server and to authenticate the expected inbound peernames, e.g.

If the NetBackup servers know the client by its backup hostname:

SERVER = mymaster-bk.backupnet.mydomain
SERVER = mymm-bk.backupnet.mydomain

If the NetBackup servers know the client by its public hostname:

SERVER = mymaster.mydomain
SERVER = mymm.mydomain

B)      Another preferred solution is to use static network routes or static host routes instead of Required Interface.  

Since network routing is the domain of the OS, use OS tools so that the configuration is reflected in the routing tables and also to eliminate the possibility of source IP spoofing.

Further, static routes can selectively use specific local interfaces when reaching specific destinations.   This is more flexible than using one Required Interface for all destinations.

Note: See the related articles for additional details about using static routes. 

C)     Another preferred solution is to use Preferred Network settings.  

This allows the source binding to be tailored for multiple specific destinations instead of using one Required Interface for all destinations.   In addition, unlike Required Interface, it does not change the requesting client in client-initiated browse/restore/expiration requests.

D)     Always make sure the server and media server lists on NetBackup hosts contain the correct peernames from which they can and should accept inbound requests from other NetBackup server hosts.   Peernames for valid NetBackup clients should be present in the policy and/or client databases on the master server.

As noted above, the Required Interface can be used when the desired results is compatible with the configuration of the operating system and networking layers and devices.   In those configurations, it allows a specific source IP to be used instead of a default one.

E)      A common NetBackup situation is network sub-interface used for specific applications.   E.g.

eth0 = 10.0.0.10/24 = myhost.mydomain (used in Standard policy)
eth0:1 = 10.0.0.100/24 = myoracle.mydomain (used in Oracle policy)

Using myoracle.mydomain as the Required Interface would cause that to be the requesting client for Oracle backup, crosscheck, and restore operations.  

Note: That setting would also affect the requesting client for client-initiated file system restores and perhaps cause them to fail.   Use server-initiated file system restores instead so that connections are all initiated inbound to the client.

Alternatively, make sure the master server recognizes both hostnames as valid clients and that the altnames database permits the peername to have access to images for both client names as with a redirected restore.

F)      A similar situation is when the default source binding order for the NICs on the client host is not as expected.  

Use Required Interface until the binding order can be corrected at the OS level, assuming that all NetBackup hosts to which this one connects are routable by the O/S via that interface.

Again, a better alternative is to make sure the master server recognizes the peername as a client, and that the altnames database allows the peername to access the client images.

G)     A similar situation is when redundant network interfaces are used on a client host instead of a bond or trunk.   Normally outbound connections would round-robin or load balance across the interfaces.

eri0 = 10.10.1.11/24 = myhost-if0
eri1 = 10.10.1.12/24 = myhost-if1 myhost (used in policies)

Using myhost (or myhost-if1) as the Required Interface would ensure that the master server only needed to know about the hostname for the eri1 interface for client-initiated operations.

Again, a better alternative is to make sure the master server recognizes both peernames and allows either to access the client images.

H)     Required interface may be useful in other configurations when situational circumstances prevent other solutions.   Simply be aware of the effects noted above when tailoring a solution.
 

 

Applies To

NetBackup 3.x – 8.2
 

Was this content helpful?