Non-unique IP addresses on connecting hosts may prevent CORBA connections

Problem

NetBackup legacy connections can be made between the hosts, but CORBA connections fail in one or both directions.   This may cause backup or restore jobs with many different status codes depending on which processes fail to connect.

Error Message

The debug log for the connecting process shows the forward profile returned from the remote host.  In this case it contains four endpoints. 

[TAO] TAO (15623|1) - Transport::handle_input(): bytes read from socket - HEXDUMP 704 bytes

47 49 ... 00 00   GIOP............

03 00 ... 4c 3a   ........-...IDL:

...snip...

31 39 ... 33 00   192.168.229.233.

14 06 ... ff ff   ........nbsl.s..

26 00 ... 32 30   &...aaaa:bbbb:cc

35 61 ... 66 3a   cc:dddd:111:2222

66 65 ... 00 00   :3333:444.......

6e 62 ... 61 70   nbsl........nbap

70 32 ... 61 6e   p247.symantec.co

74 65 ... 00 00   m...............

6e 62 ... 00 00   nbsl........P...

...snip...

31 30 ... 00 00   10.132.0.53.....

05 00 ... 64 72   ....nbsl.I....dr

0c 00 ... 63 00   ....nbsl_secsvc.

...snip...

The Connect List is searched for the endpoints from the profile, but the evaluation stops after only 2 of the 4 are reviewed.

[EndpointSelector_R2::endpoint_from_profile] epsr2: Evaluating 4 endpoints

[ConnectList::search] host 192.168.229.233 not found in cache

[ConnectList::search] host aaaa:bbbb:cccc:dddd:111:2222:3333:444 not found in cache

...These entries to create NAT bindings may not be present depending on the connection state.

[EndpointSelector_R2::evaluateNatBindings] epsr2: Target host does not advertise the address we originally connected to. Creating NAT binding of 192.168.229.233 to 10.132.0.53 in connect list.

[EndpointSelector_R2::evaluateNatBindings] epsr2: Target host does not advertise the address we originally connected to. Creating NAT binding of 10.132.0.53 to 10.132.0.53 in connect list.

[EndpointSelector_R2::evaluateNatBindings] epsr2: Target host does not advertise the address we originally connected to. Creating NAT binding of nbapp247.symantec.com to 10.132.0.53 in connect list.

[EndpointSelector_R2::evaluateNatBindings] epsr2: Target host does not advertise the address we originally connected to. Creating NAT binding of aaaa:bbbb:cccc:dddd:111:2222:3333:444 to 10.132.0.53 in connect list.
...

[ExternalResolver::set] Resolved host_name 10.132.0.53[10.132.0.53]

[EndpointSelector_R2::evaluateNatBindings] epsr2: Target host does not advertise the address we originally connected to. Replacing 192.168.229.233 with 10.132.0.53 for target endpoint.

...snip...

This will lead to a connection failure that may be either a TRANSIENT or OBJECT_NOT_FOUND.

[Orb::connectToObjectOnce] connection attempt failed: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

Cause

The failure above is expected and occurs because the same 192.168.229.233 and aaaa:bbbb:cccc:dddd:111:2222:3333:444 addresses are assigned to both of the hosts.  Because of this the endpoints within the profile that are unique to the remote host do not outnumber the endpoints within the profile that are also local to both hosts, and NetBackup cannot determine if the target host is local (itself) or remote.

This determination is necessary because NetBackup appliances ship with a common IPv4 address for the eth0 interface (either 192.168.1.1 or 192.168.229.233 depending on version).  Since the common IP address will be returned within the profile from both hosts, when they are connecting to each other they must be prevented from mistakenly connecting to the local NIC instead of the remote NIC.

Normally the forward profile would contain at least three endpoints for the remote host; the common IP address for eth0, the hostname of the remote host, and the IP address for the public interface (eth1 or bond0). 

Thus in the normal configuration, there would be at least two non-local endpoints to offset the one common endpoint, and the latter would be correctly ignored.  If additional unique network interfaces are present, the number of non-local endpoints would remain higher than the number of common endpoints.

In the example above, the same IPv6 address had been added to both hosts for some reason and a safe target for the connection could not be accurately determined from among those in the profile.

Solution

Avoid having the same IP address(es) assigned to network interfaces on more than one host.   That causes ambiguity in routed networks and also when making CORBA connections between hosts that attach to non-routed networks that utilize the same address space.

At a minimum

• Ensure that the number of unique and routable IP addresses on each host is greater than or equal to the number of IP addresses that are not unique and routable.
• Ensure that all hostnames in the NetBackup configuration resolve to the correct unique and routable IP address.

Note: Hostnames that do not resolve cannot be confirmed to be remote and are assumed to be local.

Thus the problem above can be resolved by modifying the IP addresses assigned to the hosts.   Either remove one or more of the non-unique IP addresses from one or more of the hosts, or assign additional unique and routable IP addresses to the remote target host.

Note: IP_ADDRESS_FAMILY and PREFERRED_NETWORK do not affect the determination of how many endpoints are local or non-local.

Applies To

NetBackup 7.0 – 7.6