Restore fails with status 37: operation requested by an invalid server

Problem

Restore fails with status 37 before the job goes active.
 

Error Message

The Backup, Archive, and Restore (BAR) GUI will display this error:

Operation requested by an invalid server - status 37

The bprd debug log shows the processing that occurs to authenticate the restore request.  First, a check is made to see if this is a server-directed restore.  The requesting source IP is resolved to a hostname (referred to as the peername), but it is not in the SERVER list so this is not a server-directed restore.

15:00:03.713 [13886] <2> bprd: socket fd from accept() is 10
15:00:03.720 [27641] <2> logconnections: BPRD ACCEPT FROM 3.3.3.3.43716 TO 1.1.1.1.13724
15:00:03.729 [27641] <2> connected_peer: Connection from host Node1-bk.domain, 3.3.3.3, on non-reserved port 43716
15:00:03.732 [27641] <2> nb_is_valid_master_server: checking if Node1-bk.domain is a valid server

...snip...

15:00:03.733 [27641] <2> db_valid_master_server: Node1-bk.domain is not a valid server

...snip...

The short name for the peername is in a policy, so the host sending the request is a valid client.

15:00:03.772 [27641] <2> get_ccname: determine configured name for Node1-bk.domain
15:00:03.892 [27641] <2> logconnections: BPDBM CONNECT FROM 1.1.1.1.34788 TO 1.1.1.1.13724
15:00:03.983 [27641] <2> get_ccname: configured name is: Node1-bk

...snip...

A client/user-directed restore is permitted if the requesting and destination hosts are the same.  The client provided the names for the source/browse client, the destination client, and itself in the request.  In this case they appear to be different, especially the requesting client which does not match the Client Name.

15:00:03.984 [27641] <2> process_request: As rcvd from client:
15:00:03.984 [27641] <2> process_request:   browse_clnt: Exchange
15:00:03.984 [27641] <2> process_request:   requesting_clnt: Node.default-domain
15:00:03.984 [27641] <2> process_request:   destination_clnt: Exchange
15:00:03.984 [27641] <2> process_request:   clnt_bp_conf_name: Node1

...snip...

A check of the name resolution (which is under trusted administrative control) is made to see if the requesting and destination client names resolve to the same primary hostname or same IP address.  In this case, they do not.

15:00:03.987 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node.default-domain>
15:00:03.988 [27641] <2> hosts_equal: host1 Exchange Exchange addr 3.3.3.2 (0xa526905)
15:00:03.988 [27641] <2> hosts_equal: host2 Node.default-domain Node.default-domain addr 3.3.3.1 (0xa5269ea)
15:00:03.988 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:03.988 [27641] <2> hosts_equal: name1=Exchange  name2=Node.default-domain
15:00:03.988 [27641] <2> hosts_equal: hostbuf=Exchange  hostentry->h_name=Node.default-domain
15:00:03.989 [27641] <2> hosts_equal: compare Exchange with alias Node1-alias
15:00:03.989 [27641] <2> hosts_equal: compare Node.default-domain with alias Exchange-alias
15:00:03.989 [27641] <2> hosts_equal: addresses DO NOT match
15:00:03.989 [27641] <2> hosts_equal: Exchange=0xa526905 Node.default-domain=0xa5269ea

A check of the name resolution is made to see if the peername and requesting client names resolve to the same primary host or same IP address.  In this case, they do not.

15:00:03.989 [27641] <2> hosts_equal: Comparing hosts <Node1-bk.domain> and <Node.default-domain>
15:00:03.989 [27641] <2> hosts_equal: host1 Node1-bk.domain Node1-bk.domain addr 3.3.3.3 (0xa5268f9)
15:00:03.989 [27641] <2> hosts_equal: host2 Node.default-domain Node.default-domain addr 3.3.3.1 (0xa5269ea)
15:00:03.989 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:03.989 [27641] <2> hosts_equal: name1=Node1-bk.domain  name2=Node.default-domain
15:00:03.989 [27641] <2> hosts_equal: hostbuf=Node1-bk.domain  hostentry->h_name=Node.default-domain
15:00:03.990 [27641] <2> hosts_equal: compare Node1-bk.domain with alias Node1-alias
15:00:03.990 [27641] <2> hosts_equal: compare Node.default-domain with alias Node1-bk
15:00:03.990 [27641] <2> hosts_equal: addresses DO NOT match
15:00:03.990 [27641] <2> hosts_equal: Node1-bk.domain=0xa5268f9 Node.default-domain=0xa5269ea 

Because the requesting host is not a server and does not appear to be the host it is expected to be, the requesting client (which could be spoofed) is replaced with the peername (which is trusted) and the comparison repeated. 

15:00:03.990 [27641] <2> validate_hostname: Unknown hostname Node.default-domain, switching to peername Node1-bk.domain.
15:00:03.990 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node1-bk.domain>
15:00:03.991 [27641] <2> hosts_equal: host1 Exchange Exchange addr 3.3.3.2 (0xa526905)
15:00:03.991 [27641] <2> hosts_equal: host2 Node1-bk.domain Node1-bk.domain addr 3.3.3.3 (0xa5268f9)
15:00:03.991 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:03.991 [27641] <2> hosts_equal: name1=Exchange  name2=Node1-bk.domain
15:00:03.991 [27641] <2> hosts_equal: hostbuf=Exchange  hostentry->h_name=Node1-bk.domain
15:00:03.991 [27641] <2> hosts_equal: compare Exchange with alias Node1-bk
15:00:03.991 [27641] <2> hosts_equal: compare Node1-bk.domain with alias Exchange-alias
15:00:03.991 [27641] <2> hosts_equal: addresses DO NOT match
15:00:03.991 [27641] <2> hosts_equal: Exchange=0xa526905 Node1-bk.domain=0xa5268f9 

The client attributes database is checked to see if the source/destination client has a current host entry which may match.  In this case, the hostname does not have any client attributes. 

15:00:03.991 [27641] <2> read_client: ?
15:00:03.991 [27641] <2> read_client: opendir() failed: Exchange: No such file or directory (2)

The other clients, that have attributes, are checked to see if they have a current host entry that matches.  They do not so the destination client remains unchanged.

15:00:03.992 [27641] <2> db_getCLIENT_by_hostname: accessing client /usr/openv/netbackup/db/client/myoraclient
15:00:03.993 [27641] <2> db_getCLIENT_by_hostname: hostname Exchange not configured for client myoraclient
15:00:03.993 [27641] <2> db_getCLIENT_by_hostname: accessing client /usr/openv/netbackup/db/client/mymaster
15:00:03.994 [27641] <2> db_getCLIENT_by_hostname: hostname Exchange not configured for client mymaster
15:00:03.994 [27641] <2> process_request: After get_client_and_host_names:
15:00:03.994 [27641] <2> process_request:   destination_clnt: Exchange
15:00:03.994 [27641] <2> process_request:   destination_bp_conf_name: Exchange

...snip...

A job record is created so that the NetBackup administrator will have visibility to the restore request. 

15:00:04.117 [27641] <2> logconnections: BPJOBD CONNECT FROM 1.1.1.1.34799 TO 1.1.1.1.13724
15:00:04.121 [27641] <2> set_job_details: Sending jobData jobid (911)
15:00:04.121 [27641] <2> send_structure_data: Index 4 Field m_nMainPid Value <27641>
15:00:04.121 [27641] <2> send_structure_data: Index 7 Field m_nType Value <2>
15:00:04.121 [27641] <2> send_structure_data: Index 14 Field m_szClient Value <Exchange>

...snip...

The client attributes database is checked to see if the requesting client has a current host entry which matches.  In this case, the hostname does not have any client attributes.

15:00:04.128 [27641] <2> read_client: ?
15:00:04.128 [27641] <2> read_client: opendir() failed: Node.default-domain: No such file or directory (2)
15:00:04.128 [27641] <2> get_type_of_client_free_browse: db_getCLIENT() failed: no entity was found (227) 

The other clients, that have attributes, are checked to see if they have a current host entry that matches.  They do not so the source and requesting client names remains unchanged.

15:00:04.129 [27641] <2> db_getCLIENT_by_hostname: accessing client /usr/openv/netbackup/db/client/myoraclient
15:00:04.129 [27641] <2> db_getCLIENT_by_hostname: hostname Node1-bk.domain not configured for client myoraclient
15:00:04.129 [27641] <2> db_getCLIENT_by_hostname: accessing client /usr/openv/netbackup/db/client/mymaster
15:00:04.129 [27641] <2> db_getCLIENT_by_hostname: hostname Node1-bk.domain not configured for client mymaster
15:00:04.129 [27641] <2> get_type_of_client_free_browse: db_getCLIENT_by_hostname() failed: no entity was found (227)
15:00:04.129 [27641] <2> process_request: After get_client_and_host_names:
15:00:04.129 [27641] <2> process_request:   browse_clnt: Exchange
15:00:04.129 [27641] <2> process_request:   clnt_bp_conf_name: Node1
15:00:04.129 [27641] <2> process_request:   retval: 0

...snip...

In case any adjustments have been made, the network hostname resolution is checked again to see if any of the hostnames involved are aliases for each other.

15:00:04.129 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node1-bk.domain>

...snip...

15:00:04.130 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.131 [27641] <2> hosts_equal: addresses DO NOT match

...snip...

15:00:04.131 [27641] <2> db_valid_client: comparing peer Node1-bk.domain with Node.default-domain
15:00:04.131 [27641] <2> hosts_equal: Comparing hosts <Node.default-domain> and <Node1-bk.domain>

...snip...

15:00:04.132 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.132 [27641] <2> hosts_equal: addresses DO NOT match
15:00:04.132 [27641] <2> db_valid_client: comparing peer Node1-bk.domain with Exchange
15:00:04.132 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node1-bk.domain>

...snip...

15:00:04.133 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.133 [27641] <2> hosts_equal: addresses DO NOT match
15:00:04.133 [27641] <2> db_valid_client: comparing peer Node1-bk.domain with Node1-bk.domain
15:00:04.133 [27641] <2> hosts_equal: Comparing hosts <Node1-bk.domain> and <Node1-bk.domain>
15:00:04.134 [27641] <2> hosts_equal: names are the same
15:00:04.135 [27641] <2> db_valid_client: comparing Exchange with Node1-bk.domain
15:00:04.135 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node1-bk.domain>

...snip...

15:00:04.135 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.136 [27641] <2> hosts_equal: addresses DO NOT match
15:00:04.136 [27641] <2> db_valid_client: comparing Exchange with Node.default-domain
15:00:04.136 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Node.default-domain>

...snip...

15:00:04.136 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.137 [27641] <2> hosts_equal: addresses DO NOT match
15:00:04.137 [27641] <2> db_valid_client: comparing Exchange with Exchange
15:00:04.137 [27641] <2> hosts_equal: Comparing hosts <Exchange> and <Exchange>
15:00:04.137 [27641] <2> hosts_equal: names are the same
15:00:04.137 [27641] <2> db_valid_client: ...matched!
15:00:04.137 [27641] <2> hosts_equal: Comparing hosts <Node1-bk.domain> and <Exchange>

...snip...

15:00:04.138 [27641] <2> hosts_equal: hostnames DO NOT compare
15:00:04.138 [27641] <2> hosts_equal: addresses DO NOT match

After none of the hostnames are found to be an alias for a different primary so the restore request is rejected because it is neither server-directed nor a same client restore.  The bprd main process does not log the exit status, but the parent (listener) process does.

15:00:04.138 [27641] <2> process_request: client Node.default-domain peername Node1-bk.domain is invalid for restore request
15:00:04.211 [13886] <2> childterm: pid=27641 exit=37, signo=0 core=no

The log details above and solution below can also be applied to identify and resolve related problems:

  Status Code 131: client is not validated to use the server

  Status Code 133: invalid request

  (Also check the global and client attributes for List Restore and/or Restore restrictions.)

  Status Code 135: client is not validated to perform the requested operation

The hostnames presented by the NetBackup database extensions, which support specific cluster configurations, may differ slightly from the sample log above.

Cause

NetBackup allows a client to request the restore of its own images, and also allows a server to direct the restore of the images of one client to a different client. 

If a client is presenting itself using more than one hostname, NetBackup will attempt to see if the hostnames are related/equivalent.  If the requesting and destination clients are not related and the request is not from a server, then the request is denied to prevent sensitive information from either being restored by an inappropriate host or being restored to an inappropriate host.

This rejection occurs as part of authenticating the requesting host.
Once the requesting host is authenticated, then if the requesting client and browse/source client are different then the altnames database will be checked to see if the requesting client is authorized to access the images for the browse/source client.

Solution

This is typically a simple configuration or name resolution issue.

·          On the requesting host, if Required Interface is present, does 'bpclntcmd -hn <required_interface_hostname>' resolve?   If not, the client will substitute <hostname>.<default_domain_name> as the requesting client.   The latter value may not be recognized by the master server.

·          On the requesting host, is the Client Name set correctly?

·          On the requesting host, is the Destination Client set correctly?

·          On the requesting host, is the first Server entry correct and does it resolve to the correct IP address?   If not, the client may try to connect out the wrong network interface to get to the master server.

·          On the requesting host, does the routing table direct connections to the IP for the first Server entry out the correct network interface?

·          On the master server, does the source IP from which the client host sent the request resolve to the correct peername?   This can be checked from the client host using 'bpclntcmd -pn'.   Look at the first hostname and the IP address on the second line of output.

·          On the master server, do the Client Name, Requesting Client, Destination Client, and peername resolve to the correct IP addresses and primary names?

Note:   If the requesting Client Name and Destination Client are not the same, and the peername does not have a Server entry on the master server, then it will also be necessary to authorize the requesting client to have access to the images for the browse/source client; configure the altnames database on the master server to allow the peername access to images for the browse/source client.

If using Exchange over a backup network, see 000032498 in the Related Documents section for additional details.

If the restore is urgent, then following workarounds can be used.

Option 1

Add [temporarily] a Server entry on the master server for the peername to which the master server host resolves the source IP address from which the restore request is received.   Note that while this change is in effect, the peername host has full administrative privileges to the NetBackup master server.

Option 2

On the client host, temporarily change the Client Name to match the destination client name.   That way the request will appear to be a same client restore.   Note that while this change is in effect, it may adversely affect other backup/restore operations that rely on the original value for the Client Name.