Netbackup Snapshot Manager - Unable to issue REST APIs on Snapshot manager

Problem

Not able to issue NBSM REST APIs as the Nginx container fails to resolve the flexsnap-api-gateway ip address and goes in unhealthy state.

Error Message

Below error message seen in flexsnap.log file

025/06/04 07:07:19 [warn] 25#25: *3 upstream server temporarily disabled while connecting to upstream, client: ::1, server: xx.yy.zz.com, request: "GET /cloudpoint/api HTTP/1.1", upstream: "https://xx.xx.yy.zz/cloudpoint/api";, host: "localhost"

Cause

Nginx service caches the IP of the backend API gateway and uses it to send requests to the API gateway. When the backend API gateway is restarted ( flexsnap-api-gateway), the container engine assigns a new IP on the overlay network due to which Nginx service is no longer able to communicate with the API gateway that is now listening on new IP address while former is sending requests on the old IP address.

Solution

A SIGHUP signal is all that is required for nginx to reconfigure its backend configuration.

podman restart flexsnap-nginx

To verify Ip address for the container:

   podman inspect flexsnap-api-gateway | grep -i ipadd

   "IPAddress": "",
                         "IPAddress": "xx.xx.xx.xx",

The ip address from the above command would be different from the ip address from nginx logs.

Verify if  flexsnap-api-gateway is reachable from the Nginx container using an updated ip address.

1. Exec into flexsnap-nginx container using

      podman exec -it flexsnap-nginx bash

2. Run below command to verify flexsnap-api-gateway is reachable from the Nginx container

      curl -k https://{gateway_new_ip}:8472/cloudpoint/docs