Instant access restore of virtual machine from WEB UI is failing with error "Failed to get version from the NetBackup deduplication web server"

Problem

When trying to restore a virtual machine via instant access at WEB UI, the WEB UI is showing error "Failed to get version from the NetBackup deduplication web server".

This is only applicable when appliance server(acting as backup host) is at  NBU version 8.1.2

Error Message

OID 495 logs at master server will show error:-

4/23/2020 07:17:06.303 [Debug] NB 51216 nbwebapi 495 PID:2332 TID:103 File ID:495 [No context] 1 [com.netbackup.apiversioning.annotation.VersionRangeRequestMappingHandlerMapping] Returning handler method [public org.springframework.http.ResponseEntity<java.lang.String> com.netbackup.config.hosts.controller.HostsController.getHostDetails(java.lang.String) throws com.netbackup.config.hosts.exception.UUIDDoesNotExistServiceException,com.netbackup.config.hosts.exception.HostDBUnknownErrorException,com.netbackup.config.hosts.exception.UnauthorizedHostException]
4/23/2020 07:17:06.412 [Application] NB 51216 nbwebapi 495 PID:2332 TID:94 File ID:495 [No context] [Error] :saveEntry() - hash for CA Cert from https://<master_server>:443/msdp/1.0/ping did not match provided previously provided hash
4/23/2020 07:17:06.412 [Application] NB 51216 nbwebapi 495 PID:2332 TID:94 File ID:495 [No context] [Error]      saved hash:    e4ff7f108cb1939882a565acf64bad4414418310ab13c5e019bfa46ac1u74ccf1ba8dbb0926af918f96d7ea42ae4de0c40a8133faad6fba0ae8f8fdb564a1714
4/23/2020 07:17:06.412 [Application] NB 51216 nbwebapi 495 PID:2332 TID:94 File ID:495 [No context] [Error]      computed hash: 7d31384c812da710653633506d604afabf73b369207a502148d03192d402edd7be54d5e2730deff88893cfca8052cf7f39f1c58c2ca6b4aa146cec0104394d16
4/23/2020 07:17:06.506 [Application] NB 51216 nbwebapi 495 PID:2332 TID:94 File ID:495 [No context] [Error] org.springframework.web.util.NestedServletException: Request processing failed; nested exception is com.netbackup.config.exception.MSDPCertInternalErrorException: The provided SSL certificate hash did not match the SSL certificate presented by the NetBackup deduplication web server.

Certificate path validation will show  exception in OID 495 logs at master server

4/23/2020 07:11:06.911 [Application] NB 51216 nbwebapi 495 PID:2332 TID:119 File ID:495 [No context] [Error] I/O error on GET request for "https://<master_server>:443/msdp/version": sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors; nested exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
4/23/2020 07:11:06.911 [Application] NB 51216 nbwebapi 495 PID:2332 TID:119 File ID:495 [No context] [Error] com.netbackup.recovery.exception.MsdpxException: errorCode: 4032

Below command at appliance media server will fail due with error "Could not access <API call>"

<Media_Server>:/home/maintenance # /usr/openv/pdde/vpfs/bin/nb_admin_tasks --put_nba_self_CA_cert                             
Starting new HTTPS connection (1): <master_server>
Accessing https://<master_server>:1556/netbackup/config/servers/msdp-servers/beb67e13-26a8-4ffc-bfo6-cc36bb817650 failed (500):
Could not perform the requested action; ('Could not access https://<master_server>.ho.pbcom.int:1556/netbackup/config/servers/msdp-servers/beb67e13-26a8-4ffc-bfo6-cc36bb817650', {u'errorCode': 130, u'fileUploadErrors': [], u'attributeErrors': {}, u'errorMessage': u'system error occurred', u'errorDetails': []})

API access through wget at appliance media server will also fail due to authorization failed error.

<Media_Server>:/home/maintenance # wget https://<master_server>:1556/netbackup/config/servers/msdp-servers/beb67e13-26a8-4ffc-bfo6-cc36bb817650 --no-check-certificate

Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 401
Authorization failed.

/var/log/vpfs/nb_admin_tasks.log file at appliance media server will show error

2020-04-22 13:33:34,560     INFO [227198] 735 Starting new HTTPS connection (1): <master_server>
2020-04-22 13:33:35,460    ERROR [227198] 60 Accessing https://<master_server>.int:1556/netbackup/config/servers/msdp-servers/beb67e13-26a8-4ffc-bfo6-cc36bb817650 failed (500): <Response [500]>
2020-04-22 13:33:35,460    ERROR [227198] 78 Could not perform the requested action; ('Could not access https://<master_server>:1556/netbackup/config/servers/msdp-servers/beb67e13-26a8-4ffc-bfo6-cc36bb817650', {u'errorCode': 130, u'fileUploadErrors': [], u'attributeErrors': {}, u'errorMessage': u'system error occurred', u'errorDetails': []})

Cause

Mismatch in the hash value  of the  appliance certificate available to master server and actual hash value is causing this error.

Solution

1)Please first verify if  appliance is using 3rd party certificate for appliance WEB UI.If using 3rd party certificate for appliance WEB UI ,please DO NOT perform this steps.This article is not applicable in this scenario.

You can verify if it is 3rd party certificate by checking

->HTTPS connection will be “trusted” in the browser while accessing appliance WEB UI.
->Check the certificate in the browser by using the respective browser’s functions to "view a certificate" to further confirm it is the expected certificate(not 3rd party).

2)Stop below services at appliance media server

Support > InfraServices > Stop Database

Support > InfraServices > Stop MessageQueue

Support > InfraServices > Stop Webserver

service as-alertmanager stop

service as-analyzer stop

service as-transmission stop

service as-collector stop

systemctl stop nginx

3)Backup the existing web server KeyStore file using the following command

cp /opt/apache-tomcat/security/keystore /opt/apache-tomcat/security/keystore.orig

4)Rename the vxos-ssl hostname self cert and keycert files (there are localhost cert and its keycert files, do not rename them):

# cd /etc/vxos-ssl/servers/certs
# mv  <media_server_FQDN>-self.cert.pem  <media_server_FQDN>.int-self.cert.orig
# mv <media_server_FQDN>-self.keycert.pem <media_server_FQDN>-self.keycert.orig

5)Run the 'cacert-setup.sh' script with the following syntax. This script will take care of creating a new tomcat keystore, as well as new cert and keycert files.

#/etc/vxos-ssl/bin/cacert-setup.sh -c server -i server -s <media_server_FQDN>

6) Once run, verify that there is a new keystore under /opt/apache-tomcat/security, and new hostname cert and keycert files under /etc/vxos-ssl/servers/certs

7) Start all services:

Support > InfraServices > Start Database

Support > InfraServices > Start MessageQueue

Support > InfraServices > Start Webserver

service as-alertmanager start

service as-analyzer start

service as-transmission start

service as-collector start

systemctl start nginx

8)Verify that instant access service is running by executing below command at CLISH

Support -> Test -> Software

9)Try to push the certificate to master server again from appliance

/usr/openv/pdde/vpfs/bin/nb_admin_tasks --put_nba_self_CA_cert

10)If above command is successful,then try to run instant access restore from Netbackup WEB UI and error "Failed to get version from the NetBackup deduplication web server" should not appear anymore.