NetBackup 8.0 conflicts with OpsCenter Authentication Service

Article: 100033419
Last Published: 2018-01-23
Ratings: 0 0
Product(s): NetBackup & Alta Data Protection

Problem

The installation or upgrade of a NetBackup 8.x Master Server appears to be successful, but the following message appears on-screen immediately after the installation or upgrade completes:
User-added image


The message reads:
Warning: A NetBackup security certificate could not be deployed
on this host. This will cause failure of certain NetBackup
operations. For details refer to:
https://www.veritas.com/docs/000115775

Subsequent attempts to launch the NetBackup Administration Console fail with the following message:
User-added image


The message reads:
The host <hostname> does not have NetBackup Host
ID-based security certificate installed. The certificate is
mandatory to establish a secure connection.

Click Help for information on how to install the security certificates.


At this point, if you click Help, it will direct you to a page which explains that generating and deploying a Host ID-based certificate is a two step process:
Step 1: Establish a trust with the master server by running:
  • <install_path>\NetBackup\bin\nbcertcmd -getCACertificate
Step 2: Get the certificate by running:
  • <install_path>\NetBackup\bin\nbcertcmd -getCertificate



When you attempt to follow the above two steps, you encounter the following error:
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getCACertificate
nbcertcmd: The -getCACertificate operation failed for server opscenter1.
EXIT STATUS 5969 : Response from the NetBackup Web Management Console service could not be parsed.

C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getCertificate
Request to get the certificate deployment level failed.
EXIT STATUS 5969 : Response from the NetBackup Web Management Console service could not be parsed.

NOTE:  In NetBackup 8.1, the following message may be observed when running 'nbcertcmd -getCertificate'
     EXIT STATUS 8625: Server is unavailable to process the request.  Please try later.

 

Error Message

The NetBackup Install Log captures the following messages during the installation of NetBackup 8.0:
12-17-2016,10:31:46 :  Running command: "C:\Program Files\Veritas\NetBackup\bin\\ nbcertcmd.exe" -getCACertificate -skipCAVerification.
12-17-2016,10:31:46 :  Waiting for command: nbcertcmd.exe
12-17-2016,10:31:47 :  Command produced the following output (will display up to 8192 characters):
12-17-2016,10:31:47 :  -------------------------------------------------------------------------->
12-17-2016,10:31:47 :    Request to get the certificate deployment level failed.
12-17-2016,10:31:47 :    EXIT STATUS 5969: Response from the NetBackup Web Management Console service could not be parsed.
12-17-2016,10:31:47 :  --------------------------------------------------------------------------<
12-17-2016,10:31:47 :  Command returned status 5969.
12-17-2016,10:31:47 :  "C:\Program Files\Veritas\NetBackup\bin\\nbcertcmd.exe" -getCACertificate -skipCAVerification, ERROR: Request for CA certificate failed with error status: 5969.
12-17-2016,10:31:47 :  Attempting to delete install_token...
12-17-2016,10:31:47 :  Successfully deleted install_token.
12-17-2016,10:31:47 :  WARNING: A NetBackup security certificate could not be deployed on this host. This will cause failure of certain NetBackup operations. For details refer to: http://www.veritas.com/docs/000115775


The ...\netbackup\logs\nbcert log produced during install captures the following message during the attempt to execute nbcertcmd:
10:31:46.687 [6872.7052] <2> nbcertcmd: C:\Program Files\Veritas\NetBackup\bin\\ nbcertcmd.exe -getCACertificate -skipCAVerification
10:31:46.687 [6872.7052] <2> CVssApi::initialize: Calling VssInitEx
10:31:46.702 [6872.7052] <2> GetCACertificate: Performing Get CA Certificate...
10:31:46.718 [6872.7052] <2> NBClientCURL::NBClientCURL: Using paths as caCertFilePath [C:\Program Files\Veritas\NetBackup\var\webtruststore\cacert.pem] privateKeyPath [C:\Program Files\Veritas\NetBackup\var\vxss\credentials\keystore\PrivKeyFile.pem]
10:31:46.718 [6872.7052] <2> NBClientCURL::NBClientCURL: Setting default CURL timeout = [1800] seconds
10:31:46.811 [6872.7052] <4> NBClientCURL::performOperation: HTTP response header : HTTP/1.1 500 Internal Server Error Server: Apache-Coyote/1.1 Cache-Control: private Expires: Wed, 31 Dec 1969 19:00:00 EST Content-Type: text/html;charset=utf-8 Content-Language: en <deleted for clarity>
10:31:46.811 [6872.7052] <2> NBClientCURL::performOperation: Fetched data = [xxxxx], httpcode = 500
10:31:46.811 [6872.7052] <2> curlSendRequest: actual http response : 500 expected http result: 200
10:31:46.811 [6872.7052] <16> parse_json_error_response: Unable to load json document: Failed to parse error response received from server
10:31:46.811 [6872.7052] <4> parse_json_error_response: Error payload received from server is :<!DOCTYPE html><html><head><title>Apache Tomcat/8.0.30 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;<deleted for clarity>
10:31:46.811 [6872.7052] <16> checkHttpError: Failed to parse error response from server
10:31:46.811 [6872.7052] <16> nbcert_curl_getcertdeploylevel: Failed to send request for getting certificate deployment level: 5969
10:31:46.811 [6872.7052] <2> NBClientCURL:~NBClientCURL: Performing curl_easy_cleanup()
10:31:46.811 [6872.7052] <2> NBClientCURL:~NBClientCURL: Performing curl_global_cleanup()
10:31:46.811 [6872.7052] <16> GetCACertificate: Request to get the certificate deployment level failed.
10:31:46.811 [6872.7052] <2> nbcertcmd: EXIT STATUS 5969: Response from the NetBackup Web Management Console service could not be parsed.


The following lines are captured in the Java Console log ...\netbackup\logs\user_ops\nbjlogs\jbp.xxx.log upon failing to establish a connection:
CertificateAuthenticationException encountered, The host OpsCenter1 does not have NetBackup Host ID-based security certificate installed. The certificate is mandatory to establish a secure connection.

Click Help for information on how to install the security certificates.
[12/17/16 10:56:35 AM EST {1481990195340}] [-1] [Session] abort: closing all bpjava-sessions
[12/17/16 10:56:35 AM EST {1481990195340}] [-1] [Session] nothing to deinitialize


Upon inspection, there is no GUID based Host-ID certificate located in the folder: <install_path>\NetBackup\var\VxSS\credentials
User-added image

Cause

The cause of the above situation is due to a port conflict between a processes needed to run the 'NetBackup Web Management Console' service and any other process using that port, ie., the 'Symantec NetBackup OpsCenter Authentication Service'.  If a process other than nbwmc is listening on port 3652, nbwmc will fail to start after the upgrade to NetBackup 8.x and certificates will fail to deploy.

For example, upon startup, the 'NetBackup Web Management Console' service starts two processes:

  • nbwmc.exe
  • java.exe
The java.exe process attempts to persistently listen on port 3652.

Example 'netstat -ab' output:
TCP    0.0.0.0:3652           OpsCenter1:0           LISTENING
 [java.exe]


Upon startup, the 'Symantec NetBackup OpsCenter Authentication Service' starts the process ' ops_atd.exe' which persistently listens on port 3652.

Example 'netstat -ab' output:
TCP    0.0.0.0:3652           OpsCenter1:0           LISTENING
 [ops_atd.exe]



Although it was supported to co-locate OpsCenter and NetBackup on the same host, it was never recommended for a variety of reasons.

When upgrading such an environment, it is always recommended to upgrade OpsCenter first.
If NetBackup is upgraded first, OpsCenter will be running during the installation of NetBackup and therefore it will be consuming port 3652.

Solution

Starting in OpsCenter 8.0, the service named "Veritas NetBackup OpsCenter Authentication Service" starts ops_atd.exe, but it no longer listens on port 3652 when OpsCenter is installed newly.

If OpsCenter is running a version earlier than 8.0, and is subsequently upgraded to 8.0, it is likely the "Veritas NetBackup OpsCenter Authentication Service" will continue to listen on port 3652.

It is also possible a 3rd party application is consuming port 3652 and thereby preventing NetBackup 8.x from listening on it after the NetBackup 8.x installation or upgrade.

The solution to this issue is to make port 3652 available to NetBackup upon the startup of NetBackup services.  This will allow the 'NetBackup Web Management Console' to properly setup its listening agent on port 3652.  This may be accomplished by stopping all of the OpsCenter services, if that is the application listening on that port, or, stop what ever third party application that is listening on that port.  Once confirmed that nothing is listening on 3652, restart the NetBackup services.  Confirm NetBackup Web Services (nbwmc) is running, then manually deploy the NetBackup certificates.


Once NetBackup 8.x is running properly, open a CMD prompt on the Master Server and issue the following two commands:


Step 1: Establish a trust with the master server by running:

  • <install_path>\NetBackup\bin\nbcertcmd -getCACertificate
Step 2: Get the certificate by running:
  • <install_path>\NetBackup\bin\nbcertcmd -getCertificate

Once the commands complete successfully, you will observe a GUID based Host-ID certificate located in the folder: <install_path>\NetBackup\var\VxSS\credentials
User-added image


It will now be possible to launch the NetBackup Java Administration Console and successfully connect to the Master Server.


===

It is not recommended to co-locate OpsCenter on a NetBackup Master Server.  Reference the section in the Veritas NetBackup OpsCenter Administrator's Guide which talks about Moving OpsCenter server to a different computer

 

Was this content helpful?