NetBackup Java Console hangs during launch at "Login in progress..."

Problem

A newly installed NetBackup Remote Java Administration Console is unable to connect to a NetBackup Master Server.

After entering the proper Host name, User name, Password, and clicking Login, it sits at " Login in progress..." and proceeds no further. 
The hung console window can only be closed by manually terminating the "javaw.exe" process.

Example:



Logging in from other consoles may or may not produce similar behavior.

Error Message

The jbp.xxx.log file produced by the Java Console (located here: C:\Program Files\Veritas\NetBackup\logs\user_ops\nbjlogs) captures the following final few lines:

ServerInterface:setDebugLevel:1441792
BpjavaLoginModule:Setting ServerRequest, debugLevel:1441792
BpjavaLoginModule:connectServer:[VLAB1\administrator][2012-Master.vlab1.com][0]
From Logon, Connecting to vnetd service over PBX port = 1556, authService is true
Acknowledgement from PBX1
vrts.vss.sdk.at.exception.VRTSAtException:
    at vrts.vss.sdk.at.lib.core.Authenticator.atSecConnConnectEx(Native Method)
    at vrts.vss.sdk.at.lib.core.Authenticator.vrtsAtSecConnConnectEx(Authenticator.java:2895)
    at vrts.vss.sdk.at.lib.core.Authenticator.vrtsAtSecConnConnectEx(Authenticator.java:2870)
    at vrts.shared.server.VxATSocket.sslConnect(VxATSocket.java:340)
    at vrts.shared.server.ServerInterface.secureConnect(ServerInterface.java:1393)
    at vrts.shared.server.ServerInterface.connectToServiceViaVNETD(ServerInterface.java:1587)
    at vrts.shared.server.ServerInterface.executeRun(ServerInterface.java:3750)
    at vrts.shared.server.ServerInterface.run(ServerInterface.java:2709)
Acknowledgement from PBX1
Connecting to NBATD at [ 2012-master.vlab1.com!1556!nbatd ] for establishing trust.


The bpjava-msvc log on the Master Server contains the following details:

11:09:44.855 [10428.9444] <2> poll_mainloop: ...........invoking session_dispatch (start a session on the socket) ... with fd = job = 00000000000001B0
...
11:09:44.871 [10428.9444] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:2049] found via getnameinfo OUR_HOST=2012-Master.vlab1.com IPSTR=192.168.2.100
11:09:44.871 [10428.9444] <2> host_file_match: [vnet_vxss.c:2917] match 2012-master.vlab1.com
11:09:44.871 [10428.9444] <2> populateCertificatePath: Certificate to be used for SSL [C:\Program Files\Veritas\NetBackup\var\vxss\credentials\2012-master.vlab1.com]
11:09:44.871 [10428.9444] <2> writetagEx: tag= SECURE_CHANNEL_PROCEED, sockFd = 400, tagged output = nope
11:09:44.871 [10428.9444] <2> writetagEx: tag character based, and limited to 4 chars
11:09:44.871 [10428.9444] <2> writeCountOrDataStartEx: lines = 1 , sockFd = 400
11:09:44.871 [10428.9444] <4> command_SECURE_CHANNEL_INIT: Using certificate [ C:\Program Files\Veritas\NetBackup\var\vxss\credentials\2012-master.vlab1.com] and Responding SECURE_CHANNEL_PROCEED.
...
11:09:44.949 [10428.9444] <16> VssAccept: (../../libVnbat/vss_auth.cpp,2261): vrtsAtSecConnAcceptEx returned

FAILURE
11:09:44.949 [10428.9444] <16> tls_vxss_accept: io.c.3406: VssAccept( ) failed
11:09:44.949 [10428.9444] <16> session_secure_lookup: Unexpected error occurred while establishing TLS session.Possibly this could be a NetBackup Console establishing trust with the broker or User opted not to trust issuer of Machine certificate.This may also occur due to network blip or due to expired or invalid Machine certificate.
11:09:44.949 [10428.9444] <16> session_dispatch: session_secure_lookup FAILED!!! fd =  432
11:09:44.949 [10428.9444] <2> session_forget: fd = 0000000000000190 local_session = 00000000014A66B0
11:09:44.949 [10428.9444] <2> getTokenFromFd: event = FFFFFFFFFFFFFFFF
11:09:44.949 [10428.9444] <2> getTokenFromFd: hUserToken  = FFFFFFFFFFFFFFFF
11:09:44.949 [10428.9444] <16> poll_listen: can't find file descriptor 0000000000000190 in polling table
11:09:44.949 [10428.9444] <2> session_forget: request daemon dropping session/socket 0000000000000190 (0 remaining)
11:09:44.949 [10428.9444] <2> session_forget: pid = -1, this must be an accept (request) socket on a Winodws transient master)
11:09:44.949 [10428.9444] <2> session_forget: DEAD session/socket 00000000000001B0, ShutDownUserServer = -1
11:09:44.949 [10428.9444] <2> KillSessionsJobs: getjobcount = 0
11:09:45.042 [4596.3132] <2> supportFiles: bpjava-msvc compiled on May 19 2016 at 17:11:50, NetBackup 7.7.3, level = 770000
11:09:45.042 [4596.3132] <2> supportFiles: masterDebugPathName = C:\Program Files\Veritas\NetBackup\Logs\bpjava-msvc
11:09:45.042 [4596.3132] <2> supportFiles: debug level is 5

The nbatd log folder on the Master Server is not updated with any log contents.
 

Cause

This sequence of events occurs under the following two conditions:

1. The NetBackup Authentication service is not running on the Master Server (nbatd.exe)
2. The Java Console has never successfully requested an authentication certificate from the Master Server

As with every new installation of the NetBackup Java Administration Console, until it connects to an Authentication Broker (every Master Server is its own Authentication Broker) it has no trusted security certificates present.

Upon the Java Console's first connection to a new Authentication Broker, the broker passes a security certificate containing a "Root Certificate Authority fingerprint" to the console for identify verification purposes.

Observed the first time the Java Console connects to a new Authentication Broker:


This 'handshake' between the Authentication Broker and the Java Console is driven by the NetBackup Authentication service.

The above situation occurs because the Java Console is waiting for the NetBackup Authentication service on the Master Server to pass the required certificate, so it can securely establish a connection.
The reason why some Java Consoles may successfully connect to the Master Server, is that these consoles have already received the required security certificate from the Master Server.
The initial handshake requiring nbatd to pass the certificate from the Master Server to the Java Console is no longer required.  The first-connect handshake sequence is bypassed thereby allowing the Java Console to present the certificate it already has cached, and successfully connect.

Solution:

To resolve this situation, start the NetBackup Authentication service, and then retry the launch of the problematic NetBackup Java Console.