After upgrade to NetBackup 7.7.2 or 7.7.3 unable to login to Java console as non-root AD user

After upgrade to NetBackup 7.7.2 or 7.7.3 unable to login to Java console as non-root AD user

Article: 100032684
Last Published: 2016-07-06
Ratings: 0 0
Product(s): NetBackup

Problem

After upgrade to NetBackup 7.7.2 or 7.7.3 unable to login to the Java console as non-root Active Directory (AD) user.
AD software being used is pbis (PowerBroker Open) PowerBroker Identity Services from BeyondTrust.

Getting status 511 NB-Java application server interface error when login fails.

Login to Java console works when switching to a local non-root user.

Authentication from the vssat command line on the master works fine -
# /usr/openv/netbackup/sec/at/bin/vssat authenticate -d pam -p <AD username> -b localhost:13783
Using data dir: /usr/openv/var/vxss/at
Password:

authenticate
----------------------
----------------------

Authenticated User      <AD username>

----------------------




 

Error Message

Error seen at the Java console when logging in:
Unable to login; Status: 511 NB-Java application server interface error

Error in the syslog file (/ var/log/messages):
Feb  8 17:59:59 <master hostname> lsass: [lsass] Failed to authenticate user (name =
'<AD username>') -> error = 40047, symbol = LW_ERROR_KRB5_CALL_FAILED, client pid = 13077


For further troubleshooting enable core dumps in the environment by following the procedure in the Related Articles.

Once the core is received, from the stack trace it looks like the function, getpwuid_r, used for getting the name from effective user id is leading into further calls that cause the crash.

(gdb) where
#0  0x00007fe30f317989 in raise () from /lib64/libc.so.6
#1  0x00007fe30f319098 in abort () from /lib64/libc.so.6
#2  0x00007fe3054d042e in lwmsg_connection_sendmsg (assoc=0x105b940, fd=4,
msghdr=0x7fff29cd4e30, flags=0, out_sent=0x7fff29cd4a08)
    at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-wire.c:325
#3  0x00007fe3054d0570 in lwmsg_connection_send_fragment (assoc=0x105b940,
fragment=0x105db10) at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-wire.c:394
#4  0x00007fe3054d0b5d in lwmsg_connection_send_all_fragments (assoc=0x105b940)
at /builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-wire.c:711
#5  0x00007fe3054d0a00 in lwmsg_connection_send_wrap (buffer=0x7fff29cd50c0,
needed=0) at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-wire.c:632
#6  0x00007fe3054d4ee5 in lwmsg_data_marshal (context=0x1019b80,
type=0x7fe30538e720 <gLsa2IpcFindObjectsReqSpec>, object=0x7fff29cd5260,
buffer=0x7fff29cd50c0)
    at /builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/data-marshal.c:728
#7  0x00007fe3054d1abd in lwmsg_connection_begin_send_message (assoc=0x105b940)
at /builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-wire.c:1529
#8  0x00007fe3054cfab0 in lwmsg_connection_state_established (assoc=0x105b940,
state=0x620b, event=0xffffffffffffffff)
    at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-state.c:776
#9  0x00007fe3054cf515 in lwmsg_connection_run (assoc=0x105b940,
event=CONNECTION_EVENT_NONE) at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection-state.c:248
#10 0x00007fe3054ce9b5 in lwmsg_connection_send_msg (assoc=0x620b,
message=0x620b) at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/connection.c:151
#11 0x00007fe3054ccf1b in lwmsg_assoc_send_message (assoc=0x620b,
message=0x620b) at
/builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/assoc.c:141
#12 0x00007fe3054ce1dd in lwmsg_assoc_call_dispatch (call=0x620b, in=0x620b,
out=0x7fff29cd5240, complete=0x7fe31438c740, data=0x7fe31438c740)
    at /builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/assoc-call.c:81
#13 0x00007fe3054ce57a in lwmsg_call_dispatch (call=0x620b, in=0x620b, out=0x6,
complete=0xffffffffffffffff, data=0x7fe31438c740)
    at /builder/src-buildserver/Platform-8.3/src/linux/lwmsg/src/call.c:52
#14 0x00007fe3055f14d7 in LsaTransactFindObjects (hLsa=0x8,
pszTargetProvider=0x0, FindFlags=0, ObjectType=2 '\002', QueryType=6 '\006',
dwCount=1, QueryList=...,
    pppObjects=0x7fff29cd5308) at
/builder/src-buildserver/Platform-8.3/src/linux/lsass/client/api/clientipc.c:901
#15 0x00007fe3055fbf9c in LsaFindObjects (hLsa=0x620b, pszTargetProvider=0x620b
<Address 0x620b out of bounds>, FindFlags=6, ObjectType=255 '\377', QueryType=64
'@', dwCount=79,
    QueryList=..., pppObjects=0x0) at
/builder/src-buildserver/Platform-8.3/src/linux/lsass/client/api/api2.c:55
#16 0x00007fe3055f77c0 in LsaFindUserById (hLsaConnection=0x1015570, uid=25099,
dwUserInfoLevel=0, ppUserInfo=0x7fff29cd5350)
    at /builder/src-buildserver/Platform-8.3/src/linux/lsass/client/api/users.c:227
#17 0x00007fe3017f87c2 in LsaNssCommonPasswdGetpwuid (pConnection=0x7fe3018fa280
<lsaConnection>, uid=736101502, pResultUser=0x7fff29cd5450, pszBuf=0xfbd788
"testuser", bufLen=1024,
    pErrorNumber=0x7fe31438c6a0) at
/builder/src-buildserver/Platform-8.3/src/linux/lsass/interop/nsswitch/common/nss-user.c:482
#18 0x00007fe3017f7a42 in _nss_lsass_getpwuid_r (uid=736101502,
pResultUser=0x7fff29cd5450, pszBuf=0xfbd788 "testuser", bufLen=1024,
pErrorNumber=0x7fe31438c6a0)
    at
/builder/src-buildserver/Platform-8.3/src/linux/lsass/interop/nsswitch/linux/nss-user.c:153
#19 0x00007fe30f39ec9c in getpwuid_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#20 0x00007fe30dff1c1a in at_get_os_userinfo () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#21 0x00007fe30dffc412 in utils_get_effective_user_name () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#22 0x00007fe30dfeef4d in api_library_init () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#23 0x00007fe30df95a4d in library_init () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#24 0x00007fe30df9b466 in at_client_api_init () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#25 0x00007fe30df9bb95 in vrtsAtInitEx () from
/usr/openv/netbackup/sec/at/lib/libvrtsat.so
#26 0x00007fe313c9f780 in vrtsAtInit_S () from /usr/openv/lib/libnbbaseST.so
#27 0x00007fe313ca82fd in vrtsAtInitEx () from /usr/openv/lib/libnbbaseST.so
#28 0x00007fe313c21867 in (anonymous namespace)::initializeATHandle(void**, char
const*, int*) () from /usr/openv/lib/libnbbaseST.so
#29 0x00007fe313c2dabc in (anonymous namespace)::VssAtManager::initManager() ()
from /usr/openv/lib/libnbbaseST.so
#30 0x00007fe313c2df3a in VssZap () from /usr/openv/lib/libnbbaseST.so
#31 0x000000000040d31a in createCredentialEx ()
#32 0x000000000040d535 in establishAuthorization ()
#33 0x0000000000420d40 in child_exec ()
#34 0x000000000041b759 in command_LOGON_TO_MSERVER ()
#35 0x000000000041de32 in command_exec ()
#36 0x0000000000413760 in session_dispatch ()
#37 0x0000000000410347 in poll_mainloop ()
#38 0x00000000004089a8 in main ()
 
 

Cause

strace for the bpjava-msvc process showed that pbis (Power Broker Open) PowerBroker Identity Services was re-using some the of the file descriptors used by bpjava, in particular the same FD was earlier being used to read /etc/passwd file.

The following is a sequence of the events:
1. bpjava-msvc during initialization creates VxAT (related to security infrastructure for ssl) handle. This handle in-turn opens some file descriptors (fds).

2. bpjava prepares to fork and run the process under login user. Just before it closes all open fds except a few that it needs.
   This causes the fds opened during initialization of VxAT to be closed. The VxAT handle, however has info about what handles it opened.

3. Forked process calls makes some calls which cause pbis to acquire some fds. The same fd that was earlier allocated to VxAT gets allocated to pbis.

4. VxAT termination gets called which closes the fd which was allocated to it in step 1, however that fd is currently being used by pbis.

5. Some further events cause pbis to read/write with the fd it opened. Since that fd was closed during VxAT termination, pbis crashes.

Solution

Veritas Technologies LLC has acknowledged that the above-mentioned issue (ETrack 3868503) is present in the current version(s) listed under the Product(s) Section of this article. Veritas Technologies LLC is committed to product quality and satisfied customers.
 
There are no plans to address this issue by way of a patch or hotfix in the current or previous versions of the software at the present time. However, the issue is currently scheduled to be addressed in the next major revision of the product. Please be sure to refer back to this document periodically as any changes to the status of the defect will be reflected here.  Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests.  Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.
 
Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.  For information on how to contact Veritas Sales, please see   https://www.veritas.com

Workaround:
Modify the order of lookup for password and group in /etc/nsswitch.conf file, so Active Directory lookup is done first.

# vi /etc/nsswitch.conf

Change the following lines -
passwd:     files lsass
group:      files lsass

to list "lsass" first, then "files" -
passwd:     lsass files
group:      lsass files


Then attempt to login in the Java console using the AD user id.

References

Etrack : 3868503

Was this content helpful?