Veritas Operations Manager "xprtld" doesn't startup

Problem

After installing VRTSsfmh, the xprtld process will not start on AIX.

A lslpp confirms VRTSsfmh is installed and committed

# lslpp -l | grep VRTSsf
 VRTSsfmh                 3.0.357.0  COMMITTED  Veritas Storage Foundation
 VRTSsfmh                 3.0.357.0  COMMITTED  Veritas Storage Foundation

B theutre is no xprtld seen from a ps output

# ps -ef | grep xprtld
    root 1015956       1   0 06:58:50      -  0:00 /opt/VRTSsfmh/bin/xprtld -X 1 /etc/opt/VRTSsfmh/xprtld.conf
    root 1147066  229490   0 06:59:41  pts/0  0:00 grep xprtld

Error Message

# /opt/VRTSsfmh/bin/xprtld /etc/opt/VRTSsfmh/xprtld.conf
Cannot get credential for service

A truss from the above command shows:

688378: 1527845: 0.3753:        statx("/var/VRTSat/.VRTSat/profile/certstore/keystore/KeyStore.lock", 0x2FF1FEF0, 76, 0) = 0
688378: 1527845: 0.3754:        kfcntl(12, F_SETLK, 0x2FF1FFB0) = 0
688378: 1527845: kwrite(2, " C a n n o t   g e t   c".., 33) = 33
688378: 1527845: kwrite(2, "\n", 1)                  = 1

Cause

The VRTSat Authentication debugging confirms there is an issue with the credentials:

<Date_time>:50826,18,56,778488,1,debug,EAT2,1:      Error data: Expecting: RSA PUBLIC KEY
<Date_time>:50826,18,65,778488,1,debug,EAT2,1:      Error data: Unable to read pub key
<Date_time>:50826,18,83,778488,1,debug,EAT2,1:      Error data: Expecting: ANY PRIVATE KEY
<Date_time>:50826,18,92,778488,1,debug,EAT2,1:      Error data: Unable to read priv key

Solution

1. Delete files under /var/VRTSat_lhc/
# rm /var/VRTSat_lhc/*

2. Move creds under /var/VRTSat/.VRTSat/profile/certstore/
# mkdir /tmp/backupcreds
# mv /var/VRTSat/.VRTSat/profile/certstore/*.0 /tmp/backupcreds/.

3. Delete pem keys
# rm /var/VRTSat/.VRTSat/profile/certstore/keystore/PubKeyFile.pem
# rm /var/VRTSat/.VRTSat/profile/certstore/keystore/PrivKeyFile.pem

4. Restart any shared broker process if running (/opt/VRTSat/bin/vxatd) (not running..so not required)

5. Do a localhost authentication to recreate the pem files that are deleted in step #4.
# export EAT_HOME_DIR=/opt/VRTSsfmh
# export EAT_DATA_DIR=/var/opt/VRTSsfmh/sec/root
# cd $EAT_HOME_DIR/bin
# ./vssat authenticate -d localhost

6. Start xprtld
# /opt/VRTSsfmh/adm/xprtldctrl start
 

NOTE: Do not use this article when /opt/VRTSsfmh/bin/xprtlc  -l https://localhost/world/getvitals  returns correct information. Only proceed further when /opt/VRTSsfmh/bin/xprtlc  -l https://fqdn.com/world/getvitals is not working