Unable to access web services due to MongoDB database errors in Veritas 3340 appliances

Problem

The MongoDB service is not running.

Error Message

The following error message is displayed:

Unable to authenticate user for web service access. An internal error occurred. See the Troubleshooting Guide for detailed solutions.

The following errors are written to the log files: 
Error: couldn't connect to server localhost:27017, connection attempt failed : connect@src/mongo/shell/mongo.js:231:14

Cause

Some of the reasons for the failure are listed below:

• The cfgvol volume or the ltrappcfg disk group is not created.
• VCS is not running.
• The service group appdb_svc and appdb_data are not online.

Solution

Resolving MongoDB errors when the nodes are in a cluster

If the cluster is set up, use the following methods to identify the issue:

Verifying the ltrappcfg disk group and the cfgvol volume

Use the vxprint command to verify if the Disk Group (DG) ltrappcfg is present and enabled. Volume cfgvol should be created in DG ltrappcfg:

nbapp892:~ # vxprint
Disk group: ltrappcfg

TY  NAME       ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg  ltrappcfg    ltrappcfg       -   -        -        -        -       -

dm vrts_appliances0_4 vrts_appliances0_4 - 2147417808 - -        -       -

v  cfgvol       fsgen        ENABLED  2147417808 -      ACTIVE   -       -
pl cfgvol-01    cfgvol       ENABLED  2147417808 -      ACTIVE   -       -
sd vrts_appliances0_4-01 cfgvol-01 ENABLED 2147417808 0  -        -       -

If the DG and the volume are not correct, complete the following steps:

1. If the cfgvol volume is created, verify if it is mounted to /config .
   hroaccess_01:/home/maintenance # mount | grep /config
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/system-config on /config type ext4 (rw,relatime,stripe=16,data=ordered)
/dev/vx/dsk/ltrappcfg/cfgvol on /config type vxfs(rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)
2. If the cfgvol volume or the ltrappcfg disk group is not created, the  Cluster > Configure command fails. In such situations, there is no way to manually recover from this issue, and you need to investigate the logs to identify the issue and reconfigure the cluster.
   • Collect the logs using vxlogview -p 493 -o 940 or use the debug log files located in the /log/loghandler/vxul_python.log* directory.
   • Use the Manage > Storage > Show Disk command from the appliance shell menu to verify the configuration disk status, and then use the vxdisk list command to collect the properties.

     hroaccess_01.Storage> Show Disk
- [Info] Retrieving storage disk information. This can take up to five minutes.

     Note:
- System disk is an internal disk that belongs to the node you are currently on.
- Configuration, Data, and Fencing disks are shared between all nodes of a cluster.

     ------------------------------------------------------------
Disk Name                        | Type          | Total
------------------------------------------------------------
600605B00C9D03E022232EFF341FE1A8 | System        |   1.82 TB
vrts_appliances0_0               | Data          | 127.21 TB
vrts_appliances0_1               | Configuration |      1 TB
vrts_appliances0_2               | Fencing       |    128 MB
vrts_appliances0_3               | Data          | 126.21 TB
…
hroaccess_01:/opt/IMAppliance/lib/ltrappclus # vxdisk list vrts_appliances0_1
Device:    vrts_appliances0_1
devicetag: vrts_appliances0_1
type:      auto
clusterid: hroaccess
disk:      name=vrts_appliances0_1 id=1549915977.34.localhost.localdomain
group:     name=ltrappcfg id=1549915977.36.localhost.localdomain
info:      format=cdsdisk,privoffset=208,pubslice=3,privslice=3
flags:     online ready private autoconfig shared autoimport imported

Verifying Veritas Cluster Service (VCS) status

Use the hastatus command to verify if VCS is running on both the nodes. If VCS is not running, enable VCS by referring to the Access software documentation.

# hastatus –summary
-- SYSTEM STATE
-- System               State                Frozen

A  nbapp892             RUNNING              0
A  nbapp893             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  CIFS            nbapp892             Y          N               PARTIAL
B  CIFS            nbapp893             Y          N               PARTIAL

Verifying the appdb_svc and appdb_data service groups

The appdb_data service group must be online on both the nodes. The appdb_svc service group must be online on one of the nodes and offline on the other node.

nbapp892:~ # hastatus -summary
-- SYSTEM STATE
-- System               State                Frozen

A  nbapp892             RUNNING              0
A  nbapp893             RUNNING              0
...
B  appdb_data      nbapp892             Y          N               ONLINE
B  appdb_data      nbapp893             Y          N               ONLINE
B  appdb_svc       nbapp892             Y          N               ONLINE
B  appdb_svc       nbapp893             Y          N               OFFLINE

If the status is not as displayed above, use the hagrp command to change the status of the service group. For example, 

# hagrp -online appdb_data -sys nbapp892
# hagrp -online appdb_data -sys nbapp893
# hagrp -online appdb_svc -sys nbapp892

If there is no appdb_data or appdb_svc service group, verify the VCS configuration file: 
/etc/VRTSvcs/conf/config/main.cf

In this file, check the group appdb_svc and group appdb_data sections. If there are no such groups, it implies that the appliance was not set up correctly and there is no straightforward method to recover from this situation. 

The related logs are in the /log/ and the /var/VRTSvcs/log/ directories.

Restarting related services

If the MongoDB error is resolved, restart the Tomcat server on both nodes:
nbapp892:~ # /opt/IMAppliance/scripts/infraservices.sh webserver stop
nbapp892:~ # /opt/IMAppliance/scripts/infraservices.sh webserver stop

Resolving MongoDB errors when nodes are not in a cluster

Verifying MongoDB status

access-appliance:~ # /opt/IMAppliance/scripts/infraservices.sh database status 

Restarting MongoDB

If MongoDB is not running, try to restart it by running the following command on both the nodes:
access-appliance:~ # /opt/IMAppliance/scripts/infraservices.sh database stop
access-appliance:~ # /opt/IMAppliance/scripts/infraservices.sh database start

Gathering information if MongoDB fails to restart

1. If MongoDB fails to start due to some errors, run the following commands and save the output from both the nodes: 
   access-appliance:~ # vxdisk list
access-appliance:~ # df -h
access-appliance:~ # ls -l /config
2. Save the following files on both the nodes:
   /etc/mongod.conf
/config/dblocation.conf
/config/services.conf
3. Collect all the logs from the /log directory on both the nodes.
4. Run the following commands to save the output of the last command that was run on both the nodes:
   access-appliance:~ # source /opt/IMAppliance/scripts/mongo_utils.sh
access-appliance:~ # get_mongodb_password appliancedbadmin
access-appliance:~ # $MONGO_CLIENT -u appliancedbadmin -p $pwd appliancedb