Flex Primary Instance Upgrade Fails Due to 'su nbsvcusr' Error: 'Cannot Open Session: Permission Denied'

Problem

The upgrade of the Flex Primary Instance from NetBackup (NBU) version 10.2.0.1 to 10.5 is failing. The issue occurs during the upgrade process when the system attempts to switch to the service user nbsvcusr The failure prevents the upgrade from being completed successfully.

Error Message

During the upgrade process, the following error messages are observed in the setup-server or install log files:

DATABASE_USER 'nbsvcusr' saved in bp.conf.
ServiceUser 'nbsvcusr' saved in VRTSatlocal.conf.
ServiceUser 'nbsvcusr' saved in VRTSaz.conf.
VXDBMS_NB_PGCONF '/usr/openv/db/data/instance' saved in bp.conf.
su: cannot open session: Permission denied
..
ERROR: pg_initdb_upgrade: Failed to perform initdb.
..
ERROR: The upgrade of Postgresql instance failed. Rerun [2025-05-08 01:03:14] /usr/openv/netbackup/bin/install_bp when the [2025-05-08 01:03:14] problem has been resolved.

Additionally, if you manually attempt to switch to the nbsvcusr user using the su command, the following error is displayed:

su nbsvcusr
su: cannot open session: Permission denied

This indicates that the su command is unable to open a session for the nbsvcusr user, which is critical for the upgrade process.

Cause

The cause of the issue lies in the system's configuration for the nbsvcusr user. Specifically, the su command fails because of restrictive settings in the /etc/security/limits.conf file. This file contains resource limits for users, and certain entries for the nbsvcusr user, particularly related to memlock, are causing the session to fail.

The problematic entries in the /etc/security/limits.conf file are as follows:

nbsvcusr hard memlock 4194304
nbsvcusr soft memlock 4194304

These settings restrict the memory locking limits for the nbsvcusr user, which prevents the su command from successfully opening a session for this user. As a result, the upgrade process is unable to proceed.

Solution

To resolve the issue, follow the steps below:

Step 1: Roll Back the Failed Upgrade
   Roll back the failed upgrade to ensure that NetBackup is functioning correctly on the older version (10.2.0.1).
   Verify that all services are operational and that there are no residual issues from the failed upgrade.
Step 2: Backup the /etc/security/limits.conf File
   Create a backup of the /etc/security/limits.conf file to a safe location. This ensures that you can restore the original configuration if needed.
     cp /etc/security/limits.conf /etc/security/limits.conf.bak
Step 3: Modify the /etc/security/limits.conf File
   Open the /etc/security/limits.conf file in a text editor (e.g., vi or nano).
     vi /etc/security/limits.conf
   Locate the entries for the nbsvcusr user related to memlock. They will look similar to the following:
      nbsvcusr hard memlock 4194304
   nbsvcusr soft memlock 4194304

   Comment out or remove these lines.
   For example:
      # nbsvcusr hard memlock 4194304
   # nbsvcusr soft memlock 4194304
   Save and close the file.
Step 4: Test the su Command for nbsvcusr
   Attempt to manually switch to the nbsvcusr user using the su command:
     su nbsvcusr
   If the command works without errors, the issue has been resolved.
Step 5: Retry the Upgrade
   Restart the upgrade process for NetBackup from version 10.2.0.1 to 10.5.
   Monitor the logs to ensure that the upgrade is completed successfully.