Netbackup Cloud Storage Service Container (nbcssc) not running on the Master server

Problem

Netbackup Cloud Storage Service Container (nbcssc) not running on the Master server.

Error Message

If nbcssc service will not run then following things will be impacted:

- Cloud configuration from the netbackup wizard with give error:
  "Could not receive response from Netbackp CloudStore Service Container (nbcssc). Refer to the nbcssc logs for more details"
- Commands "csconfig cldprovider -i" and "csconfig cldprovider -i" will fail with below error:
       'Failed to fetch Cloud Providers.'
- Telnet on port 5637 will fail.
 

Cause

Getting version mismatch for CloudProvider.xml in nbcssc logs.

nbcssc log: 

CldxmlController::instance Enter
CS_CheckMaster cssc : Checking if this is master server while starting nbcssc. Received=master.abc.com, NBU-
server=master.abc.com
CldxmlController::instance Initializing cloud xml
CldxmlController::instance Exception initializing cloud management: CldMgmt::CldMgmt:Version mismatch, Cloudstore version is 2.4.0 and XML version is 2.3.0
Exception initializing cldmgmt: CldMgmt::CldMgmt:Version mismatch, Cloudstore version is 2.x.x and XML version is 2.x.x
(null) cssc : Failed in cloud xml initialization due to not being able to load cloud provider details.
CS_CldSwiftInit : Enter
CS_CldSwiftInit : Exit
CSSC_Init cssc : swift intialization succesful, running in master mode!!
CSSC_Init cssc : Error during cloud management initialization, status 11. Exiting.

Solution

To troubleshooting this issue:

1. Verify hostname and host id certificates are in place.
  To verify check:

  Windows:
     Install-path\NetBackup\Var\VxSS\Credentials
   Linux /Unix:
     /usr/openv/var/vxss/credentials 

If hostname certificate is missing then create it with

/usr/openv/netbackup/bin/admincmd/

Install-path\netbackup\bin\admincmd\

bpnbaz -provisioncert <hostname>

2. Verify all the following files are in place under //install_path/Netbackup/db/cloud:
   cacert.pem
   CloudInstance.xml
   CloudPrvider.xml
   cloudstore.conf
   libstspithrottling
   
3. Verify the correct Master server name in cloudstore.conf

4. Try to start nbcssc manually by running:
nbcssc -s -a NetBackup -f -m master_name (directly on Master server)

On Media server:

bpnbaz -provisioncert media_server_name
nbcssc -s -a NetBackup -f -m master_name 

5. Enable the logs in cloudstore.conf, CSSC_LOG_LEVEL=1 (extra logging)

Note: Don't check the nbcssc logs at first place. First troubleshoot why nbcssc is not running.

6. Downloaded the updated XML and cacert.pem files from:
    https://www.veritas.com/support/pt_BR/downloads/update.UPD971796

7. Stopped the NBU services, replaced the files and started services again.

8. Verify nbcssc should be running.