The NetBackup CloudStore Service Container service stops after starting

Problem

After upgrading to NetBackup 8.0, the NetBackup CloudStore Service Container (nbcssc) service shuts down two minutes after starting.

Backups to previously configured cloud storage units fail while the service is stopped.

Error Message

In order to capture more log content around this behavior, it may be necessary to increase logging verbosity in the following way:
1. Turn up the global logging level to 5
2. Modify the cloudstore.conf file located in the following directory:
Windows: <install_path>\Veritas\NetBackup\db\cloud\cloudstore.conf
Unix: /usr/openv/netbackup/db/cloud/cloudstore.conf
Change the line CSSC_LOG_LEVEL=  from a 0 to a 1

3. Start the NetBackup CloudStore Service Container service:
Windows: net start "NetBackup CloudStore Service Container"
Unix: /usr/openv/netbackup/bin/nbcssc -a NetBackup

4. Wait for the service to stop.

5. Check the log files within the following folder:
Windows: <install_path>\Veritas\NetBackup\logs\nbcssc
Unix: /usr/openv/netbackup/logs/nbcssc

There may be several log files within this folder.  Open the log files which were modified since starting nbcssc.

If looking within csconfig.log, the first column has a number surrounded by quotes - example "1".  The number represents the severity of the message were "5" indicates an error.
Review the log for "5" entries.

Example from csconfig.log (produced when running the csconfig command):
"1" CsscSlaveHelper::CsscSlaveHelper_GetCertAndKeyPathsByName Successfully returned machine private key..
"5" CsscSlaveHelper::CsscSlaveHelper_VerifyCertificate Certificate is not valid for current time: /usr/openv/var/vxss/credentials/MyServer.example.com.
"1" CsscSlaveHelper::CsscSlaveHelper_VerifyCertificate Freeing certificate structure.
"1" CsscSlaveHelper::CsscSlaveHelper_VerifyCertificate closing certificate file.
"5" CsscSlaveHelper_GetCertAndKeyPaths failed to get cert and key path.


If looking in MMDDYY.log (windows) or log.MMDDYY (unix), (produced when setting CSSC_LOG_LEVEL=1 in cloudstore.conf), search the time column for the latest startup event.

When working properly, the log should produce the following log messages:
17/05/23 10:55:02 cssc : forming Mongoose arguments
17/05/23 10:55:02 cssc : hostname = ' MyServer'
17/05/23 10:55:02 Cluster No, cluster_name:[N/A]
17/05/23 10:55:02 resulted(clusteraware) HostName:[]
17/05/23 10:55:02 cssc : Get CA cert succeeded at retry 1 (MAX:120), cacertloc : C:\Program Files\Veritas\NetBackup\var\webtruststore\cacert.pem
17/05/23 10:55:02 cssc : using private key file: C:\Program Files\Veritas\NetBackup\var\vxss\credentials\keystore\PrivKeyFile.pem
17/05/23 10:55:02 cssc : using certificate path: C:\Program Files\Veritas\NetBackup\var\vxss\credentials\ MyServer
17/05/23 10:55:02 cssc : using chain certificate path: C:\Program Files\Veritas\NetBackup\var\vxss\credentials\ MyServer

When working improperly, you might see something like this:
17/05/22 09:30:33 cssc : forming Mongoose arguments
17/05/22 09:30:33 cssc : hostname = ' MyServer'
17/05/22 09:30:33 cssc : Key-pair generation successful getting private key...
17/05/22 09:30:34 cssc : Current retry count to check if private key deployment has been successful is 0
17/05/22 09:30:35 cssc : Current retry count to check if private key deployment has been successful is 1
17/05/22 09:30:36 cssc : Current retry count to check if private key deployment has been successful is 2
... <content removed for clarity> ...
17/05/22 09:32:32 cssc : Current retry count to check if private key deployment has been successful is 118
17/05/22 09:32:33 cssc : Current retry count to check if private key deployment has been successful is 119
17/05/22 09:32:33 cssc : Certificate handling failed, aborting

 

Cause

The above has been known to occur when the certificate which nbcssc is attempting to use, has expired and is no longer valid.

The certificate store which nbcssc is attempting to use is located here:
Windows: C:\Program Files\Veritas\NetBackup\var\VxSS\credentials
Unix: /usr/openv/var/vxss/credentials

This folder contains two sorts of certificates, Hostname certificates which have hostnames for filenames, and Host-ID certificates, which have Globally Unique Identifiers (GUIDs) for filenames.

To see if a certificate has expired, run the following command (found in the goodies folder) for each certificate found, and observe the " Not After" date
Windows: C:\Program Files\Veritas\NetBackup\bin\goodies
Unix: /usr/openv/netbackup/bin/goodies

vxsslcmd x509 -in <path to certificate> -text

Example:
C:\Program Files\Veritas\NetBackup\bin\goodies>vxsslcmd x509 -in "C:\Program Files\Veritas\NetBackup\var\VxSS\credentials\MyServer" -text
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            66:4f:fa:c4:00:00:00:0d
    Signature Algorithm: sha512WithRSAEncryption
        Issuer: CN=broker, OU=root@MyServer, O=vx
        Validity
            Not Before: Jan 31 21:59:08 2014 GMT
            Not After : Jan 31 23:14:08 2015 GMT
        Subject: CN=MyServer, OU=NBU_Machines@MyServer, O=vx
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)

If the Not After date is in the past, the certificate is expired and must be renewed.

 

Solution

To renew an expired Hostname certificate, run the following command:
Windows: <install_path>\Veritas\NetBackup\bin\admincmd
Unix: /usr/openv/netbackup/bin/admincmd

bpnbaz -ProvisionCert <hostname>   
Where <hostname> is the name of the certificate file which is expired.

NOTE: The name must be resolvable in order for the bpnbaz command to complete successfully.


To renew an expired Host-ID certificate, run the following command:
Windows: <install_path>\Veritas\NetBackup\bin
Unix: /usr/openv/netbackup/bin

nbcertcmd -renewcertificate


Once done, rerun the vxsslcmd command noted above and verify the "Not After" date has been updated accordingly.

Once the problematic certificate has been identified and updated (so it is no longer expired) the NetBackup CloudStore Service Container service should now start and remain running.