NetBackup 10.2 nbwmc does not start if timezone parameters are missing from postgres configuration

Problem

After upgrading to 10.2 nbwmc does not start and nbcertcmd -ping confirms nbwmc is not responding

nbcertcmd: The -ping operation failed.
EXIT STATUS 26: client/server handshaking failed

Additionally, the service "NetBackup Scale-Out Relational Database Manager" (pg_ctl.exe) does not start.

And the service named "NetBackup Scale-Out Relational Database Connection Pool Service" (pgbouncer.exe) is in a failing startup loop.  The process is active, but does not start properly.

Error Message

The NetBackup Web Management Console (nbwmc) service attempts to authenticate to the NetBackup  Postgres database (pg_ctl.exe), but the service is not up for the connection to succeed.

catalina.<date>log shows the following startup error 

19-Apr-2023 17:34:51.613 SEVERE [Catalina-utility-5] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener] org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessControlClientRegistry': Unsatisfied dependency expressed through field 'accessControlClientFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultClientFactory': Unsatisfied dependency expressed through field 'accessCacheService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessCacheService': Unsatisfied dependency expressed through field 'accessCacheFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultAccessCacheFactory': Unsatisfied dependency expressed through field 'managedObjectCacheLoader'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseViewManagedObjectCacheLoader': Unsatisfied dependency expressed through field 'operationCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'delegateOperationCache': Invocation of init method failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
...
Caused by: org.postgresql.util.PSQLException: FATAL: bouncer config error

The service named "NetBackup Scale-Out Relational Database Connection Pool Service", which is pgbouncer.exe under the covers, logs into the OS Application Event Log which shows the following errors upon startup

TimeGenerated : 4/19/2023 11:54:11 PM
EntryType     : Warning
Source        : vrtspgbouncersvc
InstanceId    : 0
Message       : C-000002b8e3fe75c0: NBDB/(nouser)@[::1]:59604 pooler error: server login has been failing, try again later (server_login_retry)

TimeGenerated : 4/19/2023 11:54:11 PM
EntryType     : Warning
Source        : vrtspgbouncersvc
InstanceId    : 0
Message       : C-000002b8e3fe75c0: NBDB/(nouser)@127.0.0.1:51079 pooler error: bouncer config error

TimeGenerated : 4/19/2023 11:54:11 PM
EntryType     : Error
Source        : vrtspgbouncersvc
InstanceId    : 0
Message       : varcache_apply failed: ERROR: invalid value for parameter "TimeZone": "Asia/Colombo"

Cause

Upon initial install/configuration, Postgres interrogates the OS to identify its time zone setting.
It then attempts to add time zone entries to the postgresql.conf file which instruct Postgres to use specific analogous time zone files (which should have already been installed by NetBackup).

If it fails to find the specific analogous time zone files, it creates the commented out entries for GMT within postgresql.conf.
    #log_timezone = 'GMT'
    #timezone = 'GMT'

This issue is unique to the Windows platform.

Solution

The recommended solution is to upgrade to NetBackup version 10.2.0.1 or higher.

Workaround:

To confirm if you are experiencing this problem and you are unable to upgrade, there are three items to review:

1. Examine the Tomcat startup failure messages within the  file within the <installPath>\NetBackup\wmc\webserver\logs\catalina.<date>.log for keywords which match the above log snip

2. Examine the <installPath>\Veritas\NetBackupDB\data\instance\postgresql.conf file for the two time zone rows to see if they are commented out and point to GMT (as noted above):
    log_timezone =
    timezone =

3. Examine the Application Event Log for errors produced by 'vrtspgbouncersvc' which contain the keywords documented above.  The above error occurred within the Asia/Colombo time zone, but there are a total of ten absent time zone files we know about:

• America/Blanc-Sablon
• America/Guadeloupe
• America/Hermosillo
• America/Miquelon
• America/Sao_Paulo
• America/Yellowknife
• America/Kentucky/Monticello
• Asia/Colombo
• Asia/Qyzylorda
• Europe/Oslo

If you have determined that you are experiencing the same issue, contact Veritas Technical Support and reference this Technote. Technical support can then provide you with the missing time zone file, and instructions on how to properly implement it.

Veritas Technologies has acknowledged that the above-mentioned issue (ETrack 4114644) is present in the NetBackup 10.2 on Windows platforms.. Veritas Technologies is committed to product quality and satisfied customers. This issue issue is addressed in the next release: 10.2.0.1.