Switchover Fails on 3.1.x High Availability Appliance if MSDP spoold process takes longer than 5 minutes to initialize

Article: 100042933
Last Published: 2018-05-02
Ratings: 0 0
Product(s): Appliances

Problem

In a NetBackup Appliance High Availability (HA) configuration, the switchover operation of the MSDP service may fail if the startup time for the MSDP spoold process takes longer than 5 minutes.

Error Message

The /var/VRTSvcs/log/engine_A.log shows:

2018/04/26 11:52:26 VCS NOTICE V-16-1-10166 Initiating manual online of group msdp_svc on system nbapp02
2018/04/26 11:52:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group msdp_svc on all nodes
2018/04/26 11:57:27 VCS WARNING V-16-2-13012 (nbapp02) Resource(msdp_svc): online procedure did not complete within the expected time.

Cause

During MSDP startup, the spoold process must initialize each individual datastore (mountpoint). This is done in parallel, but if any individual mount point is larger than 40TB, this process can take longer than the application online timeout set within the cluster software. The default is 5 minutes (300 seconds). The startup of spoold being the cause can be confirmed by reviewing the /msdp/data/dp1/pdvol/log/spoold/spoold.log file and looking for the time between these two lines:

April 26 11:52:28 INFO: Startup: occurred at Wed Apr 26 11:52:28 2018
April 26 11:58:22 INFO [140447991928640]: Connection Manager: initialization complete

Until "Connection Manager" has completed the initialization, the spoold process will not respond to queries.

Solution

The following ETs describe EEBs that are available through Veritas Support and can be installed on appliances to improve the MSDP startup times:

  • ET3972883 - EEB for appliance software version 3.1 (NetBackup 8.1)
  • ET3972200 - EEB for appliance software version 3.1.1 (NetBackup 8.1.1)
  • ET3972201 - EEB for appliance software version 3.1.2 (NetBackup 8.1.2)

Note: The EEBs referenced above all contain MSDP bundle fixes that address other issues not related to MSDP startup times. Make sure to ask the support representative about those details.

Alternatively, another resolution to this issue is to increase the online timeout from the default of 5 minutes to a time greater than the time take for spoold to start responding. Note other services are also started up during the online process as well. It has been observed in most cases that increasing this to 10 minutes (600 seconds) is sufficient.

As the workaround requires modifying the appliance configuration after overriding the SDCS security software please contact Veritas Technical Support and quote this article and they will be able to assist.

 

Veritas Technologies LLC has acknowledged that the above-mentioned issue is present in the current version(s) listed under the Product(s) Section of this article. Veritas Technologies LLC is committed to product quality and satisfied customers.
 
Please be sure to refer back to this document periodically as any changes to the status of the defect will be reflected here.  Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests.  Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.
 
Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.  For information on how to contact Veritas Sales, please see   https://www.veritas.com

References

JIRA : APPSOL-83872

Was this content helpful?