Important Update: Cohesity Products Knowledge Base Articles
All Cohesity Knowledge Base Articles are now managed via the Cohesity Support Portal: https://support.cohesity.com/s/searchunify. The Knowledge Base articles available here will not reflect the latest information or may no longer be accessible.
Problem
On a system reboot or service group switch, multiple resources fail to online and fault. They can be manually onlined individually after the failure. This is not limited to any specific resource and can effect any and all resources.
The cluster will be running on Suse 12 SP 2 or above.
Error Message
One or more of the following errors will be reported in the applicable resource log. i.e. DiskGroup_A.l.og, NIC_A.log etc.
2022/11/07 21:43:37 VCS ERROR V-16-2-13469 Thread(139941547460352) Agent(Application) exiting - failed to create process due to low system resources, errno(11)
2022/11/15 13:26:06 VCS WARNING V-16-1-13162 Could not create thread
2022/11/12 18:18:03 VCS INFO V-16-2-13716 Thread(140700412692224) Resource(dg_xxxxx): Output of the completed operation (monitor)
==============================================
/opt/VRTSvcs/bin/ag_i18n_inc.sh: fork: retry: No child processes
==============================================
==============================================
2022/11/12 18:23:17 VCS INFO V-16-2-13716 Thread(140700412692224) Resource(dg_xxxxx): Output of the completed operation (monitor)
==============================================
/opt/VRTSvcs/bin/ag_i18n_inc.sh: fork: retry: Resource temporarily unavailable
Cause
Suse 12 SP2 appears to have changed the default value for the parameter DefaultTasksMax to 512. This can be verified with the following command:
systemctl show --property DefaultTasksMax
This is insufficient to start anything other than a small number VCS agents and resources simultaneously. This behaviour has not been observed in Suse 15 where the default setting is much higher.
Solution
Increase the value for the parameter DefaultTasksMax.
Please consult the following Suse knowledge article for the steps to modify this value:
How to modify systemd DefaultTasksMax / TasksMax / UserTasksMax settings
Although Veritas does not have a specific recommendation, no issues have been reported with the default value in Suse 15 of 4915.