Jobs stopped getting scheduled, and the ones that were running appeared to be hung. Child jobs were never spawned.
Could not start jobs manually; they would not show up in the Activity Monitor.
After running gcore against nbpem to capture the stack trace during the time of the hang, the output revealed that none of the threads were running, indicating a deadlock condition.
The deadlock was triggered by an nbpem -predict command.
The formal resolution for this issue (Etrack 2979677) is included in the following release:
- NetBackup 7.5 Maintenance Release 5 (220.127.116.11)
NetBackup 18.104.22.168 is now available - please access the Related Article below for more information.
Please note that Symantec Corporation reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests or introduces new risks to overall code stability. Symantec's plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk. Please refer to the maintenance pack readme or contact NetBackup Enterprise Support to confirm this issue (ET2979677) was included in the maintenance pack.
If this issue is experienced, please contact Symantec technical support, referencing this document ID and Etrack 2979563 to obtain an Emergency Engineering Binary (EEB) bundle for NetBackup 22.214.171.124 containing a fix for this issue.
Running nbpemreq -subsystems screen all command to dump the PEM tables was not working.
ps -ef | grep nbpemreq never showed a daemon running.
This condition has been reported on NetBackup 126.96.36.199.