This issue has been seen to occur when using File Blocking with content checking on the file server. The handles for the FileScreenService.exe process will continue to grow until the server becomes unresponsive and faults.
Disable content checking for the File policies used on the file server.
1. Open the Vault Administration Console (VAC) and expand Directory on Sever_name>Site_name>Policies>File.
2. Highlight Volume and right-click the policy name and choose Properties.
3. Select the File Blocking Rules tab.
4. Select each rule name listed, one at a time, and click Edit.
5. Click on the File Blocking Options tab.
6. Under Options uncheck the option to Check file content.
7. Click OK two times to update the rule.
NOTE: Ensure all rules applied for that volume have this option unchecked.
8. Restart the Enterprise Vault File Blocking Service on the file server.
NOTE: If content checking is required then a restart of the Enterprise Vault File Blocking Service must be scheduled to occur periodically throughout the day to avoid the handles from growing too large.
This issue has been resolved as a part of the following:
Enterprise Vault 9.0.2 - Release Details
Enterprise Vault 10.0 - Release Details