The Vault Service account does not have access to the storage queue location

Problem

Intermittent errors are reported to the Enterprise Vault (EV) event log stating that the Vault Service account does not have access to the storage queue location,  even though permissions have been checked and it has been ensured that no other process is resetting permissions to the EV Storage Queue Location.
When the issue occurs, the affected Storage Queue file (EVSQ) may get deleted. The impact will vary depending on the type of archiving and whether EV has been configured to remove safety copies immediately after archiving.

Exchange Journaling and Mailbox Archiving
When EV is configured to Remove Safety Copy 'Immediately after Archiving',  the agent will only remove the original item or convert to a shortcut after the message  has been fully archived (i.e. ingested into the partition from the Storage Queue).
If any problem occurs, the item will stay pending in the mailbox and eventually roll backed to the original message.

SMTP Archiving
When EV is configured to Remove Safety Copy 'Immediately after Archiving',  the EML file is removed immediately after the archived item is placed on the Storage Queue. I.e. the agent does not wait for the item to be fully ingested into the partition.
If any problem occurs before the ingestion into the partition, the archiving agent will not have a copy of the original item to retry.

File System Archiving
When EV is configured to Remove Safety Copy 'Immediately after Archiving', and the folder\volume policy is further configured to create a shortcut immediately or archive and delete the file, the original file is removed immediately after the archived item is placed on the Storage Queue.  I.e. the agent does not wait for the item to be fully ingested on the partition.
If any problem occurs before the ingestion into the partition, the archiving agent will not have a copy of the original item to retry.

 

Error Message

Cause

1. The StorageOnlineOPNS (SOPNS) process is queuing savesets to a particular EVSQ file.
2. While the SOPNS queuing process is still in progress, EVStorageQueueBroker initiates a delete of that particular EVSQ File
3. As the EVSQ file has open handles against it (because it is still being populated), the file system cannot perform the delete. Instead, it marks the file as DELETE PENDING. This is handled by the File System.
4. The StorageArchive process next attempts to read the different savesets within the EVSQ File. However, because of the DELETE PENDING status of the file, any read attempt will be denied. Hence, this is the Denied, which is then logged in the Event log. This denied error will not be seen explicitly. As per the procmon screenshot, a Delete Pending Status is logged instead.
5. As EV fails to read the saveset from the queue, it then attempts to copy the saveset (not the whole queue) in the failed items folder. So we can see a 0 KB file being created, and immediately deleted, as it failed to get the content from the EVSQ file.
6. When all the handles on the EVSQ file are released, the file is then deleted.

 

Solution

Workaround
Configure EV to keep Safety Copies in the Storage Queue. This ensures that even if the race condition occurs, no deletion of the storage queue will be attempted until all items have been ingested into the partition AND backed up.