Unable to create Archive Point for NetApp C-mode SVM Volume target

Problem

The File system Archiving Task Report log shows that a previously defined Archive Point is no longer available.

This issue can also occur after removing a target and re-adding the target from the Vault Admin Console. 

Error Message

Using Dtrace.exe to collect debugging information shows that when trying to recreate the Archive Point with FSAUtility.exe, it results in error 3, which translates to "The system cannot find the path specified"

{NTFSStream.Save} Creating NTFS stream: \\?\UNC\svmsrv-cifs.domain.ext\vol1:EVArchivePoint.xml Err: 3

When using the Vault Admin Console to add a root volume, a Dtrace of EVFilesvrArcMngr and mmc will show the following:

2393    13:40:39.229     [12740]    (EVFileSvrArcMngr)    <4192>    EV-M    {NTFSStream.Load} Opening NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2394    13:40:39.229     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2395    13:40:39.229     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2396    13:40:39.229     [12740]    (EVFileSvrArcMngr)    <4192>    EV-M    {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2397    13:40:39.229     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {Attributes.CheckRetryCount} FolderAttributes:Retrying to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\' : try:(1/3)
2398    13:40:39.338     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2399    13:40:39.338     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2400    13:40:39.338     [12740]    (EVFileSvrArcMngr)    <4192>    EV-M    {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2401    13:40:39.338     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {Attributes.CheckRetryCount} FolderAttributes:Retrying to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\' : try:(2/3)
2402    13:40:39.448     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} EVFOLDERPOINT: Trying to save folder point xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2403    13:40:39.448     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {FolderAttributes.SaveXml} Folder time stamp Before processing for folder :\\svmsrv-cifs.domain.com\volume1\\ is : 3/16/2024 12:23:13 PM
2404    13:40:39.448     [12740]    (EVFileSvrArcMngr)    <4192>    EV-M    {NTFSStream.Save} Creating NTFS stream: \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml Err: 3
2405    13:40:39.448     [12740]    (EVFileSvrArcMngr)    <4192>    EV-L    {Attributes.CheckRetryCount} FolderAttributes:Will NOT retry to save xml on folder path: '\\svmsrv-cifs.domain.com\volume1\\'
2406    13:40:39.448     [2592]    (mmc)    <25448>    EV-L    {FileServerManagementStub.KVS.EnterpriseVault.FileServerArchive.IFileServerManagement.MarkRootFolder} Exception: The file path \\svmsrv-cifs.domain.com\volume1:EVFolderPoint.xml: is invalid Info: Diag: Type:System.IO.FileNotFoundException ST:|Server stack trace: |   at KVS.EnterpriseVault.Common.NTFSStream.Save(String filePath, Stream data, UInt32 flagsAndAttributes)|   at KVS.EnterpriseVault.FileServerArchive.Attributes.GetXmlWriter(String folderPath, String propsFileName, Boolean writeAsAlternateStream, MemoryFileStream& alternateStreamOut, String& filenameOut)|   at KVS.EnterpriseVault.FileServerArchive.FolderAttributes.SaveXml()|   at KVS.EnterpriseVault.FileServerArchive.Attributes.Save()|   at KVS.EnterpriseVault.FileServerArchive.FileServerArchiveManager.KVS.EnterpriseVault.FileServerArchive.IAdminClientManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove, String reserved)|   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)|   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)|Exception rethrown at [0]: |   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)|   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)|   at KVS.EnterpriseVault.FileServerArchive.IAdminClientManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove, String reserved)|   at KVS.EnterpriseVault.FileServerArchive.FileServerManagementStub.KVS.EnterpriseVault.FileServerArchive.IFileServerManagement.MarkRootFolder(String fileServerEntryId, String volumeId, String folderPath, String rfEID, Boolean remove) Inner:None
2407    13:40:39.448     [2592]    (mmc)    <25448>    EV~I    DirectoryConnection: Function call: CVersionUpdate::NewInstallation |
2408    13:40:39.448     [2592]    (mmc)    <25448>    EV:L    {CBaseDirectoryServiceWrapper::CreateDirectoryService} Directory Name [usfsa], Try Local Service [True]
2409    13:40:39.463     [2592]    (mmc)    <25448>    EV:L    {VaultCreateTrustedInstanceRequest::CreateLocalInstance} Successfully connected to local service
2410    13:40:39.463     [2592]    (mmc)    <25448>    EV:L    {VaultCreateInstanceRequest::CreateInstance} CLSID [{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A} (EnterpriseVault.DirectoryService.1)] Server Name [usfsa] Used Server Name [usfsa] Num of attempts [1] Total elapsed [0.001s] Result [Success  (0)]

2443    13:40:39.730     [2592]    (mmc)    <25448>    EV:H    {IAUIBaseImpl<class CAUIFolder,struct IAUIFolder,&struct _GUID const IID_IAUIFolder,&struct _GUID const LIBID_AUILib>::Save:#624} SaveAttributes() returned false, data not saved.
2444    13:40:39.730     [2592]    (mmc)    <25448>    EV:H    {IAUIBaseImpl<class CAUIFolder,struct IAUIFolder,&struct _GUID const IID_IAUIFolder,&struct _GUID const LIBID_AUILib>::Save:#636} Failed to save. Error: [0x80004005] [Failed to save or update details for folder '\' in the Directory Service.|Could not mark the folder with File System Archiving information.|Access is denied. ]

Cause

Further investigation showed that the environment was using the "Symlinks" option for the Shares defined at the Volume level.

Symlinks is an option that can be used to allow UNIX type access to CIFS Shares.

The issue started following an upgrade of ONTAP to a new minor version, where the "symlink-properties" option sets a different default value.

In earlier versions of ONTAP the default value was "enable", and after the upgrade, the default value for the option is "symlinks".

Volumes created prior to the upgrade would show the default value as "enable", whereas new Volumes where the CIFS shares are created with the "Symlinks" option would be set with a default value of "symlinks".

Solution

Changing the "symlink-properties" from enable to symlinks allows the hidden streams to be created / displayed, therefore allowing the FSA Task to correctly process the content within the identified Archive Point.

Note: This may require a reboot of the Enterprise Vault server.