Veritas Enterprise Vault Compliance Accelerator searches are returning unexpected results based on the message direction selection.

Problem

Compliance Accelerator (CA) searches require one of 12 message direction options to be selected.  Those options always include the Department in which the search is created or a selection of Departments when configuring an Application level search.  This Department specification is on the left side of the direction selection options and can be considered as the Anchor Department of the search.  The specification on the right side of the message direction selection options can be considered as the Target of the search and will be either

1. a selection of searchable Departments from within then CA customer environment (considered Internal) and / or entries in the "Freeform email addresses / domains" field [this is the 'other searchable departments' option],
2. the organization in which CA is deployed (i.e., all emails with SMTP addresses within the company's internal SMTP domains' scope - also considered Internal) [this is the 'any department within the organisation' option],
3. external organizations (i.e., all emails with SMTP addresses outside of the company's internal SMTP domains' scope - considered External Inbound or External Outbound) [this is the 'outside the organisation' option],
4. everyone (i.e., emails having either internal or external SMTP domains - considered Internal, External Inbound or External Outbound) [this is the 'internal AND/ OR External to organisation' option].

The CA search message direction options comprise 3 base directions:

1. To or From the Anchor Department's Monitored Employees
2. To the Anchor Department's Monitored Employees
3. From the Anchor Department's Monitored Employees

CA search configurations found to return unexpected results are, but not limited to, as follows:

1. Capture internal only items can also capture items tagged as External Inbound.
2. Capture items going only out to external recipients can also capture items tagged as External Inbound.
3. Comparison searches of inbound only versus outbound only versus inbound and outbound where the results are expected to add up for each message direction.

Cause

The unexpected search results are caused by a combination of the message direction tagging, search criteria use of message direction and / or CA's Department ID tagging.

The Enterprise Vault (EV) archiving process evaluates each message to determine and assign a message direction to be applied.  The available message direction tags and their definitions are in Table 1 below.

Table 1. Message Direction Tags

Direction Tag	Direction Tag Definition
0	Undetermined or Lotus Notes message
1	Internal Only - author and all recipients have internal SMTP addresses
2	External Inbound - author has an SMTP address that is external to the organization
3	External Outbound - author has internal SMTP address and at least 1 recipient has an SMTP address that is external to the organization or has no SMTP address

Message direction tagging is determined by the SMTP address(s) assigned to the account used by the Journal Task (usually a mail-enabled Vault Service Account).  A registry value named 'InternalSMTPDomains' can also be used to denote SMTP domains that should be considered internal. For example, the internal SMTP domain of an organization could be 'mydomain.local' so that users would have SMTP addresses of the format 'firstname.lastname@mydomain.local'.  Another SMTP domain of 'newsorg.com' could be added as an internal SMTP domain using the 'InternalSMTPDomains' registry value so that any emails having the '@mydomain.local' or '@newsorg.com' would be tagged as internal messages.

The search criteria message direction options are shown in Figure 1 below:


Figure 1: CA Message Direction Options

CA versions prior to 11.0.1 used a Journaling Connector that had to be installed on each EV server hosting one or more Journal Task instances for CA Department ID tagging to occur.  Starting with CA version 11.0.1, the CA Department ID tagging processing has been moved to the EV Storage Service processing, so the Journaling Connector is not needed (and is, in fact, uninstalled during an upgrade to 11.0.1).  CA Department ID tagging begins after at least one Monitored Employee is added to at least one Department that is set to allow monitoring and a synchronization has occurred between the CA Customer installation and the EV Storage Service to update the storage processing with the Department and Monitored Employee information.  The date on which the Journaling Connector is installed or when the first Department with at least one Monitored Employee is configured is the date and time with which a CA Configuration setting known as 'First Department Query' is populated.  Once this configuration setting is populated, any search with a date span earlier than the 'First Department Query' date will send SMTP addresses for the authors and recipients as appropriate to the EV search engine.  Any search with a date span after the 'First Department Query' date will send the Department ID tag for the author and / or recipients as appropriate to the EV search engine.  Any search with a date span that begins before the 'First Department Query' date and ends after the 'First Department Query' date will send both the Department ID tag and SMTP addresses for the authors and / or recipients as appropriate to the EV search engine.

CA Department ID tagging is based on the SMTP address(es) of each Monitored Employee's membership in one or more Departments.  CA Department ID tagging is independent of message direction tagging.  For example,

1. An "All Employees" Department has a Department ID of 5,
2. Monitored Employee "John Doe" could have SMTP addresses of 'john.doe@my.company.com' and 'user123456@newsorg.net',
3. Monitored Employee "John Doe" could be a member of the 'All Employees" Department.
4. A message from "John Doe" using his 'john.doe@mycompany.com' SMTP address would be tagged with Department ID 5 for the Department Author property (known as KVSCA.DeptAuthor).
5. Another message from "John Doe" using his 'user123456@newsorg.net' SMTP address would also be tagged with Department ID 5 for the Department Author property.
6. A message sent to "John Doe" using his 'john.doe@mycompany.com' SMTP address would be tagged with Department ID 5 for the Department Recipient property (known as KVSCA.DeptRecips).
7. A message sent to "John Doe" using his 'user123456@newsorg.net' SMTP address would also be tagged with Department ID 5 for the Department Recipient property.
8. A message sent from "John Doe" with a copy to himself using his 'john.doe@mycompany.com' SMTP address would be tagged with Department ID 5 on both the Department Author and Department Recipient properties.
9. Another message sent from "John Doe" with a copy to himself using his 'user123456@newsorg.net' SMTP address would also be tagged with Department ID 5 on both the Department Author and Department Recipient properties.
10. Another message send from "John Doe" using his 'user123456@newsorg.net' SMTP address with a copy to himself using his 'john.doe@mycompany.com' SMTP address would also be tagged with Department ID 5 on both the Department Author and Department Recipient properties.

As noted above, CA uses the Department ID tag to identify authors and recipients for searches that are configured with a date span that includes the 'First Department Query' date.  Use of SMTP addresses instead of Department ID tags can be forced by either changing a configuration option or by unchecking the Department, expanding the Department to show each of its Monitored Employees, and then selection the specific Monitored Employee(s) to use in the search criteria.  Making either of these changes causes CA to use 'legacy' search properties (SMTP address(es) instead of Department ID tag(s)).

Certain CA search direction selections do not use the message direction tag as part of the search criteria sent to the EV search engine.  Table 2 below shows how message direction tagging is used in CA searches.
 

Table 2. CA Message Direction Usage Chart

Anchor Department	Direction	Target	MsgDirection	Notes
Department or individual Monitored Employee(s)	<---->	other searchable departments	Not Used	Looks for author and recipient Department ID tag(s) or SMTP address(es) (for individually selected Monitored Employee(s)).  The message direction tag criteria is not used for either Department ID or 'legacy' searches as the author and recipient information is explicitly included in the search criteria.
Department or individual Monitored Employee(s)	<---->	any department within the organization	1 or 0	Uses the message direction tag as part of the search criteria looking for items sent between the Anchor Department's Monitored Employee(s) and other departments within the organization.
Department or individual Monitored Employee(s)	<---->	outside the organization	3 or 2 or 0	Uses the message direction tag as part of the search criteria looking for items sent between the Anchor Department's Monitored Employee(s) and anyone outside of the organization.
Department or individual Monitored Employee(s)	<---->	internal AND/ OR external to organization	3 or 2 or 1 or 0	Uses the message direction tag as part of the search criteria looking for items sent between the Anchor Department's Monitored Employee(s) and anyone inside or outside the organization.
Department or individual Monitored Employee(s)	<--	other searchable departments	Not Used	Looks for recipient Anchor Department ID tag(s) or SMTP address(es) (for individually selected Monitored Employee(s)).  The message direction tag criteria is not used for either Department ID or 'legacy' searches as the author and recipient information is explicitly included in the search criteria.
Department or individual Monitored Employee(s)	<--	any department within the organization	1 or 0	Uses the message direction tag as part of the search criteria looking for items sent to the Anchor Department's Monitored Employee(s).
Department or individual Monitored Employee(s)	<--	outside the organization	2 or 0	Uses the message direction tag as part of the search criteria looking for items sent to the Anchor Department's Monitored Employee(s).
Department or individual Monitored Employee(s)	<--	internal AND/ OR external to organization	2 or 1 or 0	Uses the message direction tag as part of the search criteria looking for items sent to the Anchor Department's Monitored Employee(s).
Department or individual Monitored Employee(s)	-->	other searchable departments	Not Used	Looks for author Anchor Department ID tag(s) or SMTP address(es) (for individually selected Monitored Employee(s)).  The message direction tag criteria is not used for either Department ID or 'legacy' searches as the author and recipient information is explicitly included in the search criteria.
Department or individual Monitored Employee(s)	-->	any department within the organization	1 or 0	Uses the message direction tag as part of the search criteria looking for items sent from the Anchor Department's Monitored Employee(s).
Department or individual Monitored Employee(s)	-->	outside the organization	3 or 0	Uses the message direction tag as part of the search criteria looking for items sent from the Anchor Department's Monitored Employee(s).
Department or individual Monitored Employee(s)	-->	internal AND/ OR external to organization	3 or 1 or 0	Uses the message direction tag as part of the search criteria looking for items sent from the Anchor Department's Monitored Employee(s).

As can be seen in the above table, CA does not use the message direction tagging for any searches between the Anchor Department and any other searchable departments (i.e., when using any of the 'Other searchable departments' options).  The inclusion of the specific author information (i.e., their Department ID tag or their SMTP address(es)) along with the specific recipient information (i.e., their Department ID tag(s) or their SMTP address(es)) precludes the need for any message direction tag  because the search is looking for the specific author and recipient information.

In addition, the 'other searchable departments' options all have the "Any of" and "All of" selection options available.  These options are used to determine how to logically process messages between the Anchor Department and the Target when multiple Departments and / or the 'Freeform email addresses / domains' field contents are selected.  Use of the "Any of" option connects the objects with the logical OR operation.  For example, when looking for messages that have authors from domain1.local, domain2.local or domain3.local, the logic applied would be
     auth:domain1.local OR auth:domain2.local OR auth:domain3.local
The use of the "All of" option connects the objects with the logical AND operation.  For example, when looking for messages that have been received by users in domain1.local, domain2.local and domain3.local, the logic applied would be
     recp:domain1.local AND recp:domain2.local AND recp:domain3.local

When the "Any of" and "All of" option is available, hovering the mouse over this selection field will cause a message pop-up to appear with the following information:
     The choice of "Any of" or "All of" will only be applied to received messages. It will be ignored for sent messages.
This message indicates that any message received by the Targets (where the Anchor Department is the author) can have the "Any of" or "All of" option used when multiple targets are involved.  This message also indicates that the "Any of" or "All of" option will not be used when the Target is sending to the Anchor Department (where the Anchor Department is the recipient) and the Target has multiple Departments selected and / or one or more 'Freeform email addresses / domains' field entries.  The use of the "All of" option for multiple senders is not valid as a message can have only 1 author with 1 SMTP address for that author.  As such, multiple Target departments and / or "Freeform email addresses / domains" field entries as authors will be bound only by the "Any of" operation that designates use of the logical OR operation between the authors (i.e., auth:domain1.local OR auth:domain2.local OR auth:domain3.local).

Message direction comparison searches could but will not likely return hit counts that will add up per message direction across all searches.  For example, a CA installation has Department ID 4 in which the following message directions were configured:

• Search 1 - Inbound to Recipients only in Department ID 4 (looking for Message Directions 1 or 2 per the table above and Department Recipients).
• Search 2 - Outbound from Authors only in Department ID 4 (looking for Message Directions 1 or 3 per the table above and Department Authors).
• Search 3 - Inbound or outbound to / from Authors and Recipients in Department ID 4 (looking for Message Directions 1, 2 or 3 per the table above and Department Authors or Department Recipients).

Search 1 will look for any message received by Monitored Employees in Department ID 4, so a message authored by a Monitored Employee in Department ID 4 with no recipients in that department will not be included in the Search 1 results.  For example, a message tagged with Message Direction 1 (internal) would be eligible for inclusion in the results for Search 1 or Search 3 but only having the Department Author from Department ID 4 would make this message ineligible for the Search 1 results.  Using the indexed attributes:
   Vault.MsgDirection: (String): 1        (Message Direction) <<< This is the reason this is message is eligible for Search 1 (recipient only), Search 2 (author only) or Search 3 (author or recipient))>>>
   KVSCA.DeptRecips: (String):        (Department Recipients) {does not exist in the message's indexed properties} <<< This is the reason this message is not eligible for Search 1 (recipient only)
   KVSCA.DeptAuthor: (String): 4        (Department Author) <<< This is the reason this is message is eligible for Search 3 (author or recipient) and Search 2 (author only) but the search criteria is looking for recipients only, thereby making Search 2 ineligible>>>

Search 2 will look for any message authored by a Monitored Employee in Department ID 4, so a message received by a Monitored Employee in Department ID 4 with the author in another Department or from outside the company will not be include in the Search 2 results.  For example, a message tagged with Message Direction 3 (external outbound) would be eligible for inclusion in the results for Search 2 or 3, but having only the Department Recipient(s) from Department ID 4 would make it ineligible for inclusion in Search 2.  Using the indexed attributes:
   Vault.MsgDirection: (String): 3        (Message Direction) <<< This is the reason this is message is eligible for Search 2 (author only) and Search 3 (author or recipient) but not Search 1 (recipient only looking for Message Directions 1 or 2)>>>
   KVSCA.DeptRecips: (String): 4      (Department Recipients) <<< This is the reason this message is eligible for Search 1 (recipient only) but the search criteria is looking for authors only, thereby making Search 2 ineligible)
   KVSCA.DeptAuthor: (String):         (Department Author) {does not exist in the message's indexed properties} <<< This is the reason this is message is eligible for Search 3 (author or recipient) but not Search 2 (author only)>>>

Search 3 will look for any message authored or received by a Monitored Employee in Department ID 4 regardless of other authors or recipients.  This search can have results that include messages authored by a Monitored Employee in Department ID 4 to a recipient in another Department or outside the company that will not be included in the Search 1 results.  Also, this search can include messages received by a Monitored Employee in Department ID 4 with the author in another Department or from outside the company which will not be included in the Search 2 results.

It is important to note the 'Freeform SMTP Address / Domain' field is directional. This means if the Message Route is right to left, the 'Freeform email addresses / domains' field will be for recipients; if the Message Route is left to right, the 'Freeform email addresses / domains' field will be for authors. If the Message Route has both directions, i.e. 'Between', then both directions are used, with the 'Freeform email addresses / domains' field being considered as recipients from left to right and as authors from right to left. An explanation of how the Freeform SMTP/Domains field works can be reviewed in Article ' Exclusion of email addresses in the Freeform Emails/Domain section of the Compliance Accelerator Search Criteria does not function as expected' (see Related Articles).

 

Solution

For searches between CA Departments that require strict adherence to message direction tagging, either

1. Use Veritas Enterprise Vault Discovery Accelerator (DA) with a Custom Attribute configured using 'Vault.MsgDirection' in the "Name:" field and the number 1 in the "Value:" field to ensure only items tagged with the internal message direction are returned, the number 2 in the "Value:" field to ensure only items tagged with the External Inbound direction are returned or the number 3 in the "Value:" field to ensure only items tagged with the External Outbound direction are returned (see Article 000090974 in the Related Articles section to allow selection of 1 or more message direction tags) -- or --
2. Understand that items returned by CA that may have been tagged as 'External Inbound' (Vault.MsgDirection 2) are still valid items as they match the requirement of the Department ID tag or SMTP address(es) for the message author and / or recipient.
3. Do not use comparison searches to determine if the message direction searches are working properly as such searches will likely not return results that will add up to the same numbers per message direction.