Analytics Conversation Analysis processing fails on certain messages.

Problem

Discovery Accelerator (DA) Analytics Conversation Analysis processing fails on certain messages.

Error Message

Errors in Dtrace log of the AnalyticsConversationAnalyzerTask process:

- {SubjectNormaliser} {C78} (TransactionId:5395) Exception while storing conversations and items in DB.
     Exception=Symantec.Accelerator.Analytics.DataAccess.DataAccessException
      ---> System.Data.SqlClient.SqlException: The error description is 'An invalid character was found in text content.'.|
      The XML parse error 0xc00ce508 occurred on line number 1, near the XML text
          "<ConversationItems>
              <Item AnalysedItemId="1234567"
               Subject="happy birthday!"
                SubjectHash="256F2873EDEC087CCA0740B104EB2916B86D10E4EDD652E3C9559AB85C2ACCE5" />
              <Item AnalysedItemId="1234568"
               Subject="your guide"
                SubjectHash="DD27F4E31222480554DFD57E376E6814591090FB164890B6050221D62C499250" />
              <Item AnalysedItemId="1234569"
               Subject="Message with emoji in the subject line"
                SubjectHash="F529B4F182D8B66251FF7E7BEBFCBC9A55E341A3BE612FE7B6562F6993491E5E" />
              <Item AnalysedItemId="1234570"
               Subject="your invoice"
                SubjectHash="A14A806C08CDA7C628929069BE160B959E4794CBAB7057D7934486D20BEF642C" />
              <Item AnalysedItemId="1234571"
               Subject="meeting today"
                SubjectHash="1AF782DBB4F62CAE2F1BF4B151DEC46241F0DFB30AEB7DB74FFB5B6315C232D2" />
              <Item AnalysedItemId="1234572"
               Subject="statement"
                SubjectHash="D9DE3CF0012769EB77F9B11715414DA8C008CBFF9E0865D2F16EE7BDF7B1110A" />
              <Item AnalysedItemId="1234573"
               Subject="lumber".|   
          at Symantec.EnterpriseVault.DatabaseAccess.EVSqlBase.Do[T](Func`1 action)|   
          at Symantec.EnterpriseVault.DatabaseAccess.EVSqlCommand.ExecuteNonQuery()|   
          at Symantec.Accelerator.Analytics.DataAccess.DBHelper.ExecuteNonQuery(EVSqlCommand cmd)|
             --- End of inner exception stack trace ---|
          at Symantec.Accelerator.Analytics.DataAccess.DBHelper.ExecuteNonQuery(EVSqlCommand cmd)|   
          at Symantec.Accelerator.Analytics.DataAccess.StoredProcedureAccessorBase.ExecuteNonQuery(EVSqlConnection conn)|   
          at Symantec.Accelerator.Analytics.DataAccess.AnalyticsDataAccessor.AddConversationsAndItems(String itemsXml)|   
          at Symantec.Accelerator.Analytics.SubjectNormaliser.NormaliseSubjects()
              XML=<ConversationItems>
                  <Item AnalysedItemId="1234567"
                   Subject="happy birthday"
                    SubjectHash="256F2873EDEC087CCA0740B104EB2916B86D10E4EDD652E3C9559AB85C2ACCE5" />
                  <Item AnalysedItemId="1234568"
                   Subject="your guide"
                    SubjectHash="DD27F4E31222480554DFD57E376E6814591090FB164890B6050221D62C499250" />
                  <Item AnalysedItemId="1234569"
                   Subject="Message with emoji in the subject line"
                    SubjectHash="F529B4F182D8B66251FF7E7BEBFCBC9A55E341A3BE612FE7B6562F6993491E5E" />
                  <Item AnalysedItemId="1234570"
                   Subject="your invoice"
                    SubjectHash="A14A806C08CDA7C628929069BE160B959E4794CBAB7057D7934486D20BEF642C" />
                  <Item AnalysedItemId="1234571"
                   Subject="meeting today"
                    SubjectHash="1AF782DBB4F62CAE2F1BF4B151DEC46241F0DFB30AEB7DB74FFB5B6315C232D2" />
                  <Item AnalysedItemId="1234572"
                   Subject="statement"
                    SubjectHash="D9DE3CF0012769EB77F9B11715414DA8C008CBFF9E0865D2F16EE7BDF7B1110A" />
                  <Item AnalysedItemId="1234573"
                   Subject="lumber"
                    SubjectHash="678682522762A7592E713C3A250DE5ACBE6749115E421B8D2ABCA3FCED35A496" />
                  <Item AnalysedItemId="1234574"
                   Subject="breaking news"
                    SubjectHash="0C080908DA6ED3B57974505056B176CE23CD2B45C722ECFBBE7964FD6D36E12F" />
                  <Item AnalysedItemId="1234575"
                   Subject="daily weather update"
                    SubjectHash="A8DC463778430B0C025ED5A53E18F897B6EFC43134D6905B9114653F1E8BA4E3" />
                  <Item AnalysedItemId="1234576"
                   Subject="✈ air show"
                    SubjectHash="2D83FFBF62F35AD6481A762682B559B3388847F04BEA4B40447678090E1154A9" />
                  <Item AnalysedItemId="1234577"
                   Subject="breaking newsupdate"
                    SubjectHash="CE2D91DA1FB19CAC5DD25DDC502F8633EF5155992C5CB2ABFC4D5D22BC6524EC" />
                  <Item AnalysedItemId="4198370
- {-} {SQLExceptionDetector} {C78} Errors[0].Number: 6602, Error Message The error description is 'An invalid character was found in text content.'.|
    The XML parse error 0xc00ce508 occurred on line number 1, near the XML text
     "<ConversationItems>
         <Item AnalysedItemId="1234567"
          Subject="happy birthday"
           SubjectHash="256F2873EDEC087CCA0740B104EB2916B86D10E4EDD652E3C9559AB85C2ACCE5" />
         <Item AnalysedItemId="1234568"
          Subject="your guide"
           SubjectHash="DD27F4E31222480554DFD57E376E6814591090FB164890B6050221D62C499250" />
         <Item AnalysedItemId="1234569"
          Subject="Message with emoji in the subject line"
           SubjectHash="F529B4F182D8B66251FF7E7BEBFCBC9A55E341A3BE612FE7B6562F6993491E5E" />
         <Item AnalysedItemId="1234570"
          Subject="your invoice"
           SubjectHash="A14A806C08CDA7C628929069BE160B959E4794CBAB7057D7934486D20BEF642C" />
         <Item AnalysedItemId="1234571"
          Subject="meeting today"
           SubjectHash="1AF782DBB4F62CAE2F1BF4B151DEC46241F0DFB30AEB7DB74FFB5B6315C232D2" />
         <Item AnalysedItemId="1234572"
          Subject="statement"
           SubjectHash="D9DE3CF0012769EB77F9B11715414DA8C008CBFF9E0865D2F16EE7BDF7B1110A" />
         <Item AnalysedItemId="1234573"
          Subject="lumber".
- {EVSqlBase.ShouldExcludeSqlException} Error 'The error description is 'An invalid character was found in text content.'.|
    The XML parse error 0xc00ce508 occurred on line number 1, near the XML text
     "<ConversationItems>
         <Item AnalysedItemId="1234567" Subject="happy birthday" SubjectHash="256F2873EDEC087CCA0740B104EB2916B86D10E4EDD652E3C9559AB85C2ACCE5" />
         <Item AnalysedItemId="1234568" Subject="your guide" SubjectHash="DD27F4E31222480554DFD57E376E6814591090FB164890B6050221D62C499250" />
         <Item AnalysedItemId="1234569" Subject="Message with emoji in the subject line" SubjectHash="F529B4F182D8B66251FF7E7BEBFCBC9A55E341A3BE612FE7B6562F6993491E5E" />
         <Item AnalysedItemId="1234570" Subject="your invoice" SubjectHash="A14A806C08CDA7C628929069BE160B959E4794CBAB7057D7934486D20BEF642C" />
         <Item AnalysedItemId="1234571" Subject="meeting today" SubjectHash="1AF782DBB4F62CAE2F1BF4B151DEC46241F0DFB30AEB7DB74FFB5B6315C232D2" />
         <Item AnalysedItemId="1234572" Subject="statement" SubjectHash="D9DE3CF0012769EB77F9B11715414DA8C008CBFF9E0865D2F16EE7BDF7B1110A" />
         <Item AnalysedItemId="1234573" Subject="lumber".
     ' is not transient. 'EVSqlCommand:
     DB:
         Data Source=chi-us-sql-31\sql31;
         Initial Catalog=LITDA2018;
         Integrated Security=True;
         Connect Timeout=3600
         - Command: usp_AddConversationsAndItems_Insert' will not be retried

Cause

This issue can occur when there are unexpected characters in the Subject field of a message that has been ingested into DA's Analytics processing.  Emoji characters have been found to cause this issue.

Solution

This issue has been addressed in the following release(s):

Enterprise Vault 12.4.1

https://www.veritas.com/support/en_US/article.100044436