Legal Holds fail to send email notices

Problem

This could happen if the Java "cacerts" keystore file or the "cert" keystore file has been damaged or the password for the keystore has changed. 

Error Message

Job Status log shows the following:

Server.yyyy-mm-dd.log has:

com.teneo.esa.litholds.service.TeneoSSLHandshakeException: [#80085] Runtime exception: Exception in installing the server certificate required for HTTPS. : java.security.UnrecoverableKeyException: Password verification failed
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
    at java.security.KeyStore.load(KeyStore.java:1214)
    at com.teneo.esa.litholds.service.CertificateInstallHelper.install (CertificateInstallHelper.java:81)
    at com.teneo.esa.litholds.service.WSClientHelper.getClient(WSClientHelper.java:61)

Cause

eDiscovery uses a copy of the certificate authority cacerts keystore file from  C:\jdk-#u##-windows-x64\jre\lib\security\ and places a copy named cert in D:\CW\V###\scratch\temp for use with sending Legal Hold notices. This certificate is used to encrypt communications between the legal hold admin and confirmation servers.

This error may occur if the default password for the cacerts file has been changed.  It may also become corrupt when multiple Repeating Status Reminders are scheduled to send at the same time, usually the default day and time of Monday at 8:00 AM.

This error may also be seen on the legal hold admin (master) server when using a standalone confirmation server and the services on the standalone confirmation server are down.  In this case, there may be nothing wrong with the cert file in D:\CW\V###\scratch\temp.

Solution

Scenario 1:  The default cacerts password has not changed:

For eDiscovery versions 9.5.x and below:
Open a command prompt in  D:\CW\V###\scratch\temp and type the following command:
Keytool -list -keystore cert -storepass changeit (Enter)

If you receive the error: Keystore was tampered with, or password was incorrect delete the cert file from D:\CW\V###\scratch\temp and copy the cacerts from C:\jdk-#u##-windows-x64\jre\lib\security to D:\CW\V###\scratch\temp and rename it cert.

For eDiscovery versions 10.x and above:
Open a command prompt in  D:\CW\V###\scratch\temp and type the following command:
keytool -list -keystore cert -storetype BCFKS -providerclass  com.safelogic.cryptocomply.jcajce.provider.CryptoComplyFipsProvider -storepass changeit

If you receive the error: Keystore was tampered with, or password was incorrect delete the cert file from D:\CW\V10#\scratch\temp and copy the cacerts from C:\jdk-#u##-windows-x64\jre\lib\security to D:\CW\V###\scratch\temp and rename it cert.

Scenario 2:  The default cacerts password has been changed

Follow the steps above to list the cert file in D:\CW\V###\scratch\temp using the changed password.  If you receive the error  Keystore was tampered with, or password was incorrect follow the above solution.

If the cert keystore opens using the changed password, check the value of the property esa.cert.keystore.password in System > Support Features > Property Browser and change it to the new password.

Scenario 3:  The cert keystore is not in the correct format

Open a command prompt in  D:\CW\V###\scratch\temp and type the following command:
Keytool -list -keystore cert -storepass changeit > C:\CertKeystoreList.txt (Enter)

Examine the first line of the newly created CertKeystoreList.txt text file.
Keystore type:
JKS is for eDP 9.x
BCFKS is for eDP 10.x

If the Keystore type is not correct for the version of eDP, delete the cert file from D:\CW\V10#\scratch\temp and copy the cacerts from C:\jdk-#u##-windows-x64\jre\lib\security to D:\CW\V###\scratch\temp and rename it cert.

Scenario 4: eDiscovery services on the confirmation server are not available.

Log into the confirmation server and check that the EsaApplicationService is started.
Check the catalina and server logs for errors.
Correct the errors so the EsaApplicationService starts and the UI is available.