Config.db Fails to Update on a Node in Data Insight 6.3.1

Problem

In Data Insight 6.3.1, the configuration database on one of the nodes fails to update, generating a Config not updated on node error in the event log for that node in the Data Insight Management Console.

Error Message

From the Management Console, checking the event logs for the affected node, the error below is seen:

V-378-1312-517: Config not updated on node node_hostname. This could be because UpdateConfigJob failed to execute on the node. Config db version on Management Server is ###. Config db Version on the node is ###.

In addition, this error may be visible in the watchdog0.0.log file on the affected node:

2022-08-23 06:20:49 WARNING: #{26} [SystemMonitorJob.isVersionSame] Config db version on node node_hostname is ### which is different from the version on MS. Config db version on MS:###

Note: The "###" in the errors shown above are placeholders for the numbers appended to the config.db files on each node, which increase every time the configuration is updated. (e.g. config.db.123)

Cause

This may be caused by intermittent network connectivity issues between the nodes, which can affect the DataInsightComm service, which is responsible for inter-node communication.

Solution

If the affected node is not configured as a Windows File Server:

1. Access both the Management Server and the affected node.
2. Navigate to the <DATADIR>\DataInsight\data\conf folder 
3. In both directories, note the number appended to the newest config.db file (e.g. config.db.123), and confirm that the number on the Management Server is higher than the number on the node.
4. On the affected node, make a backup copy of all files starting with config.db (including any .helper and .lck files), and place the copies in a destination folder other than the conf folder.
5. Copy the most recent config.db file (config.db.### with highest number) and the config.db.helper file from the conf folder on the Management Server to the conf folder on the affected node.
6. Restart the DataInsightComm and DataInsightConfig services on both the Management Server and the affected node (Start > Run > services.msc)
7. Go back to the conf folder under the data directory on both nodes and confirm that the numbers appended to both config.db files match.

If the affected node is configured in Data Insight as a Windows File Server:

1. Access both the Management Server and the Windows File Server.
2. Navigate to the <DATADIR>\DataInsight\data\conf folder on both servers. 
3. In both directories, note the number appended to the newest config.db file (e.g. config.db.123), and confirm that the number on the Management Server is higher than the number on the filer.
4. Make note of the hostname of the Collector node that is associated with the affected Windows File Server. This can be done by going to Settings > Filers in the Data Insight Console, and clicking on the name of the Windows Filer in question. The hostname of the associated Collector should be listed there.
5. Connect to the Collector node whose hostname was noted in step #4, and navigate to the <DATADIR>\DataInsight\data\conf folder. Certain files located in that directory will be needed later in this process.
6. Go back to the affected filer, make a backup copy of all files starting with config.db (including any .helper and .lck files), and place the copies in a destination folder other than the conf folder.
7. Copy the most recent config.db file (config.db.### with highest number) and the config.db.helper file from the conf folder on the associated Collector from steps #4 and #5 to the conf folder on the Windows File Server.
8. Restart the DataInsightComm and DataInsightConfig services on the Management Server, associated Collector, and Windows Filer nodes (Start > Run > services.msc)
9. Wait for about five minutes, go back to the conf folder under the data directory on the Management Server, associated Collector, and Windows Filer nodes and confirm that the numbers appended to all three config.db files match.