NetBackup Deduplication Engine (spoold) fails to start due to a problem retrieving startup modes from the Deduplication Manager (spad) process.

Problem

The NetBackup Deduplication Engine (spoold) communicates during startup with the NetBackup Deduplication Manager (spad) process. It collects the current CR (content router) modes from spad to determine what modes it should be starting in.

If a problem exists on the system which prevents this from occurring, spoold will fail to properly initialize and will endlessly loop whilst waiting for the local spad process to return the current CR mode list (as shown in the error below).

Error Message

If spoold cannot retrieve CR modes from the spad process on startup, the spoold.log will show errors like the following:

ERR [0000000001273BA0]: 25053:  Connection failed connection actively refused
WARNING [0000000001273BA0]: 25053: Failed to get startup CR modes from SPA after 15567 attempts, retrying in 10 seconds

ERR [0000000001273BA0]: 25053: Could not establish a connection to 10.20.1.240:10102: connect failed (No connection could be made because the target machine actively refused it.

The spoold process will repeatedly attempt to retrieve the CR modes from the spad process until it is successful. However, in most circumstances, the retry attempt every 10 seconds will also fail.

Using the 'crcontrol' command may also produce 'connection refused' messages, similar to those below.  This is because spoold hasn't fully initialized, and is encountering problems prior to listening for new connection requests.

Example on Unix/Linux:

# /usr/openv/pdde/pdcr/bin/crcontrol --dsstat
Error: -1: NetConnectByAddr: Failed to connect to host: Connection refused (111)
Error: -1:  NetConnectByAddr: Failed to connect to spoold on port 10082 using the following interface(s): [ ::1  ] (Connection refused) Ensure storage server services are running and operational.  V-454-92
Error: 53: Could not establish a connection to ::1:10082: connect failed (Connection refused)
Error : Connection failed connection actively refused. Note that the content router needs to be running to get a connection.

Example on Windows:

C:\Program Files\Veritas\pdde>crcontrol --dsstat
Error: -1: NetConnectByAddr: Failed to connect to host: No connection could be made because the target machine actively refused it.  (10061)
Error: -1:  NetConnectByAddr: Failed to connect to spoold on port 10082 using the following interface(s): [ ::1  ] (No connection could be made because the target machine actively refused it. )
Ensure storage server services are running and operational.  V-454-92
Error: 53: Could not establish a connection to ::1:10082: connect failed (No connection could be made because the target machine actively refused it. )
Error : Connection failed connection actively refused. Note that the content router needs to be running to get a connection.

Cause

If the spoold process is unable to retrieve the current CR (content router) modes from the spad process, spoold will fail to complete it's startup sequence.

The main possible causes for spoold being unable to retrieve CR modes from spad include the following:

• The spad process is not running (check spad.log)
• The spoold process cannot connect to the spad process on TCP port 10102 (check 'netstat -a' to ensure that spad is listening on port 10102)
• The spad database path is unmounted/inaccessible/missing or corrupt.
• The spoold process is attempting to connect to a non-local spad process.

Solution

Step through the checklist of items below to find what may be causing the issue. Continue separately through each item on the list until the spoold process is able to retrieve the CR modes from the spad process and complete its startup sequence.

1. Confirm that you are experiencing the same issue as outlined in this article by looking in the spoold.log for the string 'Failed to get startup CR modes from SPA'.
2. Check that the spad process is running by looking for the spad.exe (Windows) or spad (Unix/Appliance) process.
   • On Windows: Use 'taskmgr' or 'tasklist' to locate the spad.exe process.
   • On Unix/Linux: Use 'ps -ef | grep spad'
3. Once you have verified that the spad process is running, you should verify that it is listening on TCP port 10102 using the command: netstat -an
   • As part of this check, ensure that no firewalls are blocking local communication to TCP port 10102.
4. Check that the file containing the CR modes is accessible and not empty. This will be located in the plain ASCII file: <dedupe_location>\databases\spa\database\contentroutermode\1
   • On Windows: type E:\msdp\databases\spa\database\contentroutermode\1
   • On UNIX/Linux: cat /msdp/cat/databases/spa/database/contentroutermode/1
   • Note: The file should contain 9 fields with the pipe character as a field delimiter.
   • For example:
     1|1|1|1|1|2|1|1231907545|1231907545|
   • The first 7 fields correspond to the modes normally output from the command 'crcontrol --getmode', which looks like this:
     Mode : GET=Yes PUT=Yes DEREF=Yes SYSTEM=Yes STORAGED=Yes REROUTE=No COMPACTD=Yes
   • If the file is missing, corrupt, unreadable or otherwise not in the format described above, contact Support.
5. Check the spoold.log to ensure that the spoold process is attempting to connect to spad on the local host. If there are bad DNS or hosts file entries, the spoold process may be attempting to connect to spad on a remote host, which will fail due to authentication issues.