bptestbpcd completes with 'server not allowed access'

Problem

A  client in a NetBackup enterprise is unable to make a secure connection to the master server, or any other host.

Error Message

Execution of 'bptestbpcd -client <clientName> -verbose' from the master server to the client completes with the following language:

bptestbpcd -client <clientName> -verbose
PEER_NAME = <masterName>
HOST_NAME = <clientName>
<16>bptestbpcd main: Function ConnectToPBCD(clientName) failed: 46
<16>bptestbpcd main: Request from host <masterName> (xxx.xxx.xxx.xxx) to host <clientName> (xxx.xxx.xxx.xxx) is not allowed access.
    Host <masterName> (xxx.xxx.xxx.xxx) is not an authorized server for host <clientName>.
server not allowed access

Examination of the nbwebservice logs (OID 495) at DebugLevel = 6, shows there is a failure when attempting host validation while checking the state of mappings:

nbwebservice log location:

UNIX: /usr/openv/logs/nbwebservice
Windows: install path\netbackup\logs\nbwebservice

[No context] 5 [com.netbackup.accesscontrol.checkaccess.TracingAccessChecker] Checking access on |MANAGE|HOSTS|
[No context] 4 [com.netbackup.accesscontrol.kernel.service.DefaultAccessCacheService] getting access cache for |
[No context] 5 [com.netbackup.accesscontrol.kernel.service.DefaultAccessCacheService] Searching for cache for [|].
[No context] 4 [com.netbackup.accesscontrol.kernel.service.DefaultAccessCacheService] Found cache for [|].
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Locking cache [|] for reading.
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Locked cache [|].
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] init ReaderWriter: 0,208ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lockForReading: 1,115ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,978ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,031ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,012ms
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Unlocking cache [|].
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Locking cache [|] for reading.
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Locked cache [|].
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] init ReaderWriter: 0,086ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lockForReading: 0,147ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,042ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,034ms
[No context] 4 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] lookup: 0,031ms
[No context] 5 [com.netbackup.accesscontrol.checkaccess.cache.disk.AccessCache] Unlocking cache [|].
[No context] 5 [com.netbackup.accesscontrol.checkaccess.TracingAccessChecker] Access denied to |MANAGE|HOSTS|
[No context] 1 [com.netbackup.config.hosts.controller.ControllerExceptionHandler] Request URL - [GET] https://<mastername>:1556/netbackup/config/hosts/fc6ec7ef-0129-4b00-a112-b1bc42b3409d/hostmappings
 

Cause

This issue can be caused by a bad entry in the 'access-caches' database.
The 'access-caches' database is a cached list of hosts and their access permissions.
If the cache becomes corrupt, the corruption can cause an access failure for operations which ought to be permitted by that host - such as host validation and reviewing the mappings list for hosts.

Solution

The solution is to delete the on-disk access-caches database, and allow it to recreate itself.
This is a harmless procedure, however a restart of the 'NetBackup Web Management Console' service is required.

This is the procedure:
1. Stop the service 'NetBackup Web Management Console'
    Unix/Linux: /usr/openv/netbackup/bin/nbwmc stop
    Windows: sc stop "NetBackup Web Management Console"
2. Delete the following directory 'ac-xxxxxxxxxxxxxxxxxxx' and all its contents
    Unix/Linux: /usr/openv/wmc/webserver/temp/access-caches/ac-xxxxxxxxxxxxxxxxxxx
    Windows: <install_path>\NetBackup\wmc\webserver\temp\access-caches\ac-xxxxxxxxxxxxxxxxxxx
  Example: C:\Program Files\VERITAS\NetBackup\wmc\webserver\temp\access-caches\ac-3812899307421811156
3. Start the service 'NetBackup Web Management Console'
    Unix/Linux: /usr/openv/netbackup/bin/nbwmc start
    Windows: sc start "NetBackup Web Management Console"
4. Clear the host cache on the Master and Client
    Unix/Linux: /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache
    Windows: <install_path\VERITAS\NetBackup\bin\bpclntcmd -clear_host_cache
5. From the Master Server test connectivity to the client
    Unix/Linux: /usr/openv/netbackup/bin/admincmd/bptestbpcd -client <clientName> -verbose
    Windows: <install_path\VERITAS\NetBackup\bin\admincmd\bptestbpcd -client <clientName> -verbose