Failed to obtain SSO certificate using VMWare vCloud suite APIs

Problem

VMware type backup policies that make use of VMware Intelligent Policy queries, specifically identifying virtual machines by VMware tags, may fail with this error message.  This could happen with new policies or with polices that were previously working.  Other VMware backup policies that do not make use of VMware tags to identify virtual machines will continue to backup successfully.

Error Message

There are several variants of this error that may be observed including:

"Failed to obtain sso certificate using VMWare vCloud suite APIs"

"Failed to obtain SSO certificate using VMware vCloud Suite APIs msg com.vmware.vapi.std.errors.Unauthenticated"

"04/21/2019 18:16:01 - Error nbpem (pid=11911) Failed to obtain SSO certificate using VMware vCloud Suite APIs, msg = [Client received SOAP Fault from server: The time now Sun Apr 21 22:00:01 UTC 2019 does not fall in the request lifetime interval extended with clock tolerance of 600000 ms: [ Sun Apr 21 22:06:00 UTC 2019; Sun Apr 21 22:36:00 UTC 2019). This might be due to a clock skew problem. Please see the server log to find more detail regarding exact cause of the failure.], display name = [displayname], server = [servername]"

"May 15, 2019 1:51:02 PM - Error nbpem (pid=5000) Failed to obtain SSO certificate using VMware vCloud Suite APIs, msg = [com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
Operation Status: 200"

"scheduler found no backups due to run  (200)"

Cause

The usual connection made for a VMware agent backup is from the NetBackup Master server and/or the NetBackup VMware backup host directly to the vCenter server.  In the case of a backup policy that uses a VMware Intelligent Policy query that makes use of VMware tags an additional connection is required to a VMware server known as the VMware Platform Services Controller (PSC) server.  This connection relies on VMware Single Sign-On (SSO) certificates which are time sensitive.  If the times between the different systems involved has drifted this connection will fail.

Note the VMware Platform Services Controller (PSC) may or may not be the vCenter server.  These two roles can be deployed on a single server or on separate servers. 

Solution

Make sure the operating system time stamps on the following servers are synchronized:

- NetBackup master server.

- NetBackup VMware backup hosts

- VMware vCenter server

- VMware Platform Services Controller (PSC) server

Related articles:

VMware policies using tags show zero included and zero excluded VMs in the Test Query results