Managed hosts with duplicate virtual machine identifiers are automatically removed by Veritas Operations Manager (VOM)

Problem

Veritas Operations Manager can manage virtual hosts (configured as agents), only if they have unique virtual machine identifiers associated with them. If such hosts with duplicate identifiers (BIOSUUID for VMware virtual machine) are found, they are automatically removed from the Veritas Operations Manager database.

Error Message

The Managed Server log file is SFMdb3.dblog under /var/VRTSsfmcs (Linux) and \ProgramData\Symantec\VRTSsfmcs\logs (Windows). Then look for entries in SFMdb3.dblog starting with deprecate_host_id_list having one or more entries in the list as seen below:

I. 10/18 09:59:18. deprecate_host_id_list:({00120050-5605-11a2-0000-00001e1ee4c0},{00120050-5605-11a1-0000-00001e1ee4c0},{00120050-5605-129a-0000-00001e1ee4c0})
I. 10/18 09:59:18. SP_UPDATE_PADV({00120050-5605-11cd-0000-00001e1ee4c0})
I. 10/18 09:59:18. SFMS:SP_delete_host:ENTER:config_id({00120050-5605-11cd-0000-00001e1ee4c0}),config_type(AGENT),hostname(server2),hostid({00120050-5605-11a1-0000-00001e1ee4c0})
I. 10/18 09:59:18. SFMS:SP_delete_host_simple_host:ENTER:config_id({00120050-5605-11cd-0000-00001e1ee4c0}),config_type(AGENT),hostname(server2),hostid({00120050-5605-11a1-0000-00001e1ee4c0})

 I. 10/18 09:59:23. SFMS:SP_delete_host:ENTER:config_id({00120050-5605-11cd-0000-00001e1ee4c0}),config_type(AGENT),hostname(server1),hostid({00120050-5605-11a2-0000-00001e1ee4c0})
I. 10/18 09:59:23. SFMS:SP_delete_host_simple_host:ENTER:config_id({00120050-5605-11cd-0000-00001e1ee4c0}),config_type(AGENT),hostname(server1),hostid({00120050-5605-11a2-0000-00001e1ee4c0})
 

If the Management Server has the same UUID as a Managed Host this will be the errors seen in the Add Host Log:

[10/25/2013 10:47:20] Executing: "C:\Program Files\Veritas\VRTSsfmh\bin\xprtlc.exe" -l https://localhost:5634/agent/domain/unregister -d id={00120050-5605-0250-0000-0000943a2245}
[10/25/2013 10:47:20] Executing: "C:\Program Files\Veritas\VRTSsfmh\bin\xprtlc.exe" -u vxss:///sfm_admin -l https://managedhost/admin/cgi-bin/ping.pl
[10/25/2013 10:47:34] fancy_die
[10/25/2013 10:47:34] ADD_HOST_VERIFY_FAILED
[10/25/2013 10:47:34] fancy_dead

Cause

Multiple Managed Hosts have the same UUID

Solution

It can be observed for the following two scenarios:

When the Management Server is upgraded to version 6.0:

The hosts which are configured as agents in Veritas Operations Manager (with duplicate virtual machine identifiers) will be removed from Veritas Operations Manager during the upgrade process.

I. To identify the hosts with duplicate identifiers in Veritas Operations Manager:

1. First install one of these HotFixes:

2. Run the script installed by the HotFix to get the list of duplicate virtual machines configured in the Veritas Operations Manager domain.

For A Linux Management Server Run:

 /opt/VRTSsfmhcs/bin/check_duplicate_vmid.pl  

For A Windows Management Server Run:

c:\Program Files\Veritas\VRTSsfmh\bin\perl.exe  c:\Program Files\Veritas\VRTSsfmcs\bin\check_duplicate_vmid.pl

Sample Output:

HOST_NAME                          HOST_ID                                VM_ID                                  NOTE
---------------------------------- -------------------------------------- -------------------------------------- ----------------
Devqa-3                             {0001001a-a03b-5bf1-0000-000000000000} {423ab629-7a6f-f917-7c94-fde0cc939450} same vm_id as MS
vomrhel5u3                        {00010050-56ba-0067-0000-000000000000} {423ae0d8-0061-2694-818b-f7e8e0867929}
vomrhel5x64-vm03            {00010050-56ba-02b9-0000-000000000000} {423ae0d8-0061-2694-818b-f7e8e0867929}
vomw2k8x64-vm09            {00120050-56ba-02ba-0000-000028be49b2} {423ae0d8-0061-2694-818b-f7e8e0867929}
 

After the Management Server is upgraded to version 6.0, the managed hosts will need to be re-added to Veritas Operations Manager with the new unique virtual machine identifiers. When new host is added to Veritas Operations Manager with VMID that is already associated with an existing managed host In this case, the existing Managed Host data is removed from the Veritas Operations Manager database and only the new added host will be displayed on the Management Server console.

II. When a new Managed Host is added to Veritas Operations Manager with VMID that is already associated with an existing Managed Host:
 
In this case, the existing Managed Host data is removed from the Veritas Operations Manager database and only the new added host will be displayed on the Management Server console.

Ensure that while adding new virtual machine to Veritas Operations Manager, unique UUID is assigned to the machine. To identify each Virtual Machine has unique identifiers run one of these commands on each Managed Host:

For Windows, run this command:

wmic bios get serialnumber

Example output:
SerialNumber
VMware-11 22 33 44 55 6a a7 88-9b 7c 99 00 d1 22 3c 4b

For Linux, run this command:

dmidecode --type system

Example output:
# dmidecode 2.10
SMBIOS 2.4 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: VMware, Inc.
        Product Name: VMware Virtual Platform
        Version: None
        Serial Number: VMware-11 22 33 44 55 6a a7 88-9b 7c 99 00 d1 22 3c 4b       

If the Managed Host agent is already installed, VOM has a builtin utility that can be used called TESTVMWARE that can be run on each Managed Host: 

For Linux, run this command:

#/opt/VRTSsfmh/bin/testVMware
UUID:423a40f15e5e138d28dc8a02b8e135cc

For Windows, run this command: 

C:\Program Files\Veritas\VRTSsfmh\bin\testvmware.exe
UUID:50228f3373a4da71b5e340f00be972df

Each Virtual Machine has to have a unique identifier and if they are identical (as shown above with serial number), VOM will purge the existing duplicates from the database. This needs to be corrected on each Managed Host before it can be re-added.

VMWare has several Technical Articles on how to change the UUID and here are a few articles:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006865
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002403
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1541

 

 

 

Terms of use for this information are found in Legal Notices.

Search

Survey

Did this article answer your question or resolve your issue?

No
Yes

Did this article save you the trouble of contacting technical support?

No
Yes

How can we make this article more helpful?

Email Address (Optional)