Browse and select Virtual Machine is not updating

[danielrickard]

Hi there,

It seems that my Netbackup 7.5.0.5 servers are using old cached information about the names of my VMs.

VMs that were renamed fail to backup, because they can't be found. We use the backup select method of Virtual Machine Name.

I try to Browse and select Virtual Machine from the Policy Client tab and, after a while "Loading virtual machine list" it returns the machines with their old names, so I can't add them.

Please help, I need to back them up tonight! :)

[Konstantinos]

Hi Daniel,

what Primary VM Identifier are you using?

when you say "renamed" what exacly did you rename?

what is the date/time in the Browse Window (upper right corner)

Is what you see in the vShere Client the same as in the Browse for Virtual Machines Window?

[Dangerous_Dan]

Something you can try is removing the following files on your backup host:

Windows location: ..veritas\netbackup\online_util\fi_cntl

Unix Location: /usr/openv/netbackup/online_util/fi_cntl

vcbnames_<yourvcenter_server>_Any

<yourvcenter_server>host.xml

<yourvcenter_server>.xml

These are the files which makeup all the vCenter information collected. Sometimes it gets corrupted.

once removed, try your browse again, it should recreate these files, if it doesn't then we know there is a communication issue between your backup host and vcenter server - at which point try "telnet vcenter 443"

[danielrickard]

Hi Konstantinos,

We use the 'VM Display Name' as the primary identifier.

I renamed the VM display name in the vSphere client.

The date and time is 8/5/2015 (US format) 09:57:51 AM.

When I then browse and select I see a number of virtual machines that have recently been renamed in vSphere with their old names.

If I select them, the backup fails '156' as it cannot find the VM to take the snapshot. If I manually type the new name it fails '156' as it cannot find the machine name.

[danielrickard]

Daniel, thanks for the reply.

I have removed the vcbnames_myvirtualcenterserver from that location on my Master server and my three backup hosts.

Now when I browse it doesn't recreate the file on any of the servers. The server in question is listed in the browse window, but there is no + next to it to expand down to see the contents.

I can see the other Virtual Center servers OK still; I didn't touch their vcbnames file.

I have checked that port 443 is open from each of the servers. Also, VMs already in policies are still backing up OK.

Trying to add a new VM by typing its name still fails with a '156' error.

[GulzarShaikhAUS]

Between I hope you tried the "Refresh" button. If yes then what is the behaviour. Check the last update date mentioed there.

What is the scope of the issue, is it limittied to any ESXi server? Did u try to rename any other ESXI host guest to see if those changes are getting reflected.

To see any changes we need to refresh the list unless you want to wait till scheduled discovery happens.

[Dangerous_Dan · Accepted solution]

the vcb name files also need to be removed.

Also have you tried doing browsing in a different policy. I know in the past I experienced these symptoms when the actual policy was corrupted and browsing wasn't possible.

To simplify and track the precise issue do the following:

create a test VMware policy and in the VMware tab ensure display name is the VM identifyer and specify 1 backup host (and not media server)

ensures hosts file on selected backup host has dns entry for the vcenter server.

remove (or move if you feel uncomfortable) ALL files I mentioned above.

stop Netbackup services on backup host - Ensuring that all bpvmutil, bpfis, bpbkar, and proxy processes are killed once stopped (these are somtimes orphaned in 7.5).

create log folder bpVMutil

start Netbackup services

then try to browse (and remember to try refresh).

If it still fails - please upload bpvmutil log

[danielrickard]

SymGuy-IT, 

yes of course I refreshed :) 

The last update date and time is right now, but the display names of VMs are still from a week or so ago.

I'll check another VC and see if that has the same problem.

[danielrickard]

Thanks Daniel, I'll give that a go later today and let you know the outcome.

Out of interest, deleting the files from fi_cntl folder. Is that to be done on the Master server, from where I'm modifying the policy; or on the backups host(s) that are configured to run that policy?

For the record, I did it on both last time.

Thanks
Dan

[Dangerous_Dan]

Sorry for delay in response,

It is done on the backup host. If there was entries in the fi_cntl on the master, then at some point in the past you selected the master as the backup host.

[danielrickard]

Sorry for the long delay myself, I've been on holiday.

So, I've been through your steps on a single backup host. I see my 'VIRTUALCENTERSERVER' server in the list, but there are no VMs within it now - since I moved the LONVMMAN012 and VCBnames files.

The log says:

13:43:26.774 [9000.6960] format_virtualserver_cred: read 1 VIRTUALCENTERSERVER vcob331 ebf3eb1d2675b34d3c9375ba8143e999f386428171300faab3685f9b762c0bef0d66792e1c16340ed7edd818e6cc2c9c6b42e6d057fab4bfa4a4c9a72e01e4bf 19350 443

13:43:26.775 [9000.6960] <2> bpVMutil main: Saving VIRTUALCENTERSERVER guests to C:\Windows\TEMP\VMutl-09000441197806433960000000000-a06960
13:43:26.775 [9000.6960] <2> get_VM_names_VMware: get_VM_names_VMware: read for VIRTUALCENTERSERVER
13:43:26.776 [9000.6960]  read cache file = C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\vcbnames_VIRTUALCENTERSERVER_Any
13:43:26.776 [9000.6960] Unable to open fd=-1, errno 2

13:43:26.776 [9000.6960] checkBackupRegEntry: searching configuration for xmlCacheLimit
13:43:26.776 [9000.6960] get_integer_value failed
13:43:26.778 [9000.6960] get_vSphere_VMs: get_id_lock returned 0:  -> wait lock timeout value: 15 minutes -> WRITE LOCK VIRTUALCENTERSERVERvview_lock fd = 248 -> lockname = VIRTUALCENTERSERVERvview_lock
13:43:26.778 [9000.6960] getvSphereVmView: Getting VM's for VIRTUALCENTERSERVER
13:43:26.778 [9000.6960] refreshViewCache: using cache file C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\VIRTUALCENTERSERVER.xml 
13:43:26.778 [9000.6960] getvSphereVmView: no view needs to be rebuilt
13:43:26.778 [9000.6960] checkBackupRegEntry: searching configuration for disableIPResolution
13:43:26.778 [9000.6960] get_integer_value failed
13:43:26.778 [9000.6960] checkBackupRegEntry: searching configuration for UseNicIpaddress
13:43:26.778 [9000.6960] get_integer_value failed
13:43:26.819 [9000.6960] VirtualMachineProxyView is not valid: 
13:43:26.819 [9000.6960] <2> get_VM_names_VMware: get_vSphere_VMs returned 11
13:43:26.819 [9000.6960] <2> get_VM_names for VIRTUALCENTERSERVER failed with 11: bpVMutil

Interestingly I've logged a call with Symantec support and they have suggested upgrading to 7.7. 

I'm not sure how that would fix it, but we are planning on doing that this week, or early next, anyway.

[Dangerous_Dan]

They always ask to upgrade in case its either been fixed in a newer release or you've discovered a newer bug :)

Did you get an error when you tried to browse? 0 entries in fi_cntl suggests a communication issue between backup host and vcenter.

Things to try:

1. Reboot vcenter server
2. reboot backup host
3. Assuming your backup host is windows = delete eveything in C:\Windows\TEMP\   
4. In your VMware tab in the policy, I suggest for the test to select your backup host if you havent already and try browse again.