Important Update: Cohesity Products Documentation


All Cohesity product documentation are now managed via the Cohesity Docs Portal: https://docs.cohesity.com/HomePage/Content/home.htm. Some documentation available here may not reflect the latest information or may no longer be accessible.

NetBackup™ for Nutanix AHV Administrator's Guide

Last Published:
Product(s): NetBackup (11.1)
  1. Overview
    1.  
      Overview of configuring and protecting AHV assets in the NetBackup web UI
  2. RBAC roles for the Nutanix AHV administrator
    1.  
      RBAC roles for the Nutanix AHV administrator
    2.  
      Assign both the Default VMware Administrator and Default AHV Administrator roles to a user
    3.  
      Create a custom role for all Nutanix AHV permissions and additional VMware asset permissions
    4.  
      Create a custom role for all VMware permissions and additional Nutanix AHV asset permissions
  3. Managing AHV clusters
    1.  
      Quick configuration checklist to protect AHV virtual machines
    2.  
      Configure secure communication between the AHV cluster and NetBackup host and Nutanix Prism Central and NetBackup host
    3.  
      Enable the iSCSI initiator service on windows backup host
    4.  
      Install the iSCSI initiator package on Linux backup host
    5.  
      Migrate Java GUI/CLI added clusters into the web UI
    6.  
      Prerequisites to configure Nutanix AHV cluster
    7.  
      About support for Nutanix segmented iSCSI network
    8.  
      Configure CHAP settings for iSCSI secure communication with AHV clusters
    9.  
      About the ports that NetBackup uses to communicate with AHV
    10.  
      Add or browse an AHV cluster
    11.  
      Remove AHV Clusters
    12.  
      Add a new Nutanix Prism Central
    13.  
      Add new Prism Central server credentials
    14.  
      Remove Nutanix Prism Central
    15.  
      Create an intelligent VM group
    16.  
      Assign permissions to the intelligent VM group
    17.  
      Update the intelligent VM group
    18.  
      Remove the intelligent VM group
    19.  
      Set CHAP for iSCSI
    20.  
      Add an AHV access host
    21.  
      Remove an AHV access host
    22.  
      Change resource limits for AHV resource types
    23.  
      Change the autodiscovery frequency of AHV assets
    24. Scan for malware
      1.  
        Scanning backup images
      2.  
        Assets by workload type
  4. Managing credentials
    1. Managing AHV cluster credentials
      1.  
        Add new cluster credentials
      2.  
        Update and validate AHV cluster credentials
    2. Managing Nutanix Prism Central credentials
      1.  
        Add new Nutanix Prism Central credentials
      2.  
        Update and validate Nutanix Prism Central credentials
    3.  
      View the credential name that is applied to an asset
    4.  
      Edit or delete a named credential
  5. Instant access
    1.  
      Prerequisites of instant access
    2.  
      Things to consider and limitations before you use the instant access feature
    3.  
      Create an instant access VM
    4.  
      Download files and folders from a VM backup image
    5. Instant access Build Your Own (BYO)
      1.  
        Prerequisites of Instant Access Build Your Own (BYO)
      2.  
        Hardware configuration requirement of Instant Access Build Your Own (BYO)
      3.  
        Frequently asked questions
  6. Protecting AHV virtual machines
    1.  
      Things to know before you protect AHV virtual machines
    2.  
      Protect AHV VMs or intelligent VM groups using Protection plan
    3.  
      Backup AHV VMs or intelligent groups using Policy
    4.  
      Protect AHV VMs within VPC
    5.  
      Protect vTPM enabled AHV VMs
    6.  
      Customize protection settings for an AHV asset
    7.  
      Modify policy for an AHV asset
    8.  
      Schedules and retention
    9.  
      Backup options
    10.  
      Prerequisite to Enable virtual machine quiescing
    11.  
      Remove protection from VMs or intelligent VM groups
    12.  
      View the protection status of VMs or intelligent VM groups
  7. Recovering AHV virtual machines
    1.  
      Things to consider before you recover the AHV virtual machines
    2.  
      About the pre-recovery check
    3.  
      Recover an AHV virtual machine
    4.  
      Recover an AHV VM within VPC
    5.  
      Recover a vTPM enabled AHV VM
    6.  
      About Nutanix AHV agentless files and folders restore
    7.  
      Prerequisites for agentless files and folder recovery
    8.  
      SSH key fingerprint
    9.  
      Recover files and folders with Nutanix AHV agentless restore
    10.  
      Recovery target options
    11.  
      Pre-recovery checks for Nutanix AHV
    12.  
      About Nutanix-AHV agent-based files and folders restore
    13.  
      Prerequisites for agent-based files and folder recovery
    14.  
      Recover files and folders with Nutanix AHV agent based restore
    15.  
      Limitations
  8. Protecting Nutanix Cloud Clusters (NC2)
    1.  
      Protecting Nutanix Cloud Clusters (NC2) on AWS
    2.  
      Protecting Nutanix Cloud Clusters (NC2) on Azure
  9. Troubleshooting AHV operations
    1.  
      Troubleshooting AHV operations: Error during creation of AHV instant access virtual machines
    2.  
      Troubleshooting tips for NetBackup for AHV
    3.  
      Error during AHV credential addition
    4.  
      Error during the AHV virtual machines discovery phase
    5.  
      Errors for the Status for a newly discovered VM
    6.  
      Error run into while backing up AHV virtual machines
    7.  
      Error while restoring AHV virtual machines
  10. API and command line options for AHV
    1.  
      Using APIs and command line options to manage, protect, or recover AHV virtual machines
    2.  
      Additional NetBackup options for AHV configuration
    3.  
      Additional information about the rename file

Error while restoring AHV virtual machines

The following table describes the problem that might occur when you restore an AHV virtual machine.

Table: Error run into while restoring AHV virtual machines

Error message or cause

Explanation and recommended action

VM recovery to alternate location fails on a Windows primary server.

For a windows NetBackup primary server, ensure that the rename file ends with an empty line.

Unable to change the AHV cluster while modifying the recovery destination.

If you cannot see the list of the AHV clusters, you might not have access to the AHV clusters in RBAC.

Contact the NetBackup security administrator to resolve this issue.

Pre-recovery check runs successfully when a VM with the same UUID exists in the AHV cluster and the option to overwrite the VM is not enabled, but the VM restore fails.

The following error message is seen:

Info bpVMutil (pid=1196) FTL - Virtual machine exists and overwrite option not specified, cannot proceed with restore. End Restore; elapsed time Hypervisor policy restore error. (2822)

The pre-recovery check compares the VM display name instead of UUID to find out if the VM already exists, hence the check completes successfully. But if the overwrite option is not set, the restore job fails if a VM with the same UUID already exists.

Workaround:

Restore the VM with a new UUID.

  1. Start the recovery process.

  2. On the Recovery Options page, click Advanced.

  3. Enable Create a new VM UUID.

  4. Proceed with the recovery process and click Start recovery to restore.

Overwrite the existing VM that has the same UUID.

  1. Start the recovery process.

  2. On the Recovery Options page, enable the Overwrite existing virtual machine option.

  3. Proceed with the recovery process and click Start recovery to restore.

When you try to recover an AHV VM image that is imported from a different domain using the web UI, the pre-recovery check fails and displays that by default the recovery host is the same access host that was used during backup.

During the recovery of imported AHV VM images, select the access host in the target domain as a recovery host or select the target primary server.

MSiSCSI service is disabled. Enable the MSiSCSI service on the recovery host.

Enable the Microsoft iSCSI Initiator Service (MSiSCSI) service on the Windows backup recovery and re-run the job.

Failed to connect to the recovery host.

The recovery which is host used for agentless restore is not reachable.

Recommended action:

Ensure that the recovery host is reachable from the primary server and has the NetBackup media server or the client software installed on it.

The specified recovery host must be at NetBackup version 9.1 or later to support agentless restores.

Agentless restores of files or folders require a recovery host with NetBackup version 9.1 or later.

Recommended action:

Verify the NetBackup version on the recovery host. It should be 9.1 or later.

On UNIX NetBackup servers and clients, verify the /usr/openv/netbackup/bin/version file.

On Windows NetBackup servers, verify the install_path\netbackup\version.txt file.

Recovery host staging location does not exist.

Staging location path does not exist on recovery host for agentless restore.

Recommended action:

  • Ensure that the default staging location path or the user-configured staging location path for the recovery host is valid. NetBackup uses the following on the recovery host as the default staging location:

    • For UNIX: {installpath}/openv/tmp/staging.

    • For Windows: {installpath}\Netbackup\Temp\staging\.

  • Ensure that the staging location path that is used exists. For user-configured staging location, verify if a valid path on the recovery host is specified in bp.conf parameter AGENTLESS_RHOST_STAGING_PATH = "path".

Tar image not found at staging location on recovery host.

No tar image was found on the recovery host staging location. This image is required for agentless restore.

Recommended action:

Contact Cohesity Technical Support and share bpVMutil log from the recovery host.

Internal error has caused failure of recovery validation.

An internal error occurred while running the pre-recovery validations for agentless restore.

Recommended action:

Save the bpVMutil logs on recovery host and contact Cohesity Technical support.

Not enough space available on recovery host.

The recovery host may not have enough space to copy the selected files at the staging location for agentless restore.

Recommended action:

Ensure that sufficient free space is available on the recovery host staging location based on the total size of files or folders selected. Or select a different recovery host with sufficient free space to perform the agentless restore.

Tar utility is not present on the target host.

Failed to find the tar utility on the target host. This utility is required for agentless restore.

Recommended action:

Retry after deploying the tar utility.

Either specified staging location does not exist on target host or the user does not have required permission to access.

Recommended action:

Ensure that the target host staging location exists, and the user has sufficient permissions to access the location.

The user does not have required permission on the target host staging location.

The user does not have required permission to proceed with restore on the target host.

Recommended action:

Ensure that the target host staging location exists and the user has the minimum write and run permission on the staging location.

The user does not have root/administrator privileges. To restore files and folders, provide user with root or administrator privileges.

The user does not have required permission to proceed with restore on the target host.

Recommended action:

Provide the credential which is part of the local administrator group on the Windows target host. For Linux target host, use the credential which is root or sudo account with ALL permissions.

Admin share of the target host is not accessible from the recovery host.

The admin share of the remote host is not accessible from the recovery host to perform an agentless restore.

Recommended action:

  • Ensure that firewall exceptions are set up correctly.

  • Ensure that File and Printer Sharing is enabled.

  • Ensure that the GPO/Software Restriction Policy or Antivirus does not block access.

  • Ensure that the target host is accessible and ensure that the correct credentials are entered and have proper permissions.

For agentless files or folders restore in User Account Control (UAC) environment, provide credential of domain user which is a part of local Administrator group on the Windows target host.

Recommended action:

For an agentless restore in the User Account Control (UAC) environment, provide the credential of the domain user. This user is the part of the local Administrator group on the Windows target host.

Agentless restore is not possible.

Received an unexpected reason for agentless restore failure.

Recommended action:

Contact Cohesity Technical support and share appropriate logs.

Operating systems do not match. Ensure that the operating system of recovery host matches with the backed-up VM operating system.

Agentless restore is possible only when operating system of recovery host and backed up VM is the same.

Recommended action:

The alternate recovery host must have the same operating system as the VM that was backed up.

Failed to retrieve the backup image operating system.

Unable to retrieve operating system of backup image for performing an agentless restore. It is an internal error.

Recovery host operating system is not compatible with provided communication mode. Ensure that the operating system of recovery host and provided communication mode are compatible.

The recovery host OS type and the communication type that was provided in the agentless recovery or pre-recovery check request are not compatible.

Recommended action:

Verify that Recovery host OS type and the communication type is compatible for the recovery host:

  • Linux: The communication type must be SSH.

  • Windows: The communication type must be WMI.

Target host SSH private key is invalid.

The sshKey field of the agentless recovery or pre-recovery check request must be valid and a non-empty ssh private key of the target host.

Recommended action:

Verify that the sshKey field is specified, if the authentication type is SSH_KEY and that it is not empty.

Target host operating system is not supported for the agentless files or folders restore.

Target host operating system is not supported since agentless restore requires recovery packages to be deployed on the target host.

Recommended action:

Only SUSE Linux Enterprise Server, Microsoft Windows, Red Hat Enterprise Linux (RHEL), and Ubuntu are supported platforms.

Refer to the NetBackup Software Compatibility List for the supported platforms for this feature.

Invalid target host username or password.

The username and password fields in authentication details of the agentless recovery or pre-recovery check request must be specified.

Recommended action:

Verify that the username and password field in the authentication details of the recovery and pre-recovery check request are specified, correct, and are not empty.

Target host staging location path contains non-ASCII characters.

Target host staging location path supports ASCII characters only.

Recommended action:

Provide a custom staging location on the target host with ACSII characters only.

Specified path does not exist on the local disk.

Target host staging location should not be the network path.

Recommended action:

Specify a custom staging location on the target host which is on its local disk.

WMI connection to the target host is failed.

WMI connection to the target host is failed from recovery host.

Recommended action:

  • To connect with the WMI and the DCOM services the user must have the required permission to connect with the remote WMI service.

  • Firewall exceptions are set up to allow WMI traffic through the firewall.

  • GPO/Software restriction policy or an antivirus does not block the access.

  • Ensure that the target host is accessible. Validate the given target host credentials.

  • Ensure that the target host trust relationship with domain is intact. When you communicate across domains, a two-way trust relationship between those domains must exist.

Unable to find the specified file on the remote server.

Unable to find the specified file on the remote server.

Recommended action:

Ensure that the specified staging location on the target host exists or specify another valid staging location.

File exists with same name as the directory.

On the target host a pre-existing file with same name exists as that in the directory path of the staging location.

Recommended action:

Check for a pre-existing file on a remote host that has the same name and path as the staging location. If it exists, either rename or remove that file. Or specify an alternate staging location.

Failed to validate administrative privileges for the user.

Target host user doesn't have the administrative privileges for the agentless files and folders restore operation to proceed.

Recommended action:

Use the credential which is part of the local administrator group on the Windows target host.

For Linux target host, use the credential which is root or sudo account with ALL permissions.

Failed to connect a network resource using windows API.

The admin share of the target host is not accessible from the recovery host for agentless files or folders restore.

Recommended action:

As a part of agentless files and folders restores operation, SMB Admin share is created from recovery host on target host with the credential provided by user. This error usually occurs when the target host for agentless restore has the Windows OS and the admin share of the target host is not accessible from the recovery host. Ensure that the following requirements are met on the target host.

  • Firewall Exceptions are set up correctly.

  • File and Printer Sharing is enabled.

  • GPO/Software restriction policy or antivirus does not block the access.

  • Target host is accessible with valid credentials.

Unable to retrieve user's home directory on the target host. Specify the custom staging location.

The user's default staging location that is on the home directory can't be retrieved on the target host. Enter a valid custom staging location path.

Recommended action:

Ensure that the user's home directory exists or try with a valid custom staging location.

Failed to establish SSH session with host.

Ensure all of the following criteria are satisfied and then retry.

  • Aes256-ctr is the supported cipher used for communication. Ensure that this cipher is supported both in recovery host and target host.

  • Ensure at least one of the following the Hash-based Message Authentication Code (HMAC) protocol is supported on both recovery host and target host:

    • hmac-sha2-256

    • hmac-sha2-512

  • Ensure that the method used that is used to generate the host key is one of the following:

    • ECDSA_SHA2_NISTP256

    • ECDSA_SHA2_NISTP384

    • ECDSA_SHA2_NISTP521

    • SSH_RSA

    • SSH_DSS

Failed to verify SSH key fingerprint of host.

The SSH key fingerprint of the target host that was provided is not correct.

Recommended action:

Verify the SSH key fingerprint of the target host and retry.

Failed to authenticate the host with provided username or password.

Target host authentication is failed with the provided username and password.

Recommended action:

Verify the username or password of the target host is correct and retry.

Failed to authenticate the host with specified SSH key.

Target host authentication is failed with the provided SSH private key.

Recommended action:

Verify that the SSH private key and the key passphrase are valid that were used to generate the SSH private key of the target host. Then retry. Ensure that the corresponding public key is present in the authorized_keys file in /root/.ssh folder at the target host.

Matching SSH Key fingerprint host key method not found on target host.

Unable to find the specified SSH key fingerprint host-key method on the target host.

Recommended action:

Ensure that a supported host key method of the specified SSH key fingerprint is available on the target host. Or provide the SSH fingerprint of the host key method that is configured on the target host.

The restore fails when you restore individual files to a virtual machine that has NetBackup client software.

When you restore individual files to a virtual machine that has a NetBackup client, make sure that a firewall does not interfere with the restore. If a firewall stops the restore, turn off the firewall and retry the restore.

Mount points not available when restoring files from a Linux virtual machine.

For Linux virtual machines, only the ext2, ext3, ext4 and xfs file systems are supported for individual file restore.

If a partition is formatted with some other file system, the backup succeeds but NetBackup cannot map the file system addresses of the files. As a result, NetBackup cannot restore individual files from that partition. Only the files that were on ext2, ext3, ext4 and xfs partitions can be individually restored.

Note:

To restore individual files from their original mount points, the "/" (root) partition must be formatted as ext2, ext3, ext4 or xfs. If the "/" (root) partition is formatted with a different file system such as ButterFS, the mount points cannot be resolved. In that case, you can restore ext2, ext3, ext4 or xfs files from the /dev level (such as /dev/sda1). You cannot restore the files from their original mount point level.

For Linux VMs without persistent device naming, multiple disk controllers such as IDE, SCSI, and SATA may complicate the recovery of individual files.

This issue occurs because of non-persistent device naming, such as /dev/sda and /dev/sdb, may cause unexpected mount point changes after a restart. If the VM has a SCSI disk and SATA disk, the Restore files and folders > Add files and folders navigation interface may show incorrect mount points for the VM's files. For example, the files originally under  /vol_a might appear under /vol_b when you browse to restore them. The restore is successful, but the restored files may not be in their original directories.

Recommended action:

Search for the files on the restored VM and move them to the proper locations. To prevent this issue on Linux VMs with multiple disk controllers, Veritas recommends a persistent device-naming method for mounting the file systems. When persistent naming is in place, device mounting is consistent and this issue does not occur when you restore files from future backups. For persistent device naming, you can mount devices by UUIDs.

The following is an example of the  /etc/fstab file that contains the devices that are mounted using UUIDs:

  • UUID=93a21fe4-4c55-4e5a-8124-1e2e1460fece /boot ext4 defaults 1 2.

  • UUID=55a24fe3-4c55-4e6a-8124-1e2e1460fadf /vola ext3 defaults 0 0.

To find the device UUIDs, you can use either of the following commands:

  • blkid

  • ls -l /dev/disk/by-uuid/

For Ubuntu VMs without persistent device naming, the Restore files and folders > Add files and folders navigation interface may show incorrect mount points for the VM's files and recovery of individual file may fail.

This issue occurs because of non-persistent device naming and may cause unexpected mount points changes. For the Ubuntu VM, the Restore files and folders > Add files and folders navigation interface may show incorrect mount points for the VM's files. For example, files and folders might appear under /dev/ubuntu-vg/ubuntu-lv when you browse to restore them and recovery of individual files may fail.

Recommended action:

To prevent this issue on Ubuntu VMs, Cohesity recommends a persistent device-naming method for mounting the file systems. When persistent naming is in place, device mounting is consistent and this issue does not occur when you restore files from future backups. For persistent device naming, you can mount devices by UUID.

The following is an example of the  /etc/fstab file that contains the devices that are mounted using UUIDs:

  • UUID=93a21fe4-4c55-4e5a-8124-1e2e1460fece /boot ext4 defaults 1 2.

  • UUID=55a24fe3-4c55-4e6a-8124-1e2e1460fadf /vola ext3 defaults 0 0.

To find the device UUIDs, you can use either of the following commands:

  • blkid

  • ls -l /dev/disk/by-uuid/

Virtual machine creation failed, cannot proceed with restore.

bpVMutil

pid=3144

If a backup host with NetBackup version 10.1.1 or earlier is used to restore VMs in Virtual Private Cloud (VPC) environment, the restore job fails.

Recommended action

Use a backup host version NetBackup 10.1.1 or later to restore VMs which are in a VPC environment.

Restore from snapshot job completes with partial successful status.

Restore from snapshot job completes with partial successful status if the AHV cluster does not have the correct configuration as per the iSCSI transport option.

Workaround

Verify and fix the following error based on iSCSI transport setting:

  • For default: The iSCSI data services IP is configured.

  • For segmented: The segmented IP address is not configured.

  • For segmented_specified: The segmented iSCSI interface is not configured or the specified IP address does not match with virtual IP of any of the segmented iSCSI interfaces that are configured.

A backup host with a NetBackup version earlier than 11.0 is not supported for the Nutanix-AHV policy.

Cohesity recommends upgrading NetBackup to latest version.

NetBackup status: 213, EMM status: NetBackup media server version is too low for the operation. No storage units available for use(213).

Check the storage unit that is used in the Nutanix-AHV policy. The media server on which the storage unit was created should be at NetBackup version 11.0 or later.

For more details check the logs at path: /usr/openv/logs/nbwebservice