NetBackup™ Troubleshooting Guide
- Introduction
- Troubleshooting procedures
- About troubleshooting procedures
- Troubleshooting NetBackup problems
- Troubleshooting installation problems
- Troubleshooting configuration problems
- Device configuration problem resolution
- Testing the primary server and clients
- Testing the media server and clients
- Resolving network communication problems with UNIX clients
- Resolving network communication problems with Windows clients
- Troubleshooting vnetd proxy connections
- vnetd proxy connection requirements
- Where to begin to troubleshoot vnetd proxy connections
- Verify that the vnetd process and proxies are active
- Verify that the host connections are proxied
- Test the vnetd proxy connections
- Examine the log files of the connecting and accepting processes
- Viewing the vnetd proxy log files
- Troubleshooting security certificate revocation
- Troubleshooting cloud provider's revoked SSL certificate issues
- Troubleshooting cloud provider's CRL download issues
- How a host's CRL affects certificate revocation troubleshooting
- NetBackup job fails because of revoked certificate or unavailability of CRLs
- NetBackup job fails because of apparent network error
- NetBackup job fails because of unavailable resource
- Primary server security certificate is revoked
- Determining a NetBackup host's certificate state
- Troubleshooting issues with external CA-signed certificate revocation
- About troubleshooting networks and host names
- Verifying host name and service entries in NetBackup
- Example of host name and service entries on UNIX primary server and client
- Example of host name and service entries on UNIX primary server and media server
- Example of host name and service entries on UNIX PC clients
- Example of host name and service entries on UNIX server that connects to multiple networks
- About the bpclntcmd utility
- Using the Host properties to access configuration settings
- Resolving full disk problems
- Frozen media troubleshooting considerations
- Troubleshooting problems with the NetBackup web services
- Troubleshooting problems with the NetBackup web server certificate
- Resolving PBX problems
- Troubleshooting problems with validation of the remote host
- Troubleshooting Auto Image Replication
- Troubleshooting network interface card performance
- About SERVER entries in the bp.conf file
- About unavailable storage unit problems
- Resolving a NetBackup Administration operations failure on Windows
- Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
- Troubleshooting error messages in the NetBackup web UI and the NetBackup Administration Console
- Extra disk space required for logs and temporary files for the NetBackup Administration Console
- Unable to logon to the NetBackup Administration Console after external CA configuration
- Troubleshooting file-based external certificate issues
- Troubleshooting issues with external certificate configuration
- Troubleshooting Windows certificate store issues
- Troubleshooting backup failures
- Troubleshooting backup failure issues with NAT clients or NAT servers
- Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
- Troubleshooting issues with email notifications for Windows systems
- Troubleshooting issues with KMS configuration
- Troubleshooting issues with initiating the NetBackup CA migration because of large key size
- Troubleshooting issues with the non-privileged user (service user) account
- Troubleshooting issues with group name format in the auth.conf file
- Troubleshooting the VxUpdate add package process
- Troubleshooting issues with FIPS mode
- Troubleshooting issues with malware scanning
- Troubleshooting issues with NetBackup jobs that are enabled for data-in-transit encryption
- Troubleshooting issues with Unstructured Data Instant Access
- Troubleshooting issues with multifactor authentication
- Troubleshooting issues with multi-person authorization
- Troubleshooting connections to the NetBackup Scale-Out Relational Database
- Troubleshooting issues with private key encryption
- Troubleshooting issues with the security configuration risk feature
- Troubleshooting issues with the risk engine-based anomaly detection options
- Using NetBackup utilities
- About NetBackup troubleshooting utilities
- About the analysis utilities for NetBackup debug logs
- About the Logging Assistant
- About network troubleshooting utilities
- About the NetBackup support utility (nbsu)
- About the NetBackup consistency check utility (NBCC)
- About the NetBackup consistency check repair (NBCCR) utility
- About the nbcplogs utility
- About the robotic test utilities
- About the NetBackup Smart Diagnosis (nbsmartdiag) utility
- About log collection by job ID
- Disaster recovery
- About disaster recovery
- Recommended backup practices
- Requirements and notes for disaster recovery
- Disaster recovery packages
- About disaster recovery settings
- About disk recovery procedures for UNIX and Linux
- About clustered NetBackup server recovery for UNIX and Linux
- About disk recovery procedures for Windows
- About clustered NetBackup server recovery for Windows
- Generating a certificate on a clustered primary server after disaster recovery installation
- About the DR_PKG_MARKER_FILE environment variable
- Restoring the disaster recovery package on Windows
- Restoring the disaster recovery package on Linux
- Options to recover the NetBackup catalog
- Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files
- About NetBackup catalog recovery on Windows computers
- About NetBackup catalog recovery from disk devices
- About NetBackup catalog recovery and symbolic links
- NetBackup disaster recovery email example
- About recovering the entire NetBackup catalog
- About recovering the NetBackup catalog image files
- About recovering the NetBackup databases
- Recovering the NetBackup catalog when NetBackup Access Control is configured
- Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
- Recovering the NetBackup catalog without the disaster recovery file
- Recovering a NetBackup user-directed online catalog backup from the command line
- Restoring files from a NetBackup online catalog backup
- Unfreezing the NetBackup online catalog recovery media
- Steps to carry out when you see exit status 5988 during catalog recovery
Troubleshooting issues with malware scanning
(Applicable on scan host RHEL 8.x and NFS version 4.x) When scanning large size backup (~ 200 million files), following error is displayed on the Web UI for failed job:
Failed to get response from NetBackup malware utility.
While scan is in progress on scan host, NFS mount points are not accessible from scan host. Scan job remains in progress and timeout after two days. NFS exports on storage server are accessible.
: Ensure that you use NFS version 3 for mounting IA mounts on scan host over NFS by setting the following configuration in /etc/nfsmount.conf file on scan host:
# grep Defaultvers /etc/nfsmount.conf Defaultvers=3
SSH connection to scan host from media server failed.
: Verify the following scan host credentials:
RSA (SHA256) key
User name
Password
Refer to NetBackup Web UI Administrator's Guide for the scan host configuration.
Error can be due to unsupported scan host.
: For a complete list of supported platforms for the scan host, refer to the Software Compatibility list document.
Not enough space is available on the scan host.
SSH user does not have access to the required directories on the scan host.
On a Windows scan host, check for space availability in
C:\folder.On a Linux scan host, check for space availability in
/tmpfolder.
Media server is not able to fetch the credentials to access scan host from the Primary.
: Check that credentials for scan host are specified.
Default scan operation time out is two days. Time to scan may vary depending on the factors sch as workload type, network bandwidth, backup size.
: Scan time-out is configurable and can be changed by setting the configuration key.
Minimum value: 1 hour
Maximum value: 30 days
Mismatch between nbmalwareutil binary and the ScanManager
:
Contact NetBackup support.
Malware scanner-specific failure message.
: Refer to nbmalwarescanner logs on the media server for agentless host type pools, or on scan host if it is agent based scan.
IA share is not accessible from the scan host.
: Check IA configuration on storage server. Verify on activity monitor that IA job is successful.
IA share is busy or not accessible.
: Refer to nbmalwarescanner logs on the media server for agentless host type pools, or on scan host if it is agent based scan.
Generic failure during the scan of a backup image.
: Refer to nbmalwarescanner logs on the media server for agentless host type pools, or on scan host if it is agent based scan.
Generic failure during the scan of a backup image.
:
Verify if any scan is in progress.
If no scan is in progress, then obtain the list of such instant access mounts with ID's of the instant access mount created using the GET IA API from the following directory:
/netbackup/recovery/workloads/{workload}/instant-access-mounts
Using the DELETE API, delete the instant access mount:
/netbackup/recovery/workloads/{workload}/instant-access-mounts/{mounId}
Only five backup images can be mounted at the same time on windows scan host.
:
Ensure that scan host is not part of multiple NetBackup domains.
Check if there are any Stale mounts on the scan host by running net use.
Following drive letters are used for mounting the IA shares on the windows scan host. Ensure that they are not in use.
L:\ M:\ N:\ O:\ P:\
Microsoft Windows Defender is not installed on the scan host or not configured properly.
: Ensure that Microsoft Windows Defender is installed on scan host.
Refer NetBackup Web UI Administrator's Guide for the scan host configuration.
Symantec Protection Engine is not installed on the scan host or not configured properly.
: Ensure that Symantec Protection Engine is installed on scan host.
Refer NetBackup Web UI Administrator's Guide for the scan host configuration.
Generic error for Scan failure.
: Contact NetBackup support.
Storage server host name cannot be more than 15 characters for the SMB share support.
If Windows Server 2016 is used to set up Active Directory domain, then it does not allow a connection to a storage server with host name of length more than 15 characters.
: Ensure that the character limit is not more than 15 characters.
Generic failure during scanning backup image.
: Check for the following errors:
Refer to
nbmalwarescannerlogs on the media server for agentless host type pools, or on scan host if it is agent based scan.Check for space on media server storage.
Check for NFS service failure on media server.
Review the nbmalwarescanner to view the infected files list for the backup images in the selected date range.
: Update the date range or recovery files and folders selection to reduce the number of infected files. Retry the operation. You can also perform one of the following:
Select the option which can be used to recover selective clean files.
Skip that backup image from recovery.
There are too many infected files in the selected scan result. If the scan result has infected files greater than 5000, the following message is displayed:
Large number of infected files. To view the complete list of infected files, export the list.
: Export the infected file list in
.csvformat and download it to view it.There are many infected files in the selected scan result or the infected file paths are long to be captured in the database. Following error message is displayed:
Large number of infected files.
: This result cannot be exported or viewed.
: As the results cannot be exported or viewed, review the scan logs to view a detailed list of the infected files for the selected scan result.
For large size backup, scan operation is divided into parts. For example, if total number of files in the backup are 1,000,000, the scan operation will be divided into two parts of 500,000 files each.
Each part would be created and scanned separately. Each part can be assigned with different scan host. The Malware detection UI displays only single entry for backup.
: Each divided part details can be obtained by using the REST API.
When performing a malware scan operation with the NetBackup Malware Scanner installed on the scan host, it fails with the following error message:
Missing environment variable NB_MALWARE_SCANNER_PATH
: Ensure that NetBackup Malware Scanner is installed. Note the install location.
Login on the scan host as user using the same user credentials that were provided during scan host configuration on the primary server. Add the following lines to ~/.bashrc:
export NB_MALWARE_SCANNER_PATH=<installLocation>/savapi-sdk-linux64/bin
export PATH=$PATH:$NB_MALWARE_SCANNER_PATH
Malware scanning on Windows scan host may fail if there are cygwin mks toolkit installed.
: UNIX utilities are installed, however, defined scanuser cannot have those UNIX utilities in the PATH variable.
Error/Issue | Description | Workaround |
|---|---|---|
|
|
|
When upgrading NetBackup from previous version to NetBackup version 10.3 or later with the following options selected, the No images match the search criteria message is displayed:
Options | Fields |
|---|---|
: Backup images | : NAS-Data-Protection : Copy2 : Not scanned (Default) |
: Assets by policy type | : NAS-Data-Protection : Copy2 : Select the required scanner host pool. : Not scanned (Default) |
Workaround
To view the images that are backed up, ensure that you select the option as to scan the NAS-Data-Protection backup images created on earlier version of NetBackup media server.
When using Instant Access mount points for malware scan (traditional malware scan) in NetBackup versions prior to 10.3, performance issues were observed.
Workaround: Upgrade to NetBackup media and storage server 10.3 or later. NetBackup 10.3 introduces the feature. This improves the instant access time as well as the scan performance.
The following table provides the differences between the traditional malware scan and dynamic scan:
Key scanning procedure | Traditional malware scan using Instant Access mount points | Dynamic scan |
|---|---|---|
Instant access stage. | Analyzes the tar stream and builds each file's header and extent map file (LMDB database), which is time consuming for large number of files in the backup. | Restores TIR (catalog database) and IM (image metadata) information from fragment. |
Instant access share (NFS/SMB) is mounted and user tries to list or access the file. | Accesses it's header file and reads the attribute from it. | Query's the directory from catalog database to get all the files and directories which are under this directory. It can also query each files and directories attribute to the output. |
Scan host opens a file | Opens and loads the LMDB database. | Builds the index in memory and reads directly from data container.
|
Scan host reads a file | Searches from LMDB database and reads from data container. | If storage server is 3rd party storage vendor, it reads data through OST interface directly. If storage server is PureDisk, it searches from mapping table and reads data from data container. |
The following table provides the details for the respective log files to be viewed depending on the use case:
Table: Log file locations for Agentless scan host
Use case | Components on primary server | Components on media server | Log file path |
|---|---|---|---|
Configurations | nbwebservice | ncfnbcs | For primary server:
For media server:
|
Scan process | nbwebservice bprd | ncfnbcs nbmalwarescanner | |
Recovery | nbwebservice bprd |
Table: Log file locations for NetBackup client as the scan host
Use case | Components on primary server | Components on the scan host client | Log file path |
|---|---|---|---|
Configurations | nbwebservice | nbsubscriber |
|
Scan process | nbwebservice bprd | nbsubscriber | |
Recovery | nbwebservice bprd |
For VMWare VM backup scan, ensure that you use scan user with uid=0. SSH login is disabled by default and user may not enable it for security reasons.
Workaround
In above scenario, perform the following:
If SSH login is disabled for the root user, then non-root scan user can be added to group 0 (root) to be able to scan all the files.
For example, uid=1001(scanuser) gid=1001(scanuser) groups=1001(scanuser),0(root)