Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (9.0.0.1, 9.0)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for master servers used with Auto Image Replication and SLPs
      2. Targeted AIR trusted master server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting Windows certificate store issues
    33.  
      Troubleshooting backup failures
    34.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    35.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    36.  
      Issues with email notifications for Windows systems
    37.  
      Issues with KMS configuration
    38.  
      Issues with initiating the NetBackup CA migration because of large key size
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7.  
        Establishing a connection with NAT media server before catalog recovery
      8. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      9. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      10.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      11.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      12.  
        Recovering the NetBackup catalog without the disaster recovery file
      13.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      14.  
        Restoring files from a NetBackup online catalog backup
      15.  
        Unfreezing the NetBackup online catalog recovery media
      16.  
        Steps to carry out when you see exit status 5988 during catalog recovery
  5.  
    Index

Example of an NBCC progress display

By default, the NetBackup consistency check utility (NBCC) displays its progress numerically to standard output. The name of the output file is nbcc-info.txt.

The following example of NBCC output has been edited for brevity:

1.0  Gathering initial NBCC information
1.1  Obtaining initial NetBackup configuration information

      NBCC is being run on NetBackup master server
       server1
NBCC version 8.1 Gather mode = full
NBCC command line = C:\Veritas\NetBackup\bin\support\NBCC.exe -nozip
OS name  = MSWin32
OS version = Microsoft Windows [Version 6.1.7601]
NetBackup Install path = C:\Program Files\Veritas\
> dir output\nbcc\server1_NBCC_20130227_091747 2>&1
Parsed output for "bytes free"

               5 Dir(s)  862,367,666,176 bytes free

2.0  Gathering required NetBackup configuration information
2.1  Determining the date format to use with NetBackup commands...
       Using the date format /mm/dd/yyyy
2.2  Building EMM host configuration information...
     Detected the EMM Server hostname
         lidabl11
       Detected the EMM master server hostname
         lidabl11
       Detected the EMM Virtual Machine entry
         pambl11vm3
       Detected the EMM NDMP Host entry
         fas3240a
     ...
2.3  Obtaining EMM server aliases...
     EMM aliases for detected EMM Server
         server1
           lidabl11.acme.com
       EMM aliases for detected master server
         server1
           lidabl11.acme.com
       EMM aliases for detected media server
         server4
     ...
2.4  Obtaining Storage Server information...
       Detected FalconStor OST direct copy to tape Storage Server
             falconstorvtl5
2.5  Building NetBackup storage unit list...
     Detected Storage Unit for NetBackup for NDMP media server
         reabl3
       and NDMP Host
         falconstorvtl5
       Detected disk media storage unit host
         lidabl11
       Detected Disk Pool
         lidabl11_pdde_pool
     ...
2.6  Obtaining Disk Pool information...
       Detected Disk Pool
         lidabl11_pdde_pool
           host
             lidabl11
           Detected Disk Pool lidabl11_pdde_pool member
             lidabl11
     ...
2.7  Obtaining tpconfig Storage credential information...
       Detected the master server hostname
         lidabl11
       and associated Storage server hostname
         lidabl11
     ...
2.8  Obtaining tpconfig NDMP configuration information...
       Detected the EMM NDMP Host hostname
         fas3240a
       Detected the EMM NDMP Host hostname
         fas3240b
     ...
2.9  Analyzing EMM master and/or media servers and configured
     Storage Units...
       The following EMM server entries do not have configured
       Storage Units or Disk Pools:

       Media server - lidabl14

2.10 Obtaining NetBackup unrestricted media sharing status...
       Configuration state = NO
2.11 Obtaining NetBackup Media Server Groups...
       No Server Groups configured
2.12 Building NetBackup retention level list...
3.0  Obtaining NetBackup version from media servers
       lidabl11...
       lidabl14...
       reabl3...
       virtualization5400a...
       ... 
3.1  Gathering required NetBackup catalog information
       Start time = 2013-02-27 09:41:07
3.2  Gathering NetBackup EMM conflict table list
       Found 0 EMM conflict records
3.3 Gathering list of all tapes associated with any Active Jobs
       Building NetBackup bpdbjobs list
3.4 Gathering all TryLog file names from the
     C:\Program Files\netbackup\db\jobs\trylogs
     directory
       Found 10 TryLogs for 10 active jobs.
       TryLogs found for all Active Jobs
3.5  Building NetBackup Image database contents list
       Reading Image number 1000
       Reading Image number 2000
       Reading Image number 3000
       Reading Image number 4000
       Found 4014 images in the Image database
3.6  Building EMM database Media and Device configuration
     attribute lists
       Obtaining the EMM database Media attribute list for disk
       virtual server
         lidabl11 ...
         There were 0 bpmedialist records detected for media server
         lidabl11
           Getting device configuration data from server
         lidabl11 ...
     ...
3.7  Building EMM database Unrestricted Sharing Media attribute lists
         Found 0 Unrestricted Sharing media records in the EMM database
3.8  Building the EMM database Volume attribute list...
       Getting the EMM database Volume attributes from EMM server
         mlbnbu ...
       Found 43 Volume attribute records in the EMM database
3.9  Building NetBackup volume pool configuration list
       EMM Server lidabl11
3.10 Building NetBackup scratch pool configuration list
       EMM Server lidabl11
3.11 Gathering NetBackup EMM merge table list
       Found 0 EMM merge table records

     Summary of gathered NetBackup catalog information
       End time = 2013-02-27 09:44:16
       Number of Images gathered = 4014
       Number of database corrupt images gathered = 0
       Number of EMM database Media attribute records gathered = 38
       Number of EMM database Volume attribute records gathered = 43

       Catalog data gathering took 189 seconds to complete

     dir results for created NBCC files:
02/27/2013  09:42 AM                 8 nbcc-active-tapes

02/27/2013  09:42 AM           752,698 nbcc-bpdbjobs-most_columns

07/07/2011  09:43 AM         2,211,811 nbcc-bpimagelist-l
...

4.0  Verifying required catalog components were gathered

5.0  Beginning NetBackup catalog consistency check
     Start time = 2013-02-27 09:44:18
5.1  There were no tape media involved in active NetBackup jobs
5.3  Processing EMM database Volume attribute records, pass 1 (of 2),
     4 records to be processed
       Processed 4 EMM database Volume attribute records.
5.4  Checking for duplicate EMM server host names in Volume
     attribute data
5.5  Processing Image DB, pass 1 (of 2),
     3751 images to be processed
       3751 images processed on pass 1
       There were 0 images with at least one copy on hold detected.
5.6  Processing EMM database Media attribute records, pass 1 (of 3),
     2 records to be processed
       Processed 2 EMM database Media attribute records.
       There were 0 tape media detected that are on hold.
5.8  Check for duplicate media server names in the EMM database
       Media attribute data
5.9  Processing EMM database Media attribute records, pass 2 (of 3),
      2 records to be processed
5.10  Processing Image DB, pass 2 (of 2),
     3751 images to be processed
CONSISTENCY_ERROR Oper_7_1

5.11  NetBackup catalog consistency check completed
       End time = 2013-02-27 09:19:25

5.12  Checking for the latest NBCCR repair output directory
       C:\Program Files\Veritas\netbackup\bin\support\output\nbccr
       No repair file output directory detected.

       
       Summary of NBCC EMM Server processing
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Primary hostname:                                                 +
+ lidabl11                                                          +
+ Alias hostnames:                                                  +
+ lidabl11                                                          +
+ Sources:                                                          +
+ nbemmcmd vmoprcmd                                                 +
+ EMM Server = yes                                                  +
+ EMM NetBackup version = 8.1                                       +
+ NBCC NetBackup version = 8.1                                      +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

       Summary of NBCC Master server processing
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Primary hostname:                                                 +
+ lidabl11                                                          +
+ Alias hostnames:                                                  +
+ lidabl11                                                          +
+ Sources:                                                          +
+ nbemmcmd bpstulist nbdevquery bpgetconfig                         +
+ Master server = yes                                               +
+ EMM NetBackup version = 8.1.0.0                                   +
+ NBCC NetBackup version = 8.1                                      +
+ Tape STU detected = no - Disk STU detected = yes                  +
+ Disk Pool Host = yes                                              +
+ Associated Storage servers:                                       +
+ lidabl11 lidaclvm1                                                +
+ EMM tape media record extract attempted = yes                     +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

       Summary of NBCC Media server processing
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Primary hostname:                                                 +
+ lidabl14                                                          +
+ Alias hostnames:                                                  +
+ lidabl14.acme.com                                   +
+ Sources:                                                          +
+ nbemmcmd bpgetconfig                                              +
+ Media server = yes                                                +
+ EMM NetBackup version = 8.1.0.0                                   +
+ NBCC NetBackup version = 8.1                                      +
+ Tape STU detected = no - Disk STU detected = no                   +
+ EMM tape media record extract attempted = yes                     +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

...

     ***NBCC DETECTED A NetBackup CATALOG INCONSISTENCY!***

Report complete, closing the
.\output\nbcc\lidabl11_NBCC_20130227_094057\nbcc-info.txt
output file.

A complete description of NBCC options is in the NetBackup Commands Reference Guide.