About upgrading NetBackup MSDP and PureDisk deduplication solutions to resolve case sensitivity issues

About upgrading NetBackup MSDP and PureDisk deduplication solutions to resolve case sensitivity issues

Article: 100009707
Last Published: 2014-03-31
Ratings: 0 0
Product(s): NetBackup

Problem

 

 

Note: If all of your NetBackup media servers are running Windows, and you do not plan to add UNIX servers running MSDP, you may safely ignore the rest of this topic.

This section applies to the following deduplication solutions:

  • NetBackup Media Server Deduplication Option (master server, media server, client, or VMware backup host)

  • NetBackup appliance deduplication (master server, media server, or VMware backup host)

  • PureDisk deduplication

  • PureDisk appliance deduplication

In versions before NetBackup 7.5.0.6, a problem can occur where NetBackup cannot read an image that is stored on an MSDP or PureDisk storage unit. The issue results from inconsistencies in how entries are written and read in the deduplication database on case-sensitive file systems. The types of operations that might exhibit failure due to a read error that is associated with this issue include verify, restore, duplication, and replication. For MSDP, the issue is fixed starting in NetBackup 7.5.0.6 (appliance 2.5.3). For PureDisk, the issue is fixed starting in version 6.6.5 (appliance 1.4.4).

Error Message

 

 

Table 1-1                     Examples of the failed operations that can result from the case sensitivity issue

Operation

Activity Monitor status

Activity Monitor job details

Status in bpdm and bptm log files

Verify

(191) no images successfully processed

image open failed: error 2060018: file not found

sts_open_image failed: error 2060018

Restore

(83) media open error

Image open failed: error 2060018: file not found

sts_open_image failed: error 2060018

Duplication (MSDP source)

(83) media open error

Image open failed: error 2060018: file not found

sts_open_image failed: error 2060018

Replication (MSDP source)

(83) media open error

get image properties failed: error 2060013: no more entries

rpl_add_image_set: rpl_get_image_info() failed, error 2060013

Duplication (PDDO source)

(83) media open error

Image open failed: error 2060018: file not found

sts_open_image failed: error 2060018

Replication (PDDO source)

(83) media open error

get image properties failed: error 2060013: no more entries

rpl_add_image_set: rpl_get_image_info() failed, error 2060013

 

Cause

Example scenarios of the client name case sensitivity issue

The following list contains two examples of scenarios in which the client name case sensitivity issue can occur. These example scenarios are intended to assist you in understanding how the issue can manifest in a NetBackup environment. There may be more scenarios in which this issue can occur.

Example 1
Suppose that there is a NetBackup client named my_client running a version of NetBackup previous to version 7.5.0.6 and that there is a corresponding policy named my_policy. Suppose also that my_client performs client-side deduplication to a UNIX-based MSDP or PureDisk storage unit. Next, suppose that the following steps occur:

  1. A successful backup for my_policy is made to an MSDP or PureDisk storage unit.
     
  2. The name of the client in the policy my_policy is changed from my_client to MY_CLIENT.
     
  3. Another successful backup for my_policy is made to an MSDP or PureDisk storage unit.
     

The data for both backups is saved. However, the operations that attempt to read the data from the second backup fail if the deduplication engine is on a case-sensitive file system. The failures occur because the deduplication database entry my_client differs from MY_CLIENT and the previous version of the NetBackup client writes incorrect deduplication database entries. The issues in this specific example can be resolved by upgrading the client to NetBackup 7.5.0.6 or newer.

Example 2
Suppose that there is a UNIX media server running NetBackup 7.5.0.6 or later called media_A that is attached to an MSDP storage unit. Suppose also that there is another media server called media_B running Windows and a version of NetBackup that is older than version 7.5.0.6. In this example, media_B has the credentials that allow it to access the MSDP disk pool MSDP_POOL on media_A. Suppose that a policy named my_policy was created for a client named my_client (such as in the first example) and the following events occur:

  1. A successful backup of my_policy for my_client is made to MSDP_POOL.
     
  2. The policy my_policy is copied to MY_POLICY.
     
  3. The name of the client in the policy MY_POLICY is changed from my_client to MY_CLIENT.
     
  4. A backup of MY_POLICY is successful to MSDP_POOL.
     

In this example, the backups are saved and media_A can read them. However, one or more of the backup images cannot be read from MSDP_POOL by media_B because media_B suffers from the name case sensitivity issue. In this situation, the issue occurs whenever the master server selects media_B as the server to perform the read operations on MSDP_POOL.

This particular manifestation of the issue can be diagnosed using the following methods:

  • Use the bpimagelist command to verify that the images were stored on the desired MSDP disk pool.

  • Look for the following in the granted resource line in the Activity Monitor job details for the failed job:
    The host that is listed as the storage server is running NetBackup 7.5.0.6 or newer.
    The host that is listed as the media server is running a version of NetBackup that is older than version 7.5.0.6.

The issues in this specific example can be avoided by upgrading all of your media servers to NetBackup 7.5.0.6 or newer.

Solution

Determining the servers and clients that can encounter the issue

The case-sensitivity issue can occur if your environment performs any of the following MSDP functions on UNIX operating systems that run on NetBackup media servers or PureDisk servers (including NetBackup appliances and PureDisk appliances):

  1. Media server deduplication to attached storage
     
  2. Media server deduplication to another media server
     
  3. Media server deduplication to a PureDisk server
     
  4. Media servers for optimized MSDP duplication within the same domain
     
  5. Client-side deduplication to a media server
     
  6. Client-side deduplication to a PureDisk server
     

 

 

Note:

All of the listed operations apply to the VMware backup hosts that run the deduplication engine.

All of the listed operations also apply to the master servers that function as media servers and run the deduplication engine.

Veritas recommends that you upgrade all of the servers, clients, and VMware backup hosts in your environment that are involved in the listed operations. That includes any media server that uses credentials for an affected MSDP storage unit. Starting with NetBackup 7.5.0.6 (appliance 2.5.3) and PureDisk 6.6.5 (appliance 1.4.4), writing to the deduplication database has been updated to fix the issue. The database writes are fixed so that a change in the case of the characters that are used for the NetBackup client name and policy do not cause a read of the backup images to fail. If you do not upgrade to these minimum versions, there is a possibility that you may encounter read errors with future backup images due to this issue.

 

 

Note:

Consistent with NetBackup upgrade policy your media servers and clients must be running a minimum of NetBackup 7.5 before you can attempt to upgrade to 7.5.0.x.


Fixing the name case sensitivity issues that exist in the MSDP and PureDisk databases

If you have already encountered image read failures due to client name or policy name case sensitivity issues, your first step should be to upgrade the affected hosts to NetBackup 7.5.0.6 or newer. After the upgrade, the name case issue still affects the images that were created before the upgrade by versions of NetBackup previous to version 7.5.0.6. Use the following table to select the correct approach to repairclient and policy name case sensitivity issues with existing NetBackup images:

 

 

Table 1-2                     

Problematic name type

Storage unit type

Required action

Client

MSDP

Repair the MSDP database

Client

PureDisk

Repair the PureDisk database

Policy

MSDP

Contact Veritas Technical Support

Policy

PureDisk

Contact Veritas Technical Support

 

About the NetBackup deduplication client name case conversion utilities

The deduplication client name case conversion utilities repair the NetBackup and PureDisk deduplication database by specifically fixing the name case of each client that the name case sensitivity issue affects. If you have experienced any of the read errors associated with this issue, you should use the following procedures to repair your deduplication database.

For NetBackup 7.5.0.6 and PureDisk 6.6.5, the utility is called catalogcaseconvert and PDDODbCaseConvert, respectively.

In NetBackup 7.5.0.7 and newer, the utility is called dedupecatutil and includes several new options:

  • Running the command on its own displays the help text.

  • -h and --help
    Explicitly displays the help text.

  • --scan_client_name_case
    Scans the MSDP database for client name case sensitivity issues and writes the found issues to stdout and a log file.

  • --repair_client_name_case
    Repairs client name case sensitivity issues. Requires the MSDP services and nbsvcmon to be down.

  • --remove_empty_dirs
    Removes all empty directories from the MSDP database. Can be used at the same time as –repair_client_name_case. Requires the MSDP services and nbsvcmon to be down.

  • --force
    Optional argument to automatically stop and start the MSDP services and nbsvcmon.

  • --cleanup_case_work_space
    Cleans up the temporary directories that might remain after –repair_client_name_case is run.

 

 

Note:

The --remove_empty_dirs option is not available in NetBackup 7.6 GA.  

Scanning an MSDP deduplication database

Starting with NetBackup 7.5.0.7, the dedupecatutil utility can scan for the names of the clients that can experience the client name case sensitivity issue. The scan option lets you determine if you have the issue, which hosts to upgrade, and the approximate time that is needed to repair the deduplication database. The scan may be run at any time and does not require the NetBackup services to be shut down.

 

Note:

If you choose to run the scan utility on an active deduplication database, you might experience reduced performance with respect to NetBackup jobs that use the disk pool.


Once you have upgraded the media server that controls the MSDP storage unit to NetBackup 7.5.0.7 or newer, run the following command:
/usr/openv/pdde/pdcr/bin/dedupecatutil –scan_client_name_case

If the MSDP deduplication database does not contain any name case sensitivity issues, the following output is displayed:
The deduplication metadata is clean.

If clients are reported to have issues, the following output is displayed:
The deduplication metadata has client name case-sensitivity issues.
Note: In NetBackup 7.5.0.7, the output contains the location of the log file. For example:
/Storage/history/client_names_with_case_sensitivity_issues.xxxxxxxxxx

If any issues are reported, you can choose to proceed to the repair procedure. Logs for the scan can be found at
<MSDP_storage_location>/history/<client_names_with_case_sensitivity_issues>.<timestamp>.

 

Note:

If you choose to repair the database, the repair takes roughly twice as long as the scan.

Repairing an MSDP deduplication database

Repairing the client name case sensitivity issues requires that services be stopped on the media server that controls the MSDP storage unit. The downtime is less than one hour for most storage units. However, if you have more than 5 million files in the deduplication database, the downtime could be more than one hour. You can estimate the number of files in the deduplication database by running the following command:
find <MSDP_disk_storage_location>/databases/ –type f | wc -l

Use the following procedure to repair the deduplication database with respect to existing NetBackup images that have client name case sensitivity issues:

MSDP deduplication database repair procedure

  1. Verify that the media server has been upgraded to at least NetBackup 7.5.0.6 or NetBackup Appliance 2.5.3.
     
  2. Stop the NetBackup services on the media server. For the NetBackup appliances, you can choose to shut down the services automatically in step 3.
     
  3. For NetBackup 7.5.0.6, run the following catalogcaseconvert command to repair all existing client name case sensitivity issues on the media server:
    /usr/openv/pdde/pdcr/bin/catalogcaseconvert –y

    For NetBackup 2.5.3, run Support > MSDP > FixCaseSensitivity to repair all existing client name case sensitivity issues on the appliance media server. Alternatively, if you did not shut down the services in step 2, you can run Support > MSDP > AutoFixCaseSensitivity which automatically starts and stops the services as part of the repair.

    For NetBackup 7.5.0.7 and newer, run the following dedupecatutil command to repair all existing client name case sensitivity issues on the media server:
    /usr/openv/pdde/pdcr/bin/dedupecatutil –repair_client_name_case
     
  4. If the repair completed successfully, the command exits with the following message:
    The conversion has finished successfully

    You should then verify that the <MSDP_disk_storage_location>/databases/catalog directory exists.

    If the command did not output a successful message and the <MSDP_disk_storage_location>/databases/catalog directory does not exist, then the command may have been interrupted and you should repeat step 3.

    Note: If the repair utility reports that there were collisions, please contact Veritas Technical Support. Collisions indicate an unsuccessful repair and are reported with the following message:
    The conversion has finished with some collisions. Please check <collision_file_path> to get details
     
  5. Start the NetBackup services on the media server. If you ran Support > MSDP > AutoFixCaseSensitivity on a NetBackup appliance, the services start automatically.
     

If you have experienced the client name case sensitivity issue in the past, you may also want to remove unused policies and clients from the MSDP deduplication database. In NetBackup 7.5.0.7 and newer, the dedupecatutil utility can remove these entries by adding the –remove_empty_dirs command line option in step 3.

You may omit the –repair_client_name_case option if the scan shows that you do not have any issues.

 

Note:

The –remove_empty_dirs option is not available in NetBackup 7.6 GA.  

Repairing a PureDisk deduplication database

This section only applies to environments that use the PureDisk Deduplication Option (PDDO). Repairing the client name case sensitivity issue requires that the Metabase node service be stopped, or else any active jobs to the storage pool can fail. The downtime for the Metabase service is about half an hour for a typical PureDisk storage pool. However, the process may take more than three hours for a very large storage pool.

Use the following procedure to repair a PureDisk deduplication database with respect to existing NetBackup images that have client name case sensitivity issues:

PureDisk deduplication database repair procedure

  1. Verify that the PureDisk server has been upgraded to at least PureDisk 6.6.5 or PureDisk Appliance 1.4.4.
     
  2. Display and read the following file:
    /opt/pdconfigure/scripts/support/PDDODb_Case_Convert/README.txt
     
  3. For a cluster environment, freeze the PureDisk Resource Group.
     
  4. Run the following PDDODbCaseConvert command to repair all existing client name case sensitivity issues on the PureDisk server:/opt/pdconfigure/scripts/support/PDDODb_Case_Convert/PDDODbCaseConvert.sh –convert
     
  5. If the repair completed successfully, the command exits with the following message:
    Convert MBE PDDO database successfully.

    If the repair was unsuccessful, the command exits with the following message:
    Convert MBE PDDO database failed.

    Note: If the conversion did not succeed, you need to roll back the database using the following command:
    /opt/pdconfigure/scripts/support/PDDODb_Case_Convert/PDDODbCaseConvert.sh –rollback
     
  6. Logs can be found in the /var/log/puredisk/upgrade directory.

 

 

 

Was this content helpful?