Sign In
Forgot Password

Don’t have an account? Create One.

NetBackup 8.1.2 / 3.1.2 CloudCatalyst EEB Bundle (Etrack 3958437)

HotFix Critical

Abstract

This HotFix resolves several serious CloudCatalyst media server issues.

Description

A hotfix is available for NetBackup 8.1.2 which resolves the following issues seen after an upgrade:

  • The cache eviction takes a long time. The delete request prevents the upload request.
  • Performance improvement for uploads to cloud
  • Certain error conditions could cause failed uploads to be reported as successful resulting in potential data loss.
  • A hang could occur after several hours or days as a result of a previous non-graceful service shutdown.
  • A fix to the Azure plugin is included - vxesp crashes leading to hung CC jobs.
  • The rocksdb library has been upgraded to the latest available version to prevent vxesfsd crashes.
  • The cache eviction logic fails on some CloudCatalyst servers installed on xfs or other filesystems.
  • This results in a cache full scenario and jobs failing with write errors.
  • vxesfsd is crashing in rocksdb. Set FSDB block cache size or disable it.
  • vxesfsd crashes in del_file when deleting a file which had nlink set to 0 in FSDB.
  • vxesfsd is crashing in esfs_opendir() due to a null-pointer exception.
  • vxesfsd is crashing in boost libraries. Remove boost from vxesfsd.
  • Add more detailed logging for cache eviction.
  • Do not run cache eviction every 10 seconds if very little space is freed.
  • Do not use status 1 in image stat file because it is treated as success by MSDP.
  • Avoid crash from incorrect path in upload file list.
  • Do not use world-writable permissions on created directories.
  • Make sure that correct status is returned on failure for rmdir.
  • Do not write __EMPTYFILE__ into .bin/.bhd files
  • Download size is not the same value as fsdb. Disable download size check in case the
  • size in fsdb is incorrect.
  • Cannot remove certain directories. Change rmdir logic of checking directory is empty.
  • This change makes sure that directory with garbage data is able to be deleted.
  • vxesfsd crashing. Set max open file to limit RocksDB memory allocation. The allocated
  • RSS memory will never be larger than 1GB.

 

Version 10:

  • Implement write pause instead of fail when the cache hits the high watermark (the cache is full). Container write will be paused during cache eviction time.
    size in fsdb is incorrect.
  • ET 3989115:
    Round robin between upload and delete requests to avoid starving delete requests in very busy environments.

  • ET 3990062:

    Improve performance of opendir/readdir (remove support for optional d_type on readdir since MSDP does not use it)
    Cache eviction improvements
    Skip bhd files and recently modified files during cache eviction
    If unable to reclaim enough space, consider them for eviction the next time
    Check disk usage no more than once every 10 seconds
    Cache sys_ino for /data and /databases for performance
    Remove unnecessary lock for esfs_opendir for MSDP performance
    Improve parse_upload_filelist performance
    Cache sys_ino values while parsing the filelist to reduce stress on fsdb
    Assume root directory is ESFS_ROOT_INO instead of querying fsdb on every esfs_lookup call
    Fix error status for readonly mode for upload msdp system files
    Avoid possible duplicate ino in case of fsdb error

  • ET 3990446: 

    Propogate error properly on failed cloud delete request

 

Versions Affected

  • NetBackup 8.1.2
  • NetBackup Appliance 3.1.2

Read me

 The resolution for this issue (Etrack 3958437) is included in the following release:

 NetBackup 8.2

 

This EEB should be installed on: Cloud Catalyst Media Servers

 

Installation Steps:

  1. Stop NBU services.
  2. Uninstall any previous version of this EEB (3958437 versions 1 to 4) before installing.
  3. If not an NBU appliance, please run the EEB installer with the create option.
  4. Run the cc_touch utility (runs the touch command on the container files in the /cache/userdata/data directory) located in the bin/goodies directory as follows: cc_touch /cache/userdata/data
  5. Change esfs.json (/cache/etc/esfs.json) to have LowWatermark 50 and HighWatermark 90 values.
  6. Start NBU services.

 

Downloads:

NB_8.1.2_ET3958437_10.zip

NBAPP_EEB_ET3958437-3.1.2.0-10.x86_64.rpm

 

EEB Installer Contents

File                                                                 Checksum           Byte count

linuxR_x86/cc_touch                                 1385895309        183520
linuxR_x86/dbdump                                  2322998137        6235376
linuxR_x86/esfs_check                            1124611042        6687544
linuxR_x86/install-3958437                    1107665767        1594
linuxR_x86/librocksdb.so.6.0.2             1725966266        5219784
linuxR_x86/libstspiazure.so                  749692139        2279959
linuxR_x86/libstspiazureMT.so            2758343279        2284491
linuxR_x86/libtcmalloc.so.4.5.3            1715574939        2342432
linuxR_x86/mkesfs                                 3856799150        7350552
linuxR_x86/pre_proc_uninstall_3958437        326369049        854
linuxR_x86/preprocess_install_3958437        4060739457        879
linuxR_x86/vxesfsd                4087013991        6193344
linuxR_x86/vxesp                184583907        726328

 

Recommended service state:

Stop all NetBackup services before applying this hotfix.

Update files

File name Description Version Platform Size

Applies to the following product releases

Knowledge base

0
2020-02-11

Severity Possible Data Loss Description Duplication jobs to the cloud using NetBackup CloudCatalyst will complete with status 0, but if the CloudCatalyst cache volume becomes full, data loss can result. Versions Affected NetBackup 8.1 EEB 3958410...