Sign In
Forgot Password

Don’t have an account? Create One.

NetBackup 8.2 Hotfix - CloudCatalyst EEB Bundle (Etrack 3981837)

HotFix Critical

Abstract

This NetBackup 8.2 CloudCatalyst Hotfix EEB Bundle resolves CC Media Server issues.

Description

This Hotfix resolves the following issues:

  • 5240 CloudCatalyst Appliance hangs every 2 weeks due to memory exhaustion
  • NVE-386 CC needs to check for revoked certificates via OCSP support in go SDK code
  • CloudCatalyst starts to experience slower writes and then eventually halts
  • Make esfs use multi-object delete request when supported
  • ESFS legacy log rotation out of principle
  • Restore from CloudCatalyst-AWS-Snowball fails with error "Image warming failed 501"
  • Backups and dups to CloudCatalyst are not progressing vxesfsd and rocksdb core dumps
  • Restore of a VMware image from a CC media server is failing.
  • [SWST] NB_D2C_AMZ_AIR_Imageshare -> NB_Cloud_DR_API -> Failed to Initialize DR In Cloud
  • Amazon backup failed with media close error 87 [SWST] NB_CC_AMZ_SS_Encrypt -> Verify_Encrypt_Enabled ->
  • Failed to verify if server side encryption has been enabled
  • Datastore will not initialize on cloud catalyst server
  • CloudCatalyst vxesfsd process corrupted on nbu 8.2.
  • [CC][Glacier]Restore a image failed with error "Image warming failed 409"
  • Upgrade to the latest fuse library (version 3.6.2)
  • Fix vxesfsd crashes by removing boost library.
  • Fix vxesfsd crashes caused by invalid value of sys_nlink.
  • Remove unnecessary/obsolete entries from the fsdb database.
  • Fix vxesfsd crashes caused by a null-pointer exception in esfs_opendir.
  • Adding ocsp check to the ocsd using VerifyPeerCertificate OCSP check will happen for every tls connection made OCSP information is retrieved from the server certificate
  • Currently esfs only uses the multi-object delete feature for Amazon.
  • Adding a check for the BulkDelete attribute in CloudProvider.xml so that esfs will use the feature for all providers known to support it.
  • Honor max log size in esfs.json for ocsd
  • Fix restore failure from AWS Snowball device
  • Restore fails with error 'Image warming failed 501'.
  • esfs_storage logs an error 'NotImplemented: This operation is not supported yet. status code: 501'
  • Improve performance and reduce memory consumption for ocsd process.
  • OCSD log tool.
  • Fix import fail error in cloud NBU(for DR).
  • Add more detailed logging for cache eviction process
  • Remove orphaned entries in file list directories
  • Switch from char arrays to std::string
  • Change log write to avoid race condition
  • Refactor requestWorker to not reuse connection after getting region. Add socket log.
  • Do not create an empty zero-byte log file on startup.
  • Add OSCP caching to remove overhead of response time slowdown from OCSP server. Setting
    the default cache time to 60 Minutes.

  • ET3982970: Cannot remove certain directories. Change rmdir logic of checking directory is empty.
    This change makes sure that directory with garbage data is able to be deleted.
    vxesfsd crashing. Set max open file to limit RocksDB memory allocation. The allocated
    RSS memory will never be larger than 1GB.

  • ET3985755: Retry when there is http conflict with aws 'operation aborted' error.

  • ET 3990062: Cache sys_ino for /data and /databases for performance
    Remove unnecessary lock for esfs_opendir for MSDP performance
    Start ocsd even if vxesfsd is already running
    Check disk usage no more than once every 10 seconds

    ET 3990062:

  •     Cache eviction improvements
        Skip bhd files and recently modified files during cache eviction
        If unable to reclaim enough space, consider them for eviction the next time

  • Cache the metadata instead of release it when reference count is 0. Change ocsd to download multiple objects for one file.

  • Free cached memory in destruction method and fix a incorrect memory free.

  • ET 3989115:

  • Round robin between upload and delete requests to avoid starving delete requests in very busy environments.

  • ET 3990062:

  • Fix performance issue of image sharing when data locality is bad.

  • Fix imagesharing's issue over AIR.

  • Improve performance of opendir/readdir (remove support for optional d_type on readdir since MSDP does not use it).

  • Correct name of temp download file for Azure.

  • Prevent inode reuse and change list result for Azure.

  • Change log mechanism. There is a dedicated ocsd log routine. 

  • Problems addressed: 
  • 1. The small log file 
  • 2. Log file is unexpected closed
  • The log configuration will be more consistent.
  • Remove nbu_wrapper dependency from ocsd. It can get cloud configuration using web service.
  • Get cloud instance configuration file directly, if NB web service does not return the configuration.
  • ET 3997365: Allow esfs running for non fatal error in fsdb. Avoid crash once vxesfs cannot continue at startup.
  • ET 3994287: Ignore unrecognizable lines in bp.conf.
  • ET 3993119: Support ECA and remove '.dl' from azure download method.
  • ET 3993574: For delete requests change the ino to ext_rscn if it's not null (case of duplicate ino) for DR from cloud.
  • storage manager uses ext_rscn as real inode for download because it might be reused. The utility of DR from cloud stores inode in cloud into ext_rscn.
  • Comprehensive fsdb check at start. The allocated inode checking time is the same as metadata checking time.
  • Implement fsdb check and integrate it into vxesfsd. vxesfsd will stop when fsdb has problems. fsdb check can remove garbage entries.
  • Flush FSDB WAL at some important points.Add more info into fill_emptyfile for better analysis in future.

 

Version 15 adds:

 

ET 4006406:  When proxy server is not enabled, we shouldn't see proxy related errors in the logs. Also handled NONE auth type.
ET 3995775: Remove eof error message printed in ocsd logs.
ET 4002975: Add ReadAt function for OCSReader because AWS SDK has special logic to reduce memory allocation when ReadAt is implemented.
ET 3998016: Upgrades to go aws-sdk-go that include fixes for memory usage and other improvements
Remove libnbsqlite.so dependency from fsdb_check.
Change checking condition for socket ready.
Search for short name in certmapinfo.json if exact match is not found. Ignore case when comparing server names.
needWarm interface for msdp to know the bucket supports warming or not.
Handle warm request for Azure blob.
Update the warm stat file without warming when low latency storage type is selected.
Match the objects for MSDPCC restores.
Fix UseCRL log to identify if CRL is enabled or not.
Fix some error in calling newOCSHTTPClient and change 0,1,2 file descriptor. /var/log/ocsd.log will have info when ocsd crashes.

Read me

Versions Affected

  • NetBackup 8.2

 

This EEB should be installed on: Cloud Catalyst Media Servers

 

Downloads:

NB_8.2_ET3981837_15.zip
NBAPP_EEB_ET3981837-3.2.0.0-15.x86_64.rpm

VRTSflex-nb_EEB_ET3981837-8.2-11.x86_64.rpm

 

Installation Instructions:

1. Stop NBU services.

2. Uninstall any previous version of this EEB (3981837 versions 1

to 10) before installing version 10.

3. If not an NBU appliance, please run the EEB installer with the

create option.

4. Start NBU services.

Using the NetBackup Emergency Engineering Binary (EEB) installer

https://www.veritas.com/docs/100019405

Installing EEBs on a NetBackup 52x0 / 5330 Appliance

https://www.veritas.com/docs/100023444

 

Checksums for installed files:

 

File                                            Checksum           Byte count

linuxR_x86/cc_touch            2284086840    184224

linuxR_x86/cred_ioctl            1248017546    57448
linuxR_x86/dbdump            4081373944    7677072
linuxR_x86/esfs_check            1735544812    8528456
linuxR_x86/esfs_cleanup            3467385846    2554513
linuxR_x86/esfs_init.sh            2478374342    8224
linuxR_x86/esfs_reconfig        4192717822    472496
linuxR_x86/esfs_recover.sh        3092537070    2103
linuxR_x86/esfs_upgrade.sh        1690213188    7266
linuxR_x86/esfs_version.txt        3630179416    108
linuxR_x86/fsdb_check            3440581837    13843736
linuxR_x86/fsdbbackup            2062890755    11920
linuxR_x86/install-3981837        261667243    2823
linuxR_x86/libfuse3.so.3.6.2        2532643060    833440
linuxR_x86/mkesfs            1745604639    8119664
linuxR_x86/nbu_wrapper            1529721482    1582208
linuxR_x86/ocsd                1612611895    18177873
linuxR_x86/ocsd_log_view        97038048    128816
linuxR_x86/post_uninstall-3981837    1686394211    373
linuxR_x86/pre_proc_uninstall_3981837    3156515172    3880
linuxR_x86/preprocess_install_3981837    3993803480    1099
linuxR_x86/recoverdb            956254037    614816
linuxR_x86/setlsu_ioctl            4158014556    11944
linuxR_x86/vxesfs            3633400371    3140550
linuxR_x86/vxesfsd            2446415307    7626408

 

Recommended service state:

Stop all NetBackup services before applying this hotfix.

 

Update files

File name Description Version Platform Size

Applies to the following product releases

Knowledge base

0
2020-02-11

Severity Possible Data Loss Description Duplication jobs to the cloud using NetBackup CloudCatalyst will complete with status 0, but if the CloudCatalyst cache volume becomes full, data loss can result. Versions Affected NetBackup 8.1 EEB 3958410...

3
2020-02-14

Problem In NetBackup 8.2, Image Sharing does not support HCP as a cloud provider. You cannot configure an Image Sharing server with an HCP cloud provider. Error Message When you try to configure an Image Sharing server with an HCP cloud provider,...