Translation Notice
Please note that this content includes text that has been machine-translated from English. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.
NetBackup 8.1.2 / 3.1.2 Hotfix - CloudCatalyst EEB Package (Etrack 3958437)
Abstract
Description
A hotfix is available for NetBackup 8.1.2 which resolves the following issues seen after an upgrade:
- The cache eviction takes a long time. The delete request prevents the upload request.
- Performance improvement for uploads to cloud
- Certain error conditions could cause failed uploads to be reported as successful resulting in potential data loss.
- A hang could occur after several hours or days as a result of a previous non-graceful service shutdown.
- A fix to the Azure plugin is included - vxesp crashes leading to hung CC jobs.
- The rocksdb library has been upgraded to the latest available version to prevent vxesfsd crashes.
- The cache eviction logic fails on some CloudCatalyst servers installed on xfs or other filesystems.
- This results in a cache full scenario and jobs failing with write errors.
- vxesfsd is crashing in rocksdb. Set FSDB block cache size or disable it.
- vxesfsd crashes in del_file when deleting a file which had nlink set to 0 in FSDB.
- vxesfsd is crashing in esfs_opendir() due to a null-pointer exception.
- vxesfsd is crashing in boost libraries. Remove boost from vxesfsd.
- Add more detailed logging for cache eviction.
- Do not run cache eviction every 10 seconds if very little space is freed.
- Do not use status 1 in image stat file because it is treated as success by MSDP.
- Avoid crash from incorrect path in upload file list.
- Do not use world-writable permissions on created directories.
- Make sure that correct status is returned on failure for rmdir.
- Do not write __EMPTYFILE__ into .bin/.bhd files
- Download size is not the same value as fsdb. Disable download size check in case the
- size in fsdb is incorrect.
- Cannot remove certain directories. Change rmdir logic of checking directory is empty.
- This change makes sure that directory with garbage data is able to be deleted.
- vxesfsd crashing. Set max open file to limit RocksDB memory allocation. The allocated
- RSS memory will never be larger than 1GB.
- Implement write pause instead of fail when the cache hits the high watermark (the cache is full). Container write will be paused during cache eviction time.
size in fsdb is incorrect. -
Round robin between upload and delete requests to avoid starving delete requests in very busy environments.
-
Improve performance of opendir/readdir (remove support for optional d_type on readdir since MSDP does not use it)
-
Cache eviction improvements
-
Skip bhd files and recently modified files during cache eviction
-
If unable to reclaim enough space, consider them for eviction the next time
-
Check disk usage no more than once every 10 seconds
-
Cache sys_ino for /data and /databases for performance
-
Remove unnecessary lock for esfs_opendir for MSDP performance
-
Improve parse_upload_filelist performance
-
Cache sys_ino values while parsing the filelist to reduce stress on fsdb
-
Assume root directory is ESFS_ROOT_INO instead of querying fsdb on every esfs_lookup call
-
Fix error status for readonly mode for upload msdp system files
-
Avoid possible duplicate ino in case of fsdb error
-
Propagate error properly on failed cloud delete request
-
Prevent inode reuse and change list result for Azure.
-
Call putattr on error path.
-
Fix crash on failed start during rapid start/stop/start sequence.
- Allow esfs running for non fatal error in fsdb. Avoid crash once vxesfs cannot continue at startup.
- Add log for sts_get_physical_machine_name and sts_get_cred.
- For delete requests change the ino to ext_rscn if it's not null (case of duplicate ino) for DR from cloud.
- Storage manager uses ext_rscn as real inode for download because it might be reused. The utility of DR from cloud stores inode in cloud into ext_rscn.
- Comprehensive fsdb check at start. The allocated inode checking time is the same as metadata checking time.
- Updated pre and post install scripts for the EEB.
- Fix vxesfsd crashes by removing boost library.
- Implement fsdb check and integrate it into vxesfsd. vxesfsd will stop when fsdb has problem. fsdb check can remove garbage entries.
- Add option to check specific file consistency.
- Flush FSDB WAL at some important points. Add more info into fill_emptyfile for better analysis in future.
- Remove libnbsqlite.so dependency from fsdb_check
- Correct OS command paths from /usr/bin to /bin in pre and post-install scripts so they work on older versions of RedHat Linux.
EEB Version v16 adds:
- ET 4012807: Rebuild of rocksdb with portable=1 and remove USE_SSE=0
Versions Affected:
- NetBackup 8.1.2
- NetBackup Appliance 3.1.2
Read me
The resolution for this issue (Etrack 3958437) is included in the following release:
NetBackup 8.2
This EEB should be installed on: Cloud Catalyst Media Servers
Installation Steps:
- Stop NBU services.
- Uninstall any previous version of this EEB (3958437 versions 1 to 14) before installing.
- If not an NBU appliance, please run the EEB installer with the create option.
- Run the cc_touch utility (runs the touch command on the container files in the /
cache/userdata/data
directory) located in the bin/goodies directory as follows:cc_touch /cache/userdata/data
- Change esfs.json (
/cache/etc/esfs.json
) to have LowWatermark 50 and HighWatermark 90 values. - Start NBU services.
References:
Using the NetBackup Emergency Engineering Binary (EEB) installer https://www.veritas.com/docs/100019405
Installing EEBs on a NetBackup 52x0 / 5330 Appliance https://www.veritas.com/docs/100023444
How to install client EEB's with VxUpdate https://www.veritas.com/content/support/en_US/doc/125240132-131571482-0/v130876036-131571482
How to install Emergency Engineering Binaries on a VCS clustered NetBackup server https://www.veritas.com/docs/100021995
How to install add-ons or an EEB on NetBackup instances running on Flex 1.3 version https://www.veritas.com/content/support/en_US/doc/130821112-136840843-0/v137506948-136840843
How to install add-ons or an EEB on NetBackup instances running on the Flex 1.2 version, contact NetBackup Technical Support and reference article 100042366
Downloads:
NB_8.1.2_ET3958437_16.zip
NBAPP_EEB_ET3958437-3.1.2.0-16.x86_64.rpm
VRTSflex-nb_EEB_ET3958437-8.1.2-10.x86_64.rpm
EEB Installer Contents
File Checksum Byte count
linuxR_x86/cc_touch 1385895309 183520
linuxR_x86/dbdump 72320113 7585704
linuxR_x86/esfs_check 68013413 8667168
linuxR_x86/fsdb_check 859062949 13748560
linuxR_x86/install-3958437 1971078153 2417
linuxR_x86/librocksdb.so.6.0.2 1177832479 5731584
linuxR_x86/libstspiazure.so 3513354593 2279959
linuxR_x86/libstspiazureMT.so 82299150 2284491
linuxR_x86/libtcmalloc.so.4.5.3 1715574939 2342432
linuxR_x86/mkesfs 2702143032 8700008
linuxR_x86/post_uninstall-3958437 4168458670 441
linuxR_x86/pre_proc_uninstall_3958437 2836012658 4223
linuxR_x86/preprocess_install_3958437 86127520 2809
linuxR_x86/vxesfsd 1026060710 7543816
linuxR_x86/vxesp 3971867351 726992
Recommended service state:
Stop all NetBackup services before applying this hotfix.
Update files
|
File name | Description | Version | Platform | Size |
---|
Applies to the following product releases
Knowledge base
Duplication jobs to CloudCatalyst may complete with status 0, but if cache fills up, dataloss could result
2020-02-11Severity Possible Data Loss Description Duplication jobs to the cloud using NetBackup CloudCatalyst will complete with status 0, but if the CloudCatalyst cache volume becomes full, data loss can result. Versions Affected NetBackup 8.1 EEB 3958410...