Problem Description: This Hotfix Bundle contains multiple fixes for MSDP issues on NetBackup 8.3.0.1

Fixes Include:

ET 4009731: MSDP spoold deamon fails to start on Access 3340 with 'Too many open files'
ET 4009832: Fix the crash issue in the cachedb registry
ET 4010112: Set proper busy timeout value to forbid SQLite operations return error in case the DB is locked by another process or thread
ET 4010929: Fix client side CRL issue
ET 4010593: Add debug log for connection manager
ET 4013056: fix # in passwdfile
ET 4012328: Improve the compaction check process to make it easier to use and be more reliable
ET 4012685: Set nommap for vxfs to mitigate the impact of mmap_sem lock
ET 4014287: Don't commit garbage refops in the end of spooler tasks
ET 4016575: Improve segment encryption key strength
ET 4016704: Enable vxfs nommap in case the system is a real appliance
ET 4016995: Forbid duplicated error messages in case of refdb corruption
ET 4012233: rh-nginx116-nginx can't be stopped using systemctl
ET 4018989: Limit the error messasge number
ET 4019232: Allow only 1 spoold to start at the same time
ET 4020699: Don't call ioctl by default
ET 4020692: Race condition between load/save header file
ET 4014543: hange function compareLastImgEntry() to return false for duplicated entries to avoid std::sort coredump
ET 4020796: Potential data loss found from code inspection
ET 4021113: [vpfsd] vpfsd log has lots of similar ERR: ERROR [33098: [140655629051648]: ]: Failed to Flush all data, error: errno 2
ET 4021009: Add lock before loading lmdb for SO list
ET 4020164: Add progress info for compaction check
ET 4021534: Set time of validity for compaction precheck marker file
ET 4021632: Add version for compaction check
ET 4021750: handle exceptions for vpfs
Enable vxfs nommap by default

Fix some issues about vpfs
Handle mmap lock for crc check, compact check, fpmap, etc

ET 4023051: Support multi-threading compaction check
ET 4022549: Change the option --findgoodbackup to echo more images in catdbutil tool for the msdpcheck tool.
ET 4028943: Report deletable space during compaction check
ET 4029223: Capability to enable and disable extended attributes

ET 4029316: Delete empty container groups
ET 4019858: Universal Share deduplication Issue in Azure Cloud
ET 4030316: should delete .dcidboundary if .shareid doesn't exist, or vise versa

ET 4030877: change mode to spoold
ET 4030864: 1. fallback to use regular FS if tmpfs mount fails 2.disable directio for cloud LSU
ET 4031557: vpfsd seems dead lock and take checkpoint action hang
ET 4031938: vpfs: container was deleted unexpectedly
sync blockrock code to eeb bundle, this address the hang issue for DOE
ET 4032487: reference issue for big files of ushare in DOE
ET 4033123: ignore uldb error, later will get udcid from spoold
ET 4035175: make universal share startup faster
ET 4035182: Ingest mode for universal share
ET 4035366: vpfsd cannot start due to tid number of tlog is not sequential
ET 4033757: Glacier spoold may open a uploaded container for writing SO,if container is uploaded to Glacier and is not in esfs local cache, open will fail.
ET 4036552: Improve dcscan to dump DOs for the DAT tool
ET 4037145: vpfsck for universal share to check consistency between the metadata and data
ET 4038386: race condition between set file group and moving extent map
ET 4038646 : exception enhancement for adding reference
ET 4038770: need to check the sync references return value during universal share checkpoint to avoid potential data lost issue
ET 4023360: Avoid excessive logging of fuse_getxattr.
ET 4039335: Improve the findkms functionality
ET 4038652: For dc miss cases, try to report corrruptions when refdb header corrupt flag was set
ET 4040458: oracle incr merge file size 0 issue over dnfs
ET 4031744: Fix refdbutil touch all for cloud LSU

ET 4040877: vpfsck to check file size in header and extent

ET 4040892: 'Dcscan --fixbhd' can cause data corruption in a env with the new SO key generation algorithm

ET 4040369: add remove option for refdbutil
ET 4042556 : Universal Shares slow and unresponsive
ET 4041685 : moved lock to contain the map search when getting the encryption ctx.
ET 4043466: ushare crashed at cleanup share while reset connection
ET 4043230: cross mount point mv/rename file failed
ET 4043423: msdpcheck will not progress due to errors in CRQP relating to 'DecryptSoKey'
ET 4038864: After running msdpcheck, spad can not startup
ET 4045684: Failed to verify of the MSSQL database after the backup through the universal share

ET 4046624: need unexport all the nfs/smb shares before shutdown vpfsd
ET 4046887: inconsistent file size in the file cache for VLD

ET 4043812: Verify UShare image failed with ERR tar had an unexpected error
ET 4051431: fix vpfs mon detectes incorrect status
ET 4051681: Remove the command option 'keydictutil --refresh'
ET 4050948: universal share backword compability issue upgrade from 8.2
ET 4045989: Enhance crstats history to support cloud lsu stats
ET 4048644: Enable compaction check dispite refdb state
ET 4047809: Add a mutex around the the so key decryption
ET 4047822: fix optimizations for the VLD read
ET 4051431: fix vpfs mon dectects incorrect status
ET 4043812: Verify UShare image failed with ERR tar had an unexpected error
ET 4048644: Enable compaction check dispite refdb state
ET 4051681: Remove the command option 'keydictutil --refresh'
ET 4045989: Enhance crstats history to support cloud lsu stats
ET 4050948: universal share backword compability issue upgrade from 8.2
Improve keydictutil to add --add_key --modify_key --delete_key parameters
ET 4052988: handle SQL VLD cannot find boundary issue
ET 4052112: [Issue] After installing EEB 4013394 v23 VLD Enabled message is not showing
ET 4054876: Msdpcheck is aborting due to refdbutil segfaulting

ET 4055303: fix create file failure
ET 4056402: fix US IA delete issue with multiple instance enabled
ET 4009933: LSU level Data Selection lock pool in spad.
ET 4057843: not a valid utf8 char error for the delete reference
ET 4058092: handle create file group exception
ET 4008258: Separate cloud LSU state and alive lock.
ET 4054341: Disable VLD when Oracle Stream Handler is enabled.
ET 4022376: record policy_type/sched_type in PO of image *_HDR.img, so that spad does not have to read DO from spoold to get that information when pdplugin is new.
ET 4066657: Check the storage WORM type when expire image.
ET 4067149: Don't return and just log an error if shadow file is corrupt.
ET 4040344: Split the online checking result into small part and send it to spad
ET 4042622: Fixed the buffer size overrun caused by large amount of input data
ET 4066902: extract lsu from login param before mapping
ET 4039025: Don't support PDDO and do not try https case, so that nbu web server does not output the url request log which contain plain password
ET 4067818: Spad memory usage exceeds 100 GB
ET 4067678: Add image deletion log in history/imagedeletion from spad side Add configurable parameters to add PO/image deletion log on pdplugin side.

Version 31 added:
ET 4065036: rebase for replication to cloud lsu
ET 4060776: fix empty image name in jobstats
ET 4068537: High memory utilization more than 90% after upgrade to 9.1 and performance degraded
 

For details on the VxFS fixes:

Flex VxFS patch and MSDP EEB bundle
https://www.veritas.com/docs/100049433

Installation Instructions:  
 

Using the NetBackup Emergency Engineering Binary (EEB) installer 
https://www.veritas.com/docs/100019405

Installing EEBs on a NetBackup Appliance
https://www.veritas.com/docs/100023444

How to install client EEB's with VxUpdate
https://www.veritas.com/content/support/en_US/doc/125240132-131571482-0/v130876036-131571482

How to install Emergency Engineering Binaries on a VCS clustered NetBackup server
https://www.veritas.com/docs/100021995

How to Add a VxUpdate file (SJA) to a Flex-based NetBackup Primary Server (veritas.com)
https://www.veritas.com/content/support/en_US/article.100053208

Downloads:

NB_8.3.0.1_ET4013394_31.zip
NBAPP_EEB_ET4013394-3.3.0.1-31.x86_64.rpm

VRTSflex-nb_EEB_ET4013394-8.3.0.1-31.x86_64.rpm

Binary Package Contents and checksums for all files (cksum):

Recommended service state:

Stop all NetBackup services before applying this hotfix.