* * * READ ME * * *
                       * * * InfoScale 8.0 * * *
                         * * * Patch 3100 * * *
                         Patch Date: 2024-02-16


This document provides the following information:

   * PATCH NAME
   * OPERATING SYSTEMS SUPPORTED BY THE PATCH
   * PACKAGES AFFECTED BY THE PATCH
   * BASE PRODUCT VERSIONS FOR THE PATCH
   * SUMMARY OF INCIDENTS FIXED BY THE PATCH
   * DETAILS OF INCIDENTS FIXED BY THE PATCH
   * INSTALLATION PRE-REQUISITES
   * INSTALLING THE PATCH
   * REMOVING THE PATCH


PATCH NAME
----------
InfoScale 8.0 Patch 3100


OPERATING SYSTEMS SUPPORTED BY THE PATCH
----------------------------------------
RHEL8 x86-64


PACKAGES AFFECTED BY THE PATCH
------------------------------
VRTSaslapm
VRTSodm
VRTSvxfs
VRTSvxvm


BASE PRODUCT VERSIONS FOR THE PATCH
-----------------------------------
   * InfoScale Enterprise 8.0


SUMMARY OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
Patch ID: VRTSvxvm-8.0.0.2700
* 4154821 (4149248) Security vulnerabilities have been discovered in third-party components (OpenSSL, Curl, and libxml) employed by VxVM.
Patch ID: VRTSvxvm-8.0.0.2600
* 4067237 (4058894) Messages in /var/log/messages regarding "ignore_device".
* 4109554 (4105953) system panic due to VVR accessed a NULL pointer.
* 4111442 (4066785) create new option usereplicatedev=only to import the replicated LUN only.
* 4112549 (4112701) Nodes stuck in reconfig hang and vxconfigd coredump after rebooting all nodes with a delay of 5min in between them.
* 4113310 (4114601) Panic: in dmp_process_errbp() for disk pull scenario.
* 4113357 (4112433) Security vulnerabilities exists in third party components [openssl, curl and libxml].
* 4114963 (4114962) [NBFS-3.1][DL]:MASTER and CAT_FS got corrupted while performing multiple NVMEs failure
* 4115251 (4115195) [NBFS-3.1][DL]:MEDIA_FS got corrupted after panic loop test
* 4115252 (4115193) Data corruption observed after the node fault and cluster restart in DR environment
* 4115381 (4091783) build script and mb.sh changes in unixvm for integration of storageapi
* 4116548 (4111254) vradmind dumps core while associating a rlink to rvg because of NULL pointer reference.
* 4116551 (4108913) Vradmind dumps core because of memory corruption.
* 4116557 (4085404) Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.
* 4116559 (4091076) SRL gets into pass-thru mode because of head error.
* 4116562 (4114257) Observed IO hung and high system load average after rebooted master and one slave node rejoins cluster.
* 4116565 (4034741) The current fix from limits IO load on secondary causing deadlock situtaion
* 4116567 (4072862) Stop cluster hang because of RVGLogowner and CVMClus resources fail to offline.
* 4117110 (4113841) VVR panic in replication connection handshake request from network scan tool.
* 4118108 (4114867) systemd-udevd[2224]: invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 103 ('D')
* 4118111 (4065490) VxVM udev rules consumes more CPU and appears in "top" output when system has thousands of storage devices attached.
* 4118733 (4106689) Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95
* 4118845 (4116024) machine panic due to access illegal address.
* 4119087 (4067191) IS8.0_SUSE15_CVR: After rebooted slave node master node got panic
* 4119257 (4090772) vxconfigd/vx commands hung if fdisk opened secondary volume and secondary logowner panic'd
* 4119276 (4090943) VVR Primary RLink cannot connect as secondary reports SRL log is full.
* 4119438 (4117985) EC volume corruption due to lockless access of FPU
* 4120350 (4120878) After enabling the dmp_native_support, system failed to boot.
* 4121241 (4114927) Failed to mount /boot on dmp device after enabling dmp_native_support.
* 4121714 (4081740) vxdg flush command slow due to too many luns needlessly access /proc/partitions.
Patch ID: VRTSvxvm-8.0.0.2400
* 4110560 (4104927) Changing the attributes in vxvm-boot.service for SLES15 is causing regression in RHEL versions.
* 4113324 (4113323) VxVM Support on RHEL 8.8
* 4113661 (4091076) SRL gets into pass-thru mode because of head error.
* 4113663 (4095163) system panic due to a race freeing VVR update.
* 4113664 (4091390) vradmind service has dump core and stopped on few nodes
* 4113666 (4064772) After enabling slub debug, system could hang with IO load.
Patch ID: VRTSodm-8.0.0.3100
* 4154894 (4144269) After installing VRTSvxfs ODM fails to start.
Patch ID: VRTSodm-8.0.0.2900
* 4057432 (4056673) Rebooting the system results into emergency mode due to corruption of module dependency files. Incorrect vxgms dependency in odm service file.
Patch ID: VRTSodm-8.0.0.2700
* 4113912 (4113118) ODM support for RHEL 8.8.
Patch ID: VRTSodm-8.0.0.2600
* 4114656 (4114655) ODM support for RHEL 8.7 minor kernel 4.18.0-425.19.2.
Patch ID: VRTSodm-8.0.0.2300
* 4108585 (4107778) ODM support for RHEL 8.7 minor kernel.
Patch ID: VRTSodm-8.0.0.2200
* 4100923 (4100922) ODM module failed to load on RHEL8.7
Patch ID: VRTSvxfs-8.0.0.3100
* 4154855 (4141665) Security vulnerabilities exist in the Zlib third-party components used by VxFS.
Patch ID: VRTSvxfs-8.0.0.2900
* 4092518 (4096267) Veritas File Replication jobs might failed when there are large number of jobs run in parallel.
* 4097466 (4114176) After failover, job sync fails with error "Device or resource busy".
* 4107367 (4108955) VFR job hangs on source if thread creation fails on target.
* 4111457 (4117827) For EO compliance, there is a requirement to support 3 types of log file permissions  600, 640 and 644 with 600 being default 
new eo_perm tunable is added in vxtunefs command to manage the log file permissions.
* 4112417 (4094326) mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"
* 4118795 (4100021) Running setfacl followed by getfacl resulting in "No such device or address" error.
* 4119023 (4116329) While checking FS sanity with the help of "fsck -o full -n" command, we tried to correct the FS flag value (WORM/Softworm), but failed because -n (read-only) option was given.
* 4123143 (4123144) fsck binary generating coredump
Patch ID: VRTSvxfs-8.0.0.2700
* 4113911 (4113121) VXFS support for RHEL 8.8.
* 4114019 (4067505) invalid VX_AF_OVERLAY aflags error in fsck
* 4114020 (4083056) Hang observed while punching the smaller hole over the bigger hole.
* 4114021 (4101634) Directory inode getting incorrect file-type error in fsck.
Patch ID: VRTSvxfs-8.0.0.2600
* 4114654 (4114652) VXFS support for RHEL 8.7 minor kernel 4.18.0-425.19.2.
Patch ID: VRTSvxfs-8.0.0.2300
* 4108381 (4107777) VxFS support for RHEL 8.7 minor kernel.
Patch ID: VRTSvxfs-8.0.0.2200
* 4100925 (4100926) VxFS module failed to load on RHEL8.7
Patch ID: VRTSvxfs-8.0.0.2100
* 4095889 (4095888) Security vulnerabilities exist in the Sqlite third-party components used by VxFS.
Patch ID: VRTSvxfs-8.0.0.1800
* 4068960 (4073203) Veritas file replication might generate a core while replicating the files to target.
* 4071108 (3988752) Use ldi_strategy() routine instead of bdev_strategy() for IO's in solaris.
* 4072228 (4037035) VxFS should have the ability to control the number of inactive processing threads.
* 4078335 (4076412) Addressing Executive Order (EO) 14028,  initial requirements which is intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents.
* 4078520 (4058444) Loop mounts using files on VxFS fail on Linux systems.
* 4079142 (4077766) VxFS kernel module might leak memory during readahead of directory blocks.
* 4079173 (4070217) Command fsck might fail with 'cluster reservation failed for volume' message for a disabled cluster-mounted filesystem.
* 4082260 (4070814) Security Vulnerability observed in Zlib a third party component VxFS uses.
* 4082865 (4079622) Existing migration read/write iter operation handling is not fully functional as vxfs uses normal read/write file operation only.
* 4083335 (4076098) Fix migration issues seen with falcon-sensor.
* 4085623 (4085624) While running fsck, fsck might dump core.
* 4085839 (4085838) Command fsck may generate core due to processing of zero size attribute inode.
* 4086085 (4086084) VxFS mount operation causes system panic.
* 4088341 (4065575) Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space condition
Patch ID: VRTSvxfs-8.0.0.1700
* 4081150 (4079869) Security Vulnerability in VxFS third party components
* 4083948 (4070814) Security Vulnerability in VxFS third party component Zlib
Patch ID: VRTSvxfs-8.0.0.1200
* 4055808 (4062971) Enable partition directory on WORM file system
* 4056684 (4056682) New features information on a filesystem with fsadm(file system administration utility) from a device is not displayed.
* 4062606 (4062605) Minimum retention time cannot be set if the maximum retention time is not set.
* 4065565 (4065669) Creating non-WORM checkpoints fails when the tunables - minimum retention time and maximum retention time are set.
* 4065651 (4065666) Enable partition directory on WORM file system having WORM enabled on files with retention period not expired.
Patch ID: VRTSvxfs-8.0.0.1100
* 4061114 (4052883) VxFS support for RHEL 8.5.


DETAILS OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
This patch fixes the following incidents:

Patch ID: VRTSvxvm-8.0.0.2700

* 4154821 (Tracking ID: 4149248)

SYMPTOM:
Third-party components (OpenSSL, curl, and libxml) used by VxVM exhibit security vulnerabilities.

DESCRIPTION:
VxVM utilizes current versions of OpenSSL, curl, and libxml, which have been reported to have security vulnerabilities.

RESOLUTION:
Upgrades to newer versions of OpenSSL, curl, and libxml have been implemented to address the reported security vulnerabilities.

Patch ID: VRTSvxvm-8.0.0.2600

* 4067237 (Tracking ID: 4058894)

SYMPTOM:
After package installation and reboot , messages regarding udev rules for ignore_device are observed in /var/log/messages .
systemd-udevd[774]: /etc/udev/rules.d/40-VxVM.rules:25 Invalid value for OPTIONS key, ignoring: 'ignore_device'

DESCRIPTION:
From SLES15 Sp3 onwards , ignore_device is deprecated from udev rules and is not available for use anymore . Hence these messages are observed in system logs .

RESOLUTION:
Required changes have been done to handle this defect.

* 4109554 (Tracking ID: 4105953)

SYMPTOM:
System panic with below stack in CVR environment.

 #9 [] page_fault at 
    [exception RIP: vol_ru_check_update_done+183]
#10 [] vol_rv_write2_done at [vxio]
#11 [] voliod_iohandle at [vxio]
#12 [] voliod_loop at [vxio]
#13 [] kthread at

DESCRIPTION:
In CVR environment, when IO is issued in writeack sync mode we ack to application when datavolwrite is done on either log client or logowner depending on 
where IO is issued on. it could happen that VVR freed the metadata I/O update after SRL write is done incase of writeack sync mode, but later after freeing the update, its accessed again and hence we end up in hitting NULL ptr deference.

RESOLUTION:
Code changes have been made to avoid the accessing NULL pointer.

* 4111442 (Tracking ID: 4066785)

SYMPTOM:
When the replicated disks are in SPLIT mode, importing its disk group failed with "Device is a hardware mirror".

DESCRIPTION:
When the replicated disks are in SPLIT mode, which are readable and writable, importing its disk group failed with "Device is a hardware mirror". Third party doesn't expose disk attribute to show when it's in SPLIT mode. With this new enhancement, the replicated disk group can be imported with option `-o usereplicatedev=only`.

RESOLUTION:
The code is enhanced to import the replicated disk group with option `-o usereplicatedev=only`.

* 4112549 (Tracking ID: 4112701)

SYMPTOM:
Observed reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min in between them due to Vxconfigd core dumped on master node

DESCRIPTION:
1. reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min.
             2. This is because fork failed during command shipping which caused vxconfigd core dump on master. So all reconfigurations after that failed to 
                 process.

RESOLUTION:
Reboot master node where vold is core dumped.

* 4113310 (Tracking ID: 4114601)

SYMPTOM:
System gets panicked and rebooted

DESCRIPTION:
RCA:
Start the IO on volume device and pull out it's disk from the machine and hit below panic on RHEL8.

 dmp_process_errbp
 dmp_process_errbuf.cold.2+0x328/0x429 [vxdmp]
 dmpioctl+0x35/0x60 [vxdmp]
 dmp_flush_errbuf+0x97/0xc0 [vxio]
 voldmp_errbuf_sio_start+0x4a/0xc0 [vxio]
 voliod_iohandle+0x43/0x390 [vxio]
 voliod_loop+0xc2/0x330 [vxio]
 ? voliod_iohandle+0x390/0x390 [vxio]
 kthread+0x10a/0x120
 ? set_kthread_struct+0x50/0x50

As disk pulled out from the machine VxIO hit a IO error and it routed that IO to dmp layer via kernel-kernel IOCTL for error analysis.
following is the code path for IO routing,

voldmp_errbuf_sio_start()-->dmp_flush_errbuf()--->dmpioctl()--->dmp_process_errbuf()

dmp_process_errbuf() retrieves device number of the underlying path (os-device).
and it tries to get bdev (i.e. block_device) pointer from path-device number.
As path/os-device is removed by disk pull, linux returns fake bdev for the path-device number.
For this fake bdev there is no gendisk associated with it (bdev->bd_disk is NULL).

We are setting this NULL bdev->bd_disk to the IO buffer routed from vxio.
which leads a panic on dmp_process_errbp.

RESOLUTION:
If bdev->bd_disk found NULL then set DMP_CONN_FAILURE error on the IO buffer and return DKE_ENXIO to vxio driver

* 4113357 (Tracking ID: 4112433)

SYMPTOM:
Vulnerabilities have been reported in third party components, [openssl, curl and libxml] that are used by VxVM.

DESCRIPTION:
Third party components [openssl, curl and libxml] in their current versions,  used by VxVM have been reported with security vulnerabilities which needs

RESOLUTION:
[openssl, curl and libxml] have been upgraded to newer versions in which the reported security vulnerabilities have been addressed.

* 4114963 (Tracking ID: 4114962)

SYMPTOM:
File system data corruption with mirrored volumes in Flexible Storage Sharing (FSS) environments during beyond fault storage failure situations.

DESCRIPTION:
In FSS environments, data change object (DCO) provides functionality to track changes on detached mirrors using bitmaps. This bitmap is later used for re-sync of detached mirrors data (change delta).
When DCO volume and data volume share the same set of devices, DCO volumes last mirror failure means IOs on data volume is going to fail. In such cases instead of invaliding DCO volumes, proactively IO is failed.
This helps in protecting DCO when entire storage comes back and optimal recovery of mirrors can be performed.
When disk for one of the mirror of DCO object become available, the bug in DCO update incorrectly updates metadata of DCO which lead to ignoring valid DCO maps during actual volume recovery and hence newly recovered mirrors of volume missed blocks of valid application data. This lead to corruption when read IO were serviced from the newly recovered mirrors.

RESOLUTION:
The login of FMR map updating transaction of enabling disks is fixed to resolve the bug. This ensures all valid bitmaps are considered for recovery of mirrors and avoid data loss.

* 4115251 (Tracking ID: 4115195)

SYMPTOM:
Data corruption on file-systems with  erasure coded volumes

DESCRIPTION:
In Erasure coded (EC) volume are used in Flexible shared storage (FSS) environments, data change object (DCO) is used to tracking changes in volume with faulted columns. The DCO provides a bitmap of all changed regions during rebuild of the faulted columns. When EC volume starts with few faulted columns, the log-replay IO could not be performed on those columns. Those additional writes are expected to be tracked in DCO bitmap. Due to bug those IOs are not getting tracked in DCO bitmap as DCO bitmaps are not yet enabled when log-replay is triggered. Hence when the remaining columns are attached back, appropriate data blocks of those log-replay IOs are skipped during rebuild. This leads to data corruption when reads are serviced from those columns.

RESOLUTION:
Code changes are done to ensure log-replay on EC volume is triggered only after ensuring DCO tracking is enabled. This ensures that all IOs from log-replay operations are tracked in DCO maps for remaining faulted columns of volume.

* 4115252 (Tracking ID: 4115193)

SYMPTOM:
Data corruption on VVR primary with storage loss beyond fault tolerance level in replicated environment.

DESCRIPTION:
In Flexible Storage Sharing (FSS)  environment  any node fault can lead to storage failure. In VVR primary when last  mirror of SRL  (Storage Replicator Log) volume faulted while application writes are in progress replication is expected to go to pass-through mode.
This information is persistently recorded in the kernel log (KLOG). In the event of cascaded storage node failures, the KLOG updation protocol could not update quorum number of copies. This mis-match in on-disk v/s in-core state of VVR objects leading to data loss due to missing recovery when all storage faults are resolved.

RESOLUTION:
Code changes to handle the KLOG update failure in SRL IO failure handling is done to ensure configuration on-disk and in-core is consistent and subsequent application IO will not be allowed to avoid data corruption.

* 4115381 (Tracking ID: 4091783)

SYMPTOM:
Buildarea creation for unixvm were failing

DESCRIPTION:
unixvm build were failing as there is dependency on the storageapi while creation of build area for unixvm and veki.

This intern were causing issues in Veki packaging during unixvm builds and vxio driver compilation dependency

RESOLUTION:
Added support for storageapi build area creation and building the storageapi internally from unixvm build scripts.

* 4116548 (Tracking ID: 4111254)

SYMPTOM:
vradmind dumps core with the following stack:

#3  0x00007f3e6e0ab3f6 in __assert_fail () from /root/cores/lib64/libc.so.6
#4  0x000000000045922c in RDS::getHandle ()
#5  0x000000000056ec04 in StatsSession::addHost ()
#6  0x000000000045d9ef in RDS::addRVG ()
#7  0x000000000046ef3d in RDS::createDummyRVG ()
#8  0x000000000044aed7 in PriRunningState::update ()
#9  0x00000000004b3410 in RVG::update ()
#10 0x000000000045cb94 in RDS::update ()
#11 0x000000000042f480 in DBMgr::update ()
#12 0x000000000040a755 in main ()

DESCRIPTION:
vradmind was trying to access a NULL pointer (Remote Host Name) in a rlink object, as the Remote Host attribute of the rlink hasn't been set.

RESOLUTION:
The issue has been fixed by making code changes.

* 4116551 (Tracking ID: 4108913)

SYMPTOM:
Vradmind dumps core with the following stacks:
#3  0x00007f2c171be3f6 in __assert_fail () from /root/coredump/lib64/libc.so.6
#4  0x00000000005d7a90 in VList::concat () at VList.C:1017
#5  0x000000000059ae86 in OpMsg::List2Msg () at Msg.C:1280
#6  0x0000000000441bf6 in OpMsg::VList2Msg () at ../../include/Msg.h:389
#7  0x000000000043ec33 in DBMgr::processStatsOpMsg () at DBMgr.C:2764
#8  0x00000000004093e9 in process_message () at srvmd.C:418
#9  0x000000000040a66d in main () at srvmd.C:733

#0  0x00007f4d23470a9f in raise () from /root/core.Jan18/lib64/libc.so.6
#1  0x00007f4d23443e05 in abort () from /root/core.Jan18/lib64/libc.so.6
#2  0x00007f4d234b3037 in __libc_message () from /root/core.Jan18/lib64/libc.so.6
#3  0x00007f4d234ba19c in malloc_printerr () from /root/core.Jan18/lib64/libc.so.6
#4  0x00007f4d234bba9c in _int_free () from /root/core.Jan18/lib64/libc.so.6
#5  0x00000000005d5a0a in ValueElem::_delete_val () at Value.C:491
#6  0x00000000005d5990 in ValueElem::~ValueElem () at Value.C:480
#7  0x00000000005d7244 in VElem::~VElem () at VList.C:480
#8  0x00000000005d8ad9 in VList::~VList () at VList.C:1167
#9  0x000000000040a71a in main () at srvmd.C:743

#0  0x000000000040b826 in DList::head () at ../include/DList.h:82
#1  0x00000000005884c1 in IpmHandle::send () at Ipm.C:1318
#2  0x000000000056e101 in StatsSession::sendUCastStatsMsgToPrimary () at StatsSession.C:1157
#3  0x000000000056dea1 in StatsSession::sendStats () at StatsSession.C:1117
#4  0x000000000046f610 in RDS::collectStats () at RDS.C:6011
#5  0x000000000043f2ef in DBMgr::collectStats () at DBMgr.C:2799
#6  0x00007f98ed9131cf in start_thread () from /root/core.Jan26/lib64/libpthread.so.0
#7  0x00007f98eca4cdd3 in clone () from /root/core.Jan26/lib64/libc.so.6

DESCRIPTION:
There is a race condition in vradmind that may cause memory corruption and unpredictable result. Vradmind periodically forks a child thread to collect VVR statistic data and send them to the remote site. While the main thread may also be sending data using the same handler object, thus member variables in the handler object are accessed in parallel from multiple threads and may become corrupted.

RESOLUTION:
The code changes have been made to fix the issue.

* 4116557 (Tracking ID: 4085404)

SYMPTOM:
Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.

DESCRIPTION:
The active map flush caused RVG serialization. Once RVG gets serialized, all IOs are queued in restart queue, till the active map flush is finished. The too frequent active map flush caused the huge IO drop during flushing SRL to DCM.

RESOLUTION:
The code is modified to adjust the frequency of active map flush and balance the application IO and SRL flush.

* 4116559 (Tracking ID: 4091076)

SYMPTOM:
SRL gets into pass-thru mode when it's about to overflow.

DESCRIPTION:
Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.

RESOLUTION:
Fixed the check condition to resolve the issue.

* 4116562 (Tracking ID: 4114257)

SYMPTOM:
VxVM cmd is hung and file system was waiting for io to complete.

file system stack:
#3 [] wait_for_completion at 
#4 [] vx_bc_biowait at [vxfs]
#5 [] vx_biowait at [vxfs]
#6 [] vx_isumupd at [vxfs]
#7 [] __switch_to_asm at 
#8 [] vx_process_revokedele at [vxfs]
#9 [] vx_recv_revokedele at [vxfs]
#10 [] vx_recvdele at [vxfs]
#11 [] vx_msg_process_thread at [vxfs]

vxconfigd stack:
[<0>] volsync_wait+0x106/0x180 [vxio]
[<0>] vol_ktrans+0x9f/0x2c0 [vxio]
[<0>] volconfig_ioctl+0x82a/0xdf0 [vxio]
[<0>] volsioctl_real+0x38a/0x450 [vxio]
[<0>] vols_ioctl+0x6d/0xa0 [vxspec]
[<0>] vols_unlocked_ioctl+0x1d/0x20 [vxspec]

One of vxio thread was waiting for IO drain with below stack.

 #2 [] schedule_timeout at 
 #3 [] vol_rv_change_sio_start at [vxio]
 #4 [] voliod_iohandle at [vxio]

DESCRIPTION:
VVR rvdcm flush SIO was triggered by VVR logowner change and it would set the ru_state throttle flags which caused  MDATA_SHIP SIO got queued in rv_mdship_throttleq. As the MDATA_SHIP SIOs are active, it caused rvdcm flush SIO unable to proceed. In the end, rvdcm_flush SIO was waiting for SIOs in rv_mdship_throttleq to complete. SIOs in rv_mdship_throttleq were waiting rvdcm_flush SIO to complete. Hence a  dead lock situation.

RESOLUTION:
Code changes have been made to solve the dead lock issue.

* 4116565 (Tracking ID: 4034741)

SYMPTOM:
Due to a common RVIOmem pool being used by multiple RVG, a deadlock scenario gets created, causing high load average and system hang.

DESCRIPTION:
The current fix limits IO load on secondary by retaining the updates in NMCOM pool until the DV write done, by which RVIOMEM pool became easy to fill up and 
deadlock situtaion may occur, esp. when high work load on multiple RVGs or cross direction RVGs.Currently all RVGs share the same RVIOMEM pool, while NMCOM 
pool, RDBACK pool and network/dv update list are all per-RVGs, so the RVIOMEM pool becomes the bottle neck on secondary, which is easy to full and run into 
deadlock situation.

RESOLUTION:
Code changes to honor per-RVG RVIOMEM pool to resolve the deadlock issue.

* 4116567 (Tracking ID: 4072862)

SYMPTOM:
In case RVGLogowner resources get onlined on slave nodes, stop the whole cluster may fail and RVGLogowner resources goes in to offline_propagate state.

DESCRIPTION:
While stopping whole cluster, the racing may happen between CVM reconfiguration and RVGLogowner change SIO.

RESOLUTION:
Code changes have been made to fix these racings.

* 4117110 (Tracking ID: 4113841)

SYMPTOM:
VVR panic happened in below code path:

kmsg_sys_poll()
nmcom_get_next_mblk() 
nmcom_get_hdr_msg() 
nmcom_get_next_msg() 
nmcom_wait_msg_tcp() 
nmcom_server_main_tcp()

DESCRIPTION:
When the network scan tool send request to VVR which is unexpected, during the VVR connection handshake, the tcp connection may be terminated immediately by the network scan tool, which may lead to the sock released. Hence, VVR panic when try to refer to it as hit the NULL pointer during the processing.

RESOLUTION:
The code change has been made to check sock is valid, otherwise, return without continue with the VVR connection.

* 4118108 (Tracking ID: 4114867)

SYMPTOM:
Getting these error messages while adding new disks
[root@server101 ~]# cat /etc/udev/rules.d/41-VxVM-selinux.rules | tail -1
KERNEL=="VxVM*", SUBSYSTEM=="block", ACTION=="add", RUN+="/bin/sh -c 'if [ `/usr/sbin/getenforce` != "Disabled" -a `/usr/sbin/
[root@server101 ~]#
[root@server101 ~]# systemctl restart systemd-udevd.service
[root@server101 ~]# udevadm test /block/sdb 2>&1 | grep "invalid"
invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 104 ('D')

DESCRIPTION:
In /etc/udev/rules.d/41-VxVM-selinux.rules double quotation on Disabled and disable is the issue.

RESOLUTION:
Code changes have been made to correct the problem.

* 4118111 (Tracking ID: 4065490)

SYMPTOM:
systemd-udev threads consumes more CPU during system bootup or device discovery.

DESCRIPTION:
During disk discovery when new storage devices are discovered, VxVM udev rules are invoked for creating hardware path
symbolic link and setting SELinux security context on Veritas device files. For creating hardware path symbolic link to each
storage device, "find" command is used internally which is CPU intensive operation. If too many storage devices are attached to
system, then usage of "find" command causes high CPU consumption.

Also, for setting appropriate SELinux security context on VxVM device files, restorecon is done irrespective of SELinux is enabled or disabled.

RESOLUTION:
Usage of "find" command is replaced with "udevadm" command. SELinux security context on VxVM device files is being set
only when SELinux is enabled on system.

* 4118733 (Tracking ID: 4106689)

SYMPTOM:
Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95. The error logs are observed as below:
Mounting ZFS filesystems: cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export/home' on '/export/home': failure mounting parent dataset
cannot mount 'rpool/export/home/addm' on /export/home/addm': directory is not empty
.... ....
svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: one or more file systems failed.

DESCRIPTION:
When DMP native support is enabled and the "faulted" zpools are found, VxVM will deport the faulty zpools and re-import them. In case fs-local isn't started before vxvm-startup2, this error handling will cause a non-empty /export which further cause zfs mount failure.

RESOLUTION:
Code changes have been made to guarantee the mount order of rpool and zpools.

* 4118845 (Tracking ID: 4116024)

SYMPTOM:
kernel panicked at gab_ifreemsg with following stack:
gab_ifreemsg
gab_freemsg
kmsg_gab_send
vol_kmsg_sendmsg
vol_kmsg_sender

DESCRIPTION:
In a CVR environment there is a RVG of > 600 data volumes, enabling vxvvrstatd daemon through service vxvm-recover. vxvvrstatd calls into ioctl(VOL_RV_APPSTATS) , the latter will generate a kmsg whose length is longer than 64k and trigger a kernel panic due to GAB/LLT no support any message longer than 64k.

RESOLUTION:
Code changes have been done to add a limitation to the maximum number of data volumes for which that ioctl(VOL_RV_APPSTATS) can request the VVR statistics.

* 4119087 (Tracking ID: 4067191)

SYMPTOM:
In CVR environment after rebooting Slave node, Master node may panic with below stack:

Call Trace:
dump_stack+0x66/0x8b
panic+0xfe/0x2d7
volrv_free_mu+0xcf/0xd0 [vxio]
vol_ru_free_update+0x81/0x1c0 [vxio]
volilock_release_internal+0x86/0x440 [vxio]
vol_ru_free_updateq+0x35/0x70 [vxio]
vol_rv_write2_done+0x191/0x510 [vxio]
voliod_iohandle+0xca/0x3d0 [vxio]
wake_up_q+0xa0/0xa0
voliod_iohandle+0x3d0/0x3d0 [vxio]
voliod_loop+0xc3/0x330 [vxio]
kthread+0x10d/0x130
kthread_park+0xa0/0xa0
ret_from_fork+0x22/0x40

DESCRIPTION:
As part of CVM Master switch a rvg_recovery is triggered. In this step race
condition can occured between the VVR objects due to which the object value
is not updated properly and can cause panic.

RESOLUTION:
Code changes are done to handle the race condition between VVR objects.

* 4119257 (Tracking ID: 4090772)

SYMPTOM:
vxconfigd/vx commands hang on secondary site in a CVR environment.

DESCRIPTION:
Due to a window with unmatched SRL positions, if any application (e.g. fdisk) trying
to open the secondary RVG volume will acquire a lock and wait for SRL positions to match.
During this if any vxvm transaction kicked in will also have to wait for same lock.
Further logowner node panic'd which triggered logownership change protocol which hung
as earlier transaction was stuck. As logowner change protocol could not complete,
in absence of valid logowner SRL position could not match and caused deadlock. That lead
to vxconfigd and vx command hang.

RESOLUTION:
Added changes to allow read operation on volume even if SRL positions are
unmatched. We are still blocking write IOs and just allowing open() call for read-only
operations, and hence there will not be any data consistency or integrity issues.

* 4119276 (Tracking ID: 4090943)

SYMPTOM:
On Primary, RLink is continuously getting connected/disconnected with below message seen in secondary syslog:
  VxVM VVR vxio V-5-3-0 Disconnecting replica <rlink_name> since log is full on secondary.

DESCRIPTION:
When RVG logowner node panic, RVG recovery happens in 3 phases.
At the end of 2nd phase of recovery in-memory and on-disk SRL positions remains incorrect
and during this time if there is logowner change then Rlink won't get connected.

RESOLUTION:
Handled in-memory and on-disk SRL positions correctly.

* 4119438 (Tracking ID: 4117985)

SYMPTOM:
Memory/data corruption hit for EC volumes

DESCRIPTION:
This is a porting request original request was already reviewed:http://codereview.engba.veritas.com/r/42056/

Memory corruption hitting in EC was fixed by calling kernel_fpu_begin() for kernel version < rhel8.6. But in latest kernel kernel_fpu_begin() symbol is not 
available, We can not use it. So we have created separate Module with name 'storageapi' which is having implementation of _fpu_begin and _fpu_end
VxIO module is dependent on 'storageapi'

RESOLUTION:
take a fpu lock for FPU related operations

* 4120350 (Tracking ID: 4120878)

SYMPTOM:
System doesn't come up on taking a reboot after enabling dmp_native_support. System goes into maintenance mode.

DESCRIPTION:
"vxio.ko" is dependent on the new "storageapi.ko" module. "storageapi.ko" was missing from VxDMP_initrd file, which is created when dmp_native_support is enabled. So on reboot, without "storageapi.ko" present, "vxio.ko" fails to load.

RESOLUTION:
Code changes have been made to include "strorageapi.ko" in VxDMP_initrd.

* 4121241 (Tracking ID: 4114927)

SYMPTOM:
After enabling dmp_native_support and taking reboot, /boot is not mounted VxDMP node.

DESCRIPTION:
When dmp_native_support is enabled, vxdmproot script is expected to modify the /etc/fstab entry for /boot so that on next boot up, /boot is mounted on dmp device instead of OS device. Also, this operation modifies SELinux context of file /etc/fstab. This causes the machine to go into maintenance mode because of a read permission denied error for /etc/fstab on boot up.

RESOLUTION:
Code changes have been done to make sure SELinux context is preserved for /etc/fstab file and /boot is mounted on dmp device when dmp_native_support is enabled.

* 4121714 (Tracking ID: 4081740)

SYMPTOM:
vxdg flush command slow due to too many luns needlessly access /proc/partitions.

DESCRIPTION:
Linux BLOCK_EXT_MAJOR(block major 259) is used as extended devt for block devices. When partition number of one device is more than 15, the partition device gets assigned under major 259 to solve the sd limitations (16 minors per device), by which more partitions are allowed for one sd device. During "vxdg flush", for each lun in the disk group, vxconfigd reads file /proc/partitions line by line through fgets() to find all the partition devices with major number 259, which would cause vxconfigd to respond sluggishly if there are large amount of luns in the disk group.

RESOLUTION:
Code has been changed to remove the needless access on /proc/partitions for the luns without using extended devt.

Patch ID: VRTSvxvm-8.0.0.2400

* 4110560 (Tracking ID: 4104927)

SYMPTOM:
vxvm-boot.service fails to start on linux platforms other than SLES15

DESCRIPTION:
SLES15 specific attribute changes causes vxvm-boot.service to fail to start on other linux platforms.

RESOLUTION:
A new vxvm-boot.service file to honour vxvm-boot.service for SLES15, the existing vxvm-boot.service file will serve for other linux platforms.

* 4113324 (Tracking ID: 4113323)

SYMPTOM:
Existing package failed to load on RHEL 8.8 server.

DESCRIPTION:
RHEL 8.8 is a new release and hence VxVM module is compiled with this new kernel along with few other changes.

RESOLUTION:
Compiled VxVM code against 8.8 kernel and made changes to make it compatible.

* 4113661 (Tracking ID: 4091076)

SYMPTOM:
SRL gets into pass-thru mode when it's about to overflow.

DESCRIPTION:
Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.

RESOLUTION:
Fixed the check condition to resolve the issue.

* 4113663 (Tracking ID: 4095163)

SYMPTOM:
System panic with below stack:
 #6 [] invalid_op at 
    [exception RIP: __slab_free+414]
 #7 [] kfree at 
 #8 [] vol_ru_free_update at [vxio]
 #9 [] vol_ru_free_updateq at  [vxio]
#10 [] vol_rv_write2_done at [vxio]
#11 [] voliod_iohandle at [vxio]
#12 [] voliod_loop at [vxio]

DESCRIPTION:
The update gets freed as a part of VVR recovery. At the same time, this update also gets freed in VVR second phase of write. Hence there is a race in freeing the updates and caused the system panic.

RESOLUTION:
Code changes have been made to avoid

* 4113664 (Tracking ID: 4091390)

SYMPTOM:
vradmind hit the core dump while accessing pHdr, which is already freed.

DESCRIPTION:
While processing the config message - CFG_UPDATE, we incorrectly freed the existing config message objects. Later, objects are accessed again which dumped the vradmind core.

RESOLUTION:
Changes are done to access the correct configuration objects.

* 4113666 (Tracking ID: 4064772)

SYMPTOM:
After enabling slub debug, system could hang with IO load.

DESCRIPTION:
When creating VxVM I/O memory, VxVM does not align the cache size. This unaligned length will be treated as an invalid I/O length in SCSI layer, which causes some I/O requests are stuck in an invalid state and results in the I/Os never being able to complete. Thus system hang could be observed, especially after cache slub debug is enabled.

RESOLUTION:
Code changes have been done to align the cache size.

Patch ID: VRTSodm-8.0.0.3100

* 4154894 (Tracking ID: 4144269)

SYMPTOM:
After installing, ODM fails to start.

DESCRIPTION:
Because of the VxFS version update, the ODM module needs to be repackaged due to an
internal dependency on the VxFS version.

RESOLUTION:
As part of this fix, the ODM module has been repackaged to support the updated
VxFS version.

Patch ID: VRTSodm-8.0.0.2900

* 4057432 (Tracking ID: 4056673)

SYMPTOM:
Rebooting the system results into emergency mode.

DESCRIPTION:
Module dependency files get corrupted due to parallel invocation of depmod.

RESOLUTION:
Serialized the invocation of depmod through file lock. Corrected vxgms dependency in odm service file.

Patch ID: VRTSodm-8.0.0.2700

* 4113912 (Tracking ID: 4113118)

SYMPTOM:
The ODM module fails to load on RHEL8.8.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.8.

RESOLUTION:
Updated ODM to support RHEL 8.8.

Patch ID: VRTSodm-8.0.0.2600

* 4114656 (Tracking ID: 4114655)

SYMPTOM:
The ODM module fails to load on RHEL8.7 minor kernel 4.18.0-425.19.2.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.7 minor kernel.

RESOLUTION:
Updated ODM to support RHEL 8.7 minor kernel 4.18.0-425.19.2.

Patch ID: VRTSodm-8.0.0.2300

* 4108585 (Tracking ID: 4107778)

SYMPTOM:
The ODM module fails to load on RHEL8.7 minor kernel.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.7 minor kernel.

RESOLUTION:
Modified existing modinst-odm script to accommodate the changes in the kernel and load the correct module.

Patch ID: VRTSodm-8.0.0.2200

* 4100923 (Tracking ID: 4100922)

SYMPTOM:
ODM module failed to load on RHEL8.7

DESCRIPTION:
The RHEL8.7 is new release and it has some changes in kernel which caused ODM module failed to load
on it.

RESOLUTION:
Added code to support ODM on RHEL8.7.

Patch ID: VRTSvxfs-8.0.0.3100

* 4154855 (Tracking ID: 4141665)

SYMPTOM:
Security vulnerabilities exist in the Zlib third-party components used by VxFS.

DESCRIPTION:
VxFS uses Zlib third-party components with some security vulnerabilities.

RESOLUTION:
VxFS is updated to use a newer version of Zlib third-party components in which the security vulnerabilities have been addressed.

Patch ID: VRTSvxfs-8.0.0.2900

* 4092518 (Tracking ID: 4096267)

SYMPTOM:
Veritas File Replication jobs might failed when there are large number of jobs run in parallel.

DESCRIPTION:
File Replication Jobs might fail, with Large number of jobs configured and running in parallel with Veritas File Replication.
With large number of jobs there is a chance of referring a job which is already freed, due to which there is a core generated with replication service and
job might failed.

RESOLUTION:
updated code to handle the code to take a hold while checking invalid job configuration.

* 4097466 (Tracking ID: 4114176)

SYMPTOM:
After failover, job sync fails with error "Device or resource busy".

DESCRIPTION:
If job is in failed state on target because of job failure from source side, repld was not updating its state when it was restarted in recovery mode. Because of which job state was remaining in running state even after successful replication on target. With this state on target, if job is promoted, then replication process was not creating new ckpt for first sync after failover which was resulting in corrupting state file on new source. Because of this incorrect/corrupt state file, job sync from new source was failing with error "Device or resource busy".

RESOLUTION:
Code is modified to correct the state on target when job was started in recovery mode.

* 4107367 (Tracking ID: 4108955)

SYMPTOM:
VFR job hangs on source if thread creation fails on target.

DESCRIPTION:
On Target, if thread creation for pass completion fails because of high memory usage, repld demon doesn't send that failure reply to source. This can lead to vxfsreplicate process to remains in waiting state indefinitely for reply for pass completion from target. This will lead to job hang on source and will need manual intervention to kill the job.

RESOLUTION:
Code is modified to  retry thread creation on target and if it fails after 5 retries, target will reply to source with appropriate error.

* 4111457 (Tracking ID: 4117827)

SYMPTOM:
Without tunable change the logfile permission will always be 600 EO compliant

DESCRIPTION:
Tunable values and behavior:

Value                   Behavior
0 (default)          600 permissions, update existing file permissions on upgrade
1                    640 permissions, update existing file permissions on upgrade
2                    644 permissions, update existing file permissions on upgrade
3                    Inherit umask, update existing file permissions on upgrade
10                   600 permissions, dont touch existing file permissions on upgrade
11                   640 permissions, dont touch existing file permissions on upgrade
12                   644 permissions, dont touch existing file permissions on upgrade
13                   Inherit umask, dont touch existing file permissions on upgrade
--------------------------------------------------------------------------------------

Adding new tunable as part of vxtunefs command which is per-node global tunable (not per filesystem). 
For Executive Order, CPI will be having workflow to update the tunable during installation/upgrade/configuration 
which will take care of updating in all nodes.

RESOLUTION:
New tunable is added to vxtunefs command.
How to set tunable:
/opt/VRTS/bin/vxtunefs -D eo_perm=1

* 4112417 (Tracking ID: 4094326)

SYMPTOM:
mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"

DESCRIPTION:
In vx_sl_kmcache_init(), kmcache is initialized for each level (in this case it is 8) separately. For passing the cache name as an argument to kmem_cache_create(), we have used a macro.

#define VX_SL_KMCACHE_NAME(level)       "vx_sl_node_"#level
#define VX_SL_KMCACHE_CREATE(level)                                     \
                kmem_cache_create(VX_SL_KMCACHE_NAME(level),            \
                                  VX_KMEM_SIZE(VX_SL_KMCACHE_SIZE(level)),\
                                  0, NULL, NULL, NULL, NULL, NULL, 0);


While using this macro, we have passed "level" as an argument and that has been expanded as "vx_sl_node_level" for all the 8 levels in `for` loop. This is causing the cache allocation for all the 8 levels with same name.

RESOLUTION:
Passing separate variable value (as level value) to VX_SL_KMCACHE_NAME as it is done in  vx_wb_sl_kmcache_init().

* 4118795 (Tracking ID: 4100021)

SYMPTOM:
Running setfacl followed by getfacl resulting in "No such device or address" error.

DESCRIPTION:
When running setfacl command on some of the directories which have the VX_ATTR_INDIRECT type of acl attribute, it is not removing the existing acl attribute and adding a new one, which should  not happen ideally. This is resulting in the failure of getfacl with following "No such device or address" error.

RESOLUTION:
we have done the code chages to removal of VX_ATTR_INDIRECT type acl in setfacl code.

* 4119023 (Tracking ID: 4116329)

SYMPTOM:
fsck -o full -n command will fail with error:
"ERROR: V-3-28446:  bc_write failure devid = 0, bno = 8, len = 1024"

DESCRIPTION:
Previously, to correct the file system WORM/SoftWORM, we didn't check  if user wanted to correct the pflags or just wanted to validate if value is flag is missing or not. Also fsck was not capable to handle SOFTWORM flag.

RESOLUTION:
Code added to not try to fix the the problem if user ran fsck with -n option. Also SOFTWORM scenario is added.

* 4123143 (Tracking ID: 4123144)

SYMPTOM:
fsck binary generating coredump

DESCRIPTION:
In internal testing we found that fsck binary generates coredump due to below mentioned assert when we try to repair corrupted file system using below command:
./fsck -o full -y /dev/vx/rdsk/testdg/vol1

ASSERT(fset >= VX_FSET_STRUCT_INDEX)

RESOLUTION:
Added code to set default (primary) fileset by scanning the fset header list.

Patch ID: VRTSvxfs-8.0.0.2700

* 4113911 (Tracking ID: 4113121)

SYMPTOM:
The VxFS module fails to load on RHEL8.8.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.8.

RESOLUTION:
Updated VXFS to support RHEL 8.8.

* 4114019 (Tracking ID: 4067505)

SYMPTOM:
Fsck reports error invalid VX_AF_OVERLAY aflags

DESCRIPTION:
If the inode does not have push linkage (inode not allocated / inode and data already pushed), we skip pushing the data blocks when the inode is removed. Inode will have overlay data blocks, gen bumped up and IEREMOVE set. During extop processing size is set to 0 and bmap is cleared. This is a valid scenario.

Fsck while validating the inodes with overlay flag set, expects gen can be different only if the overlay inode has IEREMOVE set and it is last clone in the chain.

RESOLUTION:
If the push inode is not present allow gen to be different even if the clone is not last in chain.

* 4114020 (Tracking ID: 4083056)

SYMPTOM:
Hang observed while punching the smaller hole over the bigger hole.

DESCRIPTION:
We observed the hang while punching the smaller hole over the bigger hole in the file due to the tight race
while processing the punching of the hole to the file and flushing it to the disk.

RESOLUTION:
Code changes checked in.

* 4114021 (Tracking ID: 4101634)

SYMPTOM:
Fsck reports error directory block containing inode has incorrect file-type and directory contains invalid directory blocks.

DESCRIPTION:
While doing diretory sanity in fsck we skip updating new directory type ondisk in case of filetype error, hence fsck
reporting incorrect file-type error and directory contains invalid directory blocks .

RESOLUTION:
While doing diretory sanity in fsck updating new directory type ondisk in case of filetype error.

Patch ID: VRTSvxfs-8.0.0.2600

* 4114654 (Tracking ID: 4114652)

SYMPTOM:
The VxFS module fails to load on RHEL8.7 minor kernel 4.18.0-425.19.2.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.7 minor kernel.

RESOLUTION:
Updated VXFS to support RHEL 8.7 minor kernel 4.18.0-425.19.2.

Patch ID: VRTSvxfs-8.0.0.2300

* 4108381 (Tracking ID: 4107777)

SYMPTOM:
The VxFS module fails to load on RHEL8.7 minor kernel.

DESCRIPTION:
This issue occurs due to changes in the RHEL8.7 minor kernel.

RESOLUTION:
Modified existing modinst-vxfs script to accommodate the changes in the kernel and load the correct module.

Patch ID: VRTSvxfs-8.0.0.2200

* 4100925 (Tracking ID: 4100926)

SYMPTOM:
VxFS module failed to load on RHEL8.7

DESCRIPTION:
The RHEL8.7 is new release and it has some changes in kernel which caused VxFS module failed to load
on it.

RESOLUTION:
Added code to support VxFS on RHEL8.7.

Patch ID: VRTSvxfs-8.0.0.2100

* 4095889 (Tracking ID: 4095888)

SYMPTOM:
Security vulnerabilities exist in the Sqlite third-party components used by VxFS.

DESCRIPTION:
VxFS uses the Sqlite third-party components in which some security vulnerability exist.

RESOLUTION:
VxFS is updated to use newer version of this third-party components in which the security vulnerabilities have been addressed.

Patch ID: VRTSvxfs-8.0.0.1800

* 4068960 (Tracking ID: 4073203)

SYMPTOM:
Veritas file replication might generate a core while replicating the files to target when rename and unlink operation is performed on a file with FCL( file change log) mode on.

DESCRIPTION:
vxfsreplicate process of Veritas file replicator  might get a segmentation fault with File change mode on when rename and unlink operation are performed on a file.

RESOLUTION:
Addressed the issue to replicate the files, in scenarios involving rename and unlink operation with FCL mode on.

* 4071108 (Tracking ID: 3988752)

SYMPTOM:
Use ldi_strategy() routine instead of bdev_strategy() for IO's in solaris.

DESCRIPTION:
bdev_strategy() is deprecated from solaris code and was causing performance issues when used for IO's. Solaris has recommended to use LDI framework for all IO's.

RESOLUTION:
Code is modified to use ldi framework for all IO's in solaris.

* 4072228 (Tracking ID: 4037035)

SYMPTOM:
VxFS should have the ability to control the number of inactive processing threads.

DESCRIPTION:
VxFS may spawn a large number of worker threads that become inactive over time. As a result, heavy lock contention occurs during the removal of inactive threads on high-end servers.

RESOLUTION:
To avoid the contention, a new tunable, vx_ninact_proc_threads, is added. You can use vx_ninact_proc_threads to adjust the number of inactive processing threads based on your server configuration and workload.

* 4078335 (Tracking ID: 4076412)

SYMPTOM:
Addressing Executive Order (EO) 14028,  initial requirements which is intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents. Executive Order helps in improving the nation's cybersecurity and also enhance any organization's cybersecurity and software supply chain integrity.

DESCRIPTION:
Executive Order helps in improving the nation's cybersecurity and also enhance any organization's cybersecurity and software supply chain integrity, some of the  initial requirements will enable the logging which is compliant to Executive Order. This comprises of command logging,  logging unauthorised access in filesystem and logging WORM events on filesystem. Also include changes to display IP address for Veritas File replication at control plane based on tunable.

RESOLUTION:
The initial requirements of EO are addressed in this release.


As per Executive order(EO) for some of the requirements it should be Tunable based.
For example IP logging where ever applicable (for VFR it should be at control plane(not for every data transfer), and this is also tunable based.
Also for logging some kernel logs, like worm events(plan is to log those to syslog) etc are tunable based.

Introduced new tunable, eo_logging_enable. There is a protocol change because of the introduction of the tunable. 
Though the changes are planned for TOT first and then will go to Update patch on 80all maint for EO release, there is impact of this protocol change for update patch.
We might need to update protocol change with middle protocol version between existing protocol version and new protocol version(introduced because of eo)

For VFR, IP addresses of source and destination are needed to be logged as part of EO.
IP addresses will be included in the log while logging Starting/Resuming a job in VFR.
Log Location: /var/VRTSvxfs/replication/log/mount_point-job_name.log

There are 2 ways to fetch the IP address of the source and target. One is to get the IP addresses stored in the link structure of a session. These IPs are obtained by resolving the source and target hostname. It may contain both IPv4 and IPv6 for a node, and we cannot speculate on which IP actual connection has happened. The second way is to get the socket descriptor from an active connection of the session. This socket descriptor can be used to fetch the source and target IP associated with it. The second method is seems  to get the actual IP addresses used for the connection between source and target. The change contains to  fetch IP addresses from socket descriptor after establishing connections.

More details on EO Logging with respective handling for initial release for VxFS
https://confluence.community.veritas.com/pages/viewpage.action?spaceKey=VES&title=EO+VxFS+Scrum+Page

* 4078520 (Tracking ID: 4058444)

SYMPTOM:
Loop mounts using files on VxFS fail on Linux systems running kernel version 4.1 or higher.

DESCRIPTION:
Starting with the 4.1 version of the Linux kernel, the driver loop.ko uses a new API for read and write requests to the file which was not previously implemented in VxFS. This causes the virtual disk reads during mount to fail while using the -o loop option , causing the mount to fail as well. The same functionality worked in older kernels (such as the version found in RHEL7).

RESOLUTION:
Implemented a new API for all regular files on VxFS, allowing usage of the loop device driver against files on VxFS as well as any other kernel drivers using the same functionality.

* 4079142 (Tracking ID: 4077766)

SYMPTOM:
VxFS kernel module might leak memory during readahead of directory blocks.

DESCRIPTION:
VxFS kernel module might leak memory during readahead of directory blocks due to missing free operation of readahead-related structures.

RESOLUTION:
Code in readahead of directory blocks is modified to free up readahead-related structures.

* 4079173 (Tracking ID: 4070217)

SYMPTOM:
Command fsck might fail with 'cluster reservation failed for volume' message for a disabled cluster-mounted filesystem.

DESCRIPTION:
On a disabled cluster-mounted filesystem, release of cluster reservation might fail during unmount operation resulting in a  failure of command fsck with 'cluster reservation failed for volume' message.

RESOLUTION:
Code is modified to release cluster reservation in unmount operation properly even for cluster-mounted filesystem.

* 4082260 (Tracking ID: 4070814)

SYMPTOM:
Security Vulnerability observed in Zlib a third party component VxFS uses.

DESCRIPTION:
In an internal security scans vulnerabilities in Zlib were found.

RESOLUTION:
Upgrading the third party component Zlib to address these vulnerabilities.

* 4082865 (Tracking ID: 4079622)

SYMPTOM:
Migration uses normal read/write file operation instead of read/write iter functions. vxfs requires read/write iter functions from Linux kernel
5.14.

DESCRIPTION:
Starting with 5.14 version of the Linux kernel, vxfs uses a read/write iter file operation for migration.

RESOLUTION:
Developed a common function for read/write which get called for normal and iter read/write file operation.

* 4083335 (Tracking ID: 4076098)

SYMPTOM:
FS migration from ext4 to vxfs on Linux machines with falcon-sensor enabled, may fail

DESCRIPTION:
Falcon-sensor driver installed on test machines is tapping system calls such as close and is doing some 
additional vfs calls such as read. Due to this vxfs driver received read file - operation call from fsmigbgcp 
process context. Read operation is allowed only on special files from fsmigbgcp process context. Since 
the file in picture was not a special file, the vxfs debug code asserted.

RESOLUTION:
As a fix, we are now allowing the read on non special files from fsmigbgcp process context.

[Note:
 - There were other related issues fixed in this incident. But those are not likely to be hit in customer 
   environment as they are negative test scenarios (like trying to overwrite migration special file - deflist) 
   and may not be relevant to customer.
- I am not covering them in above

* 4085623 (Tracking ID: 4085624)

SYMPTOM:
While running fsck with -o and full -y on corrupted FS, fsck may dump core.

DESCRIPTION:
Fsck builds various in-core maps based on on-disk structural files, one such map is dotdotmap (which stores 
info about parent directory). For regular fset (like 999), the dotdotmap is initialized only for primary ilist
(inode list for regular inodes). It is skipped for attribute ilist (inode list for attribute inodes). This is because
attribute inodes do not have parent directories as is the case for regular inodes.

While attempting to resolve inconsistencies in FS metadata, fsck tries to clean up dotdotmap for attribute ilist. 
In the absence of a check, dotdotmap is re-initialized for attribute ilist causing segmentation fault.

RESOLUTION:
In the codepath where fsck attempts to reinitialize the dotdotmap, a check added to skip reinitialization of dotdotmap
for attribute ilist.

* 4085839 (Tracking ID: 4085838)

SYMPTOM:
Command fsck may generate core due to processing of zero size attribute inode.

DESCRIPTION:
Command fsck is modified to skip processing of zero size attribute inode.

RESOLUTION:
Command fsck fails due to allocation of memory and dereferencing it for zero size attribute inode.

* 4086085 (Tracking ID: 4086084)

SYMPTOM:
VxFS mount operation causes system panic when -o context is used.

DESCRIPTION:
VxFS mount operation supports context option to override existing extended attributes, or to specify a different, default context for file systems that do not support extended attributes. System panic observed when -o context is used.

RESOLUTION:
Required code changes are added to avoid panic.

* 4088341 (Tracking ID: 4065575)

SYMPTOM:
Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space condition

DESCRIPTION:
Write operation might be unresponsive on a local mounted VxFS filesystem in a no-space condition due to a race between two writer threads to take read-write lock the file to do a delayed allocation operation on it.

RESOLUTION:
Code is modified to allow thread which is already holding read-write lock to complete delayed allocation operation, other thread will skip over that file.

Patch ID: VRTSvxfs-8.0.0.1700

* 4081150 (Tracking ID: 4079869)

SYMPTOM:
Security Vulnerability found in VxFS while running security scans.

DESCRIPTION:
In our internal security scans we found some Vulnerabilities in VxFS third party components. The Attackers can exploit these security vulnerability 
to attack on system.

RESOLUTION:
Upgrading the third party components to resolve these vulnerabilities.

* 4083948 (Tracking ID: 4070814)

SYMPTOM:
Security Vulnerability found in VxFS while running security scans.

DESCRIPTION:
In our internal security scans we found some Vulnerabilities in VxFS third party component Zlib.

RESOLUTION:
Upgrading the third party component Zlib to resolve these vulnerabilities.

Patch ID: VRTSvxfs-8.0.0.1200

* 4055808 (Tracking ID: 4062971)

SYMPTOM:
All the operations like ls, create are blocked on file system

DESCRIPTION:
In WORM file system we do not allow directory rename. When partition directory is enabled, new directories are created and files are moved under this leaf directory based on hash. Due to WORM FS this rename operation was blocked and splitting could not complete. Blocking all the operations on file system.

RESOLUTION:
Allow directory renaming in the context of partition directory split and merge.

* 4056684 (Tracking ID: 4056682)

SYMPTOM:
New features information on a filesystem with fsadm(file system administration utility) from a device is not displayed.

DESCRIPTION:
Information about new features like WORM (Write once read many), auditlog is correctly updated with a file system mounted through the fsadm utility, but on the underlying device the new feature information is not displayed.

RESOLUTION:
Updated fsadm utility to display the new feature information correctly.

* 4062606 (Tracking ID: 4062605)

SYMPTOM:
Minimum retention time cannot be set if the maximum retention time is not set.

DESCRIPTION:
The tunable - minimum retention time cannot be set if the tunable - maximum retention time is not set. This was implemented to ensure 
that the minimum time is lower than the maximum time.

RESOLUTION:
Setting of minimum and maximum retention time is independent of each other. Minimum retention time can be set without the maximum retention time being set.

* 4065565 (Tracking ID: 4065669)

SYMPTOM:
Creating non-WORM checkpoints fails when the tunables - minimum retention time and maximum retention time are set.

DESCRIPTION:
Creation of non-WORM checkpoints fails as all WORM-related validations are extended to non-WORM checkpoints also.

RESOLUTION:
WORM-related validations restricted to WORM fsets only, allowing non-WORM checkpoints to be created.

* 4065651 (Tracking ID: 4065666)

SYMPTOM:
All the operations like ls, create are blocked on file system directory where there are WORM enabled files and retention period not expired

DESCRIPTION:
For WORM file system, files whose retention period is not expired can not be renamed. When partition directory is enabled, new directories are created and files are moved under this leaf directory based on hash. Due to WORM FS this rename operation was blocked and splitting could not complete. Blocking all the operations on file system.

RESOLUTION:
Allow directory renaming of files even if retention period is not expired in the context of partition directory split and merge.

Patch ID: VRTSvxfs-8.0.0.1100

* 4061114 (Tracking ID: 4052883)

SYMPTOM:
The VxFS module fails to load on RHEL 8.5.

DESCRIPTION:
This issue occurs due to changes in the RHEL 8.5 kernel.

RESOLUTION:
VxFS module is updated to accommodate the changes in the kernel and load as expected on RHEL 8.5.



INSTALLING THE PATCH
--------------------
Run the Installer script to automatically install the patch:
-----------------------------------------------------------
Please be noted that the installation of this P-Patch will cause downtime.

To install the patch perform the following steps on at least one node in the cluster:
1. Copy the patch infoscale-rhel8_x86_64-Patch-8.0.0.3100.tar.gz to /tmp
2. Untar infoscale-rhel8_x86_64-Patch-8.0.0.3100.tar.gz to /tmp/hf
    # mkdir /tmp/hf
    # cd /tmp/hf
    # gunzip /tmp/infoscale-rhel8_x86_64-Patch-8.0.0.3100.tar.gz
    # tar xf /tmp/infoscale-rhel8_x86_64-Patch-8.0.0.3100.tar
3. Install the hotfix(Please be noted that the installation of this P-Patch will cause downtime.)
    # pwd /tmp/hf
    # ./installVRTSinfoscale800P3100 [<host1> <host2>...]

You can also install this patch together with 8.0 base release using Install Bundles
1. Download this patch and extract it to a directory
2. Change to the Veritas InfoScale 8.0 directory and invoke the installer script
   with -patch_path option where -patch_path should point to the patch directory
    # ./installer -patch_path [<path to this patch>] [<host1> <host2>...]

Install the patch manually:
--------------------------
Manual installation is not recommended.


REMOVING THE PATCH
------------------
Manual uninstallation is not recommended.


SPECIAL INSTRUCTIONS
--------------------
Fixed Vulnerabilities: 
CVE-2022-37434 (BDSA-2022-2183),CVE-2023-38545 (BDSA-2023-2697),CVE-2023-28319 (BDSA-2023-1234),CVE-
2023-38039 (BDSA-2023-2428),CVE-2023-0464 (BDSA-2023-0610),CVE-2023-28484 (BDSA-2023-0813),CVE-2023-
29469 (BDSA-2023-0811),CVE-2023-2650 (BDSA-2023-1337),CVE-2023-28321 (BDSA-2023-1237),CVE-2023-28320 
(BDSA-2023-1233),BDSA-2022-0284,CVE-2024-0727 (BDSA-2024-0202),CVE-2023-5678 (BDSA-2023-3046),CVE-
2023-0466,CVE-2023-3817 (BDSA-2023-1972),CVE-2023-0465,BDSA-2023-1866,CVE-2023-38546 (BDSA-2023-
2699),CVE-2023-28322 (BDSA-2023-1238).


OTHERS
------
NONE