* * * READ ME * * *
                       * * * InfoScale 8.0 * * *
                         * * * Patch 2800 * * *
                         Patch Date: 2023-06-28


This document provides the following information:

   * PATCH NAME
   * OPERATING SYSTEMS SUPPORTED BY THE PATCH
   * INSTALLATION PRE-REQUISITES
   * PACKAGES AFFECTED BY THE PATCH
   * BASE PRODUCT VERSIONS FOR THE PATCH
   * SUMMARY OF INCIDENTS FIXED BY THE PATCH
   * DETAILS OF INCIDENTS FIXED BY THE PATCH
   * INSTALLING THE PATCH
   * REMOVING THE PATCH
   

PATCH NAME
----------
InfoScale 8.0 Patch 2800
(RHEL9.2 Support on IS 8.0 U2)

OPERATING SYSTEMS SUPPORTED BY THE PATCH
----------------------------------------
RHEL9 x86-64


PACKAGES AFFECTED BY THE PATCH
------------------------------
VRTSamf
VRTSaslapm
VRTScavf
VRTSdbac
VRTSfsadv
VRTSgab
VRTSglm
VRTSgms
VRTSllt
VRTSodm
VRTSperl
VRTSpython
VRTSrest
VRTSsfmh
VRTSspt
VRTSvcs
VRTSvcsag
VRTSvcsea
VRTSveki
VRTSvxfen
VRTSvxfs
VRTSvxvm


BASE PRODUCT VERSIONS FOR THE PATCH
-----------------------------------
   * InfoScale Availability 8.0
   * InfoScale Enterprise 8.0
   * InfoScale Foundation 8.0
   * InfoScale Storage 8.0


SUMMARY OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
Patch ID: VRTSvxfs-8.0.0.2800
* 4092518 (4096267) Veritas File Replication jobs might failed when there are large number of jobs run in parallel.
* 4097466 (4114176) After failover, job sync fails with error "Device or resource busy".
* 4107367 (4108955) VFR job hangs on source if thread creation fails on target.
* 4112417 (4094326) mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"
* 4118555 (4118556) VxFS support for RHEL9.2.
* 4118795 (4100021) Running setfacl followed by getfacl resulting in "No such device or address" error.
* 4119023 (4116329) While checking FS sanity with the help of "fsck -o full -n" command, we tried to correct the FS flag value (WORM/Softworm), but failed because -n (read-only) option was given.
Patch ID: VRTSvxfs-8.0.0.2500
* 4112919 (4110764) Security Vulnerability observed in Zlib a third party component used by VxFS .
Patch ID: VRTSvxfs-8.0.0.2100
* 4095889 (4095888) Security vulnerabilities exist in the Sqlite third-party components used by VxFS.
Patch ID: VRTSrest-2.0.0.1300
* 4088973 (4089451) When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error .
* 4089033 (4089453) Some VCS REST APIs were crashing the Gunicorn worker.
* 4089041 (4089449) GET resources API on empty service group was throwing an error.
* 4089046 (4089448) Logging in REST API is not EO-compliant.
Patch ID: VRTSspt-8.0.0.1400
* 4085610 (4090433) iostat and vmstat command option changes in FirstLook
* 4088066 (4090446) vxstat log collection improvements in FirstLook
* 4091983 (4092090) FirstLook should have OS flavor information stored in its log directory.
* 4096274 (4095687) While restoring version-8 metasave on sparse volume, the restore operation is not happening correctly.
Patch ID: -5.34.0.4
* 4072234 (4069607) Security vulnerability detected on VRTSperl 5.34.0.0 released with Infoscale 8.0.
* 4075150 (4075149) Security vulnerabilities detected in OpenSSL packaged with VRTSperl/VRTSpython released with Infoscale 8.0.
Patch ID: VRTSsfmh-vom-HF0800411
* 4113012 (4113011) VIOM VRTSsfmh package on Linux to fix dclid/vxlist issue with InfoScale VRTSvxvm 8.0.0.2200
Patch ID: -3.9.2.24
* 4114375 (4113851) For VRTSpython need to fix some open CVE's
Patch ID: VRTSfsadv-8.0.0.2100
* 4092150 (4088024) Security vulnerabilities exist in the OpenSSL third-party components used by VxFS.
Patch ID: VRTSfsadv-8.0.0.1800
* 4078335 (4076412) Addressing Executive Order (EO) 14028,  initial requirements intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents.
Patch ID: VRTSvcsag-8.0.0.2500
* 4118318 (4113151) VMwareDisksAgent reports resource online before VMware disk to be online is present into vxvm/dmp database.
* 4118448 (4075950) IPv6 neighbor flush logic needs to be added to IP/MultiNIC agents
* 4118455 (4118454) Process agent fails to come online when root user shell is set to /sbin/nologin.
* 4118767 (4094539) Agent resource monitor not parsing process name correctly.
Patch ID: VRTSdbac-8.0.0.2400
* 4114349 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
Patch ID: VRTSvxfen-8.0.0.2500
* 4114265 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
* 4117657 (4108561) Reading vxfen reservation not working
Patch ID: VRTSvcsea-8.0.0.2500
* 4118769 (4073508) Oracle virtual fire-drill is failing.
Patch ID: VRTSveki-8.0.0.2500
* 4114264 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
* 4118568 (4110457) Veki packaging were failing due to dependency
Patch ID: VRTSvcs-8.0.0.2500
* 4111623 (4100720) GCO fails to configure for the latest RHEL/SLES platforms.
Patch ID: VRTSvcs-8.0.0.2100
* 4103077 (4103073) Upgrading Netsnmp component to fix security vulnerabilities .
Patch ID: VRTSamf-8.0.0.2500
* 4122455 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
Patch ID: VRTSgab-8.0.0.2500
* 4122409 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
Patch ID: VRTSllt-8.0.0.2500
* 4112345 (4087662) During memory fragmentation LLT module may fail to allocate large memory leading to the node eviction or a node not being able to join.
* 4114263 (4122405) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).
Patch ID: VRTSvxvm-8.0.0.2500
* 4067237 (4058894) Messages in /var/log/messages regarding "ignore_device".
* 4109554 (4105953) system panic due to VVR accessed a NULL pointer.
* 4111442 (4066785) create new option usereplicatedev=only to import the replicated LUN only.
* 4112549 (4112701) Nodes stuck in reconfig hang and vxconfigd coredump after rebooting all nodes with a delay of 5min in between them.
* 4113310 (4114601) Panic: in dmp_process_errbp() for disk pull scenario.
* 4113357 (4112433) Security vulnerabilities exists in third party components [openssl, curl and libxml].
* 4114963 (4114962) [NBFS-3.1][DL]:MASTER and CAT_FS got corrupted while performing multiple NVMEs failure
* 4115251 (4115195) [NBFS-3.1][DL]:MEDIA_FS got corrupted after panic loop test
* 4115252 (4115193) Data corruption observed after the node fault and cluster restart in DR environment
* 4115381 (4091783) build script and mb.sh changes in unixvm for integration of storageapi
* 4116548 (4111254) vradmind dumps core while associating a rlink to rvg because of NULL pointer reference.
* 4116551 (4108913) Vradmind dumps core because of memory corruption.
* 4116557 (4085404) Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.
* 4116559 (4091076) SRL gets into pass-thru mode because of head error.
* 4116562 (4114257) Observed IO hung and high system load average after rebooted master and one slave node rejoins cluster.
* 4116565 (4034741) The current fix from limits IO load on secondary causing deadlock situtaion
* 4116567 (4072862) Stop cluster hang because of RVGLogowner and CVMClus resources fail to offline.
* 4117110 (4113841) VVR panic in replication connection handshake request from network scan tool.
* 4118108 (4114867) systemd-udevd[2224]: invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 103 ('D')
* 4118111 (4065490) VxVM udev rules consumes more CPU and appears in "top" output when system has thousands of storage devices attached.
* 4118479 (4118478) VxVM Support on RHEL9.2
* 4118733 (4106689) Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95
* 4118845 (4116024) machine panic due to access illegal address.
* 4119087 (4067191) IS8.0_SUSE15_CVR: After rebooted slave node master node got panic
* 4119257 (4090772) vxconfigd/vx commands hung if fdisk opened secondary volume and secondary logowner panic'd
* 4119276 (4090943) VVR Primary RLink cannot connect as secondary reports SRL log is full.
* 4119438 (4117985) EC volume corruption due to lockless access of FPU
* 4120350 (4120878) After enabling the dmp_native_support, system failed to boot.
* 4121241 (4114927) Failed to mount /boot on dmp device after enabling dmp_native_support.
* 4121714 (4081740) vxdg flush command slow due to too many luns needlessly access /proc/partitions.
Patch ID: VRTSvxvm-8.0.0.2100
* 4102502 (4102501) A security vulnerability exists in the third-party component libcurl
Patch ID: VRTSaslapm 8.0.0.2500
* 4101808 (4101807) VxVM with DMP is failing to detect Hitachi ShadowImage (SI) svol devices.
* 4116688 (4085145) EBSvol agent error in attach disk : RHEL 7.9 + Infoscale 8.0 on AWS instance type c6i.large with NVME devices.
* 4117385 (4117350) Import operation on disk group created on Hitachi ShadowImage (SI) disks is failing .
* 4122584 (4122583) ASLAPM rpm Support on RHEL9.2
Patch ID: VRTScavf-8.0.0.2800
* 4118779 (4074274) DR test and failover activity might not succeed for hardware-replicated disk groups.
Patch ID: VRTSgms-8.0.0.2800
* 4057427 (4057176) Rebooting the system results into emergency mode due to corruption of module dependency files.
* 4118302 (4118303) GMS support for RHEL9.2.
Patch ID: VRTSglm-8.0.0.2800
* 4118296 (4118297) GLM support for RHEL9.2.
Patch ID: VRTSodm-8.0.0.2800
* 4057432 (4056673) Rebooting the system results into emergency mode due to corruption of module dependency files. Incorrect vxgms dependency in odm service file.
* 4118467 (4118466) ODM support for RHEL9.2.
Patch ID: VRTSodm-8.0.0.2500
* 4114322 (4114321) VRTSodm driver will not load with VRTSvxfs 8.0.0.2500 patch.


DETAILS OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
This patch fixes the following incidents:

Patch ID: VRTSvxfs-8.0.0.2800

* 4092518 (Tracking ID: 4096267)

SYMPTOM:
Veritas File Replication jobs might failed when there are large number of jobs run in parallel.

DESCRIPTION:
File Replication Jobs might fail, with Large number of jobs configured and running in parallel with Veritas File Replication.
With large number of jobs there is a chance of referring a job which is already freed, due to which there is a core generated with replication service and
job might failed.

RESOLUTION:
updated code to handle the code to take a hold while checking invalid job configuration.

* 4097466 (Tracking ID: 4114176)

SYMPTOM:
After failover, job sync fails with error "Device or resource busy".

DESCRIPTION:
If job is in failed state on target because of job failure from source side, repld was not updating its state when it was restarted in recovery mode. Because of which job state was remaining in running state even after successful replication on target. With this state on target, if job is promoted, then replication process was not creating new ckpt for first sync after failover which was resulting in corrupting state file on new source. Because of this incorrect/corrupt state file, job sync from new source was failing with error "Device or resource busy".

RESOLUTION:
Code is modified to correct the state on target when job was started in recovery mode.

* 4107367 (Tracking ID: 4108955)

SYMPTOM:
VFR job hangs on source if thread creation fails on target.

DESCRIPTION:
On Target, if thread creation for pass completion fails because of high memory usage, repld demon doesn't send that failure reply to source. This can lead to vxfsreplicate process to remains in waiting state indefinitely for reply for pass completion from target. This will lead to job hang on source and will need manual intervention to kill the job.

RESOLUTION:
Code is modified to  retry thread creation on target and if it fails after 5 retries, target will reply to source with appropriate error.

* 4112417 (Tracking ID: 4094326)

SYMPTOM:
mdb invocation displays message "failed to add vx_sl_node_level walker: walk name already in use"

DESCRIPTION:
In vx_sl_kmcache_init(), kmcache is initialized for each level (in this case it is 8) separately. For passing the cache name as an argument to kmem_cache_create(), we have used a macro.

#define VX_SL_KMCACHE_NAME(level)       "vx_sl_node_"#level
#define VX_SL_KMCACHE_CREATE(level)                                     \
                kmem_cache_create(VX_SL_KMCACHE_NAME(level),            \
                                  VX_KMEM_SIZE(VX_SL_KMCACHE_SIZE(level)),\
                                  0, NULL, NULL, NULL, NULL, NULL, 0);


While using this macro, we have passed "level" as an argument and that has been expanded as "vx_sl_node_level" for all the 8 levels in `for` loop. This is causing the cache allocation for all the 8 levels with same name.

RESOLUTION:
Passing separate variable value (as level value) to VX_SL_KMCACHE_NAME as it is done in  vx_wb_sl_kmcache_init().

* 4118555 (Tracking ID: 4118556)

SYMPTOM:
VxFS module failed to load on RHEL9.2 kernel.

DESCRIPTION:
This issue occurs due to changes in RHEL9.2 kernel.

RESOLUTION:
VxFS module is updated to accommodate the changes in the kernel and load as expected on RHEL9.2 kernel.

* 4118795 (Tracking ID: 4100021)

SYMPTOM:
Running setfacl followed by getfacl resulting in "No such device or address" error.

DESCRIPTION:
When running setfacl command on some of the directories which have the VX_ATTR_INDIRECT type of acl attribute, it is not removing the existing acl attribute and adding a new one, which should  not happen ideally. This is resulting in the failure of getfacl with following "No such device or address" error.

RESOLUTION:
we have done the code chages to removal of VX_ATTR_INDIRECT type acl in setfacl code.

* 4119023 (Tracking ID: 4116329)

SYMPTOM:
fsck -o full -n command will fail with error:
"ERROR: V-3-28446:  bc_write failure devid = 0, bno = 8, len = 1024"

DESCRIPTION:
Previously, to correct the file system WORM/SoftWORM, we didn't check  if user wanted to correct the pflags or just wanted to validate if value is flag is missing or not. Also fsck was not capable to handle SOFTWORM flag.

RESOLUTION:
Code added to not try to fix the the problem if user ran fsck with -n option. Also SOFTWORM scenario is added.

Patch ID: VRTSvxfs-8.0.0.2500

* 4112919 (Tracking ID: 4110764)

SYMPTOM:
Security Vulnerability observed in Zlib a third party component VxFS uses.

DESCRIPTION:
In an internal security scans vulnerabilities in Zlib were found.

RESOLUTION:
Upgrading the third party component Zlib to address these vulnerabilities.

Patch ID: VRTSvxfs-8.0.0.2100

* 4095889 (Tracking ID: 4095888)

SYMPTOM:
Security vulnerabilities exist in the Sqlite third-party components used by VxFS.

DESCRIPTION:
VxFS uses the Sqlite third-party components in which some security vulnerability exist.

RESOLUTION:
VxFS is updated to use newer version of this third-party components in which the security vulnerabilities have been addressed.

Patch ID: VRTSrest-2.0.0.1300

* 4088973 (Tracking ID: 4089451)

SYMPTOM:
When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error.

DESCRIPTION:
When a read-only filesystem was created on a Volume, GET on mountpoint's details was throwing error as the command which was being used was not working for read-only filesystem.

RESOLUTION:
Used the appropriate command to get details of mountpoint.

* 4089033 (Tracking ID: 4089453)

SYMPTOM:
Some VCS REST APIs were crashing the Gunicorn worker.

DESCRIPTION:
Calling some VCS-related APIs were crashing the gunicorn worker handling the request. A new worker was automatically being spawned.

RESOLUTION:
Fixed the related foreign function call interface in the source code.

* 4089041 (Tracking ID: 4089449)

SYMPTOM:
GET resources API on empty service group was throwing an error.

DESCRIPTION:
When GET resources API was called on an empty service group it was giving an error, and the scenario is not handled.

RESOLUTION:
Scenario added to the code to resolve the issue.

* 4089046 (Tracking ID: 4089448)

SYMPTOM:
Logging in REST API was not in EO-compliant format.

DESCRIPTION:
Timestamp format is not EO-compliant and some attributes were missing for EO compliance.

RESOLUTION:
Changed timestamp, added new attributes like nodename, response time, source and destination IP addresses, username to REST server logs.

Patch ID: VRTSspt-8.0.0.1400

* 4085610 (Tracking ID: 4090433)

SYMPTOM:
iostat and vmstat command option changes in FirstLook

DESCRIPTION:
Time stamp for each stat collected is missing for vmstat on Linux and iostat and vmstat commands on Solaris and AIX.

RESOLUTION:
Following are the new flags introduced for each platform separately:
    1) Linux:
       vmstat:
           -t: Append timestamp to each line
    2) Solaris:
       iostat:
           -Td: Display a time stamp. Specify d for standard date format.
       vmstat:
           -Td: Display a time stamp. Specify d for standard date format.
    3) AIX:
       iostat:
           -s: Specifies the system throughput report
           -T: Displays the time stamp.
           -D: Displays the extended tape/drive utilization report.
           -l: Displays the output in long listing mode.
       vmstat:
           -t: Prints the time-stamp next to each line of output of vmstat.

* 4088066 (Tracking ID: 4090446)

SYMPTOM:
vxstat log collection improvements in FirstLook

DESCRIPTION:
Currently FirstLook collects only volume level statistics during execution through vxstat command.

RESOLUTION:
Added options in vxstat command to collect stats for volume, plex, disk and subdisk objects.

* 4091983 (Tracking ID: 4092090)

SYMPTOM:
FirstLook should have OS flavor information stored in its log directory.

DESCRIPTION:
FirstLook collects "uname -a" as part of its log collection but this does not give OS flavour information on Linux platform like Rhel7/8.

RESOLUTION:
Code changes have been done in FirstLook code base to include OS_info file inside system folder to contain OS name and its flavor.

* 4096274 (Tracking ID: 4095687)

SYMPTOM:
While restoring version-8 metasave on sparse volume, the restore operation is not happening correctly.

DESCRIPTION:
While restoring version-8 metasave on sparse volume, the restore operation is not happening correctly due to concurrent I/Os triggered on the sparse volume. While replaying the same metasave on normal volume/vset or on a sparse file (for non-MVFS), the replay is working as expected.

RESOLUTION:
Disabled the concurrent writes during metasave restore operation by default. One hidden option ("-p") has been added to enable the multi-threading during metasave restore if needed (mostly while replaying on normal volumes or volume set).

Patch ID: -5.34.0.4

* 4072234 (Tracking ID: 4069607)

SYMPTOM:
Security vulnerability detected on VRTSperl 5.34.0.0 released with Infoscale 8.0.

DESCRIPTION:
Security vulnerability detected in the Net::Netmask module.

RESOLUTION:
Upgraded Net::Netmask module and re-created VRTSperl 5.34.0.1 version to fix the vulnerability .

* 4075150 (Tracking ID: 4075149)

SYMPTOM:
Security vulnerabilities detected in OpenSSL packaged VRTSperl/VRTSpython released with Infoscale 8.0.

DESCRIPTION:
Security vulnerabilities detected in the OpenSSL.

RESOLUTION:
Upgraded OpenSSL version and re-created VRTSperl/VRTSpython version to fix the vulnerability .

Patch ID: VRTSsfmh-vom-HF0800411

* 4113012 (Tracking ID: 4113011)

SYMPTOM:
vxlist output on InfoScale server shows volume name as "-" and status Unknown.

DESCRIPTION:
vxlist output on InfoScale server shows volume name as "-" and status Unknown.

RESOLUTION:
New vxdclid plugin for VxVM has been created.

Patch ID: -3.9.2.24

* 4114375 (Tracking ID: 4113851)

SYMPTOM:
Open CVE's detected for the python programming language and other python modules being used in VRTSpython

DESCRIPTION:
Some open CVE's are exploitable in VRTSpython for IS 8.0

RESOLUTION:
VRTSpython is patched with all the open CVE's which are impacting IS 8.0.

Patch ID: VRTSfsadv-8.0.0.2100

* 4092150 (Tracking ID: 4088024)

SYMPTOM:
Security vulnerabilities exist in the OpenSSL third-party components used by VxFS.

DESCRIPTION:
VxFS uses the OpenSSL third-party components in which some security vulnerability exist.

RESOLUTION:
VxFS is updated to use newer version (1.1.1q) of this third-party components in which the security vulnerabilities have been addressed. To accommodate the changes vxfs_solutions is added with libboost_system entries in Makefile [dedup/pdde/sdk/common/Makefile].

Patch ID: VRTSfsadv-8.0.0.1800

* 4078335 (Tracking ID: 4076412)

SYMPTOM:
Addressing Executive Order (EO) 14028,  initial requirements intended to improve the Federal Governments investigative and remediation capabilities related to cybersecurity incidents. Executive Order helps in improving the nation's cybersecurity and also enhances any organization's cybersecurity and software supply chain integrity.

DESCRIPTION:
Executive Order helps in improving the nation's cybersecurity and also enhances any organization's cybersecurity and software supply chain integrity, some of the  initial requirements will enable the logging which is compliant to Executive Order. This comprises command logging,  logging unauthorised access in filesystem and logging WORM events on filesystem. Also include changes to display IP address for Veritas File replication at control plane based on tunable.

RESOLUTION:
The initial requirements of EO are addressed in this release.


As per Executive order(EO) for some of the requirements it should be Tunable based.
For example IP logging where ever applicable (for VFR it should be at control plane(not for every data transfer), and this is also tunable based.
Also for logging some kernel logs, like worm events(plan is to log those to syslog) etc are tunable based.

Introduced new tunable, eo_logging_enable. There is a protocol change because of the introduction of the tunable.
Though the changes are planned for TOT first and then will go to Update patch on 80all maint for EO release, there is impact of this protocol change for update patch.
We might need to update protocol change with middle protocol version between existing protocol version and new protocol version(introduced because of eo)

For VFR, IP addresses of source and destination are needed to be logged as part of EO.
IP addresses will be included in the log while logging Starting/Resuming a job in VFR.
Log Location: /var/VRTSvxfs/replication/log/mount_point-job_name.log

There are 2 ways to fetch the IP address of the source and target. One is to get the IP addresses stored in the link structure of a session. These IPs are obtained by resolving the source and target hostname. It may contain both IPv4 and IPv6 for a node, and we cannot speculate on which IP actual connection has happened. The second way is to get the socket descriptor from an active connection of the session. This socket descriptor can be used to fetch the source and target IP associated with it. The second method is seems  to get the actual IP addresses used for the connection between source and target. The change contains to  fetch IP addresses from socket descriptor after establishing connections.

More details on EO Logging with respective handling for initial release for VxFS
https://confluence.community.veritas.com/pages/viewpage.action?spaceKey=VES&title=EO+VxFS+Scrum+Page

Patch ID: VRTSvcsag-8.0.0.2500

* 4118318 (Tracking ID: 4113151)

SYMPTOM:
Dependent DiskGroupAgent fails to get its resource online due to disk group import failure.

DESCRIPTION:
VMwareDisksAgent reports its resource online just after VMware disk is attached to virutal machine, if dependent DiskGroup resource starts to online at the moment it must fail because VMware disk is not yet present into vxdmp database due to VxVM transaction latency. Customer used to add retry times to work around this problem but cannot apply the same to every environment.

RESOLUTION:
Added a finite period of wait for VMware disk is present into vxdmp database before online is complete.

* 4118448 (Tracking ID: 4075950)

SYMPTOM:
When IPv6 VIP switches from node1 to node2 in a cluster,
a longer time is taken to update its neighboring information and traffic to reach node2 which is on the reassigned address.

DESCRIPTION:
After the Service group switches from node1 to node2, the IPv6 VIP is not reachable from the network switch. The mac address changes after the node switch, but the network is not updated. Similar to IPv4 VIP by gracious ARP, in case of IPV6 VIP switch from node1 to node2; the network must be updated for the mac address change.

RESOLUTION:
The network devices which communicate with the VIP are not able to establish a connection with the VIP. To connect with the VIP, the VIP is pinged from the switch or from the cluster nodes 'ip -6 neighbor flush all' command is run. Neighbour flush logic is added to IP/MultiNIC agents so that the changed mac id during floating VIP switchover is updated in the network.

* 4118455 (Tracking ID: 4118454)

SYMPTOM:
when root user login shell is set to /sbin/nologin in /etc/passwd file, Process agent resource fails to come online.

DESCRIPTION:
From the engine_A.log,  the below errors were logged:
2023/05/31 11:34:52 VCS NOTICE V-16-10031-20704 Process:Process:imf_getnotification:Received notification for vxamf-group sendmail
2023/05/31 11:35:38 VCS ERROR V-16-10031-9502 Process:sendmail:online:Could not online the resource, make sure user-name is correct.
2023/05/31 11:35:39 VCS INFO V-16-2-13716 Thread(140147853162240) Resource(sendmail): Output of the completed operation (online)
==============================================
This account is currently not available.
==============================================

RESOLUTION:
The Process agent is enhanced to support nologin shell for root user. If user shell is set to /sbin/nologin, the agent starts a process using /bin/bash shell.

* 4118767 (Tracking ID: 4094539)

SYMPTOM:
The MonitorProcesses argument in the resource ArgListValues being passed to the agent (bundled ApplicationAgent) is incorrectly removing an extra needed space from the following process, as found via the recommended CLI process test.

DESCRIPTION:
In the ArgListValues under MonitorProcesses with the extra space it even shows up when displaying the resource.

RESOLUTION:
For the monitored process (not program) only remove leading and trailing spaces. Do not remove extra spaces between words.

Patch ID: VRTSdbac-8.0.0.2400

* 4114349 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

Patch ID: VRTSvxfen-8.0.0.2500

* 4114265 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

* 4117657 (Tracking ID: 4108561)

SYMPTOM:
Vxfen print keys  internal utility was not working because of overrunning of array internally

DESCRIPTION:
Vxfen print keys  internal utility will not work if the number of keys exceed 8 will then return garbage value
Overrunning array keylist[i].key of 8 bytes at byte offset 8 using index y (which evaluates to 8)

RESOLUTION:
Restricted the internal loop to VXFEN_KEYLEN. Reading reservation working fine now.

Patch ID: VRTSvcsea-8.0.0.2500

* 4118769 (Tracking ID: 4073508)

SYMPTOM:
Oracle virtual fire-drill is failing due to Oracle password file location changes from Oracle version 21c.

DESCRIPTION:
Oracle password file has been moved to $ORACLE_BASE/dbs from Oracle version 21c.

RESOLUTION:
Environment variables are used for pointing the updated path for the password file.

It is mandatory from Oracle 21c and later versions for a client to configure .env file path in EnvFile attribute. This file must have ORACLE_BASE path added to 
work Oracle virtual fire-drill feature. 

Sample EnvFile content with ORACLE_BASE path for Oracle 21c [root@inaqalnx013 Oracle]# cat /opt/VRTSagents/ha/bin/Oracle/envfile 
ORACLE_BASE="/u02/app/oracle/product/21.0.0/dbhome_1/"; export ORACLE_BASE; 

Sample attribute value EnvFile = "/opt/VRTSagents/ha/bin/Oracle/envfile"

Patch ID: VRTSveki-8.0.0.2500

* 4114264 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

* 4118568 (Tracking ID: 4110457)

SYMPTOM:
Veki packaging failure due to missing of storageapi specific files

DESCRIPTION:
While creating the build area for different components like GLM, GMS, ORAODM, unixvm, VxFS veki build area creation were failing because of storageapi changes 
were not taken care in the Veki mk-symlink and build scripts.

RESOLUTION:
Added support for creation of storageapi build area, storageapi packaging changes via veki, and storageapi build via veki from Veki makefiles.
This is helping to package the storageapi along with veki and resolving all interdependencies

Patch ID: VRTSvcs-8.0.0.2500

* 4111623 (Tracking ID: 4100720)

SYMPTOM:
GCO fails to configure for the latest RHEL/SLES platform due to an incorrect CIDR value.

DESCRIPTION:
GCO failed to configure as altname defined for latest RHEL/SLES kernel sends an incorrect CIDR value.

RESOLUTION:
Updated code to pick correct CIDR value in GCO config if altname is defined.

Patch ID: VRTSvcs-8.0.0.2100

* 4103077 (Tracking ID: 4103073)

SYMPTOM:
Security vulnerabilities present in existing version of Netsnmp.

DESCRIPTION:
Upgrading Netsnmp component to fix security vulnerabilities

RESOLUTION:
Upgrading Netsnmp component to fix security vulnerabilities for security patch IS 8.0U1_SP4.

Patch ID: VRTSamf-8.0.0.2500

* 4122455 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

Patch ID: VRTSgab-8.0.0.2500

* 4122409 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

Patch ID: VRTSllt-8.0.0.2500

* 4112345 (Tracking ID: 4087662)

SYMPTOM:
During memory fragmentation LLT module may fail to allocate large memory leading to the node eviction or a node not being able to join.

DESCRIPTION:
When system memory is heavily fragmented, LLT module fails to allocate memory in the form of Linux socket buffers (SKB) from the OS. Due to this a 
cluster node may not be able to join the cluster or a node may get evicted from the cluster.

RESOLUTION:
This HF updates LLT module so that memory is allocated from private memory pools maintained inside LLT and if pools are exhausted LLT module tries 
to allocate memory through vmalloc.

* 4114263 (Tracking ID: 4122405)

SYMPTOM:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux 9 Update 2(RHEL9.2).

DESCRIPTION:
Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL9 Update 0.

RESOLUTION:
Veritas Infoscale Availability support for Red Hat Enterprise Linux 9 Update 2(RHEL9.2) is now introduced.

Patch ID: VRTSvxvm-8.0.0.2500

* 4067237 (Tracking ID: 4058894)

SYMPTOM:
After package installation and reboot , messages regarding udev rules for ignore_device are observed in /var/log/messages .
systemd-udevd[774]: /etc/udev/rules.d/40-VxVM.rules:25 Invalid value for OPTIONS key, ignoring: 'ignore_device'

DESCRIPTION:
From SLES15 Sp3 onwards , ignore_device is deprecated from udev rules and is not available for use anymore . Hence these messages are observed in system logs .

RESOLUTION:
Required changes have been done to handle this defect.

* 4109554 (Tracking ID: 4105953)

SYMPTOM:
System panic with below stack in CVR environment.

 #9 [] page_fault at 
    [exception RIP: vol_ru_check_update_done+183]
#10 [] vol_rv_write2_done at [vxio]
#11 [] voliod_iohandle at [vxio]
#12 [] voliod_loop at [vxio]
#13 [] kthread at

DESCRIPTION:
In CVR environment, when IO is issued in writeack sync mode we ack to application when datavolwrite is done on either log client or logowner depending on 
where IO is issued on. it could happen that VVR freed the metadata I/O update after SRL write is done incase of writeack sync mode, but later after freeing the update, its accessed again and hence we end up in hitting NULL ptr deference.

RESOLUTION:
Code changes have been made to avoid the accessing NULL pointer.

* 4111442 (Tracking ID: 4066785)

SYMPTOM:
When the replicated disks are in SPLIT mode, importing its disk group failed with "Device is a hardware mirror".

DESCRIPTION:
When the replicated disks are in SPLIT mode, which are readable and writable, importing its disk group failed with "Device is a hardware mirror". Third party doesn't expose disk attribute to show when it's in SPLIT mode. With this new enhancement, the replicated disk group can be imported with option `-o usereplicatedev=only`.

RESOLUTION:
The code is enhanced to import the replicated disk group with option `-o usereplicatedev=only`.

* 4112549 (Tracking ID: 4112701)

SYMPTOM:
Observed reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min in between them due to Vxconfigd core dumped on master node

DESCRIPTION:
1. reconfig hang on 8 nodes ISO vm setup after rebooting all nodes with a delay of 5min.
             2. This is because fork failed during command shipping which caused vxconfigd core dump on master. So all reconfigurations after that failed to 
                 process.

RESOLUTION:
Reboot master node where vold is core dumped.

* 4113310 (Tracking ID: 4114601)

SYMPTOM:
System gets panicked and rebooted

DESCRIPTION:
RCA:
Start the IO on volume device and pull out it's disk from the machine and hit below panic on RHEL8.

 dmp_process_errbp
 dmp_process_errbuf.cold.2+0x328/0x429 [vxdmp]
 dmpioctl+0x35/0x60 [vxdmp]
 dmp_flush_errbuf+0x97/0xc0 [vxio]
 voldmp_errbuf_sio_start+0x4a/0xc0 [vxio]
 voliod_iohandle+0x43/0x390 [vxio]
 voliod_loop+0xc2/0x330 [vxio]
 ? voliod_iohandle+0x390/0x390 [vxio]
 kthread+0x10a/0x120
 ? set_kthread_struct+0x50/0x50

As disk pulled out from the machine VxIO hit a IO error and it routed that IO to dmp layer via kernel-kernel IOCTL for error analysis.
following is the code path for IO routing,

voldmp_errbuf_sio_start()-->dmp_flush_errbuf()--->dmpioctl()--->dmp_process_errbuf()

dmp_process_errbuf() retrieves device number of the underlying path (os-device).
and it tries to get bdev (i.e. block_device) pointer from path-device number.
As path/os-device is removed by disk pull, linux returns fake bdev for the path-device number.
For this fake bdev there is no gendisk associated with it (bdev->bd_disk is NULL).

We are setting this NULL bdev->bd_disk to the IO buffer routed from vxio.
which leads a panic on dmp_process_errbp.

RESOLUTION:
If bdev->bd_disk found NULL then set DMP_CONN_FAILURE error on the IO buffer and return DKE_ENXIO to vxio driver

* 4113357 (Tracking ID: 4112433)

SYMPTOM:
Vulnerabilities have been reported in third party components, [openssl, curl and libxml] that are used by VxVM.

DESCRIPTION:
Third party components [openssl, curl and libxml] in their current versions,  used by VxVM have been reported with security vulnerabilities which needs

RESOLUTION:
[openssl, curl and libxml] have been upgraded to newer versions in which the reported security vulnerabilities have been addressed.

* 4114963 (Tracking ID: 4114962)

SYMPTOM:
File system data corruption with mirrored volumes in Flexible Storage Sharing (FSS) environments during beyond fault storage failure situations.

DESCRIPTION:
In FSS environments, data change object (DCO) provides functionality to track changes on detached mirrors using bitmaps. This bitmap is later used for re-sync of detached mirrors data (change delta).
When DCO volume and data volume share the same set of devices, DCO volumes last mirror failure means IOs on data volume is going to fail. In such cases instead of invaliding DCO volumes, proactively IO is failed.
This helps in protecting DCO when entire storage comes back and optimal recovery of mirrors can be performed.
When disk for one of the mirror of DCO object become available, the bug in DCO update incorrectly updates metadata of DCO which lead to ignoring valid DCO maps during actual volume recovery and hence newly recovered mirrors of volume missed blocks of valid application data. This lead to corruption when read IO were serviced from the newly recovered mirrors.

RESOLUTION:
The login of FMR map updating transaction of enabling disks is fixed to resolve the bug. This ensures all valid bitmaps are considered for recovery of mirrors and avoid data loss.

* 4115251 (Tracking ID: 4115195)

SYMPTOM:
Data corruption on file-systems with  erasure coded volumes

DESCRIPTION:
In Erasure coded (EC) volume are used in Flexible shared storage (FSS) environments, data change object (DCO) is used to tracking changes in volume with faulted columns. The DCO provides a bitmap of all changed regions during rebuild of the faulted columns. When EC volume starts with few faulted columns, the log-replay IO could not be performed on those columns. Those additional writes are expected to be tracked in DCO bitmap. Due to bug those IOs are not getting tracked in DCO bitmap as DCO bitmaps are not yet enabled when log-replay is triggered. Hence when the remaining columns are attached back, appropriate data blocks of those log-replay IOs are skipped during rebuild. This leads to data corruption when reads are serviced from those columns.

RESOLUTION:
Code changes are done to ensure log-replay on EC volume is triggered only after ensuring DCO tracking is enabled. This ensures that all IOs from log-replay operations are tracked in DCO maps for remaining faulted columns of volume.

* 4115252 (Tracking ID: 4115193)

SYMPTOM:
Data corruption on VVR primary with storage loss beyond fault tolerance level in replicated environment.

DESCRIPTION:
In Flexible Storage Sharing (FSS)  environment  any node fault can lead to storage failure. In VVR primary when last  mirror of SRL  (Storage Replicator Log) volume faulted while application writes are in progress replication is expected to go to pass-through mode.
This information is persistently recorded in the kernel log (KLOG). In the event of cascaded storage node failures, the KLOG updation protocol could not update quorum number of copies. This mis-match in on-disk v/s in-core state of VVR objects leading to data loss due to missing recovery when all storage faults are resolved.

RESOLUTION:
Code changes to handle the KLOG update failure in SRL IO failure handling is done to ensure configuration on-disk and in-core is consistent and subsequent application IO will not be allowed to avoid data corruption.

* 4115381 (Tracking ID: 4091783)

SYMPTOM:
Buildarea creation for unixvm were failing

DESCRIPTION:
unixvm build were failing as there is dependency on the storageapi while creation of build area for unixvm and veki.

This intern were causing issues in Veki packaging during unixvm builds and vxio driver compilation dependency

RESOLUTION:
Added support for storageapi build area creation and building the storageapi internally from unixvm build scripts.

* 4116548 (Tracking ID: 4111254)

SYMPTOM:
vradmind dumps core with the following stack:

#3  0x00007f3e6e0ab3f6 in __assert_fail () from /root/cores/lib64/libc.so.6
#4  0x000000000045922c in RDS::getHandle ()
#5  0x000000000056ec04 in StatsSession::addHost ()
#6  0x000000000045d9ef in RDS::addRVG ()
#7  0x000000000046ef3d in RDS::createDummyRVG ()
#8  0x000000000044aed7 in PriRunningState::update ()
#9  0x00000000004b3410 in RVG::update ()
#10 0x000000000045cb94 in RDS::update ()
#11 0x000000000042f480 in DBMgr::update ()
#12 0x000000000040a755 in main ()

DESCRIPTION:
vradmind was trying to access a NULL pointer (Remote Host Name) in a rlink object, as the Remote Host attribute of the rlink hasn't been set.

RESOLUTION:
The issue has been fixed by making code changes.

* 4116551 (Tracking ID: 4108913)

SYMPTOM:
Vradmind dumps core with the following stacks:
#3  0x00007f2c171be3f6 in __assert_fail () from /root/coredump/lib64/libc.so.6
#4  0x00000000005d7a90 in VList::concat () at VList.C:1017
#5  0x000000000059ae86 in OpMsg::List2Msg () at Msg.C:1280
#6  0x0000000000441bf6 in OpMsg::VList2Msg () at ../../include/Msg.h:389
#7  0x000000000043ec33 in DBMgr::processStatsOpMsg () at DBMgr.C:2764
#8  0x00000000004093e9 in process_message () at srvmd.C:418
#9  0x000000000040a66d in main () at srvmd.C:733

#0  0x00007f4d23470a9f in raise () from /root/core.Jan18/lib64/libc.so.6
#1  0x00007f4d23443e05 in abort () from /root/core.Jan18/lib64/libc.so.6
#2  0x00007f4d234b3037 in __libc_message () from /root/core.Jan18/lib64/libc.so.6
#3  0x00007f4d234ba19c in malloc_printerr () from /root/core.Jan18/lib64/libc.so.6
#4  0x00007f4d234bba9c in _int_free () from /root/core.Jan18/lib64/libc.so.6
#5  0x00000000005d5a0a in ValueElem::_delete_val () at Value.C:491
#6  0x00000000005d5990 in ValueElem::~ValueElem () at Value.C:480
#7  0x00000000005d7244 in VElem::~VElem () at VList.C:480
#8  0x00000000005d8ad9 in VList::~VList () at VList.C:1167
#9  0x000000000040a71a in main () at srvmd.C:743

#0  0x000000000040b826 in DList::head () at ../include/DList.h:82
#1  0x00000000005884c1 in IpmHandle::send () at Ipm.C:1318
#2  0x000000000056e101 in StatsSession::sendUCastStatsMsgToPrimary () at StatsSession.C:1157
#3  0x000000000056dea1 in StatsSession::sendStats () at StatsSession.C:1117
#4  0x000000000046f610 in RDS::collectStats () at RDS.C:6011
#5  0x000000000043f2ef in DBMgr::collectStats () at DBMgr.C:2799
#6  0x00007f98ed9131cf in start_thread () from /root/core.Jan26/lib64/libpthread.so.0
#7  0x00007f98eca4cdd3 in clone () from /root/core.Jan26/lib64/libc.so.6

DESCRIPTION:
There is a race condition in vradmind that may cause memory corruption and unpredictable result. Vradmind periodically forks a child thread to collect VVR statistic data and send them to the remote site. While the main thread may also be sending data using the same handler object, thus member variables in the handler object are accessed in parallel from multiple threads and may become corrupted.

RESOLUTION:
The code changes have been made to fix the issue.

* 4116557 (Tracking ID: 4085404)

SYMPTOM:
Huge perf drop after Veritas Volume Replicator (VVR) entered Data Change Map (DCM) mode, when a large size of Storage Replicator Log (SRL) is configured.

DESCRIPTION:
The active map flush caused RVG serialization. Once RVG gets serialized, all IOs are queued in restart queue, till the active map flush is finished. The too frequent active map flush caused the huge IO drop during flushing SRL to DCM.

RESOLUTION:
The code is modified to adjust the frequency of active map flush and balance the application IO and SRL flush.

* 4116559 (Tracking ID: 4091076)

SYMPTOM:
SRL gets into pass-thru mode when it's about to overflow.

DESCRIPTION:
Primary initiated log search for the requested update sent from secondary. The search aborted with head error as a check condition isn't set correctly.

RESOLUTION:
Fixed the check condition to resolve the issue.

* 4116562 (Tracking ID: 4114257)

SYMPTOM:
VxVM cmd is hung and file system was waiting for io to complete.

file system stack:
#3 [] wait_for_completion at 
#4 [] vx_bc_biowait at [vxfs]
#5 [] vx_biowait at [vxfs]
#6 [] vx_isumupd at [vxfs]
#7 [] __switch_to_asm at 
#8 [] vx_process_revokedele at [vxfs]
#9 [] vx_recv_revokedele at [vxfs]
#10 [] vx_recvdele at [vxfs]
#11 [] vx_msg_process_thread at [vxfs]

vxconfigd stack:
[<0>] volsync_wait+0x106/0x180 [vxio]
[<0>] vol_ktrans+0x9f/0x2c0 [vxio]
[<0>] volconfig_ioctl+0x82a/0xdf0 [vxio]
[<0>] volsioctl_real+0x38a/0x450 [vxio]
[<0>] vols_ioctl+0x6d/0xa0 [vxspec]
[<0>] vols_unlocked_ioctl+0x1d/0x20 [vxspec]

One of vxio thread was waiting for IO drain with below stack.

 #2 [] schedule_timeout at 
 #3 [] vol_rv_change_sio_start at [vxio]
 #4 [] voliod_iohandle at [vxio]

DESCRIPTION:
VVR rvdcm flush SIO was triggered by VVR logowner change and it would set the ru_state throttle flags which caused  MDATA_SHIP SIO got queued in rv_mdship_throttleq. As the MDATA_SHIP SIOs are active, it caused rvdcm flush SIO unable to proceed. In the end, rvdcm_flush SIO was waiting for SIOs in rv_mdship_throttleq to complete. SIOs in rv_mdship_throttleq were waiting rvdcm_flush SIO to complete. Hence a  dead lock situation.

RESOLUTION:
Code changes have been made to solve the dead lock issue.

* 4116565 (Tracking ID: 4034741)

SYMPTOM:
Due to a common RVIOmem pool being used by multiple RVG, a deadlock scenario gets created, causing high load average and system hang.

DESCRIPTION:
The current fix limits IO load on secondary by retaining the updates in NMCOM pool until the DV write done, by which RVIOMEM pool became easy to fill up and 
deadlock situtaion may occur, esp. when high work load on multiple RVGs or cross direction RVGs.Currently all RVGs share the same RVIOMEM pool, while NMCOM 
pool, RDBACK pool and network/dv update list are all per-RVGs, so the RVIOMEM pool becomes the bottle neck on secondary, which is easy to full and run into 
deadlock situation.

RESOLUTION:
Code changes to honor per-RVG RVIOMEM pool to resolve the deadlock issue.

* 4116567 (Tracking ID: 4072862)

SYMPTOM:
In case RVGLogowner resources get onlined on slave nodes, stop the whole cluster may fail and RVGLogowner resources goes in to offline_propagate state.

DESCRIPTION:
While stopping whole cluster, the racing may happen between CVM reconfiguration and RVGLogowner change SIO.

RESOLUTION:
Code changes have been made to fix these racings.

* 4117110 (Tracking ID: 4113841)

SYMPTOM:
VVR panic happened in below code path:

kmsg_sys_poll()
nmcom_get_next_mblk() 
nmcom_get_hdr_msg() 
nmcom_get_next_msg() 
nmcom_wait_msg_tcp() 
nmcom_server_main_tcp()

DESCRIPTION:
When the network scan tool send request to VVR which is unexpected, during the VVR connection handshake, the tcp connection may be terminated immediately by the network scan tool, which may lead to the sock released. Hence, VVR panic when try to refer to it as hit the NULL pointer during the processing.

RESOLUTION:
The code change has been made to check sock is valid, otherwise, return without continue with the VVR connection.

* 4118108 (Tracking ID: 4114867)

SYMPTOM:
Getting these error messages while adding new disks
[root@server101 ~]# cat /etc/udev/rules.d/41-VxVM-selinux.rules | tail -1
KERNEL=="VxVM*", SUBSYSTEM=="block", ACTION=="add", RUN+="/bin/sh -c 'if [ `/usr/sbin/getenforce` != "Disabled" -a `/usr/sbin/
[root@server101 ~]#
[root@server101 ~]# systemctl restart systemd-udevd.service
[root@server101 ~]# udevadm test /block/sdb 2>&1 | grep "invalid"
invalid key/value pair in file /etc/udev/rules.d/41-VxVM-selinux.rules on line 20, starting at character 104 ('D')

DESCRIPTION:
In /etc/udev/rules.d/41-VxVM-selinux.rules double quotation on Disabled and disable is the issue.

RESOLUTION:
Code changes have been made to correct the problem.

* 4118111 (Tracking ID: 4065490)

SYMPTOM:
systemd-udev threads consumes more CPU during system bootup or device discovery.

DESCRIPTION:
During disk discovery when new storage devices are discovered, VxVM udev rules are invoked for creating hardware path
symbolic link and setting SELinux security context on Veritas device files. For creating hardware path symbolic link to each
storage device, "find" command is used internally which is CPU intensive operation. If too many storage devices are attached to
system, then usage of "find" command causes high CPU consumption.

Also, for setting appropriate SELinux security context on VxVM device files, restorecon is done irrespective of SELinux is enabled or disabled.

RESOLUTION:
Usage of "find" command is replaced with "udevadm" command. SELinux security context on VxVM device files is being set
only when SELinux is enabled on system.

* 4118479 (Tracking ID: 4118478)

SYMPTOM:
VxVM installation fails on RHEL9.2

DESCRIPTION:
There have been multiple changes done regarding blkcg_gq, blk_put_request, bio_clone_fast, bio_init, blk_cleanup_queue, blk_cleanup_disk, blk_execute_rq, blk_get_request, etc hence VxVM code 
is not compatible with these new code changes done in kernel .

RESOLUTION:
Required changes has been done to make VxVM compatible with RHEL9.2.

* 4118733 (Tracking ID: 4106689)

SYMPTOM:
Solaris Zones cannot be started due to Method "/lib/svc/method/fs-local" failed with exit status 95. The error logs are observed as below:
Mounting ZFS filesystems: cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export' on '/export': directory is not empty
cannot mount 'rpool/export/home' on '/export/home': failure mounting parent dataset
cannot mount 'rpool/export/home/addm' on /export/home/addm': directory is not empty
.... ....
svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: one or more file systems failed.

DESCRIPTION:
When DMP native support is enabled and the "faulted" zpools are found, VxVM will deport the faulty zpools and re-import them. In case fs-local isn't started before vxvm-startup2, this error handling will cause a non-empty /export which further cause zfs mount failure.

RESOLUTION:
Code changes have been made to guarantee the mount order of rpool and zpools.

* 4118845 (Tracking ID: 4116024)

SYMPTOM:
kernel panicked at gab_ifreemsg with following stack:
gab_ifreemsg
gab_freemsg
kmsg_gab_send
vol_kmsg_sendmsg
vol_kmsg_sender

DESCRIPTION:
In a CVR environment there is a RVG of > 600 data volumes, enabling vxvvrstatd daemon through service vxvm-recover. vxvvrstatd calls into ioctl(VOL_RV_APPSTATS) , the latter will generate a kmsg whose length is longer than 64k and trigger a kernel panic due to GAB/LLT no support any message longer than 64k.

RESOLUTION:
Code changes have been done to add a limitation to the maximum number of data volumes for which that ioctl(VOL_RV_APPSTATS) can request the VVR statistics.

* 4119087 (Tracking ID: 4067191)

SYMPTOM:
In CVR environment after rebooting Slave node, Master node may panic with below stack:

Call Trace:
dump_stack+0x66/0x8b
panic+0xfe/0x2d7
volrv_free_mu+0xcf/0xd0 [vxio]
vol_ru_free_update+0x81/0x1c0 [vxio]
volilock_release_internal+0x86/0x440 [vxio]
vol_ru_free_updateq+0x35/0x70 [vxio]
vol_rv_write2_done+0x191/0x510 [vxio]
voliod_iohandle+0xca/0x3d0 [vxio]
wake_up_q+0xa0/0xa0
voliod_iohandle+0x3d0/0x3d0 [vxio]
voliod_loop+0xc3/0x330 [vxio]
kthread+0x10d/0x130
kthread_park+0xa0/0xa0
ret_from_fork+0x22/0x40

DESCRIPTION:
As part of CVM Master switch a rvg_recovery is triggered. In this step race
condition can occured between the VVR objects due to which the object value
is not updated properly and can cause panic.

RESOLUTION:
Code changes are done to handle the race condition between VVR objects.

* 4119257 (Tracking ID: 4090772)

SYMPTOM:
vxconfigd/vx commands hang on secondary site in a CVR environment.

DESCRIPTION:
Due to a window with unmatched SRL positions, if any application (e.g. fdisk) trying
to open the secondary RVG volume will acquire a lock and wait for SRL positions to match.
During this if any vxvm transaction kicked in will also have to wait for same lock.
Further logowner node panic'd which triggered logownership change protocol which hung
as earlier transaction was stuck. As logowner change protocol could not complete,
in absence of valid logowner SRL position could not match and caused deadlock. That lead
to vxconfigd and vx command hang.

RESOLUTION:
Added changes to allow read operation on volume even if SRL positions are
unmatched. We are still blocking write IOs and just allowing open() call for read-only
operations, and hence there will not be any data consistency or integrity issues.

* 4119276 (Tracking ID: 4090943)

SYMPTOM:
On Primary, RLink is continuously getting connected/disconnected with below message seen in secondary syslog:
  VxVM VVR vxio V-5-3-0 Disconnecting replica <rlink_name> since log is full on secondary.

DESCRIPTION:
When RVG logowner node panic, RVG recovery happens in 3 phases.
At the end of 2nd phase of recovery in-memory and on-disk SRL positions remains incorrect
and during this time if there is logowner change then Rlink won't get connected.

RESOLUTION:
Handled in-memory and on-disk SRL positions correctly.

* 4119438 (Tracking ID: 4117985)

SYMPTOM:
Memory/data corruption hit for EC volumes

DESCRIPTION:
This is a porting request original request was already reviewed:http://codereview.engba.veritas.com/r/42056/

Memory corruption hitting in EC was fixed by calling kernel_fpu_begin() for kernel version < rhel8.6. But in latest kernel kernel_fpu_begin() symbol is not 
available, We can not use it. So we have created separate Module with name 'storageapi' which is having implementation of _fpu_begin and _fpu_end
VxIO module is dependent on 'storageapi'

RESOLUTION:
take a fpu lock for FPU related operations

* 4120350 (Tracking ID: 4120878)

SYMPTOM:
System doesn't come up on taking a reboot after enabling dmp_native_support. System goes into maintenance mode.

DESCRIPTION:
"vxio.ko" is dependent on the new "storageapi.ko" module. "storageapi.ko" was missing from VxDMP_initrd file, which is created when dmp_native_support is enabled. So on reboot, without "storageapi.ko" present, "vxio.ko" fails to load.

RESOLUTION:
Code changes have been made to include "strorageapi.ko" in VxDMP_initrd.

* 4121241 (Tracking ID: 4114927)

SYMPTOM:
After enabling dmp_native_support and taking reboot, /boot is not mounted VxDMP node.

DESCRIPTION:
When dmp_native_support is enabled, vxdmproot script is expected to modify the /etc/fstab entry for /boot so that on next boot up, /boot is mounted on dmp device instead of OS device. Also, this operation modifies SELinux context of file /etc/fstab. This causes the machine to go into maintenance mode because of a read permission denied error for /etc/fstab on boot up.

RESOLUTION:
Code changes have been done to make sure SELinux context is preserved for /etc/fstab file and /boot is mounted on dmp device when dmp_native_support is enabled.

* 4121714 (Tracking ID: 4081740)

SYMPTOM:
vxdg flush command slow due to too many luns needlessly access /proc/partitions.

DESCRIPTION:
Linux BLOCK_EXT_MAJOR(block major 259) is used as extended devt for block devices. When partition number of one device is more than 15, the partition device gets assigned under major 259 to solve the sd limitations (16 minors per device), by which more partitions are allowed for one sd device. During "vxdg flush", for each lun in the disk group, vxconfigd reads file /proc/partitions line by line through fgets() to find all the partition devices with major number 259, which would cause vxconfigd to respond sluggishly if there are large amount of luns in the disk group.

RESOLUTION:
Code has been changed to remove the needless access on /proc/partitions for the luns without using extended devt.

Patch ID: VRTSvxvm-8.0.0.2100

* 4102502 (Tracking ID: 4102501)

SYMPTOM:
A security vulnerability exists in the third-party component libcurl.

DESCRIPTION:
VxVM uses a third-party component named libcurl in which a security vulnerability exists.

RESOLUTION:
VxVM is updated to use a newer version of libcurl in which the security vulnerability has been addressed.

Patch ID: VRTSaslapm 8.0.0.2500

* 4101808 (Tracking ID: 4101807)

SYMPTOM:
"vxdisk -e list" does not show "svol" for Hitachi ShadowImage (SI) svol devices.

DESCRIPTION:
VxVM with DMP is failing to detect Hitachi ShadowImage (SI) svol devices.

RESOLUTION:
Hitachi ASL modified to correctly read SCSI Byte locations and recognize ShadowImage (SI) svol device.

* 4116688 (Tracking ID: 4085145)

SYMPTOM:
The issue we are discussing is with AWS environment, on-prim physical/vm host this issue does not exist.( as ioctl and sysfs is giving same values)

DESCRIPTION:
The UDID value in case of Amazon EBS devices was going beyond its limit (read from sysfs as ioctl is not supported by AWS)

RESOLUTION:
Did code changes to fetch LSN through IOCTL as we have fix for intermittent ioctl failure.

* 4117385 (Tracking ID: 4117350)

SYMPTOM:
Below error is observed when trying to import 

# vxdg -n SVOL_SIdg -o useclonedev=on -o updateid import SIdg
VxVM vxdg ERROR V-5-1-0 Disk group SIdg: import failed:
Replicated dg record is found.
Did you want to import hardware replicated LUNs?
Try vxdg [-o usereplicatedev=only] import option with -c[s]

Please refer to system log for details.

DESCRIPTION:
REPLICATED flag is used to identify a hardware replicated device so to import dg on the  REPLICATED disks , usereplicatedev option must be used . As that was not provided hence issue was observed .

RESOLUTION:
REPLICATED flag has been removed for Hitachi ShadowImage (SI) disks.

* 4122584 (Tracking ID: 4122583)

SYMPTOM:
Support for ASLAPM on RHEL9.2

DESCRIPTION:
The RHEL9.2 is new release and hence APM module 
should be recompiled with new kernel.

RESOLUTION:
Compiled APM with new kernel.

Patch ID: VRTScavf-8.0.0.2800

* 4118779 (Tracking ID: 4074274)

SYMPTOM:
DR test and failover activity might not succeed for hardware-replicated disk groups.

DESCRIPTION:
In case of hardware-replicated disks like EMC SRDF, failover of disk groups might not automatically succeed and a manual intervention might be needed. After failover, disks at the new primary site have the 'udid_mismatch' flag which needs to be updated manually for a successful failover.

RESOLUTION:
For DMP environments, the VxVM & DMP extended attributes need to be refreshed by using 'vxdisk scandisks' prior to import. VxVM has also provided a new vxdg import option '-o usereplicatedev=only' with DMP. This option selects only the hardware-replicated disks during LUN selection process.

Sample main.cf configuration for DMP managed hardware Replicated disks.
CVMVolDg srdf_dg (
CVMDiskGroup = LINUXSRDF
CVMVolume = { scott, scripts }
CVMActivation = sw
CVMDeportOnOffline = 1
ClearClone = 1
ScanDisks = 1
DGOptions = { "-t -o usereplicatedev=only" }
)
All 4 options(CVMDeportOnOffline, ClearClone, ScanDisks and DGOptions) are recommended with hardware-replicated disk groups.

Patch ID: VRTSgms-8.0.0.2800

* 4057427 (Tracking ID: 4057176)

SYMPTOM:
Rebooting the system results into emergency mode.

DESCRIPTION:
Module dependency files get corrupted due to parallel invocation of depmod.

RESOLUTION:
Serialized the invocation of depmod through file lock.

* 4118302 (Tracking ID: 4118303)

SYMPTOM:
The GMS module fails to load on RHEL9.2.

DESCRIPTION:
This issue occurs due to changes in the RHEL9.2 kernel.

RESOLUTION:
GMS module is updated to accommodate the changes in the kernel and load as expected on RHEL9.2.

Patch ID: VRTSglm-8.0.0.2800

* 4118296 (Tracking ID: 4118297)

SYMPTOM:
The GLM module fails to load on RHEL9.2.

DESCRIPTION:
This issue occurs due to changes in the RHEL9.2 kernel.

RESOLUTION:
GLM module is updated to accommodate the changes in the kernel and load as expected on RHEL9.2.

Patch ID: VRTSodm-8.0.0.2800

* 4057432 (Tracking ID: 4056673)

SYMPTOM:
Rebooting the system results into emergency mode.

DESCRIPTION:
Module dependency files get corrupted due to parallel invocation of depmod.

RESOLUTION:
Serialized the invocation of depmod through file lock. Corrected vxgms dependency in odm service file.

* 4118467 (Tracking ID: 4118466)

SYMPTOM:
ODM module failed to load on RHEL9.2 kernel.

DESCRIPTION:
This issue occurs due to changes in the RHEL9.2 kernel.

RESOLUTION:
ODM module is updated to accommodate the changes in the kernel and load as expected on RHEL9.2 kernel.

Patch ID: VRTSodm-8.0.0.2500

* 4114322 (Tracking ID: 4114321)

SYMPTOM:
VRTSodm driver will not load with VRTSvxfs patch.

DESCRIPTION:
Need recompilation of VRTSodm with latest VRTSvxfs.

RESOLUTION:
Recompiled the VRTSodm with new VRTSvxfs .

INSTALLATION PRE-REQUISITES
Following ISO/patches are needed for this patch:

1. RHEL9 image base installer for InfoScale 8.0

     Enterprise license link: https://www.veritas.com/content/support/en_US/downloads/detail.REL928974#item2 

    Availability license link: https://www.veritas.com/content/support/en_US/downloads/detail.REL150377#item2

    Storage license link: https://www.veritas.com/content/support/en_US/downloads/detail.REL919000#item2

    Foundation link: https://www.veritas.com/content/support/en_US/downloads/detail.REL216807#item2



2. This patch InfoScale 8.0 Patch 2800



INSTALLING THE PATCH
--------------------
Run the Installer script to automatically install the patch:
-----------------------------------------------------------
Please be noted that the installation of this P-Patch will cause downtime.

To install the patch perform the following steps on at least one node in the cluster:
1. Copy the patch infoscale-rhel9_x86_64-Patch-8.0.0.2800.tar.gz to /tmp
2. Untar infoscale-rhel9_x86_64-Patch-8.0.0.2800.tar.gz to /tmp/hf
    # mkdir /tmp/hf
    # cd /tmp/hf
    # gunzip /tmp/infoscale-rhel9_x86_64-Patch-8.0.0.2800.tar.gz
    # tar xf /tmp/infoscale-rhel9_x86_64-Patch-8.0.0.2800.tar
3. Install the hotfix(Please be noted that the installation of this P-Patch will cause downtime.)
    # pwd /tmp/hf
    # ./installVRTSinfoscale800P2800 [<host1> <host2>...]

You can also install this patch together with 8.0 base release using Install Bundles
1. Download this patch and extract it to a directory
2. Change to the Veritas InfoScale 8.0 directory and invoke the installer script
   with -patch_path option where -patch_path should point to the patch directory
    # ./installer -patch_path [<path to this patch>] [<host1> <host2>...]

Install the patch manually:
--------------------------
Manual installation is not recommended.


REMOVING THE PATCH
------------------
Manual uninstallation is not recommended.


KNOWN ISSUES
------------
* Tracking ID: 4085140

SYMPTOM: While doing two or more mount (of vxfs file system) operations in parallel, underneath an already
existing vxfs mount point, if a force umount is attempted on the parent vxfs mount point, then sometimes
the force unmount operation hangs permanently.

WORKAROUND: There is no workaround, except rebooting the system.

* Tracking ID: 4119974

SYMPTOM: VxFS mount binary failed to mount VxFS with SELinux context.

WORKAROUND: Use native mount command (provided by linux) instead of VxFS mount command.
mount -t vxfs /dev/vx/dsk/testdg/vol1 /mnt1 -ocontext="system_u:object_r:httpd_sys_content_t:s0"



SPECIAL INSTRUCTIONS
--------------------
1. Please check any cumulative patch (Update release) released prior to this platform patch. Install this platform patch along with the CP.
2. In case any CP is released on top of this platform patch, this platform patch will be included in that. Please check and install the latest CP.
3. In case the internet is not available, Installation of the patch must be performed concurrently with the latest CPI patch downloaded from Download Center.



OTHERS
------
NONE