* * * READ ME * * * * * * Symantec Storage Foundation HA 6.2.1 * * * * * * Patch 4100 * * * Patch Date: 2019-11-11 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Symantec Storage Foundation HA 6.2.1 Patch 4100 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- SLES12 x86-64 PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSamf VRTSdbac VRTSgab VRTSglm VRTSgms VRTSllt VRTSodm VRTSveki VRTSvxfen VRTSvxfs VRTSvxvm BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Symantec Application HA 6.2.1 * Symantec Cluster Server 6.2.1 * Symantec Dynamic Multi-Pathing 6.2.1 * Symantec File System 6.2.1 * Symantec Storage Foundation 6.2.1 * Symantec Storage Foundation Cluster File System HA 6.2.1 * Symantec Storage Foundation for Oracle RAC 6.2.1 * Symantec Storage Foundation HA 6.2.1 * Symantec Volume Manager 6.2.1 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: VRTSvxfs-6.2.1.8400 * 3972960 (3905099) VxFS unmount panicked in deactive_super(). * 3977203 (3973943) VxFS module failed to load on SLES12 SP4 * 3980789 (3980754) In function vx_io_proxy_thread(), system may hit kernel panic due to general protection fault. * 3983567 (3922986) Dead lock issue with buffer cache iodone routine in CFS. * 3983574 (3978149) FIFO file's timestamps are not updated in case of writes. * 3986836 (3987141) Vxcompress operation generates core dumps Patch ID: VRTSvxfs-6.2.1.600 * 3930621 (3927746) VxFS module failed to load on SLES12 SP3. Patch ID: VRTSvxfs-6.2.1.400 * 3915626 (3911966) VxFS module failed to load on SLES12 SP2. Patch ID: VRTSvxfs-6.2.1.300 * 3817229 (3762174) fsfreeze and vxdump commands may not work together. * 3896150 (3833816) Read returns stale data on one node of the CFS. * 3896151 (3827491) Data relocation is not executed correctly if the IOTEMP policy is set to AVERAGE. * 3896154 (1428611) 'vxcompress' can spew many GLM block lock messages over the LLT network. * 3896156 (3633683) vxfs thread consumes high CPU while running an application that makes excessive sync() calls. * 3896160 (3808033) When using 6.2.1 ODM on RHEL7, Oracle resource cannot be killed after forced umount via VCS. * 3896223 (3735697) vxrepquota reports error * 3896231 (3708836) fallocate causes data corruption * 3896261 (3855726) Panic in vx_prot_unregister_all(). * 3896267 (3861271) Missing an inode clear operation when a Linux inode is being de-initialized on SLES11. * 3896269 (3879310) File System may get corrupted after a failed vxupgrade. * 3896270 (3707662) Race between reorg processing and fsadm timer thread (alarm expiry) leads to panic in vx_reorg_emap. * 3896273 (3558087) The ls -l and other commands which uses stat system call may take long time to complete. * 3896277 (3691633) Remove RCQ Full messages * 3896281 (3830300) Degraded CPU performance during backup of Oracle archive logs on CFS vs local filesystem * 3896285 (3757609) CPU usage going high because of contention over ODM_IO_LOCK * 3896303 (3762125) Directory size increases abnormally. * 3896304 (3846521) "cp -p" fails if modification time in nano seconds have 10 digits. * 3896306 (3790721) High cpu usage caused by vx_send_bcastgetemapmsg_remaus * 3896308 (3695367) Unable to remove volume from multi-volume VxFS using "fsvoladm" command. * 3896310 (3859032) System panics in vx_tflush_map() due to NULL pointer de-reference. * 3896311 (3779916) vxfsconvert fails to upgrade layout verison for a vxfs file system with large number of inodes. * 3896312 (3811849) On cluster file system (CFS), while executing lookup() function in a directory with Large Directory Hash (LDH), the system panics and displays an error. * 3896313 (3817734) Direct command to run fsck with -y|Y option was mentioned in the message displayed to user when file system mount fails. * 3896314 (3856363) Filesystem inodes have incorrect blocks. * 3901379 (3897793) Panic happens because of race where the mntlock ID is cleared while mntlock flag still set. * 3903657 (3857254) Assert failure because of missed flush before taking filesnap of the file. * 3904841 (3901318) VxFS module failed to load on RHEL7.3. * 3905056 (3879761) Performance issue observed due to contention on vxfs spin lock vx_worklist_lk. * 3906148 (3894712) ACL permissions are not inherited correctly on cluster file system. * 3906846 (3872202) VxFS internal test hits an assert. * 3906961 (3891801) Internal test hit debug assert. * 3907350 (3817734) Direct command to run fsck with -y|Y option was mentioned in the message displayed to user when file system mount fails. * 3907359 (3907722) kernel BUG at fs/dcache.c:964. Patch ID: VRTSvxfs-6.2.1.200 * 3864963 (3853338) Files on VxFS are corrupted while running the sequential asynchronous write workload under high memory pressure. * 3865600 (3865599) VxFS module failed to load on SLES12 SP1. Patch ID: VRTSvxfs-6.2.1.100 * 3753724 (3731844) umount -r option fails for vxfs 6.2. * 3754492 (3761603) Internal assert failure because of invalid extop processing at the mount time. * 3756002 (3764824) Internal cluster file system(CFS) testing hit debug assert * 3765324 (3736398) NULL pointer dereference panic in lazy unmount. * 3765998 (3759886) In case of nested mount, force umount of parent leaves stale child entry in /etc/mtab even after subsequent umount of child. * 3769992 (3729158) Deadlock due to incorrect locking order between write advise and dalloc flusher thread. * 3793241 (3793240) Vxrestore command dumps core file because of invalid japanese strings. * 3798437 (3812914) On RHEL 6.5 and RHEL 6.4 latest kernel patch, umount(8) system call hangs if an application watches for inode events using inotify(7) APIs. * 3808285 (3808284) fsdedupadm status Japanese text includes strange character. * 3817120 (3804400) VRTS/bin/cp does not return any error when quota hard limit is reached and partial write is encountered. * 3821688 (3821686) VxFS module failed to load on SLES11 SP4. Patch ID: VRTSvxfs-6.2.1.000 * 3093833 (3093821) The system panics due to referring freed super block after the vx_unmount() function errors. * 3657150 (3604071) High CPU usage consumed by the vxfs thread process. * 3657151 (3513507) Filesystem umount() may hang due to deadlock in kernel * 3657152 (3602322) System panics while flushing the dirty pages of the inode. * 3657153 (3622323) Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics with the fsadm_vxfs(1M) command. * 3657156 (3604750) The kernel loops during the extent re-org. * 3657157 (3617191) Checkpoint creation takes a lot of time. * 3657158 (3601943) Truncating corrupted block map of a file may lead to an infinite loop. * 3657159 (3633067) While converting from ext3 file system to VxFS using vxfsconvert, it is observed that many inodes are missing.. * 3665980 (2059611) The system panics due to a NULL pointer dereference while flushing bitmaps to the disk. * 3665984 (2439261) When the vx_fiostats_tunable value is changed from zero to non-zero, the system panics. * 3665990 (3567027) During the File System resize operation, the "fullfsck flag is set. * 3666007 (3594386) On RHEL6u5, stack overflows lead to system panic. * 3666008 (3616907) System is unresponsive causing the NMI watchdog service to stall. * 3666010 (3233276) With a large file system, primary to secondary migration takes longer duration. * 3683470 (3682138) The VxVM symbols are released before the VxFS modules unload time. * 3687679 (3685391) Execute permissions for a file not honored correctly. * 3697142 (3697141) Added support for Sles12 * 3697966 (3697964) The vxupgrade(1M) command fails to retain the fs_flags after upgrading a file system. * 3698165 (3690078) The system panics at vx_dev_strategy() routine due to stack overflow. * 3706864 (3709947) The SSD cache fails to go offline due to additional slashes "//" in the dev path of the cache device. * 3710794 (3710792) Unmount fails when the mount point contains a special character(colon). * 3715567 (3715566) VxFS fails to report an error when the maxlink and nomaxlink options are set on file systems having disk layout version (DLV) lower than 10. * 3716627 (3622326) During a checkpoint promote, a file system is marked with a fullfsck flag because the corresponding inode is marked as bad. * 3717895 (2919310) During stress testing on cluster file system, an assertion failure was hit because of a missing linkage between the directory and the associated attribute inode. * 3718542 (3269553) VxFS returns inappropriate message for read of hole via Oracle Disk Manager (ODM). * 3721458 (3721466) After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock. * 3726403 (3739618) sfcache command with "-i" option maynot show filesystem cache statistic periodically. * 3727166 (3727165) Enhance RHEV support for SF devices for identification in Guest * 3729704 (3719523) 'vxupgrade' retains the superblock replica of old layout versions. * 3733811 (3729030) The fsdedupschd daemon failed to start on RHEL7. * 3733812 (3729030) The fsdedupschd daemon failed to start on RHEL7. * 3737330 (3737329) Added support for RHEL7.1 * 3743913 (3743912) Users could create sub-directories more than 64K for disk layouts having versions lower than 10. * 3744425 (3744424) Rebundled the fix "Openssl: Common Vulnerability and Exposure (CVE) CVE- 2014-3566 (POODLE)" as 6.2.1 * 3745651 (3642314) Umount operation reports error code 255 in case of write-back cache. * 3749727 (3750187) Internal noise testing hits debug assert. * 3749776 (3637636) Cluster File System (CFS) node initialization and protocol upgrade may hang during the rolling upgrade. * 3755796 (3756750) VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline. Patch ID: VRTSveki-6.2.1.7100 * 3987155 (3987156) VEKI module failed to load on SLES12 SP4. Patch ID: VRTSveki-6.2.1.500 * 3943408 (3943409) VEKI module failed to load on SLES12 SP3. Patch ID: VRTSveki-6.2.1.400 * 3916373 (3913507) VEKI module failed to load on SLES12 SP2. Patch ID: VRTSdbac-6.2.1.8200 * 3985907 (3969613) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). Patch ID: VRTSdbac-6.2.1.400 * 3943229 (3927712) Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). Patch ID: VRTSdbac-6.2.1.300 * 3916355 (3913168) Veritas Oracle Real Application Cluster does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). Patch ID: VRTSamf-6.2.1.9200 * 3985033 (3969613) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). Patch ID: VRTSamf-6.2.1.500 * 3930510 (3927712) Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). Patch ID: VRTSamf-6.2.1.400 * 3907369 (3896875) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). Patch ID: VRTSamf-6.2.1.200 * 3906412 (3896877) Veritas Infoscale Availability does not support Red Hat Enterprise Linux 7 Update 3(RHEL7.3). Patch ID: VRTSamf-6.2.1.100 * 3794201 (3794154) Veritas Cluster Server (VCS) does not support Red Hat Enterprise Linux 6 Update 7 (RHEL6.7). Patch ID: VRTSamf-6.2.1.000 * 3661452 (3652819) VxFS module fails to unload because the AMF module fails to decrement the reference count. Patch ID: VRTSvxfen-6.2.1.7300 * 3985032 (3969613) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). Patch ID: VRTSvxfen-6.2.1.500 * 3943228 (3927712) Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). Patch ID: VRTSvxfen-6.2.1.400 * 3907368 (3896875) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). Patch ID: VRTSvxfen-6.2.1.200 * 3912033 (2852872) Fencing sometimes shows "replaying" RFSM state for some nodes in the cluster. Patch ID: VRTSvxfen-6.2.1.100 * 3794201 (3794154) Veritas Cluster Server (VCS) does not support Red Hat Enterprise Linux 6 Update 7 (RHEL6.7). Patch ID: VRTSvxfen-6.2.1.000 * 3691204 (3691202) Sometimes fencing in the customized mode fails to start when the number of files present in the current working directory is large. * 3714463 (3722178) The rpm --verify command on VXFEN changes the runlevel settings for the VXFEN service. Patch ID: VRTSgab-6.2.1.9300 * 3984942 (3969613) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). Patch ID: VRTSgab-6.2.1.600 * 3943226 (3927712) Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). Patch ID: VRTSgab-6.2.1.500 * 3907367 (3896875) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). Patch ID: VRTSgab-6.2.1.300 * 3875807 (3875805) In some rare cases, if a few unicast messages are stuck in the Group Membership Atomic Broadcast (GAB) receive queue of a port, the port might receive a GAB I/O fence message. Patch ID: VRTSllt-6.2.1.9600 * 3984828 (3969613) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). Patch ID: VRTSllt-6.2.1.900 * 3930509 (3927712) Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). Patch ID: VRTSllt-6.2.1.800 * 3907366 (3896875) Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). Patch ID: VRTSllt-6.2.1.600 * 3905431 (3905430) Application IO hangs in case of FSS with LLT over RDMA during heavy data transfer. Patch ID: VRTSvxvm-6.2.1.8400 * 3914738 (3897047) Filesystems are not mounted automatically on boot through systemd on RHEL7 and SLES12. * 3921995 (3921994) Failure in the backup for disk group, Temporary files such as .bslist .cslist .perm are seen in the directory /var/temp. * 3944786 (3938549) Volume creation fails with error: Unexpected kernel error in configuration update for Rhel 7.5. * 3949907 (3949954) Dumpstack messages are printed when vxio module is loaded for the first time when called blk_register_queue. * 3956729 (3955725) Utility to clear "failio" flag on disk after storage connectivity is back. * 3964554 (3954972) Retpoline support for VxVM on Sles11 retpoline kernels * 3965707 (3960237) Irrelevant or cosmetic message is seen in dmesg. * 3967033 (3953681) Data corruption issue is seen when more than one plex of volume is detached. * 3967034 (3950199) System may panic while DMP(Dynamic Multipathing) path restoration. * 3967037 (3957549) Server panicked when tracing event because of NULL pointer check missing. * 3967038 (3932356) vxconfigd dumping core while importing DG * 3967070 (3966965) VxVM(Veritas Volume Manager) disk reclaiming failed because of getting disk array attributes incorrectly. * 3967117 (3948140) System panic can occur if size of RTPG (Report Target Port Groups) data returned by underlying array is greater than 255. * 3967121 (3946350) kmalloc-1024 and kmalloc-2048 memory consuming keeps increasing when VVR IO size is more than 256K. * 3967130 (3927493) String object get corrupted with vradmind. * 3967132 (3925377) Not all disks could be discovered by DMP after first startup. * 3967135 (3918408) Data corruption when volume grow is attempted on thin reclaimable disks whose space is just freed. * 3967139 (3927439) VxVM vxassist relayout command doesn't honor read policy. * 3967142 (3926067) vxassist relayout /vxassist commands may fail in Campus Cluster environment. * 3967147 (3936535) Poor performance due to frequent cache drops. * 3967162 (3843740) After the node leave, master node might panic while running the volume recovery. * 3967165 (3907618) vxdisk resize leads to data corruption on filesystem * 3967180 (3873123) If the disk with CDS EFI label is used as remote disk on the cluster node, restarting the vxconfigd daemon on that particular node causes vxconfigd to go into disabled state * 3967182 (3662392) In the Cluster Volume Manager (CVM) environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. * 3967185 (3945115) VxVM (Veritas Volume Manager) vxassist relayout command fails for volumes with RAID layout. * 3967186 (3913949) The DG import is failing with Split Brain after the system is rebooted or when a storage disturbance is seen. * 3967591 (3945019) vxconfigd core in vol_hash() at vol_hash.c:40 * 3972369 (3928114) Plexes inconsistency reported after rebooting node in FSS environment. * 3972434 (3917636) Filesystems from /etc/fstab file are not mounted automatically on boot through systemd on RHEL7 and SLES12. * 3972437 (3952042) vxdmp iostat memory allocation might cause memory fragmentation and pagecache drop. * 3972978 (3965962) Auto recovery gets triggered at the time of slave join. * 3978569 (3959618) VxVM support on SLES12 SP4 with kernel version 4.12 * 3982813 (3752266) 'vxfs' and 'vxportal' modules may fail to unload due to module reference count not released by 'vxio' module. * 3984659 (3955979) I/O gets hang in case of synchronous Replication. * 3984662 (3973202) A VVR primary node may panic due to accessing already freed memory. * 3984726 (3979729) System Commands may get hung when access particular directory in a certain scenario. * 3984736 (3898732) Node panic with bad mutex Patch ID: VRTSvxvm-6.2.1.600 * 3941678 (3939796) Installation of VxVM package fails on SLES12 SP3. Patch ID: VRTSvxvm-6.2.1.500 * 3920545 (3795739) In a split brain scenario, cluster formation takes very long time. * 3922253 (3919642) IO hang after switching log owner because of IOs not completely quiesced during log owner change. * 3922254 (3919640) IO hang along with vxconfigd hang on master node because of metadata request SIO(Staged IO) hogging CPU. * 3922255 (3919644) IO hang when switching log owner because of stale flags in RV(Replication Volume) kernel structure. * 3922256 (3919641) IO hang when pausing rlink because of deadlock situation. * 3922257 (3919645) IO hang when switching log owner because of stale information in RV(Replication Volume) kernel structure. * 3922258 (3919643) IO hang when switching log owner because of stale information in RV(Replication Volume) kernel structure. * 3927482 (3895950) vxconfigd hang observed due to accessing stale/un-initiliazed lock. Patch ID: VRTSvxvm-6.2.1.400 * 3913126 (3910432) When a mirror volume is created two log plexes are created by default. * 3915774 (3907800) VxVM package installation will fail on SLES12 SP2. * 3915780 (3912672) "vxddladm assign names" causes ASM disks' user-group ownership/permissions loss affecting Oracle databases on system. * 3915784 (3868154) When DMP Native Support is set to ON, dmpnode with multiple VGs cannot be listed properly in the 'vxdmpadm native ls' command * 3915785 (3906544) IO statistic information for VxVM(Veritas Volume Manager) device nodes with ODM(Oracle Disk Manager) active is not correctly handled. * 3915886 (3753345) On "SLES12 SP2", VxVM (Veritas Volume Manager) kernel modules may fail to get loaded during package installation. * 3916902 (3916799) Few VxVM [Veritas Volume Manager] and DMP [Veritas Dynamic Multipathing] tunables are not updated after rebooting the system. Patch ID: VRTSvxvm-6.2.1.300 * 3780334 (3762580) In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath co-exists with DMP (Dynamic Multi-Pathing). * 3802857 (3726110) On systems with high number of CPUs, Dynamic Multi-Pathing (DMP) devices may perform considerably slower than OS device paths. * 3803497 (3802750) VxVM (Veritas Volume Manager) volume I/O-shipping functionality is not disabled even after the user issues the correct command to disable it. * 3816222 (3816219) VxDMP event source daemon keeps reporting UDEV change event in syslog. * 3839293 (3776520) Filters are not updated properly in lvm.conf file in VxDMP initrd (initial ramdisk) while Dynamic Multipathing (DMP) Native Support is being enabled. * 3850478 (3850477) kmalloc-1024 and kmalloc-2048 memory consuming keeps increasing when reading or writing data against VxVM volume with big block size * 3851117 (3662392) In the Cluster Volume Manager (CVM) environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. * 3852148 (3852146) Shared DiskGroup(DG) fails to import when "-c" and "-o noreonline" options are specified together * 3854788 (3783356) After Dynamic Multi-Pathing (DMP) module fails to load, dmp_idle_vector is not NULL. * 3863971 (3736502) Memory leakage is found when transaction aborts. * 3868653 (3866051) Driver name over 32 bytes may cause vxconfigd unable to startup * 3871040 (3868444) Disk header timestamp is updated even if the disk group(DG) import fails. * 3871124 (3823283) While unencapsulating a boot disk in SAN environment (Storage Area etwork), Linux operating system sticks in grub after reboot. * 3873145 (3872197) vxconfigd panics when NVME devices are attached to the system * 3874737 (3874387) Disk header information is not logged to the syslog sometimes even if the disk is missing and dg import fails. * 3875933 (3737585) "Uncorrectable write error" or panic with IOHINT in VVR (Veritas Volume Replicator) environment * 3877637 (3878030) Enhance VxVM DR tool to clean up OS and VxDMP device trees without user interaction. * 3879334 (3879324) VxVM DR tool fails to handle busy device problem while LUNs are removed from OS * 3880573 (3886153) vradmind daemon core dump occurs in a VVR primary-primary configuration because of assert() failure. * 3881334 (3864063) Application I/O hangs because of a race between the Master Pause SIO (Staging I/O) and the Error Handler SIO. * 3881335 (3867236) Application IO hang happens because of a race between Master Pause SIO(Staging IO) and RVWRITE1 SIO. * 3889284 (3878153) VVR 'vradmind' deamon core dump. * 3889850 (3878911) QLogic driver returns an error due to Incorrect aiusize in FC header * 3891789 (3873625) System panicked when pulling out FC cables on SFHA6.2.1/RHEL7.2 * 3893134 (3864318) Memory consuming keeps increasing when reading/writing data against VxVM volume with big block size. * 3893362 (3881132) vxcommands hang following san change. * 3894783 (3628743) New BE takes too much time to startup during live upgrade on Solaris 11.2 * 3897764 (3741003) After removing storage from one of multiple plex in a mirrored DCO (Data Change Object) volume, entire DCO volume is detached and DCO object is having BADLOG flag marked because of a flag reset missing. * 3898129 (3790136) File system hang observed due to IO's in Ditry Region Logging (DRL). * 3898168 (3739933) Allow VxVM package installation on EFI enabled Linux machines. * 3898169 (3740730) While creating volume using vxassist CLI, dco log volume length specified at command line was not getting honored. * 3898296 (3767531) In Layered volume layout with FSS configuration, when few of the FSS_Hosts are rebooted, Full resync is happening for non-affected disks on master. * 3902626 (3795739) In a split brain scenario, cluster formation takes very long time. * 3903647 (3868934) System panic happens while deactivate the SIO (staging IO). * 3904790 (3795788) Performance degrades when many application sessions open the same data file on the VxVMvolume. * 3904796 (3853049) The display of stats delayed beyond the set interval for vxstat and multiple sessions of vxstat impacted the IO performance. * 3904797 (3857120) Commands like vxdg deport which try to close a VxVM volume might hang. * 3904800 (3860503) Poor performance of vxassist mirroring is observed on some high end servers. * 3904801 (3686698) vxconfigd was getting hung due to deadlock between two threads * 3904802 (3721565) vxconfigd hang is seen. * 3904804 (3486861) Primary node panics when storage is removed while replication is going on with heavy IOs. * 3904805 (3788644) Reuse raw device number when checking for available raw devices. * 3904806 (3807879) User data corrupts because of the writing of the backup EFT GPT disk label during the VxVM disk-group flush operation. * 3904807 (3867145) When VVR SRL occupation > 90%, then output the SRL occupation is shown by 10 percent. * 3904810 (3871750) In parallel VxVM vxstat commands report abnormal disk IO statistic data * 3904811 (3875563) While dumping the disk header information, human readable timestamp was not converted correctly from corresponding epoch time. * 3904819 (3811946) When invoking "vxsnap make" command with cachesize option to create space optimized snapshot, the command succeeds but a plex I/O error message is displayed in syslog. * 3904822 (3755209) The Veritas Dynamic Multi-pathing(VxDMP) device configured in Solaris Logical DOMains(LDOM) guest is disabled when an active controller of an ALUA array is failed. * 3904824 (3795622) With Dynamic Multipathing (DMP) Native Support enabled, Logical Volume Manager (LVM) global_filter is not updated properly in lvm.conf file. * 3904825 (3859009) global_filter of lvm.conf is not updated due to some paths of LVM dmpnode are reused during DDL(Device Discovery Layer) discovery cycle. * 3904830 (3840359) Some VxVM commands fail on using the localized messages. * 3904831 (3802075) Foreign disks with name having digit in it, defined by udev rules goes into error state after vxdisk scandisks/ * 3904833 (3729078) VVR(Veritas Volume Replication) secondary site panic occurs during patch installation because of flag overlap issue. * 3904834 (3819670) When smartmove with 'vxevac' command is run in background by hitting 'ctlr-z' key and 'bg' command, the execution of 'vxevac' is terminated abruptly. * 3904851 (3804214) VxDMP (Dynamic Multi-Pathing) path enable operation fails after the disk label is changed from guest LDOM. Open fails with error 5 on the path being enabled. * 3904858 (3899568) Adding tunable dmp_compute_iostats to start/stop the iostat gathering persistently. * 3904859 (3901633) vxrsync reports error during rvg sync because of incorrect volume end offset calculation. * 3904861 (3904538) IO hang happens during slave node leave or master node switch because of racing between RV(Replicate Volume) recovery SIO(Staged IO) and new coming IOs. * 3904863 (3851632) Some VxVM commands fail when you use the localized messages. * 3904864 (3769303) System pancis when Cluster Volume Manager (CVM) group is brought online * 3905471 (3868533) IO hang happens because of a deadlock situation. * 3906251 (3806909) Due to some modification in licensing , for STANDALONE DMP, DMP keyless license was not working. * 3906566 (3907654) Storage of cold data on dedicated SAN storage spaces increases storage cost and maintenance. Move cold data from local storage to cloud storage. * 3907017 (3877571) Disk header is updated even if the dg import operation fails * 3907593 (3660869) Enhance the Dirty region logging (DRL) dirty-ahead logging for sequential write workloads * 3907595 (3907596) vxdmpadm setattr command gives error while setting the path attribute. Patch ID: VRTSvxvm-6.2.1.200 * 3780334 (3762580) In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath co-exists with DMP (Dynamic Multi-Pathing). * 3795710 (3508122) After one node preempts SCSI-3 reservation for the other node, the I/O from the victim node does not fail. * 3802857 (3726110) On systems with high number of CPUs, Dynamic Multi-Pathing (DMP) devices may perform considerably slower than OS device paths. * 3803497 (3802750) VxVM (Veritas Volume Manager) volume I/O-shipping functionality is not disabled even after the user issues the correct command to disable it. * 3804299 (3804298) Not recording the setting/unsetting of the 'lfailed/lmissing' flag in the syslog * 3808134 (3808135) While you install the hotfix(HF) on thetop of Veritas Volume Manager(VxVM)6.2.1, the symbolic link creation is failed as old links are not cleaned from the installation scripts. * 3816222 (3816219) VxDMP event source daemon keeps reporting UDEV change event in syslog. * 3823288 (3823283) While unencapsulating a boot disk in SAN environment (Storage Area etwork), Linux operating system sticks in grub after reboot. * 3848011 (3806909) Due to some modification in licensing , for STANDALONE DMP, DMP keyless license was not working. * 3849458 (3776520) Filters are not updated properly in lvm.conf file in VxDMP initrd (initial ramdisk) while Dynamic Multipathing (DMP) Native Support is being enabled. * 3865905 (3865904) VRTSvxvm patch version 6.2.1 failed to install on new kernel update SLES12SP1 * 3870949 (3845712) On SLES12 when encapsulation is performed, System is not able to boot up through vxvm_root grub entry. Patch ID: VRTSgms-6.2.1.7200 * 3977173 (3973946) GMS module failed to load on SLES12 SP4 Patch ID: VRTSgms-6.2.1.500 * 3930628 (3930625) GMS module failed to load on SLES12 SP3. Patch ID: VRTSgms-6.2.1.400 * 3915629 (3907939) GMS module failed to load on SLES12 SP2. Patch ID: VRTSglm-6.2.1.7300 * 3977174 (3973945) GLM module failed to load on SLES12 SP4 Patch ID: VRTSglm-6.2.1.500 * 3930624 (3930622) GLM module failed to load on SLES12 SP3. Patch ID: VRTSglm-6.2.1.400 * 3915628 (3907936) GLM module failed to load on SLES12 SP2. Patch ID: VRTSodm-6.2.1.8400 * 3977175 (3973944) ODM module failed to load on SLES12 SP4 * 3988170 (3987992) Out of memory issue hit in our internal stress testing Patch ID: VRTSodm-6.2.1.600 * 3930632 (3930629) ODM module failed to load on SLES12 SP3. Patch ID: VRTSodm-6.2.1.400 * 3915627 (3907933) ODM module failed to load on SLES12 SP2. Patch ID: VRTSodm-6.2.1.300 * 3906065 (3757609) CPU usage going high because of contention over ODM_IO_LOCK DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following incidents: Patch ID: VRTSvxfs-6.2.1.8400 * 3972960 (Tracking ID: 3905099) SYMPTOM: VxFS unmount panicked in deactive_super(), the panic stack looks like following: #9 vx_fsnotify_flush [vxfs] #10 vx_softcnt_flush [vxfs] #11 vx_idrop [vxfs] #12 vx_detach_fset [vxfs] #13 vx_unmount [vxfs] #14 generic_shutdown_super #15 kill_block_super #16 vx_kill_sb #17 amf_kill_sb #18 deactivate_super #19 mntput_no_expire #20 sys_umount #21 system_call_fastpath DESCRIPTION: Suspected there is a a race between unmount, and a user-space notifier install for root inode. RESOLUTION: Added diaged code and defensive check for fsnotify_flush in vx_softcnt_flush. * 3977203 (Tracking ID: 3973943) SYMPTOM: VxFS module failed to load on SLES12 SP4 DESCRIPTION: The SLES12 SP4 is new release and it has 4.12 Linux kernel therefore VxFS module failed to load on it. RESOLUTION: Enabled VxFS support for SLES12 SP4 * 3980789 (Tracking ID: 3980754) SYMPTOM: In function vx_io_proxy_thread(), system may hit kernel panic due to general protection fault. DESCRIPTION: In function vx_io_proxy_thread(), a value is being saved into memory through the uninitialized pointer. This may result in memory corruption. RESOLUTION: Function vx_io_proxy_thread() is changed to use the pointer after initializing it. * 3983567 (Tracking ID: 3922986) SYMPTOM: System panic since Linux NMI Watchdog detected LOCKUP in CFS. DESCRIPTION: The vxfs buffer cache iodone routine interrupted the inode flush thread which was trying to acquire the cfs buffer hash lock with releasing the cfs buffer. And the iodone routine was blocked by other threads on acquiring the free list lock. In the cycle, the other threads were contending the cfs buffer hash lock with the inode flush thread. On Linux, the spinlock is FIFO tickets lock, so if the inode flush thread set ticket on the spinlock earlier, other threads cant acquire the lock. This caused a dead lock issue. RESOLUTION: Code changes are made to ensure acquiring the cfs buffer hash lock with irq disabled. * 3983574 (Tracking ID: 3978149) SYMPTOM: When the FIFO file is created on VXFS filesystem, its timestamps are not updated when writes are done to it. DESCRIPTION: In write context, Linux kernel calls update_time inode op in order to update timestamp fields.This op was not implemented in VXFS. RESOLUTION: Implemented update_time inode op in VXFS. * 3986836 (Tracking ID: 3987141) SYMPTOM: Vxcompress operation generates core dumps DESCRIPTION: Vxcompress command fails because thread 1 wants to unlock __thread_lk but fails. (gdb) bt #0 0x00007fb1b4618640 in __lll_unlock_elision () from /lib64/libpthread.so.0 #1 0x0000000000406508 in __job_cleanupHandler (arg=0x0) at jobcontrol.c:414 #2 __job_thread (arg=) at jobcontrol.c:439 #3 0x00007fb1b460e74a in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb1b3c33edd in clone () from /lib64/libc.so.6 RESOLUTION: Fixed this issue by adding trylock calls before unlocking mutexes. Patch ID: VRTSvxfs-6.2.1.600 * 3930621 (Tracking ID: 3927746) SYMPTOM: VxFS module failed to load on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES12 SP3. Patch ID: VRTSvxfs-6.2.1.400 * 3915626 (Tracking ID: 3911966) SYMPTOM: VxFS module failed to load on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES12 SP2. Patch ID: VRTSvxfs-6.2.1.300 * 3817229 (Tracking ID: 3762174) SYMPTOM: When fsfreeze is used together with vxdump, the fsfreeze command gets timeout and vxdump command fails. DESCRIPTION: The vxdump command may try to read mount list file to get information of the corresponding mount points. This behavior results in taking a file system active level, in order to synchronize with file system reinit. But in case of fsfreeze, taking the active level will never succeed, since the file system is already freezed, so this causes a deadlock and finally results in the fsfreeze timeout. RESOLUTION: Don't use fsfreeze and vxdump command together. * 3896150 (Tracking ID: 3833816) SYMPTOM: In a CFS cluster, one node returns stale data. DESCRIPTION: In a 2-node CFS cluster, when node 1 opens the file and writes to it, the locks are used with CFS_MASTERLESS flag set. But when node 2 tries to open the file and write to it, the locks on node 1 are normalized as part of HLOCK revoke. But after the Hlock revoke on node 1, when node 2 takes the PG Lock grant to write, there is no PG lock revoke on node 1, so the dirty pages on node 1 are not flushed and invalidated. The problem results in reads returning stale data on node 1. RESOLUTION: The code is modified to cache the PG lock before normalizing it in vx_hlock_putdata, so that after the normalizing, the cache grant is still with node 1.When node 2 requests PG lock, there is a revoke on node 1 which flushes and invalidates the pages. * 3896151 (Tracking ID: 3827491) SYMPTOM: Data relocation is not executed correctly if the IOTEMP policy is set to AVERAGE. DESCRIPTION: Database table is not created correctly which results in an error on the database query. This affects the relocation policy of data and the files are not relocated properly. RESOLUTION: The code is modified fix the database table creation issue. Therelocation policy based calculations are done correctly. * 3896154 (Tracking ID: 1428611) SYMPTOM: 'vxcompress' command can cause many GLM block lock messages to be sent over the network. This can be observed with 'glmstat -m' output under the section "proxy recv", as shown in the example below - bash-3.2# glmstat -m message all rw g pg h buf oth loop master send: GRANT 194 0 0 0 2 0 192 98 REVOKE 192 0 0 0 0 0 192 96 subtotal 386 0 0 0 2 0 384 194 master recv: LOCK 193 0 0 0 2 0 191 98 RELEASE 192 0 0 0 0 0 192 96 subtotal 385 0 0 0 2 0 383 194 master total 771 0 0 0 4 0 767 388 proxy send: LOCK 98 0 0 0 2 0 96 98 RELEASE 96 0 0 0 0 0 96 96 BLOCK_LOCK 2560 0 0 0 0 2560 0 0 BLOCK_RELEASE 2560 0 0 0 0 2560 0 0 subtotal 5314 0 0 0 2 5120 192 194 DESCRIPTION: 'vxcompress' creates placeholder inodes (called IFEMR inodes) to hold the compressed data of files. After the compression is finished, IFEMR inode exchange their bmap with the original file and later given to inactive processing. Inactive processing truncates the IFEMR extents (original extents of the regular file, which is now compressed) by sending cluster-wide buffer invalidation requests. These invalidations need GLM block lock. Regular file data need not be invalidated across the cluster, thus making these GLM block lock requests unnecessary. RESOLUTION: Pertinent code has been modified to skip the invalidation for the IFEMR inodes created during compression. * 3896156 (Tracking ID: 3633683) SYMPTOM: "top" command output shows vxfs thread consuming high CPU while running an application that makes excessive sync() calls. DESCRIPTION: To process sync() system call vxfs scans through inode cache which is a costly operation. If an user application is issuing excessive sync() calls and there are vxfs file systems mounted, this can make vxfs sync processing thread to consume high CPU. RESOLUTION: Combine all the sync() requests issued in last 60 second into a single request. * 3896160 (Tracking ID: 3808033) SYMPTOM: After a service group is set offline via VOM or VCSOracle process is left in an unkillable state. DESCRIPTION: Whenever ODM issues an async request to FDD, FDD is required to do iodone processing on it, regardless of how far the request gets. The forced unmount causes FDD to take one of the early error branch which misses iodone routine for this particular async request. From ODM's perspective, the request is submitted, but iodone will never be called. This has several bad consequences, one of which is a user thread is blocked uninterruptibly forever, if it waits for request. RESOLUTION: The code is modified to add iodone routine in the error handling code. * 3896223 (Tracking ID: 3735697) SYMPTOM: vxrepquota reports error like, # vxrepquota -u /vx/fs1 UX:vxfs vxrepquota: ERROR: V-3-20002: Cannot access /dev/vx/dsk/sfsdg/fs1:ckpt1: No such file or directory UX:vxfs vxrepquota: ERROR: V-3-24996: Unable to get disk layout version DESCRIPTION: vxrepquota checks each mount point entry in mounted file system table. If any checkpoint mount point entry presents before the mount point specified in the vxrepquota command, vxrepquota will report errors, but the command can succeed. RESOLUTION: Skip checkpoint mount point in the mounted file system table. * 3896231 (Tracking ID: 3708836) SYMPTOM: When using fallocate together with delayed extending write, data corruption may happen. DESCRIPTION: When doing fallocate after EOF, vxfs grows the file by splitting the last extent of the file into two parts, then converts the part after EOF to a ZFOD extent. During this procedure, a stale file size is used to calculate the start offset of the newly zeroed extent. This may overwrite the blocks which contain the unflushed data generated by the extending write and cause data corruption. RESOLUTION: The code is modified to use up-to-date file size instead of the stale file size, to make sure the new ZFOD extent is created correctly. * 3896261 (Tracking ID: 3855726) SYMPTOM: Panic happens in vx_prot_unregister_all(). The stack looks like this: - vx_prot_unregister_all - vxportalclose - __fput - fput - filp_close - sys_close - system_call_fastpath DESCRIPTION: The panic is caused by a NULL fileset pointer, which is due to referencing the fileset before it's loaded, plus, there's a race on fileset identity array. RESOLUTION: Skip the fileset if it's not loaded yet. Add the identity array lock to prevent the possible race. * 3896267 (Tracking ID: 3861271) SYMPTOM: Due to the missing inode clear action, a page can also be in a strange state. Also, inode is not fully quiescent which leads to races in the inode code. Sometime this can cause panic from iput_final(). DESCRIPTION: We're missing an inode clear operation when a Linux inode is being de-initialized on SLES11. RESOLUTION: Add the inode clear operation on SLES11. * 3896269 (Tracking ID: 3879310) SYMPTOM: The file system may get corrupted after the file system freeze during vxupgrade. The full fsck gives following errors: UX:vxfs fsck: ERROR: V-3-20451: No valid device inodes found UX:vxfs fsck: ERROR: V-3-20694: cannot initialize aggregate DESCRIPTION: The vxupgrade requires file system to be frozen during it's functional operation. It may happen that the corruption can be detected while freeze is in progress and full fsck flag can be set on the file system. However, this doesn't stop vxupgrade to proceed. At later stage of vxupgrade, after structures related to new disk layout are updated on disk, vxfs frees up and zeroes out some of the old metadata inodes. If an error occurs after this point (because of full fsck being set), the file system completely needs to go back to previous version, at the tile of full fsck. Since the metadata corresponding to previous version is already cleared, the full fsck cannot proceed and gives error. RESOLUTION: Check for full fsck flag after freezing the file system during vxupgrade. Also, disable the file system if an error occurs after writing new metadata on disk. This will force the newly written metadata to be loaded in memory on next mount. * 3896270 (Tracking ID: 3707662) SYMPTOM: Race between reorg processing and fsadm timer thread (alarm expiry) leads to panic in vx_reorg_emap with the following stack:: vx_iunlock vx_reorg_iunlock_rct_reorg vx_reorg_emap vx_extmap_reorg vx_reorg vx_aioctl_full vx_aioctl_common vx_aioctl vx_ioctl fop_ioctl ioctl DESCRIPTION: When the timer expires (fsadm with -t option), vx_do_close() calls vx_reorg_clear() on local mount which performs cleanup on reorg rct inode. Another thread currently active in vx_reorg_emap() will panic due to null pointer dereference. RESOLUTION: When fop_close is called in alarm handler context, we defer the cleaning up untill the kernel thread performing reorg completes its operation. * 3896273 (Tracking ID: 3558087) SYMPTOM: When stat system call is executed on VxFS File System with delayed allocation feature enabled, it may take long time or it may cause high cpu consumption. DESCRIPTION: When delayed allocation (dalloc) feature is turned on, the flushing process takes much time. The process keeps the get page lock held, and needs writers to keep the inode reader writer lock held. Stat system call may keeps waiting for inode reader writer lock. RESOLUTION: Delayed allocation code is redesigned to keep the get page lock unlocked while flushing. * 3896277 (Tracking ID: 3691633) SYMPTOM: Remove RCQ Full messages DESCRIPTION: Too many unnecessary RCQ Full messages were logging in the system log. RESOLUTION: The RCQ Full messages removed from the code. * 3896281 (Tracking ID: 3830300) SYMPTOM: Heavy cpu usage while oracle archive process are running on a clustered fs. DESCRIPTION: The cause of the poor read performance in this case was due to fragmentation, fragmentation mainly happens when there are multiple archivers running on the same node. The allocation pattern of the oracle archiver processes is 1. write header with O_SYNC 2. ftruncate-up the file to its final size ( a few GBs typically) 3. do lio_listio with 1MB iocbs The problem occurs because all the allocations in this manner go through internal allocations i.e. allocations below file size instead of allocations past the file size. Internal allocations are done at max 8 Pages at once. So if there are multiple processes doing this, they all get these 8 Pages alternately and the fs becomes very fragmented. RESOLUTION: Added a tunable, which will allocate zfod extents when ftruncate tries to increase the size of the file, instead of creating a hole. This will eliminate the allocations internal to file size thus the fragmentation. Fixed the earlier implementation of the same fix, which ran into locking issues. Also fixed the performance issue while writing from secondary node. * 3896285 (Tracking ID: 3757609) SYMPTOM: High CPU usage because of contention over ODM_IO_LOCK DESCRIPTION: While performing ODM IO, to update some of the ODM counters we take ODM_IO_LOCK which leads to contention from multiple of iodones trying to update these counters at the same time. This is results in high CPU usage. RESOLUTION: Code modified to remove the lock contention. * 3896303 (Tracking ID: 3762125) SYMPTOM: Directory size sometimes keeps increasing even though the number of files inside it doesn't increase. DESCRIPTION: This only happens to CFS. A variable in the directory inode structure marks the start of directory free space. But when the directory ownership changes, the variable may become stale, which could cause this issue. RESOLUTION: The code is modified to reset this free space marking variable when there's ownershipchange. Now the space search goes from beginning of the directory inode. * 3896304 (Tracking ID: 3846521) SYMPTOM: cp -p is failing with EINVAL for files with 10 digit modification time. EINVAL error is returned if the value in tv_nsec field is greater than/outside the range of 0 to 999, 999, 999. VxFS supports the update in usec but when copying in the user space, we convert the usec to nsec. So here in this case, usec has crossed the upper boundary limit i.e 999, 999. DESCRIPTION: In a cluster, its possible that time across nodes might differ.so when updating mtime, vxfs check if it's cluster inode and if nodes mtime is newer time than current node time, then accordingly increment the tv_usec instead of changing mtime to older time value. There might be chance that it, tv_usec counter got overflowed here, which resulted in 10 digit mtime.tv_nsec. RESOLUTION: Code is modified to reset usec counter for mtime/atime/ctime when upper boundary limit i.e. 999999 is reached. * 3896306 (Tracking ID: 3790721) SYMPTOM: High CPU usage on the vxfs thread process. The backtrace of such kind of threads usually look like this: schedule schedule_timeout __down down vx_send_bcastgetemapmsg_remaus vx_send_bcastgetemapmsg vx_recv_getemapmsg vx_recvdele vx_msg_recvreq vx_msg_process_thread vx_kthread_init kernel_thread DESCRIPTION: The locking mechanism in vx_send_bcastgetemapmsg_process() is inefficient. So that every time vx_send_bcastgetemapmsg_process() is called, it will perform a series of down-up operation on a certain semaphore. This can result in a huge CPU cost when multiple threads have contention on this semaphore. RESOLUTION: Optimize the locking mechanism in vx_send_bcastgetemapmsg_process(), so that it only do down-up operation on the semaphore once. * 3896308 (Tracking ID: 3695367) SYMPTOM: Unable to remove volume from multi-volume VxFS using "fsvoladm" command. It fails with "Invalid argument" error. DESCRIPTION: Volumes are not being added in the in-core volume list structure correctly. Therefore while removing volume from multi-volume VxFS using "fsvoladm", command fails. RESOLUTION: The code is modified to add volumes in the in-core volume list structure correctly. * 3896310 (Tracking ID: 3859032) SYMPTOM: System panics in vx_tflush_map() due to NULL pointer dereference. DESCRIPTION: When converting VxFS using vxconvert, new blocks are allocated to the structural files like smap etc which can contain garbage. This is done with the expectation that fsck will rebuild the correct smap. but in fsck, we have missed to distinguish between EAU fully EXPANDED and ALLOCATED. because of which, if allocation to the file which has the last allocation from such affected EAU is done, it will create the sub transaction on EAU which are in allocated state. Map buffers of such EAUs are not initialized properly in VxFS private buffer cache, as a result, these buffers will be released back as stale during the transaction commit. Later, if any file-system wide sync tries to flush the metadata, it can refer to these buffer pointers and panic as these buffers are already released and reused. RESOLUTION: Code is modified in fsck to correctly set the state of EAU on disk. Also, modified the involved code paths as to avoid using doing transactions on unexpanded EAUs. * 3896311 (Tracking ID: 3779916) SYMPTOM: vxfsconvert fails to upgrade layout verison for a vxfs file system with large number of inodes. Error message will show some inode discrepancy. DESCRIPTION: vxfsconvert walks through the ilist and converts inode. It stores chunks of inodes in a buffer and process them as a batch. The inode number parameter for this inode buffer is of type unsigned integer. The offset of a particular inode in the ilist is calculated by multiplying the inode number with size of inode structure. For large inode numbers this product of inode_number * inode_size can overflow the unsigned integer limit, thus giving wrong offset within the ilist file. vxfsconvert therefore reads wrong inode and eventually fails. RESOLUTION: The inode number parameter is defined as unsigned long to avoid overflow. * 3896312 (Tracking ID: 3811849) SYMPTOM: On cluster file system (CFS), due to a size mismatch in the cluster-wide buffers containing hash bucket for large directory hashing (LDH), the system panics with the following stack trace: vx_populate_bpdata() vx_getblk_clust() vx_getblk() vx_exh_getblk() vx_exh_get_bucket() vx_exh_lookup() vx_dexh_lookup() vx_dirscan() vx_dirlook() vx_pd_lookup() vx_lookup_pd() vx_lookup() On some platforms, instead of panic, LDH corruption is reported. Full fsck reports some meta-data inconsistencies as displayed in the following sample messages: fileset 999 primary-ilist inode 263 has invalid alternate directory index (fileset 999 attribute-ilist inode 8193), clear index? (ynq)y DESCRIPTION: On a highly fragmented file system with a file system block size of 1K, 2K or 4K, the bucket(s) of an LDH inode, which has a fixed size of 8K, can spread across multiple small extents. Currently in-core allocation for bucket of LDH inode happens in parallel to on-disk allocation, which results in small in-core buffer allocations. Combination of these small in-core allocations will be merged for final in memory representation of LDH inodes bucket. On two Cluster File System (CFS) nodes, this may result in same LDH metadata/bucket represented as in-core buffers of different sizes. This may result in system panic as LDH inodes bucket are passed around the cluster, or this may result in on-disk corruption of LDH inode's buckets, if these buffers are flushed to disk. RESOLUTION: The code is modified to separate the on-disk allocation and in-core buffer initialization in LDH code paths, so that in-core LDH bucket will always be represented by a single 8K buffer. * 3896313 (Tracking ID: 3817734) SYMPTOM: If file system with full fsck flag set is mounted, direct command message is printed to the user to clean the file system with full fsck. DESCRIPTION: When mounting file system with full fsck flag set, mount will fail and a message will be printed to clean the file system with full fsck. This message contains direct command to run, which if run without collecting file system metasave will result in evidences being lost. Also since fsck will remove the file system inconsistencies it may lead to undesired data being lost. RESOLUTION: More generic message is given in error message instead of direct command. * 3896314 (Tracking ID: 3856363) SYMPTOM: vxfs reports mapbad errors in the syslog as below: vxfs: msgcnt 15 mesg 003: V-2-3: vx_mapbad - vx_extfind - /dev/vx/dsk/vgems01/lvems01 file system free extent bitmap in au 0 marked bad. And, full fsck reports following metadata inconsistencies: fileset 999 primary-ilist inode 6 has invalid number of blocks (18446744073709551583) fileset 999 primary-ilist inode 6 failed validation clear? (ynq)n pass2 - checking directory linkage fileset 999 directory 8192 block devid/blknum 0/393216 offset 68 references free inode ino 6 remove entry? (ynq)n fileset 999 primary-ilist inode 8192 contains invalid directory blocks clear? (ynq)n pass3 - checking reference counts fileset 999 primary-ilist inode 5 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 5 clear? (ynq)n fileset 999 primary-ilist inode 8194 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 8194 clear? (ynq)n fileset 999 primary-ilist inode 8195 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 8195 clear? (ynq)n pass4 - checking resource maps DESCRIPTION: While processing the VX_IEZEROEXT extop, VxFS frees the extent without setting VX_TLOGDELFREE flag. Similarly, there are other cases where the flag VX_TLOGDELFREE is not set in the case of the delayed extent free, this could result in mapbad errors and invalid block counts. RESOLUTION: Since the flag VX_TLOGDELFREE need to be set on every extent free, modified to code to discard this flag and treat every extent free as delayed extent free implicitly. * 3901379 (Tracking ID: 3897793) SYMPTOM: Panic happens because of race where the mntlock ID is cleared while mntlock flag still set. DESCRIPTION: Panic happened because of race where mntlockid is null even after mntlock flag is set. Race is between fsadm thread and proc mount show_option thread. The fsadm thread deintialize mntlock id first and then removes mntlock flag. If other thread race with this fsadm thread, then it is possible to have mntlock flag set and mntlock id as a NULL. The fix is to remove flag first and deintialize mntlock id later. RESOLUTION: The code is modified to remove mntlock flag first. * 3903657 (Tracking ID: 3857254) SYMPTOM: Assert failure because of missed flush before taking filesnap of the file. DESCRIPTION: If the delayed extended write on the file is not completed but the snap of the file is taken, then the inode size is not updated correctly. This will trigger internal assert because of incorrect inode size. RESOLUTION: The code is modified to flush the delayed extended write before taking filesnap. * 3904841 (Tracking ID: 3901318) SYMPTOM: VxFS module failed to load on RHEL7.3. DESCRIPTION: Since RHEL7.3 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for RHEL7.3. * 3905056 (Tracking ID: 3879761) SYMPTOM: Performance issue observed due to contention on vxfs spin lock vx_worklist_lk. DESCRIPTION: ODM IOs are performed asynchronously, by queuing the ODM work items to the worker threads. It wakes up more number of worker threads than required after enqueuing the ODM work items which leads to contention of vx_worklist_lk spinlock. RESOLUTION: Modified the code such that, it will wake up one worker thread if only one workitem is enqueued. * 3906148 (Tracking ID: 3894712) SYMPTOM: ACL permissions are not inherited correctly on cluster file system. DESCRIPTION: The ACL counts stored on a directory inode gets reset every time directory inodes ownership is switched between the nodes. When ownership on directory inode comes back to the node, which previously abdicated it, ACL permissions were not getting inherited correctly for the newly created files. RESOLUTION: Modified the source such that the ACLs are inherited correctly. * 3906846 (Tracking ID: 3872202) SYMPTOM: VxFS internal test hits an assert. DESCRIPTION: In page create case VxFS was taking the ipglock twice in a thread, due to which the VxFS test hit the internal assert. RESOLUTION: Removed the ipglock from vx_wb_dio_write(). * 3906961 (Tracking ID: 3891801) SYMPTOM: Internal test hit debug assert. DESCRIPTION: Got an debug assert while creating page in shared page cache for zfod extent which is same as creating for HOLEs, which VxFS don't do. RESOLUTION: Added a check for page creation so that we don't create shared pages for zfod extent. * 3907350 (Tracking ID: 3817734) SYMPTOM: If file system with full fsck flag set is mounted, direct command message is printed to the user to clean the file system with full fsck. DESCRIPTION: When mounting file system with full fsck flag set, mount will fail and a message will be printed to clean the file system with full fsck. This message contains direct command to run, which if run without collecting file system metasave will result in evidences being lost. Also since fsck will remove the file system inconsistencies it may lead to undesired data being lost. RESOLUTION: More generic message is given in error message instead of direct command. * 3907359 (Tracking ID: 3907722) SYMPTOM: kernel BUG at fs/dcache.c:964 DESCRIPTION: There is a race between vxfs dcache pruner and umount thread in RHEL7/SLES12 kernel. This race causes kernel BUG at fs/dcache.c:964. RESOLUTION: Added code to close the race between vxfs dcache pruner and umount thread. Patch ID: VRTSvxfs-6.2.1.200 * 3864963 (Tracking ID: 3853338) SYMPTOM: Files on VxFS are corrupted while running the sequential write workload under high memory pressure. DESCRIPTION: VxFS may miss out writes sometimes under excessive write workload. Corruption occurs because of the race between the writer thread which is doing sequential asynchronous writes and the flusher thread which flushes the in-core dirty pages. Due to an overlapping write, they are serialized over a page lock. Because of an optimization, this lock is released, leading to a small window where the waiting thread could race. RESOLUTION: The code is modified to fix the race by reloading the inode write size after taking the page lock. * 3865600 (Tracking ID: 3865599) SYMPTOM: VxFS module failed to load on SLES12 SP1. DESCRIPTION: Since SLES12 SP1 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES12 SP1. Patch ID: VRTSvxfs-6.2.1.100 * 3753724 (Tracking ID: 3731844) SYMPTOM: umount -r option fails for vxfs 6.2 with error "invalid options" DESCRIPTION: Till 6.2 vxfs did not have a umount helper on linux. We added a helper in 6.2, because of this, each call to linux's umount also gets called to the umount helper binary. Due to this the -r option, which was only handled by the linux native umount, is forwarded to the umount.vxfs helper, which exits while processing the option string becase we don't support readonly remounts. RESOLUTION: To solve this, we've changed the umount.vxfs code to not exit on "-r" option, although we do not support readonly remounts, so if umount -r actually fails and the os umount attempts a readonly remount, the mount.vxfs binary will then exit with an error. This solves the problem of linux's default scripts not working for our fs. * 3754492 (Tracking ID: 3761603) SYMPTOM: Full fsck flag will be set incorrectly at the mount time. DESCRIPTION: There might be possibility that extop processing will be deferred during umount (i.e. in case of crash or disk failure) and will be kept on disk, so that mount can process them. During mount, inode can have multiple extop set. Previously if inode has trim and reorg extop set during mount, we were incorrectly setting fullfsck. This patch avoids this situation. RESOLUTION: Code is modified to avoid such unnecessary setting of fullfsck. * 3756002 (Tracking ID: 3764824) SYMPTOM: Internal cluster file system(CFS) testing hit debug assert DESCRIPTION: Internal debug assert is seen when there is a glm recovery while one of the secondary nodes is doing mount, specifically when glm recovery happens between attaching a file system and mounting file system. RESOLUTION: Code is modified to handle glm reconfiguration issue. * 3765324 (Tracking ID: 3736398) SYMPTOM: Panic in the lazy unmount path during deinit of VxFS-VxVM API. DESCRIPTION: The panic is caused when an exiting thread drops the last reference to a lazy-unmounted VxFS file-system where that fs is the last VxFS mount in the system. The exiting thread does unmount, which then calls into VxVM to de-initialize the private FS-VM API(as it is the last VxFS mounted fs). The function to be called in VxVM is looked-up via the files under /proc, this requires an opening of a file but the exit processing has removed the structs needed by the thread to open a file. RESOLUTION: The solution is to cache the de-init function (vx_fsvm_api_deinit) when the VxFS-VxVM API is initialized, so no function look-up is needed during an unmount. The cached function pointer can then be called during the last unmount bypassing the need to open the file by the exiting thread. * 3765998 (Tracking ID: 3759886) SYMPTOM: In case of nested mount, force umount of parent leaves stale child entry in /etc/mtab even after subsequent umount of child. DESCRIPTION: On rhel6 and sles11, in case of nested mount, if parent mount (say /mnt1) was removed/umounted forcefully, then child mounts (like /mnt1/dir) also get umounted but the "/etc/mtab" entry was not getting updated accordingly for child mount. Previously it was possible to remove such child entries from "/etc/mtab" by using os's umount binary. But from shikra on words, we have added helper umount binary in "/sbin/umount.vxfs". So now os's umount binary will call this helper binary which in turn call vxumount for child umount which will fail since path was not present. Hence mtab entry will not get updated and will show child as mounted. RESOLUTION: Code is modified to update mnttab when ENOENT error is returned by umount() system call. * 3769992 (Tracking ID: 3729158) SYMPTOM: fuser and other commands hang on vxfs file systems. DESCRIPTION: The hang is seen while 2 threads contest for 2 locks -ILOCK and PLOCK. The writeadvise thread owns the ILOCK but is waiting for the PLOCK. The dalloc thread owns the PLOCK and is waiting for the ILOCK. RESOLUTION: Correct order of locking is PLOCK followed by the ILOCK. * 3793241 (Tracking ID: 3793240) SYMPTOM: Vxrestore command dumps core file because of invalid japanese strings. DESCRIPTION: Vxrestore command dumps core file because of invalid characters such as %, $ etc. are present in the japanese strings. RESOLUTION: code is modified to remove the extra characters from the Japanese message strings. * 3798437 (Tracking ID: 3812914) SYMPTOM: On RHEL 6.5 and RHEL 6.4 latest kernel patch, umount(8) system call hangs if an application watches for inode events using inotify(7) APIs. DESCRIPTION: On RHEL 6.5 and RHEL 6.4 latest kernel patch, additional OS counters were added in the super block to track inotify Watches. These new counters were not implemented in VxFS for RHEL6.5/RHEL6.4 kernel. Hence, while doing umount, the operation hangs until the counter in the superblock drops to zero, which would never happen since they are not handled in VxFS. RESOLUTION: The code is modified to handle additional counters added in super block of RHEL6.5/RHEL6.4 latest kernel. * 3808285 (Tracking ID: 3808284) SYMPTOM: fsdedupadm status Japanese text includes strange character. DESCRIPTION: The translation of FAILED string in English is incorrect in Japanese and is "I/O" which stands for The failed I / O.So the translation from English to Japanese is not correct. RESOLUTION: Corrected the translation for "FAILED" string in Japanese. * 3817120 (Tracking ID: 3804400) SYMPTOM: VRTS/bin/cp does not return any error when quota hard limit is reached and partial write is encountered. DESCRIPTION: When quota hard limit is reached, VRTS/bin/cp may encounter a partial write, but it may not return any error to up layer application in such situation. RESOLUTION: Adjust VRTS/bin/cp to detect the partial write caused by quota limit, and return a proper error to up layer application. * 3821688 (Tracking ID: 3821686) SYMPTOM: VxFS module might not get loaded on SLES11 SP4. DESCRIPTION: Since SLES11 SP4 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES11 SP4. Patch ID: VRTSvxfs-6.2.1.000 * 3093833 (Tracking ID: 3093821) SYMPTOM: The system panics due to referring freed super block after the vx_unmount() function errors. DESCRIPTION: In the file system unmount process, once Linux VFS calls in to VxFS specific unmount processing vx_unmount(), it doesn't expect error from this call. So, once the vx_unmount()function returns, Linux frees the file systems corresponding super_block object. But if any error is observed during vx_unmount(), it may leave file system Inodes in vxfs inode cache as it is. Otherwise, when there is no error, the file system inodes are processed and dropped on vx_unmount(). As file systems inodes left in VxFS inode cache would still point to freed super block object, so now when these inodes on Inode cache are cleaned up to free or reuse, they may refer freed super block in certain cases, which might lead to Panic due to NULL pointer de-reference. RESOLUTION: Do not return EIO or ENXIO error from vx_detach_fset() when you unmounts the filesystem. Insted of returning error process, drop inode from the inode cache. * 3657150 (Tracking ID: 3604071) SYMPTOM: With the thin reclaim feature turned on, you can observe high CPU usage on the vxfs thread process. The backtrace of such kind of threads usually look like this: - vx_dalist_getau - vx_recv_bcastgetemapmsg - vx_recvdele - vx_msg_recvreq - vx_msg_process_thread - vx_kthread_init DESCRIPTION: In the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, the locking mechanism is inefficient. Thus every time when this routine is called, it will perform a series of down-up operation on a certain semaphore. This can result in a huge CPU cost when many threads calling the routine in parallel. RESOLUTION: The code is modified to optimize the locking mechanism in the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, so that it only does down-up operation on the semaphore once. * 3657151 (Tracking ID: 3513507) SYMPTOM: Filesystem umount() may hang due to deadlock in kernel with the following stack trace and other threads(denoted as T1): vx_async_waitmsg vx_msg_broadcast vx_cwfa_step vx_cwfreeze_common vx_cwfreeze_all vx_freeze vx_detach_fset vx_unmount generic_shutdown_super sys_umount In the meanwhile, a thread(T2) is stuck at down_read: rwsem_down_failed_common rwsem_down_read_failed call_rwsem_down_read_failed down_read do_page_fault page_fault filldir64 vx_readdir_int mntput_no_expire vx_irwlock2 vx_readdir vfs_readdir sys_getdents64 A thread(T3) is waiting for down_write rwsem_down_failed_common rwsem_down_write_failed call_rwsem_down_write_failed down_write sys_munmap A thread(T4) is doing vx_read? vx_active_common_flush vx_read activate_task vfs_read sys_read system_call_fastpath DESCRIPTION: This is a dead lock which involves multiple threads. When the umount thread(T1) is initiated, it blocks all file operations from coming into the filesystem and waits for those active threads to drop the active levels they have. One of the active thread(T2) hits a pagecache which attempts a rw semaphore of the process memory map structure with read mode. However, this semaphore is held by another thread(T4) with read mode, whilst a write request(T3) is waiting in the queue that prevents other read requests from getting the semaphore. The current read user of the semaphore(T4) is requiring the active level of the file system while holding read semaphore. But the file system is frozen by the first thread(T1). RESOLUTION: The code is modified to adjust the active level to avoid the deadlock. * 3657152 (Tracking ID: 3602322) SYMPTOM: System may panic while flushing the dirty pages of the inode. The following stack traces are observed: vx_iflush_list() vx_workitem_process() vx_worklist_process() vx_worklist_thread() and vx_vn_cache_deinit() vx_inode_deinit vx_ilist_chunkclean() vx_inode_free_list() vx_ifree_scan_list() vx_workitem_process() vx_worklist_process() vx_worklist_thread() DESCRIPTION: Panic may occur due to the synchronization problem between one thread that flushes the inode, and the other thread that frees the chunks that contain the inodes on the freelist. The thread that frees the chunks of inodes on the freelist grabs an inode, and clears/dereference the inode pointer while deinitializing the inode. This may result in the pointer dereference, if the flusher thread is working on the same inode. RESOLUTION: The code is modified to resolve the race condition by taking proper locks on the inode and freelist, whenever a pointer in the inode is dereferenced. If the inode pointer is already de-initialized to NULL, then the flushing is attempted on the next inode. * 3657153 (Tracking ID: 3622323) SYMPTOM: Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics using the fsadm_vxfs(1M) command with the following stack: - vx_irwlock - vx_clust_fset_curused - vx_getcompstats - vx_aioctl_getcompstats - vx_aioctl_common - vx_aioctl - vx_unlocked_ioctl - vx_ioctl - vfs_ioctl - do_vfs_ioctl - sys_ioctl - system_call_fastpath DESCRIPTION: When file system is mounted as read-only, part of the initial setup is skipped, including loading of few internal structures. These structures are referenced while gathering statistics for sharing and/or compression. As a result, panic occurs. RESOLUTION: The code is modified to only allow "fsadm -HS all" to gather sharing and/or compression statistics on read-write file systems. On read-only file systems, this command fails. * 3657156 (Tracking ID: 3604750) SYMPTOM: The kernel loops during the extent re-org with the following stack trace: vx_bmap_enter() vx_reorg_enter_zfod() vx_reorg_emap() vx_extmap_reorg() vx_reorg() vx_aioctl_full() $cold_vx_aioctl_common() vx_aioctl() vx_ioctl() vno_ioctl() ioctl() syscall() DESCRIPTION: The extent re-org minimizes the file system fragmentation. When the re-org request is issued for an inode with a lot of ZFOD extents, it reallocates the extents of the original inode to the re-org inode. During this, the ZFOD extent are preserved and enter the re-org inode in a transaction. If the extent allocated is big, the transaction that enters the ZFOD extents becomes big and returns an error. Even when the transaction is retried the same issue occurs. As a result, the kernel loops during the extent re-org. RESOLUTION: The code is modified to enter the Bmap (block map) of the allocated extent and then perform the ZFOD processing. If you get a committable error during the ZFOD enter, then commit the transaction and continue with the ZFOD enter. * 3657157 (Tracking ID: 3617191) SYMPTOM: Checkpoint creation may take hours. DESCRIPTION: During checkpoint creation, with an inode marked for removal and being overlaid, there may be a downstream clone and VxFS starts pulling all the data. With Oracle it's evident because of temporary files deletion during checkpoint creation. RESOLUTION: The code is modified to selectively pull the data, only if a downstream push inode exists for file. * 3657158 (Tracking ID: 3601943) SYMPTOM: The block map tree of a file is corrupted across the levels, and during truncating, inode for the file may lead to an infinite loop. There are various DESCRIPTION: For files larger than 64G, truncation code first walks through the bmap tree to find the optimal offset from which to begin the truncation. If this truncation falls within corrupted range of the bmap, actual truncation code which relies on binary search to find this offset. As a result, the truncation cannot find the offset, thus it returns empty. The output makes the truncation code to submit dummy transaction, which updates the inode of file with latest ctime, without freeing the extents allocated. RESOLUTION: The truncation code is modified to detect the corruption, mark the inode bad and, mark the file system for full-fsck. The modification makes the truncation possible for full fsck. Next time when it runs, the truncation code is able to throw out the inode and free the extents. * 3657159 (Tracking ID: 3633067) SYMPTOM: While converting from ext3 file system to VxFS using vxfsconvert, it is observed that many inodes are missing. DESCRIPTION: When vxfsconvert(1M) is run on an ext3 file system, it misses an entire block group of inodes. This happens because of an incorrect calculation of block group number of a given inode in border case. The inode which is the last inode for a given block group is calculated to have the correct inode offset, but is calculated to be in the next block group. This causes the entire next block group to be skipped when the code attempts to find the next consecutive inode. RESOLUTION: The code is modified to correct the calculation of block group number. * 3665980 (Tracking ID: 2059611) SYMPTOM: The system panics due to a NULL pointer dereference while flushing the bitmaps to the disk and the following stack trace is displayed: vx_unlockmap+0x10c vx_tflush_map+0x51c vx_fsq_flush+0x504 vx_fsflush_fsq+0x190 vx_workitem_process+0x1c vx_worklist_process+0x2b0 vx_worklist_thread+0x78 DESCRIPTION: The vx_unlockmap() function unlocks a map structure of the file system. If the map is being used, the hold count is incremented. The vx_unlockmap() function attempts to check whether this is an empty mlink doubly linked list. The asynchronous vx_mapiodone routine can change the link at random even though the hold count is zero. RESOLUTION: The code is modified to change the evaluation rule inside the vx_unlockmap() function, so that further evaluation can be skipped over when map hold count is zero. * 3665984 (Tracking ID: 2439261) SYMPTOM: When the vx_fiostats_tunable is changed from zero to non-zero, the system panics with the following stack trace: vx_fiostats_do_update vx_fiostats_update vx_read1 vx_rdwr vno_rw rwuio pread DESCRIPTION: When vx_fiostats_tunable is changed from zero to non-zero, all the incore-inode fiostats attributes are set to NULL. When these attributes are accessed, the system panics due to the NULL pointer dereference. RESOLUTION: The code has been modified to check the file I/O stat attributes are present before dereferencing the pointers. * 3665990 (Tracking ID: 3567027) SYMPTOM: During the File System resize operation, the "fullfsck flag is set with the following message:vxfs: msgcnt 183168 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol file system fullfsck flag set - vx_fs_upgrade_reorg DESCRIPTION: File system resize requires some temporary inodes to swap the old inode and the converted inode. However, before a structural inode ise processed, the "fullfsck flag is set when a failure occurs during the metadata change. The flag is cleared after the swap is successfully completed. If the temporary inode allocation fails, VxFS leaves the fullfsck flag on the disk. However, all temporary inodes can be cleaned up when not being in use, thus these temporary inodes do not result in corruption. RESOLUTION: The code is modified to clear the fullfsck flag if the structural inode conversion cannot create its temporary inode. * 3666007 (Tracking ID: 3594386) SYMPTOM: On installing VxFS 5.1SP1RP4P1HF1 on RHEL6u5, system panic occurs when ftrace(1M) is enabled and multiple files are created when large amount of space is consumed. DESCRIPTION: The system panic occurs when the stack overflow happens through the bio_alloc() function that was initiated through a routine. The routine is where the bio_alloc() function exits VxFS and call through the block layer. RESOLUTION: The code is modified by adding a new handoff routine that would hand off the bio_alloc() task to a new worker thread in case the remaining stack was insufficient to proceed. * 3666008 (Tracking ID: 3616907) SYMPTOM: While performing the garbage collection operation, VxFS causes the non-maskable interrupt (NMI) service to stall. DESCRIPTION: With a highly fragmented Reference Count Table (RCT), when a garbage collection operation is performed, the CPU could be used for a longer duration. The CPU could be busy if a potential entry that could be freed is not identified. RESOLUTION: The code is modified such that the CPU is released after a when it is idle after a specified time interval. * 3666010 (Tracking ID: 3233276) SYMPTOM: On a 40 TB file system, the fsclustadm setprimary command consumes more than 2 minutes for execution. And, the unmount operation consumes more time causing a primary migration. DESCRIPTION: The old primary needs to process the delegated allocation units while migrating from primary to secondary. The inefficient implementation of the allocation unit list is consuming more time while removing the element from the list. As the file system size increases, the allocation unit list also increases, which results in additional migration time. RESOLUTION: The code is modified to process the allocation unit list efficiently. With this modification, the primary migration is completed in 1 second on the 40 TB file system. * 3683470 (Tracking ID: 3682138) SYMPTOM: The VxVM symbols are released before the VxFS modules unload time. DESCRIPTION: On RHEL7, the VxFS module receives or adds VxVM symbol as follows: 1. Get or imports all required VxVM symbols on first access or on first mount. Refcnt/hold for VxVM module is incremented through VEKI on symbol_get request so that module doesnt get unloaded. 2. The symbols (imported in step 1) are used as required. 3. VxFS puts/releases VxVM symbols during the VxFS unload time. Refcnt is decremented/hold is released. As a result, VxVM module unloads until the VxFS module is unloaded. Manual upgrade of VxVM pkg cannot happen until the VxFS modules are unloaded RESOLUTION: The code is modified to release VxVM symbols before the VxFS modules unload time. * 3687679 (Tracking ID: 3685391) SYMPTOM: Execute permissions for a file not honored correctly. DESCRIPTION: The user was able to execute the file regardless of not having the execute permissions. RESOLUTION: The code is modified such that an error is reported when the execute permissions are not applied. * 3697142 (Tracking ID: 3697141) SYMPTOM: Added support for Sles12 DESCRIPTION: Added support for Sles12 RESOLUTION: Added support for Sles12 * 3697966 (Tracking ID: 3697964) SYMPTOM: When a file system is upgraded, the upgraded layout clears the superblock flags (fs_flags). DESCRIPTION: When a file system is upgraded, the new superblock structure gets populated with the field values. Most of these values are inherited from the old superblock. As a result, the fs_flags values are overwritten and the flags such as VX_SINGLEDEV are deleted from the superblock. RESOLUTION: The code is modified to restore the old superblock flags while upgrading the disk layout of a file system. * 3698165 (Tracking ID: 3690078) SYMPTOM: The system panics at vx_dev_strategy() routine with the following stack trace: vx_snap_strategy() vx_logbuf_write() vx_logbuf_io() vx_logbuf_flush() vx_logflush() vx_mapstrategy() vx_snap_strategy() vx_clonemap() vx_unlockmap() vx_holdmap() vx_extmaptran() vx_extmapchange() vx_extprevfind() vx_extentalloc() vx_te_bmap_alloc() vx_bmap_alloc_typed() vx_bmap_alloc() vx_get_alloc() vx_cfs_pagealloc() vx_alloc_getpage() vx_do_getpage() vx_internal_alloc() vx_write_alloc() vx_write1() vx_write_common_slow() vx_write_common() vx_vop_write() vx_writev() vx_naio_write_v2() vfs_writev() DESCRIPTION: The issue was observed due to low handoff limit of vx_extprevfind. RESOLUTION: The code is modified to avoid the stack overflow. * 3706864 (Tracking ID: 3709947) SYMPTOM: The SSD cache fails to go offline due to additional slashes "//" in the dev path of the cache device. DESCRIPTION: The conv_to_bdev() function authenticates whether a path contains rdsk (character device) or dsk (disk device). If rdsk is present, the fs_convto_bspec() function calls the fs_pathcanon() function to remove the additional slashes. In case the path contains dsk (disk device) then VxFS refuses to call the functions. As a result, the additional slashes appear in the final output path. RESOLUTION: The code is modified to enable VxFS to call the correct functions. * 3710794 (Tracking ID: 3710792) SYMPTOM: Unmount fails when the mount point contains a special character (colon). DESCRIPTION: For clones, the pseudo device name must contain <devicepath>:<clone name>. Thus, when using the cloned block device, the umount codepath performs an exclusive processing to unmount the device. Sometimes, the mount point itself can be created using the special character (colon). As a result, the mount point splits at the colon. RESOLUTION: The code is modified to unsplit the mount point if it is not a block device and it is a real mount point that can be searched in the mounted file systems table (mtab). * 3715567 (Tracking ID: 3715566) SYMPTOM: VxFS fails to report an error when the maxlink and nomaxlink options are set for disk layout version (DLV) lower than 10. DESCRIPTION: The maxlink and nomaxlink options allow you to enable and disable the maxlink support feature respectively. The maxlink support feature operates only on DLV version 10. Due to an issue, the maxlink and nomaxlink options wrongly appear in DLV versions lower than 10. However, when selected the options do not take effect. RESOLUTION: The code is modified such that VxFS reports an error when you attempt to set the maxlink and nomaxlink options for DLV version lower than 10. * 3716627 (Tracking ID: 3622326) SYMPTOM: During a checkpoint promote, a file system is marked with a fullfsck flag because the corresponding inode is marked as bad. DESCRIPTION: The issue was observed because VxFS skipped moving the data to clone inode. As a result, the inode is marked bad during a checkpoint promote and as a consequence the file system is marked with a fullfsck flag. RESOLUTION: The code is modified to move the correct data to the clone inode. * 3717895 (Tracking ID: 2919310) SYMPTOM: During stress testing on cluster file system, an assertion failure was hit because of a missing linkage between the directory and the associated attribute inode. DESCRIPTION: As per the designed behavior, the node which owns the inode of the file, receives the request to remove the file from the directory. If the directory has an alternate index (hash directory) present, then in the file remove receive handler, the attribute inode is read from the disk. However, VxFS does not create a linkage between the directory and the corresponding inode, which results in an assert failure. RESOLUTION: The code is modified to set the directory inodes i_dirhash field to attribute inode. This change is exercised while bringing the inode incore during file or directory removal. * 3718542 (Tracking ID: 3269553) SYMPTOM: VxFS returns inappropriate message for read of hole via ODM. DESCRIPTION: Sometimes sparse files containing temp or backup/restore files are created outside the Oracle database. And, Oracle can read these files only using the ODM. As a result, ODM fails with an ENOTSUP error. RESOLUTION: The code is modified to return zeros instead of an error. * 3721458 (Tracking ID: 3721466) SYMPTOM: After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock. DESCRIPTION: The VX_SINGLEDEV flag was introduced in disk layout version 7. The purpose of the flag is to indicate whether a file system resides only on a single device or a volume. When the disk layout is upgraded from version 6 to 7, the flag is not inherited along with the other values since it was not supported in version 6. RESOLUTION: The code is modified to set the VX_SINGLEDEV flag when the disk layout is upgraded from version 6 to 7. * 3726403 (Tracking ID: 3739618) SYMPTOM: sfcache command with the "-i" option may not show filesystem cache statistic periodically. DESCRIPTION: sfcache command with the "-i" option may not show filesystem cache statistic periodically. RESOLUTION: The code is modified to add a loop to print sfcache statistics at the specified interval. * 3727166 (Tracking ID: 3727165) SYMPTOM: Enhance RHEV support for SF devices for identification in Guest DESCRIPTION: We need a provision of assigning serial no to a device while attach. This would be passed on as the serial no of the scsi device to the guest. As a result, scsi inquiry on the device should return the provided serial no in Vendor specific data. RESOLUTION: Corresponding changes for supplying the serial number to the vxfs hook is done * 3729704 (Tracking ID: 3719523) SYMPTOM: 'vxupgrade' does not clear the superblock replica of old layout versions. DESCRIPTION: While upgrading the file system to a new layout version, a new superblock inode is allocated and an extent is allocated for the replica superblock. After writing the new superblock (primary + replica), VxFS frees the extent of the old superblock replica. Now, if the primary superblock corrupts, the full fsck searches for replica to repair the file system. If it finds the replica of old superblock, it restores the file system to the old layout, instead of creating a new one. This behavior is wrong. In order to take the file system to a new version, we should clear the replica of old superblock as part of vxupgrade, so that full fsck won't detect it later. RESOLUTION: Clear the replica of old superblock as part of vxupgrade. * 3733811 (Tracking ID: 3729030) SYMPTOM: The fsdedupschd daemon failed to start on RHEL7. DESCRIPTION: The dedup service daemon failed to start because RHEL 7 changed the service management mechanism. The daemon uses the new systemctl to start and stop the service. For the systemctl to properly start, stop, or query the service, it needs a service definition file under the /usr/lib/systemd/system. RESOLUTION: The code is modified to create the fsdedupschd.service file while installing the VRTSfsadv package. * 3733812 (Tracking ID: 3729030) SYMPTOM: The fsdedupschd daemon failed to start on RHEL7. DESCRIPTION: The dedup service daemon failed to start because RHEL 7 changed the service management mechanism. The daemon uses the new systemctl to start and stop the service. For the systemctl to properly start, stop, or query the service, it needs a service definition file under the /usr/lib/systemd/system. RESOLUTION: The code is modified to create the fsdedupschd.service file while installing the VRTSfsadv package. * 3737330 (Tracking ID: 3737329) SYMPTOM: Added support for RHEL7.1 DESCRIPTION: Added support for RHEL7.1 RESOLUTION: Added support for RHEL7.1 * 3743913 (Tracking ID: 3743912) SYMPTOM: Users could create sub-directories more than 64K for disk layouts having versions lower than 10. DESCRIPTION: In this release, the maxlink feature enables users to create sub-directories larger than 64K.This feature is supported on disk layouts whose versions are higher than or equal to 10. The macro VX_TUNEMAXLINK denotes the maximum limitation on sub-directories. And, its value was changed from 64K to 4 billion. Due to this, users could create more than 64K sub-directories for layout versions < 10 as well, which is undesirable. This fix is applicable only on platforms other than AIX. RESOLUTION: The code is modified such that now you can set the value of sub-directory limitation to 64K for layouts whose versions are lower than 10. * 3744425 (Tracking ID: 3744424) SYMPTOM: Rebundled the fix "Openssl: Common Vulnerability and Exposure (CVE) CVE-2014-3566 (POODLE)" as 6.2.1. DESCRIPTION: VRTSfsadv package uses old versions of OpenSSL which are vulnerable to POODLE(CVE-2014-3566) and Hearbleed(CVE-2014-0160). By upgrading to OpenSSL 0.9.8zc, many security vulnerabilities have been fixed. RESOLUTION: The VRTSfsadv package is built with OpenSSL 0.9.8zc.. * 3745651 (Tracking ID: 3642314) SYMPTOM: Umount operation reports an error code 255 in case of write-back cache. DESCRIPTION: Umount helper binary uses umount or umount2 system calls to unmount VxFS file system. In case of error, these system calls return -1. The umount helper returns this value to the system(OSes) umount binary which interpreted it as 255. RESOLUTION: The code is modified to maintain consistency with the operating systems behavior. In case of umount failure, it must return MOUNT_EX_FAIL which is defined as 32 for RHEL7 and 1 for RHEL6 or SLES11. * 3749727 (Tracking ID: 3750187) SYMPTOM: Internal noise testing hits debug assert. DESCRIPTION: The issue is observed becasue Linux inodes are not initialized properly. Initialization of Linux inodes are done at some VxFS entry points like vx_lookup. In write back replay, an inode is created based on the inode number in log and similar initialization is required here as well. RESOLUTION: The code is modified to have proper initialization of inodes. * 3749776 (Tracking ID: 3637636) SYMPTOM: Cluster File System (CFS) node initialization and protocol upgrade may hang during rolling upgrade with the following stack trace: vx_svar_sleep_unlock() vx_event_wait() vx_async_waitmsg() vx_msg_broadcast() vx_msg_send_join_version() vx_msg_send_join() vx_msg_gab_register() vx_cfs_init() vx_cfs_reg_fsckd() vx_cfsaioctl() vxportalunlockedkioctl() vxportalunlockedioctl() And vx_delay() vx_recv_protocol_upgrade_intent_msg() vx_recv_protocol_upgrade() vx_ctl_process_thread() vx_kthread_init() DESCRIPTION: CFS node initialization waits for the protocol upgrade to complete. Protocol upgrade waits for the flag related to the CFS initialization to be cleared. As the result, the deadlock occurs. RESOLUTION: The code is modified so that the protocol upgrade process does not wait to clear the CFS initialization flag. * 3755796 (Tracking ID: 3756750) SYMPTOM: VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline. DESCRIPTION: When FDD module is unloaded before the cache file system is taken offline, few FDD related structures in the cache file system remains to be free. As a result, memory leak is observed. RESOLUTION: The code is modified such that FDD related structure is not initialized for cache file systems. Patch ID: VRTSveki-6.2.1.7100 * 3987155 (Tracking ID: 3987156) SYMPTOM: VEKI module failed to load on SLES12 SP4. DESCRIPTION: Since SLES12 SP4 is new release therefore VEKI module failed to load on it. RESOLUTION: Added VEKI support for SLES12 SP4. Patch ID: VRTSveki-6.2.1.500 * 3943408 (Tracking ID: 3943409) SYMPTOM: VEKI module failed to load on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 is new release therefore VEKI module failed to load on it. RESOLUTION: Added VEKI support for SLES12 SP3. Patch ID: VRTSveki-6.2.1.400 * 3916373 (Tracking ID: 3913507) SYMPTOM: VEKI module failed to load on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 is new release therefore VEKI module failed to load on it. RESOLUTION: Added VEKI support for SLES12 SP2. Patch ID: VRTSdbac-6.2.1.8200 * 3985907 (Tracking ID: 3969613) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP3. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP4 is now introduced. Patch ID: VRTSdbac-6.2.1.400 * 3943229 (Tracking ID: 3927712) SYMPTOM: Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). DESCRIPTION: Veritas Cluster Server did not support SUSE Linux Enterprise Server versions released after SLES 12 SP2. RESOLUTION: Veritas Cluster Server support for SUSE Linux Enterprise Server 12 SP3 is now introduced. Patch ID: VRTSdbac-6.2.1.300 * 3916355 (Tracking ID: 3913168) SYMPTOM: Veritas Oracle Real Application Cluster does not work with SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2) and is unable to load the vcsmm module . DESCRIPTION: Veritas Oracle Real Application Cluster did not support SUSE Linux Enterprise Server versions released after SLES 12 SP1. RESOLUTION: Veritas Oracle Real Application Cluster support for SUSE Linux Enterprise Server 12 SP2 is now introduced. Patch ID: VRTSamf-6.2.1.9200 * 3985033 (Tracking ID: 3969613) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP3. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP4 is now introduced. Patch ID: VRTSamf-6.2.1.500 * 3930510 (Tracking ID: 3927712) SYMPTOM: Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). DESCRIPTION: Veritas Cluster Server did not support SUSE Linux Enterprise Server versions released after SLES 12 SP2. RESOLUTION: Veritas Cluster Server support for SUSE Linux Enterprise Server 12 SP3 is now introduced. Patch ID: VRTSamf-6.2.1.400 * 3907369 (Tracking ID: 3896875) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP1. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP2 is now introduced. Patch ID: VRTSamf-6.2.1.200 * 3906412 (Tracking ID: 3896877) SYMPTOM: Veritas Infoscale Availability does not support Red Hat Enterprise Linux 7 Update 3(RHEL7.3). DESCRIPTION: Veritas Infoscale Availability does not support Red Hat Enterprise Linux versions later than RHEL7 Update 2. RESOLUTION: Veritas Infoscale Availability support for Red Hat Enterprise Linux 7 Update 3(RHEL7.3) is now introduced. Patch ID: VRTSamf-6.2.1.100 * 3794201 (Tracking ID: 3794154) SYMPTOM: Veritas Cluster Server (VCS) does not support Red Hat Enterprise Linux 6 Update 7 (RHEL6.7). DESCRIPTION: VCS did not support RHEL versions released after RHEL6 Update 6. RESOLUTION: VCS support for Red Hat Enterprise Linux 6 Update 7 (RHEL6.7) is now introduced. Patch ID: VRTSamf-6.2.1.000 * 3661452 (Tracking ID: 3652819) SYMPTOM: VxFS module fails to unload because the Asynchronous Monitoring Framework(AMF) module does not decrement the reference count of VxFS module. DESCRIPTION: Sometimes due to a race between AMF unregister (or get-notification) and unmount of the file systems, the AMF module fails to decrement the reference count of VxFS module. RESOLUTION: The AMF module code is modified to resolve the reference count issue. Patch ID: VRTSvxfen-6.2.1.7300 * 3985032 (Tracking ID: 3969613) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP3. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP4 is now introduced. Patch ID: VRTSvxfen-6.2.1.500 * 3943228 (Tracking ID: 3927712) SYMPTOM: Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). DESCRIPTION: Veritas Cluster Server did not support SUSE Linux Enterprise Server versions released after SLES 12 SP2. RESOLUTION: Veritas Cluster Server support for SUSE Linux Enterprise Server 12 SP3 is now introduced. Patch ID: VRTSvxfen-6.2.1.400 * 3907368 (Tracking ID: 3896875) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP1. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP2 is now introduced. Patch ID: VRTSvxfen-6.2.1.200 * 3912033 (Tracking ID: 2852872) SYMPTOM: Veritas Fencing command "vxfenadm -d" sometimes shows "replaying" RFSM state for some nodes in the cluster. DESCRIPTION: During cluster startup, sometimes fencing RFSM keeps showing "replaying" state for a node, but in fact the node has entered "running" state. RESOLUTION: The code is modified so that now fencing does not show incorrect RFSM state for a node. Patch ID: VRTSvxfen-6.2.1.100 * 3794201 (Tracking ID: 3794154) SYMPTOM: Veritas Cluster Server (VCS) does not support Red Hat Enterprise Linux 6 Update 7 (RHEL6.7). DESCRIPTION: VCS did not support RHEL versions released after RHEL6 Update 6. RESOLUTION: VCS support for Red Hat Enterprise Linux 6 Update 7 (RHEL6.7) is now introduced. Patch ID: VRTSvxfen-6.2.1.000 * 3691204 (Tracking ID: 3691202) SYMPTOM: Sometimes fencing fails to start and reports an error that it was unable to register with majority of coordination points. DESCRIPTION: The failure was observed due to an issue in the fencing startup script where a wildcard expansion in an echo expression produced some undesired value. The undesired value caused the fencing startup failure. RESOLUTION: The fencing code is modified so that the wildcard in the echo expression is does not get expanded. * 3714463 (Tracking ID: 3722178) SYMPTOM: The runlevel setting for VXFEN service changes upon running the rpm --verify command on VXFEN. DESCRIPTION: The unexpected behavior is observed due to an issue in the VXFEN spec file which was turning on the VXFEN service. RESOLUTION: The VXFEN spec file code is modified so that the runlevel settings of VXFEN service are not modified during the verification phase. Patch ID: VRTSgab-6.2.1.9300 * 3984942 (Tracking ID: 3969613) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP3. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP4 is now introduced. Patch ID: VRTSgab-6.2.1.600 * 3943226 (Tracking ID: 3927712) SYMPTOM: Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). DESCRIPTION: Veritas Cluster Server did not support SUSE Linux Enterprise Server versions released after SLES 12 SP2. RESOLUTION: Veritas Cluster Server support for SUSE Linux Enterprise Server 12 SP3 is now introduced. Patch ID: VRTSgab-6.2.1.500 * 3907367 (Tracking ID: 3896875) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP1. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP2 is now introduced. Patch ID: VRTSgab-6.2.1.300 * 3875807 (Tracking ID: 3875805) SYMPTOM: A port on a node receives an I/O fence message when the membership for that port changes. This is caused by some unicast messages being stuck in the GAB receive queue of that port. DESCRIPTION: Under certain rare situation, a few unicast messages which belong to a future generation get stuck in the GAB receive queue of a port. This causes unintended consequences like preventing a RECONFIG message from being delivered to that port. In this case, the port receives an I/O fence message from the GAB to ensure the consistency in membership. RESOLUTION: The code is modified to ensure that the unicast messages belonging to future generation are never left pending in the GAB receive queue of a port. Patch ID: VRTSllt-6.2.1.9600 * 3984828 (Tracking ID: 3969613) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP3. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP4 is now introduced. Patch ID: VRTSllt-6.2.1.900 * 3930509 (Tracking ID: 3927712) SYMPTOM: Veritas Cluster Server (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3). DESCRIPTION: Veritas Cluster Server did not support SUSE Linux Enterprise Server versions released after SLES 12 SP2. RESOLUTION: Veritas Cluster Server support for SUSE Linux Enterprise Server 12 SP3 is now introduced. Patch ID: VRTSllt-6.2.1.800 * 3907366 (Tracking ID: 3896875) SYMPTOM: Veritas Infoscale Availability (VCS) does not support SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2). DESCRIPTION: Veritas Infoscale Availability did not support SUSE Linux Enterprise Server versions released after SLES 12 SP1. RESOLUTION: Veritas Infoscale Availability support for SUSE Linux Enterprise Server 12 SP2 is now introduced. Patch ID: VRTSllt-6.2.1.600 * 3905431 (Tracking ID: 3905430) SYMPTOM: Application IO hangs in case of FSS with LLT over RDMA during heavy data transfer. DESCRIPTION: In case of FSS using LLT over RDMA, sometimes IO may hang because of race conditions in LLT code. RESOLUTION: LLT module is modified to fix the race conditions arising due to heavy load with multiple application threads. Patch ID: VRTSvxvm-6.2.1.8400 * 3914738 (Tracking ID: 3897047) SYMPTOM: Filesystems are not mounted automatically on boot through systemd on RHEL7 and SLES12. DESCRIPTION: When systemd service tries to start all the FS in /etc/fstab, the Veritas Volume Manager (VxVM) volumes are not started since vxconfigd is still not up. The VxVM volumes are started a little bit later in the boot process. Since the volumes are not available, the FS are not mounted automatically at boot. RESOLUTION: Registered the VxVM volumes with UDEV daemon of Linux so that the FS would be mounted when the VxVM volumes are started and discovered by udev. * 3921995 (Tracking ID: 3921994) SYMPTOM: Temporary files such as .bslist .cslist .perm are seen in the directory /var/temp. DESCRIPTION: When ADD and REMOVE operations of disks of a disk group are done between the interval of two backups, a failure in the next backup of the same disk group is observed, which is why the files are left behind in the directory as specified in RESOLUTION: Corrected the syntax errors in the code, to handle the vxconfigbackup issue. * 3944786 (Tracking ID: 3938549) SYMPTOM: Command vxassist -g make vol fails with error: Unexpected kernel error in configuration update for Rhel 7.5. DESCRIPTION: Due to changes in Rhel 7.5 source code the vxassist make volume command failed to create volume and returned with error "Unexpected kernel error in configuration update". RESOLUTION: Changes are done in VxVM code to solve the issue for volume creation. * 3949907 (Tracking ID: 3949954) SYMPTOM: Dumpstack messages are printed when vxio module is loaded for the first time when called blk_register_queue. DESCRIPTION: In RHEL 7.5 a new check was added in kernel code in blk_register_queue where if QUEUE_FLAG_REGISTERED was already set on the queue a dumpstack warning message was printed. In vxvm the flag was already set as the flag got copied from the device queue which was earlier registered by the OS. RESOLUTION: Changes are done in VxVM code to avoid copying of QUEUE_FLAG_REGISTERED fix the dumpstack warnings. * 3956729 (Tracking ID: 3955725) SYMPTOM: Utility to clear "failio" flag on disk after storage connectivity is back. DESCRIPTION: If I/Os to the disks timeout due to some hardware failures like weak Storage Area Network (SAN) cable link or Host Bus Adapter (HBA) failure, VxVM assumes that disk is bad or slow and it sets "failio" flag on the disk. Because of this flag, all the subsequent I/Os fail with the No such device error. After the connectivity is back, the "failio" needs to clear using "vxdisk failio=off". We have come up with a utility "vxcheckfailio" which will clear the "failio" flag for all the disks whose all paths are enabled. RESOLUTION: Code changes are done to add utility "vxcheckfailio" that will clear the "failio" flag on the disks. * 3964554 (Tracking ID: 3954972) SYMPTOM: Retpoline support for VxVM on Sles11 retpoline kernels DESCRIPTION: The Sles11 is new release and it has Retpoline kernel. The VxVM module should be recompiled with retpoline aware GCC to support retpoline kernel. RESOLUTION: Compiled VxVM with retpoline GCC. * 3965707 (Tracking ID: 3960237) SYMPTOM: Below message in seen frequently in dmesg for every I/O. "VxVM vxio 0 Entering voldiskiodone" DESCRIPTION: Due to some bug in the code message is seen constantly in dmesg. RESOLUTION: Code changes are done to remove the message. * 3967033 (Tracking ID: 3953681) SYMPTOM: Data corruption issue is seen when more than one plex of volume is detached. DESCRIPTION: When a plex of volume gets detached, DETACH map gets enabled in the DCO (Data Change Object). The incoming IO's are tracked in DRL (Dirty Region Log) and then asynchronously copied to DETACH map for tracking. If one more plex gets detached then it might happen that some of the new incoming regions are missed in the DETACH map of the previously detached plex. This leads to corruption when the disk comes back and plex resync happens using corrupted DETACH map. RESOLUTION: Code changes are done to correctly track the IO's in the DETACH map of previously detached plex and avoid corruption. * 3967034 (Tracking ID: 3950199) SYMPTOM: System may panic with following stack while DMP(Dynamic Mulitpathing) path restoration: #0 [ffff880c65ea73e0] machine_kexec at ffffffff8103fd6b #1 [ffff880c65ea7440] crash_kexec at ffffffff810d1f02 #2 [ffff880c65ea7510] oops_end at ffffffff8154f070 #3 [ffff880c65ea7540] no_context at ffffffff8105186b #4 [ffff880c65ea7590] __bad_area_nosemaphore at ffffffff81051af5 #5 [ffff880c65ea75e0] bad_area at ffffffff81051c1e #6 [ffff880c65ea7610] __do_page_fault at ffffffff81052443 #7 [ffff880c65ea7730] do_page_fault at ffffffff81550ffe #8 [ffff880c65ea7760] page_fault at ffffffff8154e2f5 [exception RIP: _spin_lock_irqsave+31] RIP: ffffffff8154dccf RSP: ffff880c65ea7818 RFLAGS: 00210046 RAX: 0000000000010000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000200246 RSI: 0000000000000040 RDI: 00000000000000e8 RBP: ffff880c65ea7818 R8: 0000000000000000 R9: ffff8824214ddd00 R10: 0000000000000002 R11: 0000000000000000 R12: ffff88302d2ce400 R13: 0000000000000000 R14: ffff880c65ea79b0 R15: ffff880c65ea79b7 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff880c65ea7820] dmp_open_path at ffffffffa07be2c5 [vxdmp] #10 [ffff880c65ea7980] dmp_restore_node at ffffffffa07f315e [vxdmp] #11 [ffff880c65ea7b00] dmp_revive_paths at ffffffffa07ccee3 [vxdmp] #12 [ffff880c65ea7b40] gendmpopen at ffffffffa07cbc85 [vxdmp] #13 [ffff880c65ea7c10] dmpopen at ffffffffa07cc51d [vxdmp] #14 [ffff880c65ea7c20] dmp_open at ffffffffa07f057b [vxdmp] #15 [ffff880c65ea7c50] __blkdev_get at ffffffff811d7f7e #16 [ffff880c65ea7cb0] blkdev_get at ffffffff811d82a0 #17 [ffff880c65ea7cc0] blkdev_open at ffffffff811d8321 #18 [ffff880c65ea7cf0] __dentry_open at ffffffff81196f22 #19 [ffff880c65ea7d50] nameidata_to_filp at ffffffff81197294 #20 [ffff880c65ea7d70] do_filp_open at ffffffff811ad180 #21 [ffff880c65ea7ee0] do_sys_open at ffffffff81196cc7 #22 [ffff880c65ea7f30] compat_sys_open at ffffffff811eee9a #23 [ffff880c65ea7f40] symev_compat_open at ffffffffa0c9b08f DESCRIPTION: System panic can be encounter due to race condition. There is a possibility that a path picked by DMP restore daemon for processing may be deleted before the restoration process is complete. Hence when the restoration daemon tries to access the path properties it leads to system panic as the path properties are already freed. RESOLUTION: Code changes are done to handle the race condition. * 3967037 (Tracking ID: 3957549) SYMPTOM: Server panicked when resyncing mirror volume with the following stack: voliot_object_event+0x2e0 vol_oes_sio_start+0x80 voliod_iohandle+0x30 voliod_loop+0x248 thread_start+4 DESCRIPTION: In case IO error happened during mirror resync, need to log trace event for the IO error. As the IO is from mirror resync, KIO should be NULL. But NULL pointer check for KIO is missed during logging trace event, hence the panic. RESOLUTION: Code changes have been made to fix the issue. * 3967038 (Tracking ID: 3932356) SYMPTOM: In a two node cluster vxconfigd dumps core while importing the DG - dapriv_da_alloc () in setup_remote_disks () in volasym_remote_getrecs () req_dg_import () vold_process_request () start_thread () from /lib64/libpthread.so.0 from /lib64/libc.so.6 DESCRIPTION: The vxconfigd is dumping core due to address alignment issue. RESOLUTION: The alignment issue is fixed. * 3967070 (Tracking ID: 3966965) SYMPTOM: Reclaiming disks from Infinibox disk array failed. DESCRIPTION: The disk array attributes should be composed as name-value pair separated by '=', but separator '=' was missed, hence disk reclaiming reports wrong attribute and fails. RESOLUTION: Code changes have been made to correct this problem. * 3967117 (Tracking ID: 3948140) SYMPTOM: System may panic if RTPG data returned by the array is greater than 255 with below stack: dmp_alua_get_owner_state() dmp_alua_get_path_state() dmp_get_path_state() dmp_check_path_state() dmp_restore_callback() dmp_process_scsireq() dmp_daemons_loop() DESCRIPTION: The size of the buffer given to RTPG SCSI command is currently 255 bytes. But the size of data returned by underlying array for RTPG can be greater than 255 bytes. As a result incomplete data is retrieved (only the first 255 bytes) and when trying to read the RTPG data, it causes invalid access of memory resulting in error while claiming the devices. This invalid access of memory may lead to system panic. RESOLUTION: The RTPG buffer size has been increased to 1024 bytes for handling this. * 3967121 (Tracking ID: 3946350) SYMPTOM: kmalloc-1024 and kmalloc-2048 memory consuming keeps increasing when Veritas Volume Replicator (VVR) IO size is more than 256K. DESCRIPTION: In case of VVR , if I/O size is more than 256K, then the IO is broken into child IOs. Due to code defect, the allocated space doesn't got freed when splited IOs are completed. RESOLUTION: The code is modified to free VxVM allocated memory after split IOs competed. * 3967130 (Tracking ID: 3927493) SYMPTOM: VVR replication cannot be established between primary and secondary. DESCRIPTION: Through debugging, we found that during vradmind execution, string object might be unexpectedly corrupted (overwritten). As a result, the vradmind connection status cannot be corrected interpreted. Hence, the replication between primary and secondary cannot be established. The reason that string object get corrupted is, vradmind is built as single-thread program; However, during execution, it may (depending on run-time environment) load multi-thread module, which causes behavioral change and corruption of String object. RESOLUTION: Change build option to compile vradmind in thread-safe manner. * 3967132 (Tracking ID: 3925377) SYMPTOM: Not all disks could be discovered by Dynamic Multi-Pathing(DMP) after first startup.. DESCRIPTION: DMP is started too earlier in the boot process if iSCSI and raw haven't been installed. Till that point the FC devices are not recognized by OS, hence DMP misses FC devices. RESOLUTION: The code is modified to make sure DMP get started after OS disk discovery. * 3967135 (Tracking ID: 3918408) SYMPTOM: Data corruption when volume grow is attempted on thin reclaimable disks whose space is just freed. DESCRIPTION: When the space in the volume is freed by deleting some data or subdisks, the corresponding subdisks are marked for reclamation. It might take some time for the periodic reclaim task to start if not issued manually. In the meantime, if same disks are used for growing another volume, it can happen that reclaim task will go ahead and overwrite the data written on the new volume. Because of this race condition between reclaim and volume grow operation, data corruption occurs. RESOLUTION: Code changes are done to handle race condition between reclaim and volume grow operation. Also reclaim is skipped for those disks which have been already become part of new volume. * 3967139 (Tracking ID: 3927439) SYMPTOM: VxVM(Veritas Volume Manager) vxassist relayout command doesn't honor read policy. Read policy will be forcibly reset to SELECT when relayout finishes, regardless of volume's read policy before relayout operation. DESCRIPTION: There're some places in vxassist relayout code that SELECT policy is hard coded. RESOLUTION: Code changes have been made to inherit volume's read policy before relayout operation. * 3967142 (Tracking ID: 3926067) SYMPTOM: In a Campus Cluster environment, vxassist relayout command may fail with following error: VxVM vxassist ERROR V-5-1-13124 Site offline or detached VxVM vxassist ERROR V-5-1-4037 Relayout operation aborted. (20) vxassist convert command also might fail with following error: VxVM vxassist ERROR V-5-1-10128 No complete plex on the site. DESCRIPTION: For vxassist "relayout" and "convert" operations in Campus Cluster environment, VxVM (Veritas Volume Manager) needs to sort the plexes of volume according to sites. When the number of plexes of volumes are greater than 100, the sorting of plexes fail due to a bug in the code. Because of this sorting failure, vxassist relayout/convert operations fail. RESOLUTION: Code changes are done to properly sort the plexes according to site. * 3967147 (Tracking ID: 3936535) SYMPTOM: The IO performance is poor due to frequent cache drops on snapshots configured system. DESCRIPTION: On VxVM snapshots configured system, along with the IO going, DCO map update will happen and it could allocate lots of chucks of pages memory, which triggered kswapd to swap the cache memory out, so cache drops were seen. RESOLUTION: Code changes are done to allocate big size memory for DCO map update without triggering memory swap out. * 3967162 (Tracking ID: 3843740) SYMPTOM: Master node panics with the below stack. voldco_get_dco_offset+0x26e/0x300 [vxio] voldco_initialize_header_30+0x52/0x120 [vxio] voldco_get_toc_entry+0xc3/0x1c0 [vxio] voldco_write_new_map_30+0x298/0x790 [vxio] voldco_write_new_map+0x61a/0x730 [vxio] voldco_write_pervol_maps_instant0x588/0xed0 [vxio] DESCRIPTION: In case of transaction abort with version 30 DCO, detach map is getting incorrectly deleted through one of the code path, leading to the system panic. RESOLUTION: Code changes are done to make sure that such maps do not get deleted. * 3967165 (Tracking ID: 3907618) SYMPTOM: vxdisk resize leads to data corruption on filesystem with MSDOS labelled disk having VxVM sliced format. DESCRIPTION: vxdisk resize changes the geometry on the device if required. When vxdisk resize is in progress, absolute offsets i.e offsets starting from start of the device are used. For MSDOS labelled disk, the full disk is devoted on Slice 4 but not slice 0. Thus when IO is scheduled on the device an extra 32 sectors gets added to the IO which is not required since we are already starting the IO from start of the device. This leads to data corruption since the IO on the device shifted by 32 sectors. RESOLUTION: Code changes have been made to not add 32 sectors to the IO when vxdisk resize is in progress to avoid corruption. * 3967180 (Tracking ID: 3873123) SYMPTOM: When remote disk on node is EFI disk, vold enable fails. And following message get logged, and eventually causing the vxconfigd to go into disabled state: Kernel and on-disk configurations don't match; transactions are disabled. DESCRIPTION: This is becasue one of the cases of EFI remote disk is not properly handled in disk recovery part when vxconfigd is enabled. RESOLUTION: Code changes have been done to set the EFI flag on darec in recovery code * 3967182 (Tracking ID: 3662392) SYMPTOM: In the CVM environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. DESCRIPTION: During the first stage of resize transaction, the master node re-adjusts the disk offsets and public/private partition device numbers. On a slave node, the public/private partition device numbers are not adjusted properly. Because of this, the partition starting offset is are added twice and causes the corruption. The window is small during which public/private partition device numbers are adjusted. If I/O occurs during this window then only corruption is observed. After the resize operation completes its execution, no further corruption will happen. RESOLUTION: The code has been changed to add partition starting offset properly to an I/O on slave node during execution of a resize command. * 3967185 (Tracking ID: 3945115) SYMPTOM: VxVM vxassist relayout command fails for volumes with RAID layout with the following message: VxVM vxassist ERROR V-5-1-2344 Cannot update volume VxVM vxassist ERROR V-5-1-4037 Relayout operation aborted. (7) DESCRIPTION: During relayout operation, the target volume inherits the attributes from original volume. One of those attributes is read policy. In case if the layout of original volume is RAID, it will set RAID read policy. RAID read policy expects the target volume to have appropriate log required for RAID policy. Since the target volume is of different layout, it does not have the log present and hence relayout operation fails. RESOLUTION: Code changes have been made to set the read policy to SELECT for target volumes rather than inheriting it from original volume in case original volume is of RAID layout. * 3967186 (Tracking ID: 3913949) SYMPTOM: The DG import is failing with Split Brain after the system is rebooted or when a storage disturbance is seen. The DG import may fail due to split brain with following messages in syslog: V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm B000F8BF40FF000043042DD4A5 V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm B000F8BF40FF00004003FE9356 DESCRIPTION: When a disk is detached, the SSB ID of the remaining DA and DM records shall be incremented. Unfortunately for some reason, the SSB ID of DA record is only incremented, but the SSB ID of DM record is NOT updated. One probable reason may be because the disks get detached before updating the DM records. RESOLUTION: A work-around option is provided to bypass the SSB checks while importing the DG, the user can import the DG with 'vxdg -o overridessb import ' command if a false split brain happens. For using '-o overridessb', one should confirm that all DA records of the DG are available in ENABLED state and are differing with DM records against SSB by 1. * 3967591 (Tracking ID: 3945019) SYMPTOM: Core Dump observed in ./cvm/scripts/admin/clone/clone_new.tc#7 DESCRIPTION: (gdb) p &((struct darecnfo *)(slave_resp_data[1] + 4))->recs $43 = 0x7fc2240abc94 "_348d" (gdb) p &((struct darecnfo *)(slave_resp_data[1]))->recs $44 = 0x7fc2240abc90 "emc0_348d" This causes improper darec and disk_attrs pointers sent to vde_table_add() and eventually to vol_hash(). But sizeof(int) is added to skip the return value in the response. But when we interpret the response on master, we actually don't skip it, using struct darecnfo. RESOLUTION: Instead of darecnfo, used generic struct da_transfer_info for interpreting the response. It also avoided the multiple declaration of structures for same purpose. * 3972369 (Tracking ID: 3928114) SYMPTOM: Filesystem was corrupted rightly after node joining the cluster in FSS configuration. DESCRIPTION: When rebooting node in FSS environment, corresponding plex would be detached. Subsequent I/O would be marked dirty on corresponding detach map. Due to code error, the offset of detach map was wrongly calculated, hence regions that should be dirtied are not correctly marked. Hence, during the recovery after reboot, dirty regions that should be synced after reattach are not synced, and this result in plexes inconsistency and data corruption. RESOLUTION: Code change has been done to calculate correct offset of detach map. * 3972434 (Tracking ID: 3917636) SYMPTOM: Filesystems from /etc/fstab file are not mounted automatically on boot through systemd on RHEL7 and SLES12. DESCRIPTION: While bootup, when systemd tries to mount using the devices mentioned in /etc/fstab file on the device, the device is not accessible leading to the failure of the mount operation. As the device discovery happens through udev infrastructure, the udev-rules for those devices need to be run when volumes are created so that devices get registered with systemd. In the case udev rules are executed even before the devices in "/dev/vx/dsk" directory are created. Since the devices are not created, devices will not be registered with systemd leading to the failure of mount operation. RESOLUTION: Run "udevadm trigger" to execute all the udev rules once all volumes are created so that devices are registered. * 3972437 (Tracking ID: 3952042) SYMPTOM: dmpevents.log is flooding with below messages: Tue Jul 11 09:28:36.620: Lost 12 DMP I/O statistics records Tue Jul 11 10:05:44.257: Lost 13 DMP I/O statistics records Tue Jul 11 10:10:05.088: Lost 6 DMP I/O statistics records Tue Jul 11 11:28:24.714: Lost 6 DMP I/O statistics records Tue Jul 11 11:46:35.568: Lost 1 DMP I/O statistics records Tue Jul 11 12:04:10.267: Lost 13 DMP I/O statistics records Tue Jul 11 12:04:16.298: Lost 5 DMP I/O statistics records Tue Jul 11 12:44:05.656: Lost 31 DMP I/O statistics records Tue Jul 11 12:44:38.855: Lost 2 DMP I/O statistics records DESCRIPTION: when DMP (Dynamic Multi-Pathing) expand iostat table, DMP allocates a new larger table, replaces the old table with the new one and frees the old one. This increases the possibility of memory fragmentation. RESOLUTION: The code is modified to increase the initial value for iostat table. * 3972978 (Tracking ID: 3965962) SYMPTOM: Following process is seen in the "ps -ef" output /bin/sh - /usr/sbin/auto_recover DESCRIPTION: When there are plexes in the configuration which needs recovery, then if there are events such as master initialization , master takeover , auto recovery gets triggered at the time of slave join. RESOLUTION: Code changes are done, to allow admins to prevent auto recovery at the time of slave join. * 3978569 (Tracking ID: 3959618) SYMPTOM: Compilation/functionality was not working with 4.12 kernel. DESCRIPTION: There are lot of changes related to BIO and SCSI structures because of which the compilation/functionality was not working with 4.12 kernel. RESOLUTION: The diff contains all the changes required to make it functional with 4.12 kernel. * 3982813 (Tracking ID: 3752266) SYMPTOM: During package upgrade/un-installation or while stopping the services using CPI installer, the 'vxfs' or 'vxportal' modules may fail to unload if VVR replication was enabled earlier. DESCRIPTION: If VVR (Veritas Volume Replicator) replication is enabled, VxVM (Veritas Volume Manager) gets a reference count of vxportal device in order to get some information from VxFS. The module reference count is not released even after replication is stopped. Because of this, the vxfs and vxportal modules can not be unloaded while stopping the services or during package un-installation or upgrade. RESOLUTION: Code changes are done in VxVM to properly release the reference count of vxportal module in order to fix the module unload failure. * 3984659 (Tracking ID: 3955979) SYMPTOM: In case of Synchronous mode of replication with TCP , if there are any network related issues, I/O's get hang for upto 15-30 mins. DESCRIPTION: When synchronous replication is used , and if because of some network issues secondary is not being able to send the network ack's to the primary, I/O gets hang on primary waiting for these network ack's. In case of TCP mode we depend on TCP for timeout to happen and then I/O's get drained out, since in this case there is no handling from VVR side, I/O's get hang until TCP triggers its timeout which in normal case happens within 15-30 mins. RESOLUTION: Code changes are done to allow user to set the time for tcp within which timeout should get triggered. * 3984662 (Tracking ID: 3973202) SYMPTOM: A VVR primary node may panic with below stack due to accessing the freed memory: nmcom_throttle_send() nmcom_sender() kthread () kernel_thread() DESCRIPTION: After sending the data to VVR (Veritas Volume Replicator) secondary site, the code was accessing some variables for which the memory was already released due to the data ACK getting processed quite early. This was a rare race condition which may happen due to accessing the freed memory. RESOLUTION: Code changes have been made to avoid the incorrect memory access. * 3984726 (Tracking ID: 3979729) SYMPTOM: System Commands may get hung when access particular directory in the scenario that the volume is instant ready. DESCRIPTION: When the volume is instant ready, IO may get queued but without a chance to be driven until wait for another DRL flush. RESOLUTION: Code changes have been made to avoid such hang situation. * 3984736 (Tracking ID: 3898732) SYMPTOM: Node gets panic with panic string bad mutex. DESCRIPTION: Plex attach or volume mirroring operations were accessing volume device fields incorrectly during a transaction which lead to freed memory access. This resulted in node panic. RESOLUTION: Modified the plex attach/volume mirroring code to properly interlock access to volume devices with transactions. Patch ID: VRTSvxvm-6.2.1.600 * 3941678 (Tracking ID: 3939796) SYMPTOM: Installation of VxVM package fails on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 has many kernel changes, package installation fails. Added code changes to provide SLES12 SP3 platform support for VxVM. RESOLUTION: Added code changes to provide SLES12 SP3 platform support. Patch ID: VRTSvxvm-6.2.1.500 * 3920545 (Tracking ID: 3795739) SYMPTOM: In a split brain scenario, cluster formation takes very long time. DESCRIPTION: In a split brain scenario, the surviving nodes in the cluster try to preempt the keys of nodes leaving the cluster. If the keys have been already preempted by one of the surviving nodes, other surviving nodes will receive UNIT Attention. DMP (Dynamic Multipathing) then retries the preempt command after a delayof 1 second if it receives Unit attention. Cluster formation cannot complete untill PGR keys of all the leaving nodes are removed from all the disks. If the number of disks are very large, the preemption of keys takes a lot of time, leading to the very long time for cluster formation. RESOLUTION: The code is modified to avoid adding delay for first couple of retries when reading PGR keys. This allows faster cluster formation with arrays that clear the Unit Attention condition sooner. * 3922253 (Tracking ID: 3919642) SYMPTOM: IO hang may happen after log owner switch. DESCRIPTION: IOs are partly quiesced during log owner change, result in some updates' log end process lost hence upcoming IO hang. RESOLUTION: Code changes have been mode to quiesce IO completely during log owner switch. * 3922254 (Tracking ID: 3919640) SYMPTOM: When log owner on slave and SRL overflow triggered, IO hang and vxconfigd hang may happen if there're heavy IO load from master node. DESCRIPTION: After SRL(Serialized Replicate Log) overflow, log owner on slave node sent DCM(Data Change Map) active request to master node. Master node process the request when therere IO daemons idle. But all IO daemons are busy in sending and retry sending metadata request to log owner node, as log owner is migrating to DCM mode, the request couldnt be processed, hence the hang. RESOLUTION: Code changes have been mode to fix the issue. * 3922255 (Tracking ID: 3919644) SYMPTOM: When switching log owner in case rlink is DCM(Data Change Map) mode, IO hang may happen when the log owner switches back. DESCRIPTION: In case log owner switch happens in DCM mode, as some RV kernel flag reset is missed. When the log owner switched back after DCM replay, data inconsistent make DCM activated again and IO hang may happen. RESOLUTION: Code changes have been mode to clear stale information during log owner switch. * 3922256 (Tracking ID: 3919641) SYMPTOM: In case of log owner configured on slave node, when pausing rlink from master node and adding IO from log owner node, IO hang on log owner node. DESCRIPTION: Deadlock may happen between Master Pause SIO(Staged IO) and Error Handler SIO. Master pause SIO needs to disconnect rlink, results in RV serialized by invoking error handler SIO. Serialized RV(Replicate Volume) prevents master pause SIO from re-starting. As error handler SIO depends on master pause SIO done to complete so that RV can't get out of serialized state. RESOLUTION: Code changes have been mode to fix the deadlock issue. * 3922257 (Tracking ID: 3919645) SYMPTOM: When switching log owner in case rlink is DCM(Data Change Map) mode, IO hang may happen when the log owner switches back. DESCRIPTION: In case log owner switch happens in DCM mode, as some RV kernel information reset is missed. When the log owner switched back after DCM replay, data inconsistent make DCM activated again and IO hang may happen. RESOLUTION: Code changes have been mode to clear stale information during log owner switch. * 3922258 (Tracking ID: 3919643) SYMPTOM: When switching log owner in case rlink is in SRL(Serialized Replicate Log) flush state, IO hang may happen when the log owner switches back. DESCRIPTION: SRL flush start/end position is stored in RV kernel structure and reset of these positions is missed during log owner switch, so when log owner switched back this node will continue to perform SRL flush operation. As SRL flush has been completed by other node and SRL may contain new updates, readback from SRL may contain invalid updates, SRL flush couldn't continue, hence the hang. RESOLUTION: Code changes have been mode to clear stale information during log owner switch. * 3927482 (Tracking ID: 3895950) SYMPTOM: vxconfigd hang may be observed all of a sudden. The following stack will be seen as part of threadlist: slock() .disable_lock() volopen_isopen_cluster() vol_get_dev() volconfig_ioctl() volsioctl_real() volsioctl() vols_ioctl() rdevioctl() spec_ioctl() vnop_ioctl() vno_ioctl() common_ioctl(??, ??, ??, ??) DESCRIPTION: Some of the critical structures in the code are protected with lock to avoid simultaneous modification. A particular lock structure gets copied to the local stack memory. In this case the structure might have information about the state of the lock and also at the time of copy that lock structure might be in an intermediate state. When function tries to access such type of lock structure, the result could lead to panic or hang since that lock structure might be in some unknown state. RESOLUTION: When we make local copy of the structure, no one is going to modify the new local structure and hence acquiring lock is not required while accessing this copy. Patch ID: VRTSvxvm-6.2.1.400 * 3913126 (Tracking ID: 3910432) SYMPTOM: When a mirror volume is created two log plexes are created by default. DESCRIPTION: - For a mirror/RAID 5 volumes a log plex is reqquired. Due to a bug in the code we are creating two log plex records while creation of a volume. RESOLUTION: Code changes are done to create a single log plex by default. * 3915774 (Tracking ID: 3907800) SYMPTOM: VxVM package installation will fail on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 has lot of kernel changes, package installation fails. Added code changes to provide SLES12 SP2 platform support for VxVM. RESOLUTION: Added code changes to provide SLES12 SP2 platform support. * 3915780 (Tracking ID: 3912672) SYMPTOM: "vxddladm assign names" command results in the ASM disks losing their ownership/permission settings which may affect the Oracle databases DESCRIPTION: The command "vxddladm assign names" calls a function which creates raw and block device nodes and set the user-group ownership/permissions as per the mode stored in in-memory record structure. The in-memory records are not getting created (it is NULL) before going to that function. Hence no setting of permissions after device nodes' creation. RESOLUTION: Code changes are done to make sure that in-memory records of user-group ownership/permissions of each dmpnode from vxdmprawdev file gets created before the function call which creates device nodes and sets permissions on them. * 3915784 (Tracking ID: 3868154) SYMPTOM: When DMP Native Support is set to ON, and if a dmpnode has multiple VGs, 'vxdmpadm native ls' shows incorrect VG entries for dmpnodes. DESCRIPTION: When DMP Native Support is set to ON, multiple VGs can be created on a disk as Linux supports creating VG on a whole disk as well as on a partition of a disk.This possibility was not handled in the code, hence the display of 'vxdmpadm native ls' was getting messed up. RESOLUTION: Code now handles the situation of multiple VGs of a single disk * 3915785 (Tracking ID: 3906544) SYMPTOM: Iostat command is not showing any activity for VxVM device nodes with ODM active. DESCRIPTION: ODM IO is processed by private interface between VxVM and VxFS(Veritas File System), in which only IO sectors is accumulated, but IO count is missed. RESOLUTION: Code changes have been done to accumulate IO count as well. * 3915886 (Tracking ID: 3753345) SYMPTOM: On "SLES12 SP2", VxVM (Veritas Volume Manager) kernel modules may fail to get loaded during package installation. DESCRIPTION: While installing the VxVM package, VxVM kernels modules may fail to get loaded because module dependencies are not correctly updated during package installation. RESOLUTION: Added code fix to ensure VxVM modules get correctly loaded on "SLES12 SP2" OS. * 3916902 (Tracking ID: 3916799) SYMPTOM: Few VxVM [Veritas Volume Manager] and DMP [Veritas Dynamic Multipathing] tunables are not updated after rebooting the system. DESCRIPTION: Few VxVM/DMP tunables require system reboot to happen in order to change their values. However, values of such tunables are NOT changed even after rebooting the system. The problem happens due to failure in reading the tunables file due to some changes in the 4.4 kernel API related to reading a file. RESOLUTION: Added code changes to correctly read the tunables file as per 4.4 kernel API. Patch ID: VRTSvxvm-6.2.1.300 * 3780334 (Tracking ID: 3762580) SYMPTOM: In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath co-exists with DMP (Dynamic Multi-Pathing). The following logs are printed while setting up fencing for the cluster. VXFEN: vxfen_reg_coord_pt: end ret = -1 vxfen_handle_local_config_done: Could not register with a majority of the coordination points. DESCRIPTION: In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the interface used by DMP to send the SCSI commands to block devices does not transfer the data to or from the device. Therefore, the SCSI-3 PR keys do not get registered. RESOLUTION: The code is modified to use SCSI request_queue to send the SCSI commands to the underlying block device. Additional patch is required from EMC to support processing SCSI commands via the request_queue mechanism on EMC PowerPath devices. Please contact EMC for patch details for a specific kernel version. * 3802857 (Tracking ID: 3726110) SYMPTOM: On systems with high number of CPUs, DMP devices may perform considerably slower than OS device paths. DESCRIPTION: In high CPU configuration, I/O statistics related functionality in DMP takes more CPU time because DMP statistics are collected on per CPU basis. This stat collection happens in DMP I/O code path hence it reduces the I/O performance. Because of this, DMP devices perform slower than OS device paths. RESOLUTION: The code is modified to remove some of the stats collection functionality from DMP I/O code path. Along with this, the following tunable need to be turned off: 1. Turn off idle lun probing. #vxdmpadm settune dmp_probe_idle_lun=off 2. Turn off statistic gathering functionality. #vxdmpadm iostat stop Notes: 1. Please apply this patch if system configuration has large number of CPU and if DMP performs considerably slower than OS device paths. For normal systems this issue is not applicable. * 3803497 (Tracking ID: 3802750) SYMPTOM: Once VxVM (Veritas Volume Manager) volume I/O-shipping functionality is turned on, it is not getting disabled even after the user issues the correct command to disable it. DESCRIPTION: VxVM (Veritas Volume Manager) volume I/O-shipping functionality is turned off by default. The following two commands can be used to turn it on and off: vxdg -g <dgname> set ioship=on vxdg -g <dgname> set ioship=off The command to turn off I/O-shipping is not working as intended because I/O-shipping flags are not reset properly. RESOLUTION: The code is modified to correctly reset I/O-shipping flags when the user issues the CLI command. * 3816222 (Tracking ID: 3816219) SYMPTOM: VxDMP (Veritas Dynamic Multi-Pathing) event source daemon (vxesd) keeps reporting a lot of messages in syslog as below: "vxesd: Device sd*(*/*) is changed" DESCRIPTION: The vxesd daemon registers with the UDEV framework and keeps VxDMP up-to-date with devices' status. Due to some change at device, vxesd keeps reporting this kind of change-event listened by udev. VxDMP only cares about "add" and "remove" UDEV events. For UDEV "change" event, we can avoid logging for these events to VxDMP. RESOLUTION: The code is modified to stop logging UDEV change-event related messages in syslog. * 3839293 (Tracking ID: 3776520) SYMPTOM: Filters are not updated properly in lvm.conf file in VxDMP initrd while DMP Native Support is being enabled. As a result, root Logical Volume (LV) is mounted on OS device upon reboot. DESCRIPTION: From LVM version 105, global_filter was introduced as part of lvm.conf file. VxDMP updates initird lvm.conf file with the filters required for DMP Native Support to function. While updating the lvm.conf, VxDMP checks for the filter field to be updated, but ideally we should check for global_filter field to be updated in the latest LVM version. This leads to lvm.conf file not updated with the proper filters. RESOLUTION: The code is modified to properly update global_filter field in lvm.conf file in VxDMP initrd. * 3850478 (Tracking ID: 3850477) SYMPTOM: kmalloc-1024 and kmalloc-2048 memory consuming keeps increasing when reading or writing data against VxVM volume with big block size DESCRIPTION: In case the incoming I/O size is too big for a disk to handle, VxVM splits it into smaller ones to move forward. VxVM then allocates memory to backup those splited I/Os. Due to code issue, the allocated space is not freed when I/O splitting completes. RESOLUTION: The code is modified to free VxVM allocated memory after I/O splitting completes. * 3851117 (Tracking ID: 3662392) SYMPTOM: In the CVM environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. DESCRIPTION: During the first stage of resize transaction, the master node re-adjusts the disk offsets and public/private partition device numbers. On a slave node, the public/private partition device numbers are not adjusted properly. Because of this, the partition starting offset is are added twice and causes the corruption. The window is small during which public/private partition device numbers are adjusted. If I/O occurs during this window then only corruption is observed. After the resize operation completes its execution, no further corruption will happen. RESOLUTION: The code has been changed to add partition starting offset properly to an I/O on slave node during execution of a resize command. * 3852148 (Tracking ID: 3852146) SYMPTOM: In a CVM cluster, when importing a shared diskgroup specifying both -c and -o noreonline options, the following error may be returned: VxVM vxdg ERROR V-5-1-10978 Disk group : import failed: Disk for disk group not found. DESCRIPTION: The -c option will update the disk ID and disk group ID on the private region of the disks in the disk group being imported. Such updated information is not yet seen by the slave because the disks have not been re-onlined (given that noreonline option is specified). As a result, the slave cannot identify the disk(s) based on the updated information sent from the master, causing the import to fail with the error Disk for disk group not found. RESOLUTION: The code is modified to handle the working of the "-c" and "-o noreonline" options together. * 3854788 (Tracking ID: 3783356) SYMPTOM: After DMP module fails to load, dmp_idle_vector is not NULL. DESCRIPTION: After DMP module load failure, DMP resources are not cleared off from the system memory, so some of the resources are in NON-NULL value. When system retries to load, it frees invalid data, leading to system panic with error message BAD FREE, because the data being freed is not valid at that point. RESOLUTION: The code is modified to clear up the DMP resources when module failure happens. * 3863971 (Tracking ID: 3736502) SYMPTOM: When FMR is configured in VVR environment, 'vxsnap refresh' fails with below error message: "VxVM VVR vxsnap ERROR V-5-1-10128 DCO experienced IO errors during the operation. Re-run the operation after ensuring that DCO is accessible". Also, multiple messages of connection/disconnection of replication link(rlink) are seen. DESCRIPTION: Inherently triggered rlink connection/disconnection causes the transaction retries. During transaction, memory is allocated for Data Change Object(DCO) maps and is not cleared on abortion of a transaction. This leads to a problem of memory leak and eventually to exhaustion of maps. RESOLUTION: The fix has been added to clear the allocated DCO maps when transaction aborts. * 3868653 (Tracking ID: 3866051) SYMPTOM: After we load a driver with name over 32 bytes in kernel, we will not be able to restart vxconfigd. DESCRIPTION: In Kernel, if we have any driver with name over 32 bytes AND when we restart vxconfigd. Then due to a defect in our code about size of driver name we accept, the process stack will be corrupted. Hence, vxconfigd becomes unable to startup. RESOLUTION: Code changes are made to fix the memory corruption issue. * 3871040 (Tracking ID: 3868444) SYMPTOM: Disk header timestamp is updated even if the disk group import fails. DESCRIPTION: While doing dg import operation, during join operation disk header timestamps are updated. This makes difficult for support to understand which disk is having latest config copy if dg import is failed and decision is to be made if force dg import is safe or not. RESOLUTION: Dump the old disk header timestamp and sequence number in the syslog which can be referred on deciding if force dg import would be safe or not * 3871124 (Tracking ID: 3823283) SYMPTOM: Linux operating system sticks in grub after reboot. Manual kernel load is required to make operating system functional. DESCRIPTION: During unencapsulation of a boot disk in SAN environment, multiple entries corresponding to root disk are found in by-id device directory. As a result, a parse command fails, leading to the creation of an improper menu file in grub directory. This menu file defines the device path to load kernel and other modules. RESOLUTION: The code is modified to handle multiple entries for SAN boot disk. * 3873145 (Tracking ID: 3872197) SYMPTOM: vxconfigd panics when NVME devices are attached to the system with the following stack: panic+0xa7/0x16f oops_end+0xe4/0x100 no_context+0xfb/0x260 __bad_area_nosemaphore+0x125/0x1e0 bad_area+0x4e/0x60 __do_page_fault+0x473/0x500 dmp_rel_shared_lock+0x20/0x30 [vxdmp] dmp_send_scsipkt+0xd8/0x120 [vxdmp] do_page_fault+0x3e/0xa0 page_fault+0x25/0x30 elv_may_queue+0xd/0x20 get_request+0x49/0x3c0 get_request_wait+0x2a/0x1d0 swiotlb_map_sg_attrs+0x79/0x130 blk_get_request+0x46/0xa0 dmp_kernel_scsi_ioctl+0x11d/0x3a0 [vxdmp] dmp_scsi_ioctl+0xae/0x2a0 [vxdmp] __wake_up+0x53/0x70 dmp_send_scsireq+0x5f/0xc0 [vxdmp] dmp_do_scsi_gen+0xab/0x1b0 [vxdmp] dmp_pr_check_aptpl+0xcd/0x150 [vxdmp] dmp_make_mp_node+0x239/0x280 [vxdmp] dmp_decode_add_disk+0x816/0x1110 [vxdmp] dmp_decipher_instructions+0x270/0x350 [vxdmp] dmp_process_instruction_buffer+0x1be/0x1d0 [vxdmp] dmp_reconfigure_db+0x6e/0xf0 [vxdmp] gendmpioctl+0x2c2/0x610 [vxdmp] dmpioctl+0x35/0x70 [vxdmp] dmp_ioctl+0x2b/0x50 [vxdmp] dmp_compat_ioctl+0x56/0x70 [vxdmp] DESCRIPTION: When vxconfigd is started it tries to send a SGIO IOCTL to send a SCSI command to NVME devices using request queue mechanism. The NVME devices does not have elevator set leading to the failure of SGIO command leading to panic of the system. RESOLUTION: Code changes have been done to bypass the SGIO command for NVME devices. * 3874737 (Tracking ID: 3874387) SYMPTOM: Disk header information is not logged to the syslog sometimes even if the disk is missing and dg import fails. DESCRIPTION: In scenarios where disk has config copy enabled and get active disk record, then disk header information was not getting logged even though the disk is missing thereafter dg import fails. RESOLUTION: Dump the disk header information even if the disk record is active and attached to the disk group. * 3875933 (Tracking ID: 3737585) SYMPTOM: Customer encounters following "Uncorrectable write error or below panic in VVR environment: [000D50D0]xmfree+000050 [051221A0]vol_tbmemfree+0000B0 [05122294]vol_memfreesio_start+00001C [05131324]voliod_iohandle+000050 [05131740]voliod_loop+0002D0 [05126438]vol_kernel_thread_init+000024 DESCRIPTION: IOHINT structure allocated from VxFS is also freed by VxFS after IO done from VxVM. IOs to VxVM with VVR needs 2 phases, SRL(Serial Replication Log) write and Data volume write, VxFS gets IO done after SRL write and doesnt wait for Data Volume write completion, so if Data volume write gets started or done after VxFS frees IOHINT, it may cause write IO error or double free panic due to freeing memory wrongly as per corrupted IOHINT info. RESOLUTION: Code changes were done to clone the IOHINT structure before writing to data volume. * 3877637 (Tracking ID: 3878030) SYMPTOM: Enhance VxVM(Veritas Volume Manager) DR(Dynamic Reconfiguration) tool to clean up OS and VxDMP(Veritas Dynamic Multi-Pathing) device trees without user interaction. DESCRIPTION: When users add or remove LUNs, stale entries in OS or VxDMP device trees can prevent VxVM from discovering changed LUNs correctly. It even causes VxVM vxconfigd process core dump under certain conditions, users have to reboot system to let vxconfigd restart again. VxVM has DR tool to help users adding or removing LUNs properly but it requires user inputs during operations. RESOLUTION: Enhancement has been done to VxVM DR tool. It accepts '-o refresh' option to clean up OS and VxDMP device trees without user interaction. * 3879334 (Tracking ID: 3879324) SYMPTOM: VxVM(Veritas Volume Manager) DR(Dynamic Reconfiguration) tool fails to handle busy device problem while LUNs are removed from OS DESCRIPTION: OS devices may still be busy after removing them from OS, it fails 'luxadm - e offline ' operation and leaves staled entries in 'vxdisk list' output like: emc0_65535 auto - - error emc0_65536 auto - - error RESOLUTION: Code changes have been done to address busy devices issue. * 3880573 (Tracking ID: 3886153) SYMPTOM: In a VVR primary-primary configuration, if 'vrstat' command is runing, vradmind core dump may occur with the stack like below: __assert_c99 StatsSession::sessionInitReq StatsSession::processOpReq StatsSession::processOpMsgs RDS::processStatsOpMsg DBMgr::processStatsOpMsg process_message DESCRIPTION: vrstat command initiates StatSession which need to send initilization request to secondary. On Secondary there is assert() to ensure it's secondary that processing the request. In primary-primary configuration it leads to core dump. RESOLUTION: The code changes have been made to fix the issue by returning failure to StatSession initiator. * 3881334 (Tracking ID: 3864063) SYMPTOM: Application I/O hangs after the Master Pause command is issued. DESCRIPTION: Some flags (VOL_RIFLAG_DISCONNECTING or VOL_RIFLAG_REQUEST_PENDING) in VVR (Veritas Volume Replicator) kernel are not cleared because of a race between the Master Pause SIO and the Error Handler SIO. This causes the RU (Replication Update) SIO to fail to proceed, which leads to I/O hang. RESOLUTION: The code is modified to handle the race condition. * 3881335 (Tracking ID: 3867236) SYMPTOM: Application IO hang happens after issuing Master Pause command. DESCRIPTION: The flag VOL_RIFLAG_REQUEST_PENDING in VVR(Veritas Volume Replicator) kernel is not cleared because of a race between Master Pause SIO and RVWRITE1 SIO resulting in RU (Replication Update) SIO to fail to proceed thereby causing IO hang. RESOLUTION: Code changes have been made to handle the race condition. * 3889284 (Tracking ID: 3878153) SYMPTOM: VVR (Veritas Volume Replicator) 'vradmind' deamon core dump in following stack. #0 __kernel_vsyscall () #1 raise () from /lib/libc.so.6 #2 abort () from /lib/libc.so.6 #3 __libc_message () from /lib/libc.so.6 #4 malloc_printerr () from /lib/libc.so.6 #5 _int_free () from /lib/libc.so.6 #6 free () from /lib/libc.so.6 #7 operator delete(void*) () from /usr/lib/libstdc++.so.6 #8 operator delete[](void*) () from /usr/lib/libstdc++.so.6 #9 inIpmHandle::~IpmHandle (this=0x838a1d8, __in_chrg=) at Ipm.C:2946 #10 IpmHandle::events (handlesp=0x838ee80, vlistsp=0x838e5b0, ms=100) at Ipm.C:644 #11 main (argc=1, argv=0xffffd3d4) at srvmd.C:703 DESCRIPTION: Under certain circumstances 'vradmind' daemon may core dump freeing a variable allocated in stack. RESOLUTION: Code change has been done to address the issue. * 3889850 (Tracking ID: 3878911) SYMPTOM: QLogic driver returns following error due to Incorrect aiusize in FC header FC_ELS_MALFORMED, cnt=c60h, size=314h DESCRIPTION: When creating CT pass-through command to be sent, the ct_aiusize we specify in request header does not conform to FT standard. Hence during the sanity check of FT header in OS layer, it reports error and get_topology() failed. RESOLUTION: Code changes have been done so that ct_aiusize is in compliance with FT standard. * 3891789 (Tracking ID: 3873625) SYMPTOM: System panicked when pulling out FC cables on SFHA6.2.1/RHEL7.2, the stack trace of the panic is like following: #8 [ffff880fecb23a90] page_fault at ffffffff8163d408 [exception RIP: blk_rq_map_kern+31] RIP: ffffffff812cfd8f RSP: ffff880fecb23b48 RFLAGS: 00010296 RAX: ffffffffffffffed RBX: ffff880fcf847230 RCX: 0000000000001010 RDX: 0000000000001010 RSI: ffff880fd10d2000 RDI: ffff880fcf847230 RBP: ffff880fecb23b70 R8: 0000000000000010 R9: ffff880fcf7a5b40 R10: ffff88080f803b00 R11: 0000000000000001 R12: 0000000000000000 R13: ffffffffffffffed R14: ffffffffffffffed R15: ffff8807fcfd5b00 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff880fecb23b78] dmp_kernel_scsi_ioctl at ffffffffa0a4e499 [vxdmp] #10 [ffff880fecb23bb8] dmp_scsi_ioctl at ffffffffa0a908a9 [vxdmp] #11 [ffff880fecb23c50] dmp_send_scsireq at ffffffffa0a91580 [vxdmp] #12 [ffff880fecb23c70] dmp_do_scsi_gen at ffffffffa0a868b4 [vxdmp] #13 [ffff880fecb23c98] dmp_pr_send_cmd at ffffffffa0a8eec3 [vxdmp] #14 [ffff880fecb23d30] dmp_pr_do_read at ffffffffa0a61d30 [vxdmp] #15 [ffff880fecb23da8] dmp_def_get_reservation at ffffffffa0a63284 [vxdmp] #16 [ffff880fecb23db8] dmp_pgr_read at ffffffffa0a8d3c4 [vxdmp] #17 [ffff880fecb23df0] gendmpioctl at ffffffffa0a5abf3 [vxdmp] #18 [ffff880fecb23e18] dmpioctl at ffffffffa0a5b181 [vxdmp] #19 [ffff880fecb23e30] dmp_ioctl at ffffffffa0a811aa [vxdmp] #20 [ffff880fecb23e50] blkdev_ioctl at ffffffff812d8da3 #21 [ffff880fecb23ea8] block_ioctl at ffffffff81219701 #22 [ffff880fecb23eb8] do_vfs_ioctl at ffffffff811f1ef5 #23 [ffff880fecb23f30] sys_ioctl at ffffffff811f2171 #24 [ffff880fecb23f80] system_call_fastpath at ffffffff81645909 DESCRIPTION: Linux kernel function __get_request() may fail under memory pressure and returns negative value, DMP doesn't check this and dereference it as a valid req pointer, hence the panic. RESOLUTION: Modified the code to check if the req is valid or error code. * 3893134 (Tracking ID: 3864318) SYMPTOM: Memory consuming keeps increasing when reading/writing data against VxVM volume with big block size. DESCRIPTION: In case incoming IO size is too big for a disk to handle, VxVM will split it into smaller ones to move forward. VxVM will allocate memory to backup the those split IOs. Due to code defect, the allocated space doesn't got freed when split IOs are completed. RESOLUTION: The code is modified to free VxVM allocated memory after split IOs competed. * 3893362 (Tracking ID: 3881132) SYMPTOM: vxcommands hangs following the SAN change. OS device handles and DMP devices are not cleaned up, causing kernel core dump. DESCRIPTION: Prior to discovering NEW LUNs, we need to check the PQ values of the OS device handles and deletes them when non-zero. OS and VM Device Trees should be in-Sync after adding/removing Luns. Devices were not getting cleaned up. Fix is to clean-up device-tree to make them in-sync. RESOLUTION: Code changes made in DMPDR tool, which will delete stale OS device handles to fix the issue. * 3894783 (Tracking ID: 3628743) SYMPTOM: On Solaris 11.2, New boot environment takes long time to start up during live upgrade. Here deadlock is seen in ndi_devi_enter( ), when loading VxDMP driver and Deadlocks caused by VXVM drivers due to use of Solaris ddi_pathname_to_dev_t or ddi_hold_devi_by_path private interfaces. DESCRIPTION: Here deadlocks caused by VXVM drivers due to use of Solaris ddi_pathname_to_dev_t or e_ddi_hold_devi_by_path private interface and ddi_pathname_to_dev_t/e_ddi_hold_devi_by_path are Solaris internal use only routine and is not multi-thread safe. Normally this is not a problem as the various VXVM drivers don't unload or detach, however there are certain conditions where our _init routines might be called which can expose this deadlock condition. RESOLUTION: Code is modified to resolve deadlock. * 3897764 (Tracking ID: 3741003) SYMPTOM: In CVM (Cluster Volume Manager) environment, after removing storage from one of multiple plex in a mirrored DCO volume, the DCO volume is detached and DCO object is having BADLOG flag marked. DESCRIPTION: When one plexs storage of a mirrored volume is removed, only that plex should be detached instead of entire volume. While IO reading is undergoing on a failed DCO plex, the local failed IO gets restarted and shiped to other nodes for retry, which also gets failed, since the storage is removed from other nodes as well. Because a flag reset is missing, the failed IO returns error result in entire volume is detached and marked as BADLOG flag even though the IO is successful from an alternate plex. RESOLUTION: Code changes are added to handle this case for the Resiliency of VxVM in Partial-Storage outage scenario. * 3898129 (Tracking ID: 3790136) SYMPTOM: File system hang can be observed sometimes due to IO's hung in DRL. DESCRIPTION: There might be some IO's hung in DRL of mirrored volume due to incorrect calculation of outstanding IO's on volume and number of active IO's which are currently in progress on DRL. The value of the outstanding IO on volume can get modified incorrectly leading to IO's on DRL not to progress further which in turns results in a hang kind of scenario. RESOLUTION: Code changes have been done to avoid incorrect modification of value of outstanding IO's on volume and prevent the hang. * 3898168 (Tracking ID: 3739933) SYMPTOM: VxVM package installation fails when Linux-server is having EFI support enabled. DESCRIPTION: In LINUX, VxVM install scripts assumes GRUB bootloader in BIOS mode, and tries to locate corresponding grub config file. In case system has GRUB bootloader in EFI mode, VxVM fails to locate required grub config file and so installation gets aborted. RESOLUTION: Code changes added to allow VxVM installation on LINUX machine, where EFI support is Enabled. * 3898169 (Tracking ID: 3740730) SYMPTOM: While creating volume using vxassist CLI, dco-log length specified as command line parameter was not getting honored. E.g. -> bash # vxassist -g make logtype=dco dcolen= VxVM vxassist ERROR V-5-1-16707 Specified dcologlen() is less than minimum dcologlen(17152) DESCRIPTION: While creating volume, using dcologlength attribute of dco-volume in vxassist CLI, the size of dcolog specified is not correctly parsed in the code, because of which it internally compares the size with incorrectly calculated size & throws the error indicating that size specified isn't sufficient. So the values in comparison was incorrect. Hence changed the code to compare the user-specified value passes the minimum-threshold value or not. RESOLUTION: The code is changed to Fix the issue, which honors the length value of dcolog volume specified by user in vxassist CLI. * 3898296 (Tracking ID: 3767531) SYMPTOM: In Layered volume layout with FSS configuration, when few of the FSS_Hosts are rebooted, Full resync is happening for non-affected disks on master. DESCRIPTION: In configuration, where there are multiple FSS-Hosts, with layered volume created on the hosts. When the slave nodes are rebooted , few of the sub-volumes of non-affected disks are fully getting synced on master. RESOLUTION: Code-changes have been made to sync only needed part of sub- volume. * 3902626 (Tracking ID: 3795739) SYMPTOM: In a split brain scenario, cluster formation takes very long time. DESCRIPTION: In a split brain scenario, the surviving nodes in the cluster try to preempt the keys of nodes leaving the cluster. If the keys have been already preempted by one of the surviving nodes, other surviving nodes will receive UNIT Attention. DMP (Dynamic Multipathing) then retries the preempt command after a delayof 1 second if it receives Unit attention. Cluster formation cannot complete untill PGR keys of all the leaving nodes are removed from all the disks. If the number of disks are very large, the preemption of keys takes a lot of time, leading to the very long time for cluster formation. RESOLUTION: The code is modified to avoid adding delay for first couple of retries when reading PGR keys. This allows faster cluster formation with arrays that clear the Unit Attention condition sooner. * 3903647 (Tracking ID: 3868934) SYMPTOM: System panic in the stack like below, while deactivate the VVR(VERITAS Volume Replicator) batch write SIO: panic_trap+000000 vol_cmn_err+000194 vol_rv_inactive+000090 vol_rv_batch_write_start+001378 voliod_iohandle+000050 voliod_loop+0002D0 vol_kernel_thread_init+000024 DESCRIPTION: When VVR do batch write SIO, if it fails to reserve VVR IO memory, the SIO will be put on queue for restart and then will be deactivated. If the deactivation blocked for some time since cannot get the lock, and during this period, the SIO is restarted due to the IO memory reservation request satisfied, the SIO would be corrupted due to be deactivated twice. Hence it causes the system panic. RESOLUTION: Code changes have been made to remove the unnecessary SIO deactivation after VVR IO memory reservation failed. * 3904790 (Tracking ID: 3795788) SYMPTOM: Performance degradation is seen when many application sessions open the same data file on Veritas Volume Manager (VxVM) volume. DESCRIPTION: This issue occurs because of the lock contention. When many application sessions open the same data file on the VxVM volume, the exclusive lock is occupied on all CPUs. If there are a lot of CPUs in the system, this process could be quite time- consuming, which leads to performance degradation at the initial start of applications. RESOLUTION: The code is modified to change the exclusive lock to the shared lock when the data file on the volume is open. * 3904796 (Tracking ID: 3853049) SYMPTOM: On a server with more number CPUs, the stats output of vxstat is delayed beyond the set interval. Also multiple sessions of vxstat is impacting the IO performance. DESCRIPTION: The vxstat acquires an exclusive lock on each CPU in order to gather the stats. This would affect the consolidation and display of stats in an environment with huge number of CPUs and disks. The output of stats for interval of 1 second can get delayed beyond the set interval. Also the acquisition of lock happens in the IO path which would affect the IO performance due contention of these locks. RESOLUTION: The code modified to remove the exclusive spin lock. * 3904797 (Tracking ID: 3857120) SYMPTOM: If the volume is shared in a CVM configuration, the following stack traces will be seen under a vxiod daemon suggesting an attempt to drain I/O. In this case, CVM halt will be blocked and eventually time out. The stack trace may appear as: sleep+0x3f0 vxvm_delay+0xe0 volcvm_iodrain_dg+0x150 volcvmdg_abort_complete+0x200 volcvm_abort_sio_start+0x140 voliod_iohandle+0x80 or cv_wait+0x3c() delay_common+0x6c vol_mv_close_check+0x68 vol_close_device+0x1e4 vxioclose+0x24 spec_close+0x14c fop_close+0x8c closef2+0x11c closeall+0x3c proc_exit+0x46c exit+8 post_syscall+0x42c syscall_trap+0x188 Since the vxconfigd would be busy in transaction of trying to close the volume or drain the IO then all the other threads which send a request to vxconfigd will hang. DESCRIPTION: VxVM maintains an I/O count of the in-progress I/O on the volume. When two threads from VxVM asynchronously manipulate the I/O count on the volume, the race between these threads might lead to stale I/O count remaining on the volume even though the volume has actually completed all I/Os . Since there is an invalid pending I/O count on the volume due to the race condition, the volume cannot be closed. RESOLUTION: This issue has been fixed in the VxVM code manipulating the I/O count to avoid the race condition between the two threads. * 3904800 (Tracking ID: 3860503) SYMPTOM: Poor performance of vxassist mirroring is observed compared to using raw dd utility to do mirroring . DESCRIPTION: There is huge lock contention on high end server with large number of cpus, because doing copy on each region needs to obtain some unnecessary cpu locks. RESOLUTION: VxVM code has been changed to decrease the lock contention. * 3904801 (Tracking ID: 3686698) SYMPTOM: vxconfigd was getting hung due to deadlock between two threads DESCRIPTION: Two threads were waiting for same lock causing deadlock between them. This will lead to block all vx commands. untimeout function will not return until pending callback is cancelled (which is set through timeout function) OR pending callback has completed its execution (if it has already started). Therefore locks acquired by callback routine should not be held across call to untimeout routine or deadlock may result. Thread 1: untimeout_generic() untimeout() voldio() volsioctl_real() fop_ioctl() ioctl() syscall_trap32() Thread 2: mutex_vector_enter() voldsio_timeout() callout_list_expire() callout_expire() callout_execute() taskq_thread() thread_start() RESOLUTION: Code changes have been made to call untimeout outside the lock taken by callback handler. * 3904802 (Tracking ID: 3721565) SYMPTOM: vxconfigd hang is seen with below stack. genunix:cv_wait_sig_swap_core genunix:cv_wait_sig_swap genunix:pause unix:syscall_trap32 DESCRIPTION: In FMR environment, write is done on a source volume having space-optimized(SO) snapshot. Memory is acquired first and then ILOCKs are acquired on individual SO volumes for pushed writes. On the other hand, a user write on SO snapshot will first acquire ILOCK and then acquire memory. This causes deadlock. RESOLUTION: Code is modified to resolve deadlock. * 3904804 (Tracking ID: 3486861) SYMPTOM: Primary node panics with below stack when storage is removed while replication is going on with heavy IOs. Stack: oops_end no_context page_fault vol_rv_async_done vol_rv_flush_loghdr_done voliod_iohandle voliod_loop DESCRIPTION: In VVR environment, when write to data volume failson primary node, error handling is initiated. As a part of it, SRL header will be flushed. As primary storage is removed, flushing will fail. Panic will be hit as invalid values will be accessed while logging error message. RESOLUTION: Code is modified to resolve the issue. * 3904805 (Tracking ID: 3788644) SYMPTOM: When DMP (Dynamic Multi-Pathing) native support enabled for Oracle ASM environment, if we constantly adding and removing DMP devices, it will cause error like: /etc/vx/bin/vxdmpraw enable oracle dba 775 emc0_3f84 VxVM vxdmpraw INFO V-5-2-6157 Device enabled : emc0_3f84 Error setting raw device (Invalid argument) DESCRIPTION: There is a limitation (8192) of maximum raw device number N (exclusive) of /dev/raw/rawN. This limitation is defined in boot configuration file. When binding a raw device to a dmpnode, it uses /dev/raw/rawN to bind the dmpnode. The rawN is calculated by one-way incremental process. So even if we unbind the device later on, the "released" rawN number will not be reused in the next binding. When the rawN number is increased to exceed the maximum limitation, the error will be reported. RESOLUTION: Code has been changed to always use the smallest available rawN number instead of calculating by one-way incremental process. * 3904806 (Tracking ID: 3807879) SYMPTOM: Writing the backup EFI GPT disk label during the disk-group flush operation may cause data corruption on volumes in the disk group. The backup label could incorrectly get flushed to the disk public region and overwrite the user data with the backup disk label. DESCRIPTION: For EFI disks initialized under VxVM (Veritas Volume Manager), it is observed that during a disk-group flush operation, vxconfigd (veritas configuration daemon) could stop writing the EFI GPT backup label to the volume public region, thereby causing user data corruption. When this issue happens, the real user data are replaced with the backup EFI disk label RESOLUTION: The code is modified to prevent the writing of the EFI GPT backup label during the VxVM disk-group flush operation. * 3904807 (Tracking ID: 3867145) SYMPTOM: When VVR SRL occupation > 90%, then output the SRL occupation is shown by 10 percent. DESCRIPTION: This is kind of enhancement, to show the SRL Occupation when it's more than 90% is previously shown with 10 percentage gap. Here the enhancement is to show the logs with 1 percentage granularity. RESOLUTION: Changes are done to show the syslog messages wih 1 percent granularity, when SRL is filled > 90%. * 3904810 (Tracking ID: 3871750) SYMPTOM: In parallel VxVM(Veritas Volume Manager) vxstat commands report abnormal disk IO statistic data. Like below: # /usr/sbin/vxstat -g -u k -dv -i 1 -S ...... dm emc0_2480 4294967210 4294962421 -382676k 4294967.38 4294972.17 ...... DESCRIPTION: After VxVM IO statistics was optimized for huge CPUs and disks, there's a race condition when multiple vxstat commands are running to collect disk IO statistic data. It causes disk's latest IO statistic value become smaller than previous one, hence VxVM treates the value overflow so that abnormal large IO statistic value is printed. RESOLUTION: Code changes are done to eliminate such race condition. * 3904811 (Tracking ID: 3875563) SYMPTOM: While dumping the disk header information, human readable timestamp was not converted correctly from corresponding epoch time. DESCRIPTION: When disk group import fails if one of the disk is missing while importing the disk group, it will dump the disk header information the syslog. But, human readable time stamp was not getting converted correctly from corresponding epoch time. RESOLUTION: Code changes done to dump disk header information correctly. * 3904819 (Tracking ID: 3811946) SYMPTOM: When invoking "vxsnap make" command with cachesize option to create space optimized snapshot, the command succeeds but the following error message is displayed in syslog: kernel: VxVM vxio V-5-0-603 I/O failed. Subcache object <subcache-name> does not have a valid sdid allocated by cache object <cache-name>. kernel: VxVM vxio V-5-0-1276 error on Plex <plex-name> while writing volume <volume-name> offset 0 length 2048 DESCRIPTION: When space optimized snapshot is created using "vxsnap make" command along with cachesize option, cache and subcache objects are created by the same command. During the creation of snapshot, I/Os from the volumes may be pushed onto a subcache even though the subcache ID has not yet been allocated. As a result, the I/O fails. RESOLUTION: The code is modified to make sure that I/Os on the subcache are pushed only after the subcache ID has been allocated. * 3904822 (Tracking ID: 3755209) SYMPTOM: The Dynamic Multi-pathing(DMP) device configured in Solaris LDOM guest is disabled when an active controller of an ALUA array is failed. DESCRIPTION: DMP in guest environment monitors cached target port ID of virtual paths in LDOM. If a controller of an ALUA array fails for some reason, active/primary target port ID of an ALUA array will be changed in I/O domain resulting in stale entry in the guest. DMP in the guest wrongly interprets this target port change to mark the path as unavailable. This causes I/O on the path to be failed. As a result the DMP device is disabled in LDOM. RESOLUTION: The code is modified to not use the cached target port IDs for LDOM virtual disks. * 3904824 (Tracking ID: 3795622) SYMPTOM: With Dynamic Multi-Pathing (DMP) Native Support enabled, LVM global_filter is not updated properly in lvm.conf file to reject the newly added paths. DESCRIPTION: With DMP Native Support enabled, when new paths are added to existing LUNs, LVM global_filter is not updated properly in lvm.conf file to reject the newly added paths. This can lead to duplicate PV (physical volumes) found error reported by LVM commands. RESOLUTION: The code is modified to properly update global_filter field in lvm.conf file when new paths are added to existing disks. * 3904825 (Tracking ID: 3859009) SYMPTOM: pvs command will show the duplicate PV messages since global_filter of lvm.conf is not updated after fiber switch or storage controller get rebooted. DESCRIPTION: When fiber switch or storage controller reboot, some paths dev No. may get reused during DDL reconfig cycle, in this case VxDMP(Veritas Dynamic Multi-Pathing) wont treat them as newly added devices. For those devices belong to LVM dmpnode, VxDMP will not trigger lvm.conf update for them. As a result, the global_filter of lvm.conf will not be updated. Hence the issue. RESOLUTION: The code has been changed to update lvm.conf correctly. * 3904830 (Tracking ID: 3840359) SYMPTOM: On using localized messages, some VxVM commands fail while executing vxrootadm. The error message is as follows: VxVM vxmkrootmir ERROR V-5-2-3943 The Master Boot Record (MBR) could not be copied to the root disk mirror.To manually install it, follow the procedures in the VxVM Boot Disk Recovery chapter of the VxVM Trouble Shooting Guide. DESCRIPTION: The issue occurs when the output of the sfdisk command appears in the localized format. When the output is not translated into English language, a mismatch of messages is observed and command fails. RESOLUTION: The code is modified to convert the output of necessary commands in the scripts into English language before comparing it with the expected output. * 3904831 (Tracking ID: 3802075) SYMPTOM: Disks having digit in its name and which are added as foreign path using "vxddladm addforeign" goes into ERROR state after running vxdisk scandisks. DESCRIPTION: When the disk is added as foreign using 'vxddladm addforeign' , and after performing device-discovery using vxdisk scandisks, we use disk whole disk name, which is not the exact name of the disk. When digits are added in the name of the disk using udev rule, we now use the actual name of disk instead of whole disk name. RESOLUTION: The code is modified to use the exact disk-device name which adds the foreign disk successfully. * 3904833 (Tracking ID: 3729078) SYMPTOM: In VVR environment, the panic may occur after SF(Storage Foundation) patch installation or uninstallation on the secondary site. DESCRIPTION: VXIO Kernel reset invoked by SF patch installation removes all Disk Group objects that have no preserved flag set, because the preserve flag is overlapped with RVG(Replicated Volume Group) logging flag, the RVG object won't be removed, but its rlink object is removed, result of system panic when starting VVR. RESOLUTION: Code changes have been made to fix this issue. * 3904834 (Tracking ID: 3819670) SYMPTOM: When smartmove with 'vxevac' command is run in background by hitting 'ctlr-z' key and 'bg' command, the execution of 'vxevac' is terminated abruptly. DESCRIPTION: As part of "vxevac" command for data movement, VxVM submits the data as a task in the kernel, and use select() primitive on the task file descriptor to wait for task finishing events to arrive. When "ctlr-z" and bg is used to run vxevac in background, the select() returns -1 with errno EINTR. VxVM wrongly interprets it as user termination action and hence vxevac is terminated. Instead of terminating vxevac, the select() should be retried untill task completes. RESOLUTION: The code is modified so that when select() returns with errno EINTR, it checks whether vxevac task is finished. If not, the select() is retried. * 3904851 (Tracking ID: 3804214) SYMPTOM: VxDMP (Dynamic Multi-Pathing) path enable operation fails after the disk label is changed from guest LDOM. Open fails with error 5 (EIO) on the path being enabled. Following error messages can be seen in /var/adm/messages: vxdmp: [ID 808364 kern.notice] NOTICE: VxVM vxdmp V-5-3-0 dmp_open_path: Open failed with 5 for path 237/0x30 vxdmp: [ID 382146 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 [Warn] disabled path 237/0x30 belonging to the dmpnode 307/0x38 due to open failure DESCRIPTION: While a disk is exported to the Solaris LDOM, Solaris OS in the control/IO domain holds NORMAL mode open on the existing partitions of the DMP node. If the disk partitions/label is changed from LDOM such that some of the older partitions are removed, Solaris OS in the control/IO domain does not know about this change and continues to hold NORMAL mode open on those deleted partitions. If a disabled DMP path is enabled in this scenario, the NORMAL mode open the path fails and path enable operation errors out. This can be worked around by detaching and reattaching the disk to the LDOM. Due to a problem in DMP code, the stale NORMAL mode open flag was not being reset even when the DMP disk was detached from the LDOM. This was preventing the DMP path to be enabled even after the DMP disk was detached from the LDOM. RESOLUTION: Code was fixed to reset NORMAL mode open when the DMP disk is detached from the LDOM. With this fix, DMP disk will have to reattached to the LDOM only once after the disk labels change. When the disk is reattached, it will get the correct open mode (NORMAL/NDELAY) on the partitions that exist after label change. * 3904858 (Tracking ID: 3899568) SYMPTOM: "vxdmpadm iostat stop" as per design cannot stop the iostat gathering persistently. To avoid Performance & Memory crunch related issues, it is generally recommended to stop the iostat gathering.There is a requirement to provide such ability to stop/start the iostat gathering persistently in those cases. DESCRIPTION: Today DMP iostat daemon is stopped using - "vxdmpadm iostat stop". but this is not persistent setting. After reboot this would be lost and hence customer needs to also have to put this in init scripts at appropriate place for persistent effect. RESOLUTION: Code is modified to provide a tunable "dmp_compute_iostats" which can start/stop the iostat gathering persistently. Notes: Use following command to start/stop the iostat gathering persistently. # vxdmpadm settune dmp_compute_iostats=on/off. * 3904859 (Tracking ID: 3901633) SYMPTOM: Lots of error messages like the following are reported while performing RVG sync. VxVM VVR vxrsync ERROR V-5-52-2027 getdigest response err [192.168.10.101:/dev/vx/dsk/testdg/v1 <- 192.168.10.105:/dev/vx/dsk/testdg/v1] [[ndigests sent=-1 ndigests received=0]] VxVM VVR vxrsync ERROR V-5-52-2027 getdigest response err [192.168.10.101:/dev/vx/dsk/testdg/v1 <- 192.168.10.105:/dev/vx/dsk/testdg/v1] [[ndigests sent=-2 ndigests received=0]] DESCRIPTION: While performing last volume region read and sync, volume end offset calculation is not correct, which may lead to over volume end read and sync, result in an internal variable became negative number and vxrsync reports error. It can happen if volume size is not multiple of 512KB, plus the last 512KB volume region is partly in use by VxFS. RESOLUTION: Code changes have been done to fix the issue. * 3904861 (Tracking ID: 3904538) SYMPTOM: RV(Replicate Volume) IO hang happens during slave node leave or master node switch. DESCRIPTION: RV IO hang happens because of SRL(Serial Replicate Log) header is updated by RV recovery SIO. After slave node leave or master node switch, RV recovery could be initiated. During RV recovery, all new coming IOs should be quiesced by setting NEED RECOVERY flag on RV to avoid racing. Due to a code defect, this flag is removed by transaction commit, result in conflicting between new IOs and RV recovery SIO. RESOLUTION: Code changes have been made to fix this issue. * 3904863 (Tracking ID: 3851632) SYMPTOM: When you use the localized messages, some VxVM commands fail while mirroring the volume through vxdiskadm. The error message is similar to the following: ? [y, n, q,?] (: y) y /usr/lib/vxvm/voladm.d/bin/disk.repl: test: unknown operator 1 DESCRIPTION: The issue occurs when the output of the vxdisk list command appears in the localized format. When the output is not translated into English language, a mismatch of messages is observed and command fails. RESOLUTION: The code is modified to convert the output of the necessary commands in the scripts into English language before comparing it with the expected output. * 3904864 (Tracking ID: 3769303) SYMPTOM: System pancis when CVM group is brought online with below stack: voldco_acm_pagein voldco_write_pervol_maps_instant voldco_map_update voldco_write_pervol_maps volfmr_copymaps_instant vol_mv_get_attmir vol_subvolume_get_attmir vol_plex_get_attmir vol_mv_fmr_precommit vol_mv_precommit vol_commit_iolock_objects vol_ktrans_commit volconfig_ioctl ns_capable volsioctl_real mntput path_put vfs_fstatat from_kgid_munged read_tsc vols_ioctl vols_compat_ioctl compat_sys_ioctl sysenter_dispatch voldco_get_accumulator DESCRIPTION: In case of layered volumes, when 'vxvol' comamnd is triggered through 'vxrecover' command with '-Z vols(implicit) option, only the volumes passed through CLI are started, the respective top level volumes remain unstarted. As a result, associated DCO volumes also remain unstarted. At this point of time, if any of the plex of sub-volume needs to be attached back, vxrecover will trigger it. With DCO version 30, vxplex command tries to perform some map manipulation as a part of plex-attach transaction. If the DCO volume is not started before plex attach, the in-core DCO contents are improperly loaded and this leads to panic. RESOLUTION: The code is modified to handle the starting of appropriate associated volumes of a layered volume group. * 3905471 (Tracking ID: 3868533) SYMPTOM: IO hang happens when starting replication. VXIO deamon hang with stack like following: vx_cfs_getemap at ffffffffa035e159 [vxfs] vx_get_freeexts_ioctl at ffffffffa0361972 [vxfs] vxportalunlockedkioctl at ffffffffa06ed5ab [vxportal] vxportalkioctl at ffffffffa06ed66d [vxportal] vol_ru_start at ffffffffa0b72366 [vxio] voliod_iohandle at ffffffffa09f0d8d [vxio] voliod_loop at ffffffffa09f0fe9 [vxio] DESCRIPTION: While performing DCM replay in case Smart Move feature is enabled, VxIO kernel needs to issue IOCTL to VxFS kernel to get file system free region. VxFS kernel needs to clone map by issuing IO to VxIO kernel to complete this IOCTL. Just at the time RLINK disconnection happened, so RV is serialized to complete the disconnection. As RV is serialized, all IOs including the clone map IO form VxFS is queued to rv_restartq, hence the deadlock. RESOLUTION: Code changes have been made to handle the dead lock situation. * 3906251 (Tracking ID: 3806909) SYMPTOM: During installation of volume manager installation using CPI in key-less mode, following logs were observed. VxVM vxconfigd DEBUG V-5-1-5736 No BASIC license VxVM vxconfigd ERROR V-5-1-1589 enable failed: License has expired or is not available for operation transactions are disabled. DESCRIPTION: While using CPI for STANDALONE DMP installation in key less mode, volume manager Daemon(vxconfigd) cannot be started due to a modification in a DMP NATIVE license string that is used for license verification and this verification was failing. RESOLUTION: Appropriate code changes are incorporated to resolve the DMP keyless License issue to work with STANDALONE DMP. * 3906566 (Tracking ID: 3907654) SYMPTOM: Storage of cold data on dedicated SAN storage spaces increases storage cost and maintenance. DESCRIPTION: The local storage capacity is consumed by cold or legacy files which are not consumed or processed frequently. These files occupy dedicated SAN storage space, which is expensive. Moving such files to public or private S3 cloud storage services is a better cost-effective solution. Additionally, cloud storage is elastic allowing varying service levels based on changing needs. Operational charges apply for managing objects in buckets for public cloud services using the Storage Transfer Service. RESOLUTION: You can now migrate or move legacy data from local SAN storage to a target Private or public cloud. * 3907017 (Tracking ID: 3877571) SYMPTOM: Disk header is updated even if the dg import operation fails DESCRIPTION: When dg import fails because of the disk failure, importing dg forcefully needs checking the disks having latest configuration copy. But, it is very difficult to decide which disk to choose without disk header update logs. RESOLUTION: Improved the logging to track the disk header changes. * 3907593 (Tracking ID: 3660869) SYMPTOM: Enhance the DRL dirty-ahead logging for sequential write workloads. DESCRIPTION: With the current DRL implementation, when sequential hints are passed by the above FS layer, further regions in the DRL are dirtied to ensure that the write on the DRL is saved when the new IO on the region comes. But with the current design, there is a flaw and the number of IO's on the DRL are similar to the number of IO's on the data volume. Because of the flaw, same region is being dirtied again and again as part of the DRL IO. This can lead to performance hit as well. RESOLUTION: In order to improve the performance, the number of IO's on the DRL are reduced by enhancing the implementation of Dirty-ahead logging with DRL. * 3907595 (Tracking ID: 3907596) SYMPTOM: vxdmpadm setattr command gives the below error while setting the path attribute: "VxVM vxdmpadm ERROR V-5-1-14526 Failed to save path information persistently" DESCRIPTION: Device names on linux change once the system is rebooted. Thus the persistent attributes of the device are stored using persistent hardware path. The hardware paths are stored as symbolic links in the directory /dev/vx/.dmp. The hardware paths are obtained from /dev/disk/by-path using the path_id command. In SLES12, the command to extract the hardware path changes to path_id_compat. Since the command changed, the script was failing to generate the hardware paths in /dev/vx/.dmp directory leading to the persistent attributes not being set. RESOLUTION: Code changes have been made to use the command path_id_compat to get the hardware path from /dev/disk/by-path directory. Patch ID: VRTSvxvm-6.2.1.200 * 3780334 (Tracking ID: 3762580) SYMPTOM: In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath co-exists with DMP (Dynamic Multi-Pathing). The following logs are printed while setting up fencing for the cluster. VXFEN: vxfen_reg_coord_pt: end ret = -1 vxfen_handle_local_config_done: Could not register with a majority of the coordination points. DESCRIPTION: In Linux kernels greater than or equal to RHEL6.6 (e.g. RHEL7 and SLES11SP3), the interface used by DMP to send the SCSI commands to block devices does not transfer the data to or from the device. Therefore, the SCSI-3 PR keys do not get registered. RESOLUTION: The code is modified to use SCSI request_queue to send the SCSI commands to the underlying block device. Additional patch is required from EMC to support processing SCSI commands via the request_queue mechanism on EMC PowerPath devices. Please contact EMC for patch details for a specific kernel version. * 3795710 (Tracking ID: 3508122) SYMPTOM: When running vxfentsthdw during the preempt key operation, the I/O on the victim node is expected to fail, but sometimes it doesn't fail DESCRIPTION: After node 1 preempts the SCSI-3 reservations for node 2, it is expected that write I/Os from victim node 2 will fail. It is observed that sometimes the storage does not preempt all the keys of the victim node fast enough, but it fails the I/O with reservation conflict. In such case, the victim node could not correctly identify that it has been preempted and can still do re-registration of keys to perform the I/O. RESOLUTION: The code is modified to correctly identify that SCSI-3 keys of a node has been preempted. * 3802857 (Tracking ID: 3726110) SYMPTOM: On systems with high number of CPUs, DMP devices may perform considerably slower than OS device paths. DESCRIPTION: In high CPU configuration, I/O statistics related functionality in DMP takes more CPU time because DMP statistics are collected on per CPU basis. This stat collection happens in DMP I/O code path hence it reduces the I/O performance. Because of this, DMP devices perform slower than OS device paths. RESOLUTION: The code is modified to remove some of the stats collection functionality from DMP I/O code path. Along with this, the following tunable need to be turned off: 1. Turn off idle lun probing. #vxdmpadm settune dmp_probe_idle_lun=off 2. Turn off statistic gathering functionality. #vxdmpadm iostat stop Notes: 1. Please apply this patch if system configuration has large number of CPU and if DMP performs considerably slower than OS device paths. For normal systems this issue is not applicable. * 3803497 (Tracking ID: 3802750) SYMPTOM: Once VxVM (Veritas Volume Manager) volume I/O-shipping functionality is turned on, it is not getting disabled even after the user issues the correct command to disable it. DESCRIPTION: VxVM (Veritas Volume Manager) volume I/O-shipping functionality is turned off by default. The following two commands can be used to turn it on and off: vxdg -g <dgname> set ioship=on vxdg -g <dgname> set ioship=off The command to turn off I/O-shipping is not working as intended because I/O-shipping flags are not reset properly. RESOLUTION: The code is modified to correctly reset I/O-shipping flags when the user issues the CLI command. * 3804299 (Tracking ID: 3804298) SYMPTOM: For CVM (Cluster Volume Manager) environment, the setting/unsetting of the 'lfailed/lmissing' flag is not recorded in the syslog. DESCRIPTION: For CVM environment, when VxVM (Verita's volume Manager) discovers that a diskis not accessible from a node of CVM cluster, it marks theLFAILED (locally failed) flag on the disk. And when VxVM discovers that a disk isnot discovered by DMP (Dynamic Multipathing) on a node of CVM cluster, it marks the LMISSING (locally missing) flag on the disk. Messages of the setting andunsetting of the 'lmissing/lfailed' flag are not recorded in the syslog. RESOLUTION: The code is modified to record the setting and unsetting of the 'lfailed/lmissing' flag in syslog. * 3808134 (Tracking ID: 3808135) SYMPTOM: While you install the HF on the top of VxVM 6.2.1, following errors are reported. ln: failed to create symbolic link '/etc/rc.d/init.d/vras-vradmind': File exists ln: failed to create symbolic link '/etc/rc.d/init.d/vxrsyncd': File exists DESCRIPTION: During the installation of the HF, symbolic links are recreated, but the stale links from the previously installed VxVM are not deleted, therefore the errors occur. RESOLUTION: The code is modified for a smooth HFinstallation session on the top of VxVM 6.2.1. * 3816222 (Tracking ID: 3816219) SYMPTOM: VxDMP (Veritas Dynamic Multi-Pathing) event source daemon (vxesd) keeps reporting a lot of messages in syslog as below: "vxesd: Device sd*(*/*) is changed" DESCRIPTION: The vxesd daemon registers with the UDEV framework and keeps VxDMP up-to-date with devices' status. Due to some change at device, vxesd keeps reporting this kind of change-event listened by udev. VxDMP only cares about "add" and "remove" UDEV events. For UDEV "change" event, we can avoid logging for these events to VxDMP. RESOLUTION: The code is modified to stop logging UDEV change-event related messages in syslog. * 3823288 (Tracking ID: 3823283) SYMPTOM: Linux operating system sticks in grub after reboot. Manual kernel load is required to make operating system functional. DESCRIPTION: During unencapsulation of a boot disk in SAN environment, multiple entries corresponding to root disk are found in by-id device directory. As a result, a parse command fails, leading to the creation of an improper menu file in grub directory. This menu file defines the device path to load kernel and other modules. RESOLUTION: The code is modified to handle multiple entries for SAN boot disk. * 3848011 (Tracking ID: 3806909) SYMPTOM: During installation of volume manager installation using CPI in key-less mode, following logs were observed. VxVM vxconfigd DEBUG V-5-1-5736 No BASIC license VxVM vxconfigd ERROR V-5-1-1589 enable failed: License has expired or is not available for operation transactions are disabled. DESCRIPTION: While using CPI for STANDALONE DMP installation in key less mode, volume manager Daemon(vxconfigd) cannot be started due to a modification in a DMP NATIVE license string that is used for license verification and this verification was failing. RESOLUTION: Appropriate code changes are incorporated to resolve the DMP keyless License issue to work with STANDALONE DMP. * 3849458 (Tracking ID: 3776520) SYMPTOM: Filters are not updated properly in lvm.conf file in VxDMP initrd while DMP Native Support is being enabled. As a result, root Logical Volume (LV) is mounted on OS device upon reboot. DESCRIPTION: From LVM version 105, global_filter was introduced as part of lvm.conf file. VxDMP updates initird lvm.conf file with the filters required for DMP Native Support to function. While updating the lvm.conf, VxDMP checks for the filter field to be updated, but ideally we should check for global_filter field to be updated in the latest LVM version. This leads to lvm.conf file not updated with the proper filters. RESOLUTION: The code is modified to properly update global_filter field in lvm.conf file in VxDMP initrd. * 3865905 (Tracking ID: 3865904) SYMPTOM: The installation failed while installing VxVM 6.2.1 on SLES12SP1 superm2028-49472:/sles12sp0 # rpm -ivh VRTSvxvm-6.2.1.000- SLES12.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:VRTSvxvm-6.2.1.000-SLES12 ################################# [100%] Installing file /etc/init.d/vxvm-boot vxvm-boot 0:off 1:off 2:on 3:on 4:on 5:on 6:off creating VxVM device nodes under /dev ERROR: No appropriate modules found. Error in loading module "vxdmp". See documentation. warning: %post(VRTSvxvm-6.2.1.000-SLES12.x86_64) scriptlet failed, exit status 1 DESCRIPTION: Installation of VRTSvxvm patch version 6.2.1 failed on SLES12SP1 due to change in kernel version. RESOLUTION: The VxVM package has been re-compiled with SLES12SP1 build environment. * 3870949 (Tracking ID: 3845712) SYMPTOM: After booting through vxvm_root entry of the grub, Following message is seen at the console - "/dev/disk/by-uuid/**UUID** doesn't exist." DESCRIPTION: This is because, when the VxVM-initrd image is created as part of root device encapsulation, the SWAP device has old Swap UUID and when vxvm creates swap-volumes, it uses new UUID for swap device, which will not be present in VxVM-initrd image. Dracut has checks for mapping of swap UUID to get the device-id while booting up, and it doesn't match with the UUID of swap device present in the VxVM-initrd image, hence this issue. RESOLUTION: Code changes are implemented so that UUID checks for swap device in dracut passes properly. Patch ID: VRTSgms-6.2.1.7200 * 3977173 (Tracking ID: 3973946) SYMPTOM: GMS module failed to load on SLES12 SP4 DESCRIPTION: The SLES12 SP4 is new release and it has 4.12 Linux kernel therefore GMS module failed to load on it. RESOLUTION: Enabled GMS support for SLES12 SP4 Patch ID: VRTSgms-6.2.1.500 * 3930628 (Tracking ID: 3930625) SYMPTOM: GMS module failed to load on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 is new release therefore GMS module failed to load on it. RESOLUTION: Added GMS support for SLES12 SP3. Patch ID: VRTSgms-6.2.1.400 * 3915629 (Tracking ID: 3907939) SYMPTOM: GMS module failed to load on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 is new release therefore GMS module failed to load on it. RESOLUTION: Added GMS support for SLES12 SP2. Patch ID: VRTSglm-6.2.1.7300 * 3977174 (Tracking ID: 3973945) SYMPTOM: GLM module failed to load on SLES12 SP4 DESCRIPTION: The SLES12 SP4 is new release and it has 4.12 Linux kernel therefore GLM module failed to load on it. RESOLUTION: Enabled GLM support for SLES12 SP4 Patch ID: VRTSglm-6.2.1.500 * 3930624 (Tracking ID: 3930622) SYMPTOM: GLM module failed to load on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 is new release therefore GLM module failed to load on it. RESOLUTION: Added GLM support for SLES12 SP3. Patch ID: VRTSglm-6.2.1.400 * 3915628 (Tracking ID: 3907936) SYMPTOM: GLM module failed to load on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 is new release therefore GLM module failed to load on it. RESOLUTION: Added GLM support for SLES12 SP2. Patch ID: VRTSodm-6.2.1.8400 * 3977175 (Tracking ID: 3973944) SYMPTOM: ODM module failed to load on SLES12 SP4 DESCRIPTION: The SLES12 SP4 is new release and it has 4.12 Linux kernel therefore ODM module failed to load on it. RESOLUTION: Enabled ODM support for SLES12 SP4 * 3988170 (Tracking ID: 3987992) SYMPTOM: System panic due to Out of memory issue DESCRIPTION: The odm module might leak allocated pids in few conditions which cause system panic due to the OOM if panic_on_oom is set. RESOLUTION: Code changes added to free up allocated pids in case it is not used. Patch ID: VRTSodm-6.2.1.600 * 3930632 (Tracking ID: 3930629) SYMPTOM: ODM module failed to load on SLES12 SP3. DESCRIPTION: Since SLES12 SP3 is new release therefore ODM module failed to load on it. RESOLUTION: Added ODM support for SLES12 SP3. Patch ID: VRTSodm-6.2.1.400 * 3915627 (Tracking ID: 3907933) SYMPTOM: ODM module failed to load on SLES12 SP2. DESCRIPTION: Since SLES12 SP2 is new release therefore ODM module failed to load on it. RESOLUTION: Added ODM support for SLES12 SP2. Patch ID: VRTSodm-6.2.1.300 * 3906065 (Tracking ID: 3757609) SYMPTOM: High CPU usage because of contention over ODM_IO_LOCK DESCRIPTION: While performing ODM IO, to update some of the ODM counters we take ODM_IO_LOCK which leads to contention from multiple of iodones trying to update these counters at the same time. This is results in high CPU usage. RESOLUTION: Code modified to remove the lock contention. INSTALLING THE PATCH -------------------- Run the Installer script to automatically install the patch: ----------------------------------------------------------- Please be noted that the installation of this P-Patch will cause downtime. To install the patch perform the following steps on at least one node in the cluster: 1. Copy the patch sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz to /tmp 2. Untar sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz to /tmp/hf # mkdir /tmp/hf # cd /tmp/hf # gunzip /tmp/sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz # tar xf /tmp/sfha-sles12_x86_64-Patch-6.2.1.4100.tar 3. Install the hotfix(Please be noted that the installation of this P-Patch will cause downtime.) # pwd /tmp/hf # ./installSFHA621P41 [ ...] You can also install this patch together with 6.2.1 maintenance release using Install Bundles 1. Download this patch and extract it to a directory 2. Change to the Veritas InfoScale 6.2.1 directory and invoke the installmr script with -patch_path option where -patch_path should point to the patch directory # ./installmr -patch_path [] [ ...] Install the patch manually: -------------------------- Manual installation is not recommended. REMOVING THE PATCH ------------------ Manual uninstallation is not recommended. SPECIAL INSTRUCTIONS -------------------- INSTALLING THE PATCH ON SLES12SP4 DIRECTLY ------------------------------------------------------------- Run the Installer script to automatically install the patch: ---------------------------------------------------------------- Please be noted that the installation of this P-Patch will cause downtime. To install the patch perform the following steps on at least one node in the cluster: 1. Copy the patch sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz to /tmp 2. Untar sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz to /tmp/hf # mkdir /tmp/hf # cd /tmp/hf # gunzip /tmp/sfha-sles12_x86_64-Patch-6.2.1.4100.tar.gz # tar xf /tmp/sfha-sles12_x86_64-Patch-6.2.1.4100.tar 3. Download the CPI patch cpi-Patch-6.2.1.1900 from SORT 4. Untar the CPI patch # mkdir /tmp/cpi_patch # cp cpi-Patch-6.2.1.1900.tar.gz /tmp/cpi_patch # cd /cpi_patch # tar -xvzf cpi-Patch-6.2.1.1900.tar.gz 5. Install the hotfix(Please be noted that the installation of this P-Patch will cause downtime.) Run the installer from the 6.2.1MR-GA media: #./installmr -patch_path /tmp/hf/ -require /tmp/cpi_patch/patches/CPI_6.2.1_P19.pl [ ...] OTHERS ------ NONE