* * * READ ME * * * * * * Symantec Volume Manager 6.1.1 * * * * * * Patch 6.1.1.200 * * * Patch Date: 2015-10-13 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Symantec Volume Manager 6.1.1 Patch 6.1.1.200 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- SLES11 x86-64 PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxvm BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Symantec Dynamic Multi-Pathing 6.1 * Symantec Storage Foundation 6.1 * Symantec Storage Foundation Cluster File System HA 6.1 * Symantec Storage Foundation for Oracle RAC 6.1 * Symantec Storage Foundation HA 6.1 * Symantec Volume Manager 6.1 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: 6.1.1.200 * 3607232 (3596330) 'vxsnap refresh' operation fails with `Transaction aborted waiting for IO drain` error * 3662397 (3662392) In the Cluster Volume Manager (CVM) environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. * 3682022 (3648719) The server panics while adding or removing LUNs or HBAs. * 3729172 (3726110) On systems with high number of CPU's, Dynamic Multipathing (DMP) devices may perform considerably slower than OS device paths. * 3736352 (3729078) VVR(Veritas Volume Replication) secondary site panic occurs during patch installation because of flag overlap issue. * 3778391 (3565212) IO failure is seen during controller giveback operations on Netapp Arrays in ALUA mode. * 3790099 (3581646) Logical Volumes fail to migrate back to OS devices Dynamic Multipathing (DMP) when DMP native support is disabled while root("/") is mounted on LVM. * 3790106 (3488071) The command "vxdmpadm settune dmp_native_support=on" fails to enable Dynamic Multipathing (DMP) Native Support * 3790115 (3623617) The command "vxdmpadm settune dmp_native_support=on" may fail with a perl error * 3790117 (3776520) Filters are not updated properly in lvm.conf file in VxDMP initrd (initial ramdisk) while Dynamic Multipathing (DMP) Native Support is being enabled. * 3795623 (3795622) With Dynamic Multipathing (DMP) Native Support enabled, LVM global_filter is not updated properly in lvm.conf file. * 3800388 (3581264) VxVM package uninstallation succeeds even if DMP Native Support is enable and IO's are in progress on LV. * 3800421 (3762580) In Linux kernels greater than or equal to RHEL6.6 like RHEL7 and SLES11SP3, the vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath exists in coexistence with DMP (Dynamic Multipathing). * 3808593 (3795788) Performance degradation seen when many application session open the same data files on VxVM(Veritas Volume Manager) volume. * 3812272 (3811946) When invoking "vxsnap make" command with cachesize option to create space optimized snapshot, the command succeeds but a plex IO error message is displayed in syslog. * 3852806 (3823283) While unencapsulating a boot disk in SAN environment (Storage Area etwork), Linux operating system sticks in grub after reboot. * 3852809 (3825467) SLES11-SP4 build fails. * 3852831 (3806909) Due to some modification in licensing , for STANDALONE DMP, DMP keyless license was not working. DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following Symantec incidents: Patch ID: 6.1.1.200 * 3607232 (Tracking ID: 3596330) SYMPTOM: 'vxsnap refresh' operation fails with following indicants: Errors occur from DR (Disaster Recovery) Site of VVR (Veritas Volume Replicator): o vxio: [ID 160489 kern.notice] NOTICE: VxVM vxio V-5-3-1576 commit: Timedout waiting for rvg [RVG] to quiesce, iocount [PENDING_COUNT] msg 0 o vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-8011 Internal transaction failed: Transaction aborted waiting for io drain At the same time, following errors occur from Primary Site of VVR: vxio: [ID 218356 kern.warning] WARNING: VxVM VVR vxio V-5-0-267 Rlink [RLINK] disconnecting due to ack timeout on update message DESCRIPTION: VM (Volume Manager) Transactions on DR site get aborted as pending IOs could not be drained in stipulated time leading to failure of FMR (Fast-Mirror Resync) 'snap' operations. These IOs could not be drained because of IO throttling. A bug/race in conjunction with timing in VVR causes a miss in clearing this throttling condition/state. RESOLUTION: Code changes have been done to fix the race condition which ensures clearance of throttling state at appropriate time. * 3662397 (Tracking ID: 3662392) SYMPTOM: In the CVM environment, if I/Os are getting executed on slave node, corruption can happen when the vxdisk resize(1M) command is executing on the master node. DESCRIPTION: During the first stage of resize transaction, the master node re-adjusts the disk offsets and public/private partition device numbers. On a slave node, the public/private partition device numbers are not adjusted properly. Because of this, the partition starting offset is are added twice and causes the corruption. The window is small during which public/private partition device numbers are adjusted. If I/O occurs during this window then only corruption is observed. After the resize operation completes its execution, no further corruption will happen. RESOLUTION: The code has been changed to add partition starting offset properly to an I/O on slave node during execution of a resize command. * 3682022 (Tracking ID: 3648719) SYMPTOM: The server panics with a following stack trace while adding or removing LUNs or HBAs: dmp_decode_add_path() dmp_decipher_instructions() dmp_process_instruction_buffer() dmp_reconfigure_db() gendmpioctl() vxdmpioctl() DESCRIPTION: While deleting a dmpnode, Dynamic Multi-Pathing (DMP) releases the memory associated with the dmpnode structure. In case the dmpnode doesn't get deleted for some reason, and if any other tasks access the freed memory of this dmpnode, then the server panics. RESOLUTION: The code is modified to avoid the tasks from accessing the memory that is freed by the dmpnode, which is deleted. The change also fixed the memory leak issue in the buffer allocation code path. * 3729172 (Tracking ID: 3726110) SYMPTOM: On systems with high number of CPU's, Dynamic Multipathing (DMP) devices may perform considerably slower than OS device paths. DESCRIPTION: In high CPU configuration, IO statistics related functionality in DMP takes more CPU time as DMP statistics are collected on per CPU basis. This stat collection happens in DMP IO code path hence it reduces the IO performance. Because of this DMP devices perform slower than OS device paths. RESOLUTION: Code changes are made to remove some of the stats collection functionality from DMP IO code path. Along with this, following tunable need to be turned off. 1. Turn off idle lun probing. #vxdmpadm settune dmp_probe_idle_lun=off 2. Turn off statistic gathering functionality. #vxdmpadm iostat stop Notes: 1. Please apply this patch if system configuration has large number of CPU and if DMP is performing considerably slower than OS device paths. For normal systems this issue is not applicable. * 3736352 (Tracking ID: 3729078) SYMPTOM: In VVR environment, the panic may occur after SF(Storage Foundation) patch installation or uninstallation on the secondary site. DESCRIPTION: VXIO Kernel reset invoked by SF patch installation removes all Disk Group objects that have no preserved flag set, because the preserve flag is overlapped with RVG(Replicated Volume Group) logging flag, the RVG object won't be removed, but its rlink object is removed, result of system panic when starting VVR. RESOLUTION: Code changes have been made to fix this issue. * 3778391 (Tracking ID: 3565212) SYMPTOM: While performing controller giveback operations on NetApp ALUA arrays, the below messages are observed in /etc/vx/dmpevents.log [Date]: I/O error occured on Path belonging to Dmpnode [Date]: I/O analysis done as DMP_PATH_BUSY on Path belonging to Dmpnode [Date]: I/O analysis done as DMP_IOTIMEOUT on Path belonging to Dmpnode DESCRIPTION: During the asymmetric access state transition, DMP puts the buffer pointer in the delay queue based on the flags observed in the logs. This delay resulted in timeout and thereby filesystem went into disabled state. RESOLUTION: DMP code is modified to perform immediate retries instead of putting the buffer pointer in the delay queue for transition in progress case. * 3790099 (Tracking ID: 3581646) SYMPTOM: Sometimes Logical Volumes may fail to migrate back to OS devices when Dynamic Multipathing (DMP) Native Support is disabled when the root is mounted on LVM. DESCRIPTION: lvmetad caches open count on devices which are present in accept section of filter in lvm.conf file. When DMP Native Support is enabled, all non-VxVM devices are put in reject section of filter so that only "/dev/vx/dmp" devices remain in accept section of filter in lvm.conf file. So lvmetad caches open count on "/dev/vx/dmp" devices. When DMP Native Support is disabled "/dev/vx/dmp" devices are not put in reject section of filter causing a stale open count for lvmetad which is causing physical volumes to point to stale devices even when DMP Native Support is disabled. RESOLUTION: Code changes have been made to add "/dev/vx/dmp" devices in reject section of filter in lvm.conf file so lvmetad releases open count on these devices. * 3790106 (Tracking ID: 3488071) SYMPTOM: The command "vxdmpadm settune dmp_native_support=on" fails to enable Dynamic Multipathing (DMP) Native Support and fails with the below error: VxVM vxdmpadm ERROR V-5-1-15690 Operation failed for one or more volume groups DESCRIPTION: From LVM version 105, global_filters were introduced as part of lvm.conf file. RHEL 6.6 and RHEL 6.6 onwards has LVM version 111 and hence supports global_filters. The code changes were not done to handle global_filter in lvm.conf file while 6.1.1.100 was released. RESOLUTION: Code changes have been done to handle global_filter in lvm.conf file and allow DMP Native Support to work. * 3790115 (Tracking ID: 3623617) SYMPTOM: The command "vxdmpadm settune dmp_native_support=on" may fail with a below perl error: Can't locate Sys/Syslog.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/lib/vxvm/bin/vxupdatelvm line 73. BEGIN failed--compilation aborted at /usr/lib/vxvm/bin/vxupdatelvm line 73. DESCRIPTION: In one of VxVM specific script used for enabling Dynamic Multipathing (DMP) Native Support, perl syslog module is used. It can happen sometimes that minimal package installation of OS might not contain perl syslog module leading to this error. RESOLUTION: Code changes have been done to use internally developed perl module and avoid the dependency. * 3790117 (Tracking ID: 3776520) SYMPTOM: Filters are not updated properly in lvm.conf file in VxDMP initrd while DMP Native Support is being enabled. As a result, root Logical Volume (LV) is mounted on OS device upon reboot. DESCRIPTION: From LVM version 105, global_filter was introduced as part of lvm.conf file. VxDMP updates initird lvm.conf file with the filters required for DMP Native Support to function. While updating the lvm.conf, VxDMP checks for the filter field to be updated, but ideally we should check for global_filter field to be updated in the latest LVM version. This leads to lvm.conf file not updated with the proper filters. RESOLUTION: The code is modified to properly update global_filter field in lvm.conf file in VxDMP initrd. * 3795623 (Tracking ID: 3795622) SYMPTOM: With Dynamic Multipathing (DMP) Native Support enabled, LVM global_filter is not updated properly in lvm.conf file to reject the newly added paths. DESCRIPTION: With Dynamic Multipathing (DMP) Native Support enabled, when new paths are added to existing LUNs, LVM global_filter is not updated properly in lvm.conf file to reject the newly added paths. This can lead to duplicate PV (physical volumes) found error reported by LVM commands. RESOLUTION: Code changes have been made to properly update global_filter field in lvm.conf file when new paths are added to existing disks. * 3800388 (Tracking ID: 3581264) SYMPTOM: VxVM package uninstallation succeeds even When DMP (Dynamic Multipathing) Native Support is on and IO's are in progress on LV (Logical Volume). DESCRIPTION: While uninstalling the VxVM package, no precaution is taken to log an error message if DMP native support could not be successfully disabled. As a result, the uninstallation proceeds without any error message. RESOLUTION: Code changes have been made to display an error message if DMP native support could not be disabled during uninstallation. Additionally, if the rootvg is enabled, the uninstallation will proceed with an error message; otherwise uninstallation will fail. * 3800421 (Tracking ID: 3762580) SYMPTOM: Below logs were seen while setting up fencing for the cluster. VXFEN: vxfen_reg_coord_pt: end ret = -1 vxfen_handle_local_config_done: Could not register with a majority of the coordination points. DESCRIPTION: It is observed that in Linux kernels greater than or equal to RHEL6.6, RHEL7 and SLES11SP3, the interface used by DMP to send the SCSI commands to block devices does not transfer the data to/from the device. Therefore the SCSI-3 PR keys does not get registered. RESOLUTION: Code change is done to use scsi request_queue to send the SCSI commands to the underlying block device. Additional patch is required from EMC to support processing SCSI commands via the request_queue mechanism on EMC PowerPath devices. Please contact EMC for patch details for a specific kernel version. * 3808593 (Tracking ID: 3795788) SYMPTOM: Performance degradation seen when many application session open the same data files on VxVM(Veritas Volume Manager) volume. DESCRIPTION: This issue occurs because of lock contention while opening the volume.While opening the volume, exclusive lock is taken on all CPU's. If there are a lot of CPU's in the system, this process could be quite time consuming, which leads to performance degradation seen during the initial start of applications. RESOLUTION: Code changes have been made to change exclusive lock to shared lock while opening the volume. * 3812272 (Tracking ID: 3811946) SYMPTOM: When invoking "vxsnap make" command with cachesize option to create space optimized snapshot, the command succeeds but following error message is displayed in syslog: kernel: VxVM vxio V-5-0-603 I/O failed. Subcache object does not have a valid sdid allocated by cache object . kernel: VxVM vxio V-5-0-1276 error on Plex while writing volume offset 0 length 2048 DESCRIPTION: When space optimized snapshot is created using "vxsnap make" command along with cachesize option, cache and subcache objects get created with the help of same command. During the creation of snapshot, it can happen that IO from the volumes are pushed onto a subcache even though the subcache ID has not yet been allocated. Since the subcache ID is not allocated, the IO fails. RESOLUTION: Code changes have been made to make sure that the IO's on the subcache are pushed only after the subcache ID has been allocated. * 3852806 (Tracking ID: 3823283) SYMPTOM: Linux operating system sticks in grub after reboot. Manual kernel load is required to make operating system functional. DESCRIPTION: During unencapsulation of a boot disk in SAN environment, multiple entries corresponding to root disk are found in by-id device directory. As a result, a parse command fails, leading to the creation of an improper menu file in grub directory. This menu file defines the device path to load kernel and other modules. RESOLUTION: The code is modified to handle multiple entries for SAN boot disk. * 3852809 (Tracking ID: 3825467) SYMPTOM: Error showing about symbol d_lock. DESCRIPTION: This symbol is already present in this kernel, so its duplicate getting declared in kernel code of VxVM. RESOLUTION: Code is modified to remove the definition, hence solved the issue * 3852831 (Tracking ID: 3806909) SYMPTOM: During installation of volume manager installation using CPI in key-less mode, following logs were observed. VxVM vxconfigd DEBUG V-5-1-5736 No BASIC license VxVM vxconfigd ERROR V-5-1-1589 enable failed: License has expired or is not available for operation transactions are disabled. DESCRIPTION: While using CPI for STANDALONE DMP installation in key less mode, volume manager Daemon(vxconfigd) cannot be started due to a modification in a DMP NATIVE license string that is used for license verification and this verification was failing. RESOLUTION: Appropriate code changes are incorporated to resolve the DMP keyless License issue to work with STANDALONE DMP. INSTALLING THE PATCH -------------------- Run the Installer script to automatically install the patch: ----------------------------------------------------------- To install the patch perform the following steps on at least one node in the cluster: 1. Copy the hot-fix vm-sles11_x86_64-Patch-6.1.1.200.tar.gz to /tmp 2. Untar vm-sles11_x86_64-Patch-6.1.1.200.tar.gz to /tmp/hf # mkdir /tmp/hf # cd /tmp/hf # gunzip /tmp/vm-sles11_x86_64-Patch-6.1.1.200.tar.gz # tar xf /tmp/vm-sles11_x86_64-Patch-6.1.1.200.tar 3. Install the hotfix # pwd /tmp/hf # ./installVRTSvxvm611P200 [ ...] Install the patch manually: -------------------------- o Before-the-upgrade :- (a) Stop I/Os to all the VxVM volumes. (b) Umount any filesystems with VxVM volumes. (c) Stop applications using any VxVM volumes. o Select the appropriate RPMs for your system, and upgrade to the new patch. # rpm -Uhv VRTSvxvm-6.1.1.200-SLES11.x86_64.rpm REMOVING THE PATCH ------------------ # rpm -e SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE