* * * READ ME * * * * * * Veritas Volume Manager 5.1 SP1 RP3 * * * * * * P-patch 1 * * * Patch Date: 2012-11-15 This document provides the following information: * PATCH NAME * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * OPERATING SYSTEMS SUPPORTED BY THE PATCH * INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Veritas Volume Manager 5.1 SP1 RP3 P-patch 1 PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxvm BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Veritas Storage Foundation for Oracle RAC 5.1 SP1 * Veritas Storage Foundation Cluster File System 5.1 SP1 * Veritas Storage Foundation 5.1 SP1 * Veritas Storage Foundation High Availability 5.1 SP1 * Veritas Storage Foundation Cluster File System for Oracle RAC 5.1 SP1 * Veritas Dynamic Multi-Pathing 5.1 SP1 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- SLES11 x86-64 INCIDENTS FIXED BY THE PATCH ---------------------------- This patch fixes the following Symantec incidents: Patch ID: 5.1.133.100 * 2485252 (Tracking ID: 2910043) SYMPTOM: Frequent swapin/swapout seen due to higher order memory requests DESCRIPTION: In VxVM operations such as plex attach, snapshot resync/reattach issue ATOMIC_COPY IOCTL's. Default I/O size for these operation is 1MB and VxVM allocates this memory from operating system. Memory allocations of such large size can results into swapin/swapout of pages and are not very efficient. In presence of lot of such operations , system may not work very efficiently. RESOLUTION: VxVM has its own I/O memory management module, which allocates pages from operating system and efficiently manage them. Modified ATOMIC_COPY code to make use of VxVM's internal I/O memory pool instead of directly allocating memory from operating system. * 2711758 (Tracking ID: 2710579) SYMPTOM: Data corruption can be observed on a CDS (Cross-platform Data Sharing) disk, as part of operations like LUN resize, Disk FLUSH, Disk ONLINE etc. The following pattern would be found in the data region of the disk. cyl alt 2 hd sec DESCRIPTION: The CDS disk maintains a SUN VTOC in the zeroth block and a backup label at the end of the disk. The VTOC maintains the disk geometry information like number of cylinders, tracks and sectors per track. The backup label is the duplicate of VTOC and the backup label location is determined from VTOC contents. If the content of SUN VTOC located in the zeroth sector are incorrect, this may result in the wrong calculation of the backup label location. If the wrongly calculated backup label location falls in the public data region rather than the end of the disk as designed, data corruption occurs. RESOLUTION: Suppressed writing the backup label to prevent the data corruption. * 2847333 (Tracking ID: 2834046) SYMPTOM: VxVM dynamically reminors all the volumes during DG import if the DG base minor numbers are not in the correct pool. This behaviour cases NFS client to have to re-mount all NFS file systems in an environment where CVM is used on the NFS server side. DESCRIPTION: Starting from 5.1, the minor number space is divided into two pools, one for private disk groups and another for shared disk groups. During DG import, the DG base minor numbers will be adjusted automatically if not in the correct pool, and so do the volumes in the disk groups. This behaviour reduces many minor conflicting cases during DG import. But in NFS environment, it makes all file handles on the client side stale. Customers had to unmount files systems and restart applications. RESOLUTION: A new tunable, "autoreminor", is introduced. The default value is "on". Most of the customers don't care about auto-reminoring. They can just leave it as it is. For a environment that autoreminoring is not desirable, customers can just turn it off. Another major change is that during DG import, VxVM won't change minor numbers as long as there is no minor conflicts. This includes the cases that minor numbers are in the wrong pool. * 2860208 (Tracking ID: 2859470) SYMPTOM: The EMC SRDF-R2 disk may go in error state when you create EFI label on the R1 disk. For example: R1 site # vxdisk -eo alldgs list | grep -i srdf emc0_008c auto:cdsdisk emc0_008c SRDFdg online c1t5006048C5368E580d266 srdf-r1 R2 site # vxdisk -eo alldgs list | grep -i srdf emc1_0072 auto - - error c1t5006048C536979A0d65 srdf-r2 DESCRIPTION: Since R2 disks are in write protected mode, the default open() call (made for read-write mode) fails for the R2 disks, and the disk is marked as invalid. RESOLUTION: As a fix, DMP was changed to be able to read the EFI label even on a write protected SRDF-R2 disk. * 2881862 (Tracking ID: 2878876) SYMPTOM: vxconfigd, VxVM configuration daemon dumps core with the following stack. vol_cbr_dolog () vol_cbr_translog () vold_preprocess_request () request_loop () main () DESCRIPTION: This core is a result of a race between two threads which are processing the requests from the same client. While one thread completed processing a request and is in the phase of releasing the memory used, other thread is processing a request "DISCONNECT" from the same client. Due to the race condition, the second thread attempted to access the memory which is being released and dumped core. RESOLUTION: The issue is resolved by protecting the common data of the client by a mutex. * 2883606 (Tracking ID: 2189812) SYMPTOM: While executing 'vxdisk updateudid' on a disk which is in 'online invalid' state causes vxconfigd to dump core with following stack: priv_join() req_disk_updateudid() request_loop() main() DESCRIPTION: While updating udid, nullity check was not done for an internal data structure. This lead vxconfigd to dump core. RESOLUTION: Code changes are done to add nullity checks for internal data structure. * 2906832 (Tracking ID: 2398954) SYMPTOM: Machine panics while doing I/O on a VxFS mounted instant snapshot with ODM smartsync enabled. The panic has the following stack. panic: post_hndlr(): Unresolved kernel interruption cold_vm_hndlr bubbledown as_ubcopy privlbcopy volkio_to_kio_copy vol_multistepsio_overlay_data vol_multistepsio_start voliod_iohandle voliod_loop kthread_daemon_startup DESCRIPTION: VxVM uses the fields av_back and av_forw of io buf structure to store its private information. VxFS also uses these fields to chain io buffers before passing I/O to VxVM. When an I/O is received at VxVM layer it always resets these fields. But if ODM smartsync is enabled, VxFS uses a special strategy routine to pass on hints to VxVM. Due to a bug in the special strategy routine, the fields av_back and av_forw are not reset and could be pointing to a valid buffer in VxFS io buffer chain. So VxVM interprets these fields wrongly and modifies its contents, it corrupts the next buffer in the chain leading to this panic. RESOLUTION: The fields av_back and av_forw of io buf structure are reset in the special strategy routine. * 2919718 (Tracking ID: 2919714) SYMPTOM: On a THIN lun, vxevac returns 0 without migrating unmounted VxFS volumes. The following error messages are displayed when an unmounted VxFS volumes is processed: VxVM vxsd ERROR V-5-1-14671 Volume v2 is configured on THIN luns and not mounted. Use 'force' option, to bypass smartmove. To take advantage of smartmove for supporting thin luns, retry this operation after mounting the volume. VxVM vxsd ERROR V-5-1-407 Attempting to cleanup after failure ... DESCRIPTION: On a THIN lun, VM will not move or copy data on an unmounted VxFS volumes unless smartmove is bypassed. The vxevac command fails needs to be enhanced to detect unmounted VxFS volumes on THIN luns and to support a force option that allows the user to bypass smartmove. RESOLUTION: The vxevac script has be modified to check for unmounted VxFS volumes on THIN luns prior to performing the migration. If an unmounted VxFS volume is detected the command fails with a non-zero return code and displays a message notifying the user to mount the volumes or bypass smartmove by specifying the force option: VxVM vxevac ERROR V-5-2-0 The following VxFS volume(s) are configured on THIN luns and not mounted: v2 To take advantage of smartmove support on thin luns, retry this operation after mounting the volume(s). Otherwise, bypass smartmove by specifying the '-f' force option. * 2928768 (Tracking ID: 2928764) SYMPTOM: If the tunable dmp_fast_recovery is set to off, PGR( Persistent Group Reservation) key registration fails except for the first path i.e. only for the first path PGR key gets registered. Consider we are registering keys as follows. # vxdmpadm settune dmp_fast_recovery=off # vxdmpadm settune dmp_log_level=9 # vxdmppr read -t REG /dev/vx/rdmp/hitachi_r7000_00d9 Node: /dev/vx/rdmp/hitachi_r7000_00d9 ASCII-KEY HEX-VALUE ----------------------------- # vxdmppr register -s BPGR0000 /dev/vx/rdmp/hitachi_r7000_00d9 # vxdmppr read -t REG /dev/vx/rdmp/hitachi_r7000_00d9 Node: /dev/vx/rdmp/hitachi_r7000_00d9 ASCII-KEY HEX-VALUE ----------------------------- BPGR0000 0x4250475230303030 This being a multipathed disk, only first path gets PGR key registered through it. You will see the log messages similar to following: Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-0-0 SCSI error opcode=0x5f returned rq_status=0x12 cdb_status=0x1 key=0x6 asc=0x2a ascq=0x3 on path 8/0x90 Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-3-0 dmp_scsi_ioctl: SCSI ioctl completed host_byte = 0x0 rq_status = 0x8 Sep 6 11:29:41 clabcctlx04 kernel: sd 4:0:0:4: reservation conflict Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-3-0 dmp_scsi_ioctl: SCSI ioctl completed host_byte = 0x11 rq_status = 0x17 Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-0-0 SCSI error opcode=0x5f returned rq_status=0x17 cdb_status=0x0 key=0x0 asc=0x0 ascq=0x0 on path 8/0xb0 Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-3-0 dmp_pr_send_cmd failed with transport error: uscsi_rqstatus = 23ret = -1 status = 0 on dev 8/0xb0 Sep 6 11:29:41 clabcctlx04 kernel: VxVM vxdmp V-5-3-0 dmp_scsi_ioctl: SCSI ioctl completed host_byte = 0x0 rq_status = 0x8 DESCRIPTION: After key for first path gets registered successfully, the second path gets a reservation conflict which is expected. But in case of synchronous mode i.e. when dmp_fast_recovery is off, we don't set the proper reservation flag, due to which the registration command fails with the transport error and PGR keys on other paths don't get registered. In asynchronous mode we set it correctly, hence don't see the issue there. RESOLUTION: Set the proper reservation flag so that the key can be registered for other paths as well. * 2929003 (Tracking ID: 2928987) SYMPTOM: vxconfigd hung is observed when IO failed by OS layer. DESCRIPTION: DMP is supposed to do number of IO retries that are defined by user. When it receives IO failure from OS layer, due to bug it restarts IO without checking IO retry count, thus IO gets stuck in loop infinitely RESOLUTION: Code changes are done in DMP to use the IO retry count defined by user. * 2940448 (Tracking ID: 2940446) SYMPTOM: I/O can hang on volume with space optimized snapshot if the underlying cache object is of very large size. It can also lead to data corruption in cache- object. DESCRIPTION: Cache volume maintains B+ tree for mapping the offset and its actual location in cache object. Copy-on-write I/O generated on snapshot volumes needs to determine the offset of particular I/O in cache object. Due to incorrect type- casting the value calculated for large offset truncates to smaller value due to overflow, leading to data corruption. RESOLUTION: Code changes are done to avoid overflow during offset calculation in cache object. * 2946948 (Tracking ID: 2406096) SYMPTOM: vxconfigd, VxVM configuration daemon, dumps core with the following stack: vol_cbr_oplistfree() vol_clntaddop() vol_cbr_translog() vold_preprocess_request() request_loop() main() DESCRIPTION: vxsnap utility forks a child process and the parent process exits. The child process continues the remaining work as a background process. It does not create a new connection with vxconfigd and continues to use the parent's connection. Since the parent is dead, vxconfigd cleans up its client structure. Corresponding to further requests from child process, vxconfigd tries accessing the client structure that was already freed and hence, dumps core. RESOLUTION: The issue is solved by initiating a separate connection with vxconfigd from the forked child. * 2949855 (Tracking ID: 2943637) SYMPTOM: System panicked after the process of expanding DMP IO statistic queue size. The following stack message can be observed in syslog before panic: oom_kill_process select_bad_process out_of_memory __alloc_pages_nodemask alloc_pages_current __vmalloc_area_node dmp_alloc __vmalloc_node dmp_alloc vmalloc_32 dmp_alloc dmp_zalloc dmp_iostatq_add dmp_iostatq_op dmp_process_stats dmp_daemons_loop DESCRIPTION: In the process of expanding DMP IO statistic queue size, memory is allocated in sleep/block way. When Linux kernel can't satisfy the memory allocation request, i.e. system under high load and the amount of per-CPU memory chunk can be large since amounts of CPU, it will invoke OOM killer to kill other processes/threads to free more memory, which may cause system panic. RESOLUTION: The code changes were made to allocate memory in non-sleep way in the process of expanding DMP IO statistic queue size, hence, it will return fail quickly if the system can't satisfy the request but not invoke OOM killer. * 2957608 (Tracking ID: 2671241) SYMPTOM: When the DRL log plex is configured in a volume, vxnotify doesn't report volume enabled message. DESCRIPTION: When the DRL log plex is configured in a volume, we will make a two phase start of the volume; The first is to start plexes and make the volume state DETACHED, then make the volume state ENABLED in the second phase after the log recovery. However when we are notifying the configuration change to the interested client, we are only checking the status change from DISABLED to ENABLED. RESOLUTION: With fix, notification is generated on state change of volume from any state to 'ENABLED' (and any state to 'DISABLED'). * 2962269 (Tracking ID: 2193755) SYMPTOM: The system panics with back trace similar to : notifier_call_chain notify_die __die no_context __bad_area_nosemaphore page_fault dmp_send_scsipkt_req dmp_send_scsipkt dmp_send_scsireq dmp_bypass_strategy dmp_path_okay dmp_error_action dmp_daemons_loop child_rip DESCRIPTION: In Linux, every IO buffer for a device has to be within the limit of maximum IO size which that device can handle. In addition the numbers of vectors pointing to the actual data segments within the IO buffer also have to be less than the maximum value. When IO cannot start on a page boundary one chunk of IO gets split into two IO vectors. As the number of vectors in the IO buffer exceeds the maximum IO vector limit on the subsystem, panic happens. RESOLUTION: Code changes are made to ensure that the IO size and the number of vectors in the IO buffer are within the allowed limit. * 2978414 (Tracking ID: 2416341) SYMPTOM: Even though paths of a DMP(Dynamic multi-pathing) device are not manually disabled using "vxdmpadm disable" CLI, some times they can be in "DISABLED(M)" state, if these paths are disconnected and re-connected back due to SAN reconfiguration. Even DMP device can be in "DISABLED" state if all the paths of the DMP device are in "DISABLED(M)" state. "vxdmpadm getsubpaths" CLI will show the paths in DISABLED(M) state as shown below: # vxdmpadm getsubpaths dmpnodename=emc0_00ec sdo DISABLED(M) - emc0_00ec emc0 c2 - sdu DISABLED(M) - emc0_00ec emc0 c1 - sdx DISABLED(M) - emc0_00ec emc0 c3 - DESCRIPTION: DMP listens to UDEV device ADD/REMOVE events. In response to device UDEV "REMOVE" events corresponding paths under DMP device are marked as "DISABLED(M)" to avoid sending further i/os on the path. Now when UDEV ADD events are raised on device reconnect, some times these events are not properly communicated to DMP. Thus paths remains in "DISABLED(M)" state. RESOLUTION: Done code changes to properly communicate UDEV Device "ADD" events to DMP so that paths are enabled back. * 2979692 (Tracking ID: 2575051) SYMPTOM: In a CVM environment, Master switch or master takeover operations results in panic with below stack. volobject_iogen vol_cvol_volobject_iogen vol_cvol_recover3_start voliod_iohandle voliod_loop kernel_thread DESCRIPTION: Panic happens while accessing fields of stale cache object. The cache recovery process gets initiated by master takeover or master switch operation. In the recovery process VxVM do not take I/O count on cache objects. In meanwhile, same cache object can go through transaction while recovery is still in progress. Therefore cache object gets changed as part of transaction and in recovery code path VxVM try to access stale cache object resulting in panic. RESOLUTION: This issue is addressed by code changes in cache recovery code path. INSTALLING THE PATCH -------------------- (sles11_x86_64) # rpm -Uhv VRTSvxvm-5.1.132.100-SP1RP3P1_SLES11.x86_64.rpm REMOVING THE PATCH ------------------ # rpm -e SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE