* * * READ ME * * * * * * Symantec Volume Manager 6.1 * * * * * * Patch 6.1.0.100 * * * Patch Date: 2014-05-21 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH * KNOWN ISSUES PATCH NAME ---------- Symantec Volume Manager 6.1 Patch 6.1.0.100 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- AIX 6.1 ppc AIX 7.1 ppc PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxvm BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Symantec Dynamic Multi-Pathing 6.1 * Symantec Storage Foundation 6.1 * Symantec Storage Foundation Cluster File System HA 6.1 * Symantec Storage Foundation for Oracle RAC 6.1 * Symantec Storage Foundation HA 6.1 * Symantec Volume Manager 6.1 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: 6.1.0.100 * 3444900 (3399131) For PowerPath (PP) enclosure, both DA_TPD and DA_COEXIST_TPD flags are set. * 3445234 (3358904) System with ALUA enclosures sometimes panics during path fault scenarios * 3446126 (3338208) writes from fenced out node on Active-Passive (AP/F) shared storage device fail with unexpected error * 3447306 (3424798) Veritas Volume Manager (VxVM) mirror attach operations (e.g., plex attach, vxassist mirror, and third-mirror break-off snapshot resynchronization) may take longer time under heavy application I/O load. * 3447894 (3353211) A. After EMC Symmetrix BCV (Business Continuance Volume) device switches to read-write mode, continuous vxdmp (Veritas Dynamic Multi Pathing) error messages flood syslog. B. DMP metanode/path under DMP metanode gets disabled unexpectedly. * 3449714 (3417185) Rebooting host, after the exclusion of a dmpnode while I/O is in progress on it, leads to vxconfigd core dump. * 3452709 (3317430) The vxdiskunsetup utility throws error after upgradation from 5.1SP1RP4. * 3452727 (3279932) The vxdisksetup and vxdiskunsetup utilities were failing on disk which is part of a deported disk group (DG), even if "-f" option is specified. * 3452811 (3445120) Change tunable VOL_MIN_LOWMEM_SZ value to trigger early readback. * 3455455 (3409612) The value of reclaim_on_delete_start_time cannot be set to values outside the range: 22:00-03:59 * 3456729 (3428025) System running Symantec Replication Option (VVR) and configured as VVR primary crashes when heavy parallel I/Os load are issued * 3458036 (3418830) Node boot-up is getting hung while starting vxconfigd * 3458799 (3197987) vxconfigd dumps core, when 'vxddladm assign names file=' is executed and the file has one or more invalid values for enclosure vendor ID or product ID. * 3470346 (3377383) The vxconfigd crashes when a disk under Dynamic Multi-pathing (DMP) reports device failure. DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following Symantec incidents: Patch ID: 6.1.0.100 * 3444900 (Tracking ID: 3399131) SYMPTOM: The following command fails with an error for a path managed by Third Party Driver (TPD) which co-exists with DMP. # vxdmpadm -f disable path= VxVM vxdmpadm ERROR V-5-1-11771 Operation not supported DESCRIPTION: The Third Party Drivers manage the devices with or without the co-existence of Dynamic Multi Pathing driver. Disabling the paths managed by a third party driver which does not co-exist with DMP is not supported. But due to bug in the code, disabling the paths managed by a third party driver which co-exists with DMP also fails. The same flags are set for all third party driver devices. RESOLUTION: The code has been modified to block this command only for the third party drivers which cannot co-exist with DMP. * 3445234 (Tracking ID: 3358904) SYMPTOM: System panics with following stack: dmp_alua_get_owner_state() dmp_alua_get_path_state() dmp_get_path_state() dmp_get_enabled_ctlrs() dmp_info_ioctl() gendmpioctl() dmpioctl() vol_dmp_ktok_ioctl() dmp_get_enabled_cntrls() vx_dmp_config_ioctl() quiescesio_start() voliod_iohandle() voliod_loop() kernel_thread() DESCRIPTION: System running with Asymmetric Logical Unit Access (ALUA) LUNs sometimes panics during path fault scenarios. This happens due to possible NULL pointer access in some of cases due to bug in the code . RESOLUTION: Code changes have been made to fix the bug. * 3446126 (Tracking ID: 3338208) SYMPTOM: writes from fenced out LDOM guest node on Active-Passive (AP/F) shared storage device fail. I/O failure messages seen in system log.. .. Mon Oct 14 06:03:39.411: I/O retry(6) on Path c0d8s2 belonging to Dmpnode emc_clariion0_48 Mon Oct 14 06:03:39.951: SCSI error occurred on Path c0d8s2: opcode=0x2a reported device not ready (status=0x2, key=0x2, asc=0x4, ascq=0x3) LUN not ready, manual intervention required Mon Oct 14 06:03:39.951: I/O analysis done as DMP_PATH_FAILURE on Path c0d8s2 belonging to Dmpnode emc_clariion0_48 Mon Oct 14 06:03:40.311: Marked as failing Path c0d1s2 belonging to Dmpnode emc_clariion0_48 Mon Oct 14 06:03:40.671: Disabled Path c0d8s2 belonging to Dmpnode emc_clariion0_48 due to path failure .. .. DESCRIPTION: write SCSI commands from fenced out host should fail with reservation conflict from shared device. This error code needs to be propagated to upper layers for appropriate action. In DMP ioctl interface, DMP first sends command through available active paths. If command fails then command was unnecessarily tried on passive paths. This causes command to be failed with not ready error code and this error code getting propagated to upper layer instead of reservation conflict. RESOLUTION: Added code changes to not retry IO SCSI commands on passive paths in case of Active-Passive (AP/F) shared storage device. * 3447306 (Tracking ID: 3424798) SYMPTOM: Veritas Volume Manager (VxVM) mirror attach operations (e.g., plex attach, vxassist mirror, and third-mirror break-off snapshot resynchronization) may take longer time under heavy application I/O load. The vxtask list command shows tasks are in the 'auto-throttled (waiting)' state for a long time. DESCRIPTION: With the AdminIO de-prioritization feature, VxVM administrative I/O's (e.g. plex attach, vxassist mirror, and third-mirror break-off snapshot resynchronization) are de-prioritized under heavy application I/O load, but this can lead to very slow progress of these operations. RESOLUTION: The code is modified to disable the AdminIO de-prioritization feature. * 3447894 (Tracking ID: 3353211) SYMPTOM: A. After EMC Symmetrix BCV (Business Continuance Volume) device switches to read-write mode, continuous vxdmp (Veritas Dynamic Multi Pathing) error messages flood syslog as shown below: NOTE VxVM vxdmp V-5-3-1061 dmp_restore_node: The path 18/0x2 has not yet aged - 299 NOTE VxVM vxdmp 0 dmp_tur_temp_pgr: open failed: error = 6 dev=0x24/0xD0 NOTE VxVM vxdmp V-5-3-1062 dmp_restore_node: Unstable path 18/0x230 will not be available for I/O until 300 seconds NOTE VxVM vxdmp V-5-3-1061 dmp_restore_node: The path 18/0x2 has not yet aged - 299 NOTE VxVM vxdmp V-5-0-0 [Error] i/o error occurred (errno=0x6) on dmpnode 36/0xD0 .. .. B. DMP metanode/path under DMP metanode gets disabled unexpectedly. DESCRIPTION: A. DMP caches the last discovery NDELAY open for the BCV dmpnode paths. BCV device switching to read-write mode is an array side operation. Typically in such cases, the system administrators are required to run the following command: 1. vxdisk rm OR In case of parallel backup jobs, 1. vxdisk offline 2. vxdisk online This causes DMP to close cached open, and during the next discovery, the device is opened in read-write mode. If the above steps are skipped then it causes the DMP device to go in state where one of the paths is in read-write mode and the others remain in NDELAY mode. If the above layers request for NORMAL open, then the DMP has the code to close NDELAY cached open and reopen in NORMAL mode. When the dmpnode is online, this happens only for one of the paths of dmpnode. B. DMP performs error analysis for paths on which I/O has failed. In some cases, the SCSI probes are sent, failed with the return value/sense codes that are not handled by DMP. This causes the paths to get disabled. RESOLUTION: A. The code is modified for the DMP EMC ASL (Array Support Library) to handle case A for EMC Symmetrix arrays. B. The DMP code is modified to handle the SCSI conditions correctly for case B. * 3449714 (Tracking ID: 3417185) SYMPTOM: Rebooting host, after the exclusion of a dmpnode while I/O is in progress on it, leads to vxconfigd core dump. DESCRIPTION: The function which deletes the path after exclusion, does not update the corresponding data structures properly. Consequently, rebooting the host, after the exclusion of dmpnode while I/O is in progress on it, leads to vxconfigd core dump with following stack ddl_find_devno_in_table () ddl_get_disk_policy () devintf_add_autoconfig_main () devintf_add_autoconfig () mode_set () req_vold_enable () request_loop () main () RESOLUTION: Code changes have been made to update the data structure properly. * 3452709 (Tracking ID: 3317430) SYMPTOM: "vxdiskunsetup" utility throws error during execution. Following error message is observed. "Device unexport failed: Operation is not supported". DESCRIPTION: In vxdisksetup vxunexport is called, without checking if the disk is exported or the CVM protocol version. When the bits are upgraded from 5.1SP1RP4, the CVM protocol version is not updated. Hence the error. RESOLUTION: Code changes done so that the vxunexport is called after doing proper checks. * 3452727 (Tracking ID: 3279932) SYMPTOM: The vxdisksetup and vxdiskunsetup utilities were failing on disk which is part of deported disk group (DG), even if '-f' option is specified. The vxdisksetup command fails with following error: VxVM vxedpart ERROR V-5-1-10089 partition modification failed : Device or resource busy The vxdiskunsetup command fails following error: VxVM vxdisk ERROR ERROR V-5-1-0 Device appears to be owned by disk group . Use -f option to force destroy. VxVM vxdiskunsetup ERROR V-5-2-5052 : Disk destroy failed. DESCRIPTION: The vxdisksetup and vxdiskunsetup utilities internally call the 'vxdisk' utility. Due to a defect in vxdisksetup and vxdiskunsetup, the vxdisk operation used to fail on disk which is part of deported DG, even if '-f' operation is requested by user. RESOLUTION: Code changes are done to the vxdisksetup and vxdiskunsetup utilities so that when "-f" option is specified the operation succeeds. * 3452811 (Tracking ID: 3445120) SYMPTOM: Default value of tunable 'vol_min_lowmem_sz' is not consistent on all platforms. DESCRIPTION: Default value of tunable 'vol_min_lowmem_sz' is not consistent on all platforms. RESOLUTION: Set the default value of tunable 'vol_min_lowmem_sz' to 32MB on all platforms. * 3455455 (Tracking ID: 3409612) SYMPTOM: Fails to run "vxtune reclaim_on_delete_start_time " if the specified value is outside the range of 22:00-03:59 (E.g. setting it to 04:00 or 19:30 fails). DESCRIPTION: Tunable reclaim_on_delete_start_time can be set to any time value within 00:00 to 23:59. But because of the wrong regular expression to parse time, it cannot be set to all values in 00:00 - 23:59. RESOLUTION: The regular expression has been updated to parse time format correctly. Now all values in 00:00-23:59 can be set. * 3456729 (Tracking ID: 3428025) SYMPTOM: When heavy parallel I/O load is issued, system running Symantec Replication Option (VVR) and configured as VVR primary, panics with the below stack: schedule vol_alloc vol_zalloc volsio_alloc_kc_types vol_subdisk_iogen_base volobject_iogen vol_rv_iogen volobject_iogen vol_rv_batch_write_start volkcontext_process vol_rv_start_next_batch vol_rv_batch_write_done [...] vol_rv_batch_write_done volkcontext_process vol_rv_start_next_batch vol_rv_batch_write_done volkcontext_process vol_rv_start_next_batch vol_rv_batch_kio volkiostart vol_linux_kio_start vol_fsvm_strategy DESCRIPTION: Heavy parallel I/O load leads to I/O throttling in Symantec Replication Option (VVR). Improper throttle handling leads to kernel stack overflow. RESOLUTION: Handled I/O throttle correctly which avoids stack overflow and subsequent panic. * 3458036 (Tracking ID: 3418830) SYMPTOM: Node boot-up is getting hung while starting vxconfigd if '/etc/VRTSvcs/conf/sysname' file or '/etc/vx/vxvm-hostprefix' file is not present. User will see following messages on console. # Starting up VxVM ... # VxVM general startup... After these messages, it will hung. DESCRIPTION: While generating unique prefix, we were calling scanf instead of sscanf for fetching prefix which resulted in this hang. So while starting vxconfigd, it was waiting for some user input because of scanf which resulted in hang while booting up the node. RESOLUTION: Code changes are done to address this issue. * 3458799 (Tracking ID: 3197987) SYMPTOM: vxconfigd dumps core, when 'vxddladm assign names file= is executed and the file has one or more invalid values for enclosure vendor ID or product ID. DESCRIPTION: When the input file provided to 'vxddladm assign names file=' has invalid vendor ID or product ID, the vxconfigd daemon is unable to find the corresponding enclosure being referred to and makes an invalid memory reference. Following stack trace can be seen- strncasecmp () from /lib/libc.so.6 ddl_load_namefile () req_ddl_set_names () request_loop () main () RESOLUTION: As a fix the vxconfigd daemon verifies the validity of input vendor ID and product ID before making a memory reference to the corresponding enclosure in its internal data structures. * 3470346 (Tracking ID: 3377383) SYMPTOM: vxconfigd crashes when a disk under DMP reports device failure. After this, the following error will be seen when a VxVM (Veritas Volume Manager)command is excuted:- "VxVM vxdisk ERROR V-5-1-684 IPC failure: Configuration daemon is not accessible" DESCRIPTION: If a disk fails and reports certain failure to DMP (Veritas Dynamic Multipathing), then vxconfigd crashes because that error is not handled properly. RESOLUTION: The code is modified to properly handle the device failures reported by a failed disk under DMP. INSTALLING THE PATCH -------------------- Run the Installer script to automatically install the patch: ----------------------------------------------------------- To install the patch perform the following steps on at least one node in the cluster: 1. Copy the hot-fix vm-aix-6.1.0.100-patches.tar.gz to /tmp 2. Untar vm-aix-6.1.0.100-patches.tar.gz to /tmp/hf # mkdir /tmp/hf # cd /tmp/hf # gunzip /tmp/vm-aix-6.1.0.100-patches.tar.gz # tar xf /tmp/vm-aix-6.1.0.100-patches.tar 3. Install the hotfix # pwd /tmp/hf # ./installVRTSvxvm610HF100 [ ...] Install the patch manually: -------------------------- If the currently installed VRTSvxvm is below 6.1.0.0 level, upgrade VRTSvxvm to 6.1.0.0 level before installing this patch. AIX maintenance levels and APARs can be downloaded from the IBM web site: http://techsupport.services.ibm.com 1. Since the patch process will configure the new kernel extensions, a) Stop I/Os to all the VxVM volumes. b) Ensure that no VxVM volumes are in use or open or mounted before starting the installation procedure. c) Stop applications using any VxVM volumes. 2. Check whether root support or DMP native support is enabled. If it is enabled, it will be retained after patch upgrade. # vxdmpadm gettune dmp_native_support If the current value is 'on', DMP native support is enabled on this machine. # vxdmpadm native list vgname=rootvg If the output is some list of hdisks, root support is enabled on this machine 3. Proceed with patch installation as mentioned below a. Before applying this VxVM 6.1.0.100 patch, stop the VEA Server's vxsvc process: # /opt/VRTSob/bin/vxsvcctrl stop b. If your system has Veritas Operation Manager(VOM) configured then check whether vxdclid daemon is running, if it is running then stop vxdclid daemon. Command to check the status of vxdclid daemon #/opt/VRTSsfmh/etc/vxdcli.sh status Command to stop the vxdclid daemon #/opt/VRTSsfmh/etc/vxdcli.sh stop c. To apply this patch, use following command: # installp -ag -d ./VRTSvxvm.bff VRTSvxvm d. To apply and commit this patch, use following command: # installp -acg -d ./VRTSvxvm.bff VRTSvxvm NOTE: Please refer installp(1M) man page for clear understanding on APPLY & COMMIT state of the package/patch. e. Reboot the system to complete the patch upgrade. #reboot f. If you have stopped vxdclid daemon before upgrade then re-start vxdclid daemon using following command #/opt/VRTSsfmh/etc/vxdcli.sh start REMOVING THE PATCH ------------------ Run the Uninstaller script to automatically remove the patch: ------------------------------------------------------------ To uninstall the patch perform the following step on at least one node in the cluster: # /opt/VRTS/install/uninstallVRTSvxvm610HF100 [ ...] Remove the patch manually: ------------------------- 1. Check whether root support or DMP native support is enabled or not: # vxdmpadm gettune dmp_native_support If the current value is "on", DMP native support is enabled on this machine. # vxdmpadm native list vgname=rootvg If the output is some list of hdisks, root support is enabled on this machine If disabled: goto step 3. If enabled: goto step 2. 2. If root support or DMP native support is enabled: a. It is essential to disable DMP native support. Run the following command to disable DMP native support as well as root support # vxdmpadm settune dmp_native_support=off b. If only root support is enabled, run the following command to disable root support # vxdmpadm native disable vgname=rootvg c. Reboot the system # reboot 3. a. Before backing out patch, stop the VEA server's vxsvc process: # /opt/VRTSob/bin/vxsvcctrl stop b. If your system has Veritas Operation Manager(VOM) configured then check whether vxdclid daemon is running, if it is running then stop vxdclid daemon. Command to check the status of vxdclid daemon #/opt/VRTSsfmh/etc/vxdcli.sh status Command to stop the vxdclid daemon #/opt/VRTSsfmh/etc/vxdcli.sh stop c. To reject the patch if it is in "APPLIED" state, use the following command and re-enable DMP support # installp -r VRTSvxvm 6.1.0.100 d. # reboot e. If you have stopped vxdclid daemon before upgrade then re-start vxdclid daemon using following command #/opt/VRTSsfmh/etc/vxdcli.sh start KNOWN ISSUES ------------ * Tracking ID: 3480154 SYMPTOM: The problem occurs when on Veritas Operation Manager(VOM)configured server. VxVM stops the daemons and unloads the VxVM kernel modules at time of patch upgrade.On VOM configured server vxdclid daemon uses one of the VxVM config file (/dev/vx/info)" which prevent to unload VxVM module like vxio, vxspec, vxdmp from kernel . WORKAROUND: Before upgrade, stop the vxdclid daemon. #/opt/VRTSsfmh/etc/vxdcli.sh stop After upgrade completes, start vxdclid daemon using command #/opt/VRTSsfmh/etc/vxdcli.sh start * Tracking ID: 3515765 SYMPTOM: LVM does SCSI2 reservation on disk devices to avoid concurrency violation. But SCSI2 reservation doesn't support multi-pathing operations. If DMP is enabled, the I/O operations to the given LVM disk are allowed only on the path containing volume groups. During disk initialization, if you try to do I/O operations to LVM disks over the paths which do not contain volume groups, those I/O operations fail. Hence, the disk write failure occurs. WORKAROUND: Before conversion, manually disable the paths which are to LVM disks but do not contain volume groups: 1. List the Logical Volume Group (LVG) on the machine: # lsvg 2. Find the path that the LVG is created on: # lspv | grep volume_group_name 3. Find vxdisks that are using this LVG: # vxdmpadm getsubpaths all | grep path_name Where path_name specifies the name of the path that the LVG is created on. 4. Find all the paths of the vxdisk: # vxdisk path | grep vxdisk_name 5. Disable the paths which are to LVM disks but don't contain volume groups: # vxdmpadm disable path=path_name 6. To confirm all paths are disabled except the one which has volume groups, check vxdisk path: # vxdisk path | grep vxdisk_name SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE