OS: AIX OS Version: 5.3, 6.1 Fixes Applied for Products: VRTSvxfen - Veritas I/O Fencing by Symantec Additional Instructions: Please read the following instructions before installing the patch. PATCH VRTSvxfen 5.1SP1P1 for VxFEN 5.1SP1 ========================================== Patch Date: Nov 29, 2010 This README provides information on: * BEFORE GETTING STARTED * CRC AND BYTE COUNT * FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH * PACKAGES AFFECTED BY THE PATCH * INSTALLING THE PATCH IN THE VCS ENVIRONMENT * UNINSTALLING THE PATCH FROM THE VCS ENVIRONMENT * INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT * UNINSTALLING THE PATCH FROM THE SFRAC ENVIRONMENT BEFORE GETTING STARTED ---------------------- This patch applies only to: VRTSvxfen 5.1.100.0 running on AIX 5.3 or 6.1 Ensure that you are running the supported configurations before installing this patch. CRC AND BYTE COUNT ------------------ Ensure that the patch file you downloaded matches the following checksum and byte count. To ascertain the checksum and byte count, use the following command: # cksum VRTSvxfen.bff 540951639 3430400 VRTSvxfen.bff FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH -------------------------------------------- Etrack Incidents: 2177011 Symptom-Description-Resolution (SDRs) of Fixed Symantec Incidents ------------------------------------------------------------------ Symptom: All the nodes in a running cluster may panic under the following circumstances: 1. You configure VxFEN with mode=scsi3 and disk_policy=dmp 2. The nodes are disconnected from one or more coordinator disks 3. If one of the nodes then crashes, or is rebooted, or if a network partition occurs further, the VxFEN RACER node displays the following stack at the time of panic: pvthread+01B300 STACK: .simple_lock+000000 () closef+000024 (??) fp_close+00004C (??) .kernel_add_gate_cstack+000030 () .vxfen_plat_dummy_read_io+0000F4 () .vxfen_dummy_read_io+0001FC () .vxfen_grab_coord_disks+000210 () .vxfen_grab_coord_pt+0000C4 () .vxfen_msg_node_left_ack+0001E0 () .vxfen_process_client_msg+0005DC () .vxfen_vrfsm_cback+000FE0 () vrfsm_step+000374 (0000000400000004, F100070006E94460) vrfsm_msg_dispatch+000264 (F100070007844500) vrfsm_recv_thread+000164 (0000000000000000) vxplat_aix_thread_base+000020 (0FFFFFFFF4017FE0) Description: If storage is disconnected, then the fp_opendev() function call on the disk device returns failure as expected; the VxFEN driver nevertheless attempts to access the invalid device file pointer and therefore panics. Resolution: Symantec has modified the code to ensure that the VxFEN driver accesses the device file pointer, only if the fp_opendev() function call returns success. PACKAGES AFFECTED BY THE PATCH ------------------------------- This patch updates the following VCS package(s) VRTSvxfen from 5.1.100.0 to 5.1.100.1 INSTALLING THE PATCH IN THE VCS ENVIRONMENT --------------------------------------- Execute the following steps on all the nodes in the VCS cluster: Stopping VCS on the cluster node -------------------------------- 1. Take offline all applications, which are configured on CVM/CFS and are outside VCS control. After you take offline all applications that use CFS and CVM, run 'slibclean' to unload the libraries from memory. 2. Stop VCS on the current node: # /opt/VRTSvcs/bin/hastop -local Verify that ports 'f' (CFS), 'v' and 'w' (CVM), and 'h' (VCS) are closed: # /sbin/gabconfig -a The display must not include port 'f', 'v', 'w' and 'h'. 3. Unconfigure VxFEN: # /etc/init.d/vxfen.rc stop Verify that port 'b' is closed: # /sbin/gabconfig -a The display must not include port 'b'. 4. Unload the VxFEN driver: # /etc/methods/vxfenext -stop 5. Verify that the VxFEN driver is unloaded: # /etc/methods/vxfenext -status If the VxFEN driver is still loaded, the output includes "yes". Installing the patch -------------------- 1. Un-compress the patch downloaded from Symantec. Change directory to the unzipped patch location. Install the VRTSvxfen (5.1.100.1) patch by using the following command: # installp -a -d [extract dir]/patches/VRTSvxfen.bff VRTSvxfen 2. Verify that the new patch has been installed: # lslpp -l VRTSvxfen If the patch is properly installed, the following output appears: Path: /usr/lib/objrepos VRTSvxfen 5.1.100.1 APPLIED Veritas I/O Fencing by Symantec 5.1.100.100-5.1SP1P1-2010-10-2 9_18.30.00 Build Env:AIX ogma 3 5 0001D4CAD300 IBM, 9115-505 Re-starting VCS on the cluster node ------------------------------------ 1. Verify that the new VxFEN driver is loaded: # /etc/methods/vxfenext -status 2. If not already loaded, load the newly installed VxFEN driver: # /etc/methods/vxfenext -start 3. Configure VxFEN: # /etc/init.d/vxfen.rc start Verify that vxfen is configured: # /sbin/gabconfig -a The output must display port 'b'. On one of the nodes, type: # vxfenadm -d I/O Fencing Cluster Information: ================================ Fencing Protocol Version: 201 Fencing Mode: SCSI3 Fencing SCSI3 Disk Policy: dmp Cluster Members: * 0 (galaxy) 1 (nebula) RFSM State Information: node 0 in state 8 (running) node 1 in state 8 (running) 4. Start VCS: # /opt/VRTSvcs/bin/hastart Verify that VCS is up and running: # /sbin/gabconfig -a The output must include ports 'h'. If CVM and CFS are configured, the 'f', 'v' and 'w' ports are also listed . 5. Start the applications (stopped earlier) that are outside VCS control. Committing the patch -------------------- 1. To commit the patch: (Note: The patch cannot be backed out once committed.) # installp -c VRTSvxfen 2. Verify that the fileset is committed: # lslpp -l VRTSvxfen The following output must appear: Path: /usr/lib/objrepos VRTSvxfen 5.1.100.1 COMMITTED Veritas I/O Fencing by Symantec 5.1.100.100-5.1SP1P1-2010-10-2 9_18.30.00 Build Env:AIX ogma 3 5 0001D4CAD300 IBM, 9115-505 by Symantec UNINSTALLING THE PATCH FROM THE VCS ENVIRONMENT ----------------------------------------------- The VRTSvxfen patch can only be backed out if it is not committed. Steps to remove the patch from a cluster node --------------------------------------------- 1. To stop VCS on the node and unload any drivers, as required, execute the steps described in the "Stopping VCS on the cluster node" section. 2. Back out the patch by using the following command: # installp -r VRTSvxfen 5.1.100.1 3. Verify that the patch is removed from the system: (Note: The previously installed fileset(s) will be in the committed state again, unless a hotfix is installed on top of VCS 5.1SP1) # lslpp -l VRTSvxfen If the patch is properly removed, the following output appears: Path: /usr/lib/objrepos VRTSvxfen 5.1.100.0 APPLIED Veritas I/O Fencing by Symantec 5.1.100.000-5.1SP1-2010-09-30_ 23.30.00 Build Env:AIX ogma 3 5 0001D4CAD300 IBM, 9115-505 4. Restart the node by executing the steps described in the "Re-starting VCS on the cluster node" section. INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT: --------------------------------------------- Execute the following steps on all the nodes in the cluster that has the SFRAC stack installed: 1. Take offline all applications that are configured on CVM/CFS and are outside VCS control. 2. If the Oracle database is not configured under VCS control, stop it by using the following command: $ srvctl stop instance -d [database name] -i [instance name] 3. If the CRS is not configured under VCS control, stop it by using the following command: # crsctl stop crs 4. After all the Oracle instances and other applications using CFS and CVM are stopped, run 'slibclean' to unload the libraries from memory. 5. Stop VCS on the current node: # /opt/VRTSvcs/bin/hastop -local 6. Verify that ports 'h', 'f', 'v' and 'w' are closed: # /sbin/gabconfig -a The display must not include ports 'h', 'f', 'v', and 'w'. 7. Unconfigure VCSMM: # /sbin/vcsmmconfig -U Verify that port 'o' is closed: # /sbin/gabconfig -a The display must not include port 'o'. This also ensures that all Oracle instances are offline. 8. Unconfigure VxFEN: # /etc/init.d/vxfen.rc stop Verify that port 'b' is closed: # /sbin/gabconfig -a The display must not include port 'b'. 9. Unload the VxFEN driver: # /etc/methods/vxfenext -stop 10. Verify that the VxFEN driver is unloaded: # /etc/methods/vxfenext -status If the VxFEN driver is still loaded, the output includes "yes". 11. Execute the steps listed in the "Installing the patch" section of the "INSTALLING THE PATCH FROM THE VCS ENVIRONMENT" chapter. 12. Verify that the new VxFEN driver is loaded: # /etc/methods/vxfenext -status 13. If not already loaded, load the newly installed VxFEN driver: # /etc/methods/vxfenext -start 14. Configure VxFEN: # /etc/init.d/vxfen.rc start Verify that VxFEN is configured: # /sbin/gabconfig -a The output must list port 'b'. On one of the nodes, type: # vxfenadm -d I/O Fencing Cluster Information: ================================ Fencing Protocol Version: 201 Fencing Mode: SCSI3 Fencing SCSI3 Disk Policy: dmp Cluster Members: * 0 (galaxy) 1 (nebula) RFSM State Information: node 0 in state 8 (running) node 1 in state 8 (running) 15. Configure VCSMM: # /sbin/vcsmmconfig -c Verify that VCSMM is configured: # /sbin/gabconfig -a The output must list port 'o'. 16. Start VCS: # /opt/VRTSvcs/bin/hastart 17. Check if all ports are now open: # /sbin/gabconfig -a The output must list ports 'a', 'b', 'd', 'f', 'h', 'o', 'v', and 'w'. 18 If the CRS is not configured under VCS control, start it by using the following command. # crsctl start crs 19. If the Oracle database is not configured in VCS, start it by using the following command. $ srvctl start instance -d [database name] -i [instance name] 20. Bring online all applications that are configured on CVM/CFS and are outside VCS control (stopped earlier). 21. To commit the patches, perform the steps described in the "Committing the Patch" section in the "INSTALLING THE PATCH IN THE VCS ENVIRONMENT" chapter. UNINSTALLING THE PATCH FROM THE SFRAC ENVIRONMENT -------------------------------------------------- The VRTSvxfen patch can only be backed out if it has not been committed. Steps to back out the patch: ----------------------------------------- 1. To stop the drivers, perform steps 1 through 9 of the "INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT" chapter. 2. Back out the patch by using the following command: # installp -r VRTSvxfen 5.1.100.1 3. Verify that the patch has been removed from the system: (Note: The previously installed fileset(s) will be in committed state again, unless a hotfix is installed on top of SFRAC 5.1Sp1) # lslpp -l VRTSvxfen If the patch is properly removed, the following output appears: Path: /usr/lib/objrepos VRTSvxfen 5.1.100.0 APPLIED Veritas I/O Fencing by Symantec 5.1.100.000-5.1SP1-2010-09-30_ 23.30.00 Build Env:AIX ogma 3 5 0001D4CAD300 IBM, 9115-505 4. To configure VxFEN and bring up SFRAC, perform steps 12 through 20 above of chapter "INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT").