OS: AIX OS Version: 5.2, 5.3, 6.1 Fixes Applied for Products: VRTSgab - Veritas Group Membership and Atomic Broadcast by Symantec Additional Instructions: Please read the instructions below before installing the patch. PATCH VRTSgab 5.0MP3RP2HF1 for GAB 5.0MP3RP2 =============================================================== Patch Date: November, 2009 This README provides information on: * BEFORE GETTING STARTED * CRC AND BYTE COUNT * FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH * PACKAGES AFFECTED BY THE PATCH * INSTALLING THE PATCH IN VCS ENVIRONMENT * UNINSTALLING THE PATCH IN VCS ENVIRONMENT * INSTALLING THE PATCH IN SFRAC ENVIRONMENT * UNINSTALLING THE PATCH IN SFRAC ENVIRONMENT BEFORE GETTING STARTED: ---------------------- This patch only applies to: VRTSgab 5.0.3.200 running on AIX 5.2, 5.3 or 6.1 Ensure that you are running the supported configurations before installing this patch. CRC AND BYTE COUNT: ------------------ Ensure that the patch file you have downloaded matches the following checksum and byte count. The following command can be used to ascertain this: # cksum VRTSgab.rte.bff 1501532012 7321600 VRTSgab.rte.bff FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH: -------------------------------------------- Etrack Incidents: 1851614 SDR's of Fixed Symantec Incidents: -------------------------------- Symantec Incident : 1851614 Symptom: Node panics with panic string : "GAB: Port h halting system due to internal protocol error". Description: There was a race condition in the GAB's sequence recovery protocol, which can cause node panic when there delay in response of messages and disconnect from GAB Master. Resolution: Fixed the race condition in the GAB's sequence recovery protocol. PACKAGES AFFECTED BY THE PATCH: ------------------------------- This patch updates the following VCS package(s) VRTSgab.rte from 5.0.3.200 to 5.0.3.201 INSTALLING THE PATCH IN VCS ENVIRONMENT : --------------------------------------- The following steps should be run on all nodes in the VCS cluster: Stopping VCS on the cluster node: -------------------------------- 1. Offline all applications, which are configured on CVM/CFS and are outside VCS control. After all applications using CFS and CVM have been taken down, run 'slibclean' to unload the libraries from memory. 2. Stop VCS on the current node. # /opt/VRTSvcs/bin/hastop -local Verify that ports 'f' (CFS), 'v' and 'w' (CVM), 'h' (VCS) have been closed, # /sbin/gabconfig -a The display should not have port 'f', 'v', 'w' and 'h' listed. 3. If VXFEN is not configured, please go to step 5 4. Unconfigure VxFen: # /sbin/vxfenconfig -U Verify that port 'b' has been closed # /sbin/gabconfig -a The display should not have port 'b' listed 5. Unconfigure GAB: # /sbin/gabconfig -U 6. Unload the GAB driver: # /etc/methods/gabkext -stop 7. Verify that the GAB driver has been unloaded: # /etc/methods/gabkext -status gab: unloaded NOTE: If you are unable to successfully unload either the GAB driver, the server must be rebooted AFTER the installation of the patch. This is so that the new GAB driver gets loaded in the AIX kernel. Installing the Patch: -------------------- 1. Un-compress the downloaded patch from Symantec. Change directory to the unzipped patch location. Install the VRTSgab (5.0.3.201) patch using the following command: # installp -a -d [extract dir]/patches/VRTSgab.rte.bff VRTSgab.rte 2. Verify that the new patch has been installed: # lslpp -l VRTSgab.rte You will find the following output on display with the patch installed properly: VRTSgab.rte 5.0.3.201 APPLIED Veritas Group Membership and Atomic Broadcast by Symantec Re-starting VCS on the cluster node: ----------------------------------- 1. Verify that the new GAB driver has been loaded: # /etc/methods/gabkext -status gab: loaded 2. If not already loaded, load the newly installed GAB driver: # /etc/methods/gabkext -start 3. Configure GAB: # sh /etc/gabtab 4. Verify that the GAB membership shows up correctly: # /sbin/gabconfig -a The display should have Port 'a' listed 5. Configure VxFen (if VxFEN was configured previously) # /sbin/vxfenconfig -c Verify that vxfen has been configured # /sbin/gabconfig -a The output should list port 'b' 6. Start VCS: # /opt/VRTSvcs/bin/hastart Verify that VCS is up and running: # /sbin/gabconfig -a The display should show port 'f', 'v', 'w' and 'h' listed. The 'f', 'v' and 'w' port will be listed if CVM and CFS are configured. 7. Start applications (stopped earlier), which are outside VCS control. Committing the Patch: -------------------- 1. To commit the patch: (Note: That the patch cannot be backed out once it is committed) # installp -c VRTSgab.rte 2. Verify that the fileset is committed: # lslpp -l VRTSgab.rte VRTSgab.rte 5.0.3.201 COMMITTED Veritas Group Membership and Atomic Broadcast by Symantec UNINSTALLING THE PATCH IN VCS ENVIRONMENT : ----------------------------------------- The VRTSgab.rte patch can ONLY be backed out if it has NOT been committed. Steps to remove the Patch from a cluster node: --------------------------------------------- 1. Follow the steps provided under "Stopping VCS on the cluster node" section above, to stop VCS on the node & unload any drivers, as required. 2. Remove/Backout the patch by the following command: # installp -r VRTSgab.rte 5.0.3.201 3. Verify that the patch has been removed from the system: (Note: The previously installed fileset(s) will be in committed state again. It may differ from the mentioned, if a Hotfix was installed on top of VCS 5.0MP3RP2) # lslpp -l VRTSgab.rte You may find the following output on display with the patch removed properly: VRTSgab.rte 5.0.3.200 COMMITTED Veritas Group Membership and Atomic Broadcast by Symantec 4. Restart the node following the steps under "Re-starting VCS on the cluster node" section above. Note: The gab drivers will now refer to the old ones. INSTALLING THE PATCH IN SFRAC ENVIRONMENT : ----------------------------------------- The following steps should be run on all nodes in the cluster, with SFRAC stack installed: 1. Offline all applications, which are configured on CVM/CFS and are outside VCS control. 2. If Oracle database is not configured in VCS, stop it using following command: $ srvctl stop instance -d [database name] -i [instance name] 3(a). For Oracle 9iR2, stop 'gsd' using the following command as Oracle user: $ gsdctl stop To check the status of gsdctl, run the following command: $ gsdctl stat The gsdctl command is typically found in $ORACLE_HOME/bin. 3(b). For Oracle 10gR1 and 10gR2, Stop CRS manually, if CRS is not under VCS control. # /etc/init.crs stop 4. After all the oracle instances and other applications using CFS and CVM have been stopped, run 'slibclean' to unload the libraries from memory. 5. Stop VCS on the current node. # /opt/VRTSvcs/bin/hastop -local 6. Verify that ports 'h', 'f', 'v' and 'w' have been closed # /sbin/gabconfig -a The display should not have ports 'h', 'f', 'v' and 'w' listed 7. Unconfigure VCSMM: # /sbin/vcsmmconfig -U Verify that port 'o' has been closed # /sbin/gabconfig -a The display should not have port 'o' listed. If it does ensure that Oracle instances are offline. 8. Unconfigure LMX: # /sbin/lmxconfig -U 9. Unconfigure VxFen: # /sbin/vxfenconfig -U Verify that port 'b' has been closed # /sbin/gabconfig -a The display should not have port 'b' listed 10. Unmount ODM: # umount /dev/odm Verify that port 'd' has been closed # /sbin/gabconfig -a The display should not have port 'd' listed 11. At this point all gab ports except port 'a' should have been closed Verify this as follows: # /sbin/gabconfig -a 12. Follow steps 5 to 7 of "Stopping VCS on the cluster" section from "INSTALLING THE PATCH IN VCS ENVIRONMENT" chapter above. 13. Follow all the instruction in "Installing the patch" section from "INSTALLING THE PATCH IN VCS ENVIRONMENT" chapter above. 14. Follow steps 1 to 5 of "Re-starting VCS on the cluster" section from "INSTALLING THE PATCH IN VCS ENVIRONMENT" chapter above. 15. Configure LMX: # /sbin/lmxconfig -c 16. Configure VCSMM: # /sbin/vcsmmconfig -c Verify that vxfen has been configured # /sbin/gabconfig -a The output should list port 'o' 17. Mount ODM: # mount /dev/odm 18. Start VCS: # /opt/VRTSvcs/bin/hastart 19. Check if all ports are now open # /sbin/gabconfig -a The output should list ports 'a', 'b', 'd', 'f', 'h', 'o', 'v', and 'w'. 20(a). For Oracle 10gR1 and 10gR2, start CRS manually, if CRS is not under VCS control. # /etc/init.crs start 20(b). For Oracle 9iR2, start 'gsd' using the following command as Oracle user $ gsdctl start To check the status of gsdctl, run the following command: $ gsdctl stat The gsdctl command is typically found in $ORACLE_HOME/bin. 21. If Oracle database is not configured in VCS, start it using following procedure. $ srvctl start instance -d [database name] -i [instance name] 22. Online all applications, which are configured on CVM/CFS and are outside VCS control (stopped earlier). 23. To commit the patches follow "Committing the Patch" section from "INSTALLING THE PATCH IN VCS ENVIRONMENT" chapter above. UNINSTALLING THE PATCH IN SFRAC ENVIRONMENT: ------------------------------------------- The VRTSgab.rte.bff patch can ONLY be backed out if it has not been committed. Steps to Backout the Patch: -------------------------- 1. Follow the steps outlined 1 through 17 of chapter "INSTALLING THE PATCH IN SFRAC ENVIRONMENT" to stop and unload the drivers. 2. Remove/Backout the patch by the following command: # installp -r VRTSgab.rte 5.0.3.201 3. Verify that the patch has been removed from the system: (Note: The previously installed fileset(s) will be in committed state again. It may differ from the mentioned, if a Hotfix was installed on top of VCS 5.0MP3RP2) # lslpp -l VRTSgab.rte You may find the following output on display with the patch removed properly: VRTSgab.rte 5.0.3.200 COMMITTED Veritas Group Membership and Atomic Broadcast by Symantec 4. Next as before go through the process of loading and configuring GAB and bringing up SFRAC (steps 19 through 27 above of chapter "INSTALLING THE PATCH IN SFRAC ENVIRONMENT"). Note: The gab drivers will now refer to the old ones.