OS: AIX
OS Version: 5.3, 6.1
Fixes Applied for Products:
VRTSvxfen - Veritas I/O Fencing by Symantec
Additional Instructions:
Please read the following instructions before installing the patch.
PATCH VRTSvxfen 5.1SP1P1 for VxFEN 5.1SP1
==========================================
Patch Date: Nov 29, 2010
This README provides information on:
* BEFORE GETTING STARTED
* CRC AND BYTE COUNT
* FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH
* PACKAGES AFFECTED BY THE PATCH
* INSTALLING THE PATCH IN THE VCS ENVIRONMENT
* UNINSTALLING THE PATCH FROM THE VCS ENVIRONMENT
* INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT
* UNINSTALLING THE PATCH FROM THE SFRAC ENVIRONMENT
BEFORE GETTING STARTED
----------------------
This patch applies only to:
VRTSvxfen 5.1.100.0 running on AIX 5.3 or 6.1
Ensure that you are running the supported configurations before
installing this patch.
CRC AND BYTE COUNT
------------------
Ensure that the patch file you downloaded matches the following
checksum and byte count.
To ascertain the checksum and byte count, use the following command:
# cksum VRTSvxfen.bff
540951639 3430400 VRTSvxfen.bff
FIXES AND ENHANCEMENTS INCLUDED IN THE PATCH
--------------------------------------------
Etrack Incidents: 2177011
Symptom-Description-Resolution (SDRs) of Fixed Symantec Incidents
------------------------------------------------------------------
Symptom:
All the nodes in a running cluster may panic under the
following circumstances:
1. You configure VxFEN with mode=scsi3 and disk_policy=dmp
2. The nodes are disconnected from one or more coordinator disks
3. If one of the nodes then crashes, or is rebooted, or if a
network partition occurs further, the VxFEN RACER node
displays the following stack at the time of panic:
pvthread+01B300 STACK:
.simple_lock+000000 ()
closef+000024 (??)
fp_close+00004C (??)
.kernel_add_gate_cstack+000030 ()
.vxfen_plat_dummy_read_io+0000F4 ()
.vxfen_dummy_read_io+0001FC ()
.vxfen_grab_coord_disks+000210 ()
.vxfen_grab_coord_pt+0000C4 ()
.vxfen_msg_node_left_ack+0001E0 ()
.vxfen_process_client_msg+0005DC ()
.vxfen_vrfsm_cback+000FE0 ()
vrfsm_step+000374 (0000000400000004, F100070006E94460)
vrfsm_msg_dispatch+000264 (F100070007844500)
vrfsm_recv_thread+000164 (0000000000000000)
vxplat_aix_thread_base+000020 (0FFFFFFFF4017FE0)
Description:
If storage is disconnected, then the fp_opendev() function
call on the disk device returns failure as expected; the VxFEN
driver nevertheless attempts to access the invalid device
file pointer and therefore panics.
Resolution:
Symantec has modified the code to ensure that the VxFEN driver
accesses the device file pointer, only if the fp_opendev()
function call returns success.
PACKAGES AFFECTED BY THE PATCH
-------------------------------
This patch updates the following VCS package(s)
VRTSvxfen from 5.1.100.0 to 5.1.100.1
INSTALLING THE PATCH IN THE VCS ENVIRONMENT
---------------------------------------
Execute the following steps on all the nodes in the VCS cluster:
Stopping VCS on the cluster node
--------------------------------
1. Take offline all applications, which are configured on CVM/CFS
and are outside VCS control.
After you take offline all applications that use CFS and CVM,
run 'slibclean' to unload the libraries from memory.
2. Stop VCS on the current node:
# /opt/VRTSvcs/bin/hastop -local
Verify that ports 'f' (CFS), 'v' and 'w' (CVM), and 'h' (VCS) are closed:
# /sbin/gabconfig -a
The display must not include port 'f', 'v', 'w' and 'h'.
3. Unconfigure VxFEN:
# /etc/init.d/vxfen.rc stop
Verify that port 'b' is closed:
# /sbin/gabconfig -a
The display must not include port 'b'.
4. Unload the VxFEN driver:
# /etc/methods/vxfenext -stop
5. Verify that the VxFEN driver is unloaded:
# /etc/methods/vxfenext -status
If the VxFEN driver is still loaded, the output includes "yes".
Installing the patch
--------------------
1. Un-compress the patch downloaded from Symantec.
Change directory to the unzipped patch location.
Install the VRTSvxfen (5.1.100.1) patch by using the following command:
# installp -a -d [extract dir]/patches/VRTSvxfen.bff VRTSvxfen
2. Verify that the new patch has been installed:
# lslpp -l VRTSvxfen
If the patch is properly installed, the following output appears:
Path: /usr/lib/objrepos
VRTSvxfen 5.1.100.1 APPLIED Veritas I/O Fencing by
Symantec
5.1.100.100-5.1SP1P1-2010-10-2
9_18.30.00 Build Env:AIX ogma
3 5 0001D4CAD300 IBM, 9115-505
Re-starting VCS on the cluster node
------------------------------------
1. Verify that the new VxFEN driver is loaded:
# /etc/methods/vxfenext -status
2. If not already loaded, load the newly installed VxFEN driver:
# /etc/methods/vxfenext -start
3. Configure VxFEN:
# /etc/init.d/vxfen.rc start
Verify that vxfen is configured:
# /sbin/gabconfig -a
The output must display port 'b'.
On one of the nodes, type:
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
4. Start VCS:
# /opt/VRTSvcs/bin/hastart
Verify that VCS is up and running:
# /sbin/gabconfig -a
The output must include ports 'h'.
If CVM and CFS are configured, the 'f', 'v' and 'w' ports are also listed .
5. Start the applications (stopped earlier) that are outside VCS control.
Committing the patch
--------------------
1. To commit the patch:
(Note: The patch cannot be backed out once committed.)
# installp -c VRTSvxfen
2. Verify that the fileset is committed:
# lslpp -l VRTSvxfen
The following output must appear:
Path: /usr/lib/objrepos
VRTSvxfen 5.1.100.1 COMMITTED Veritas I/O Fencing by
Symantec
5.1.100.100-5.1SP1P1-2010-10-2
9_18.30.00 Build Env:AIX ogma
3 5 0001D4CAD300 IBM, 9115-505
by Symantec
UNINSTALLING THE PATCH FROM THE VCS ENVIRONMENT
-----------------------------------------------
The VRTSvxfen patch can only be backed out if it is not committed.
Steps to remove the patch from a cluster node
---------------------------------------------
1. To stop VCS on the node and unload any drivers, as required, execute
the steps described in the "Stopping VCS on the cluster node"
section.
2. Back out the patch by using the following command:
# installp -r VRTSvxfen 5.1.100.1
3. Verify that the patch is removed from the system:
(Note: The previously installed fileset(s) will be in the committed
state again, unless a hotfix is installed on top of VCS 5.1SP1)
# lslpp -l VRTSvxfen
If the patch is properly removed, the following output appears:
Path: /usr/lib/objrepos
VRTSvxfen 5.1.100.0 APPLIED Veritas I/O Fencing by
Symantec
5.1.100.000-5.1SP1-2010-09-30_
23.30.00 Build Env:AIX ogma 3
5 0001D4CAD300 IBM, 9115-505
4. Restart the node by executing the steps described in the
"Re-starting VCS on the cluster node" section.
INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT:
---------------------------------------------
Execute the following steps on all the nodes in the cluster that has
the SFRAC stack installed:
1. Take offline all applications that are configured on CVM/CFS
and are outside VCS control.
2. If the Oracle database is not configured under VCS control,
stop it by using the following command:
$ srvctl stop instance -d [database name] -i [instance name]
3. If the CRS is not configured under VCS control, stop it by using
the following command:
# crsctl stop crs
4. After all the Oracle instances and other applications using
CFS and CVM are stopped, run 'slibclean' to unload
the libraries from memory.
5. Stop VCS on the current node:
# /opt/VRTSvcs/bin/hastop -local
6. Verify that ports 'h', 'f', 'v' and 'w' are closed:
# /sbin/gabconfig -a
The display must not include ports 'h', 'f', 'v', and 'w'.
7. Unconfigure VCSMM:
# /sbin/vcsmmconfig -U
Verify that port 'o' is closed:
# /sbin/gabconfig -a
The display must not include port 'o'. This also ensures that
all Oracle instances are offline.
8. Unconfigure VxFEN:
# /etc/init.d/vxfen.rc stop
Verify that port 'b' is closed:
# /sbin/gabconfig -a
The display must not include port 'b'.
9. Unload the VxFEN driver:
# /etc/methods/vxfenext -stop
10. Verify that the VxFEN driver is unloaded:
# /etc/methods/vxfenext -status
If the VxFEN driver is still loaded, the output includes "yes".
11. Execute the steps listed in the "Installing the patch" section
of the "INSTALLING THE PATCH FROM THE VCS ENVIRONMENT" chapter.
12. Verify that the new VxFEN driver is loaded:
# /etc/methods/vxfenext -status
13. If not already loaded, load the newly installed VxFEN driver:
# /etc/methods/vxfenext -start
14. Configure VxFEN:
# /etc/init.d/vxfen.rc start
Verify that VxFEN is configured:
# /sbin/gabconfig -a
The output must list port 'b'.
On one of the nodes, type:
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
15. Configure VCSMM:
# /sbin/vcsmmconfig -c
Verify that VCSMM is configured:
# /sbin/gabconfig -a
The output must list port 'o'.
16. Start VCS:
# /opt/VRTSvcs/bin/hastart
17. Check if all ports are now open:
# /sbin/gabconfig -a
The output must list ports
'a', 'b', 'd', 'f', 'h', 'o', 'v', and 'w'.
18 If the CRS is not configured under VCS control, start it by
using the following command.
# crsctl start crs
19. If the Oracle database is not configured in VCS,
start it by using the following command.
$ srvctl start instance -d [database name] -i [instance name]
20. Bring online all applications that are configured on CVM/CFS
and are outside VCS control (stopped earlier).
21. To commit the patches, perform the steps described in the
"Committing the Patch" section in the
"INSTALLING THE PATCH IN THE VCS ENVIRONMENT" chapter.
UNINSTALLING THE PATCH FROM THE SFRAC ENVIRONMENT
--------------------------------------------------
The VRTSvxfen patch can only be backed out if it has not been committed.
Steps to back out the patch:
-----------------------------------------
1. To stop the drivers, perform steps 1 through 9 of the
"INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT" chapter.
2. Back out the patch by using the following command:
# installp -r VRTSvxfen 5.1.100.1
3. Verify that the patch has been removed from the system:
(Note: The previously installed fileset(s) will be in committed state again,
unless a hotfix is installed on top of SFRAC 5.1Sp1)
# lslpp -l VRTSvxfen
If the patch is properly removed, the following output appears:
Path: /usr/lib/objrepos
VRTSvxfen 5.1.100.0 APPLIED Veritas I/O Fencing by
Symantec
5.1.100.000-5.1SP1-2010-09-30_
23.30.00 Build Env:AIX ogma 3
5 0001D4CAD300 IBM, 9115-505
4. To configure VxFEN and bring up SFRAC, perform steps 12 through 20 above of
chapter "INSTALLING THE PATCH IN THE SFRAC ENVIRONMENT").
