The following procedure can be use to troubleshoot a mixed I/O fencing configuration (configuration using both coordinator disks and CP server for I/O fencing). This procedure involves using the following commands to obtain I/O fencing information:
To obtain I/O fencing cluster information on the coordinator disks, run the following command on one of the cluster nodes:
# vxfenadm -s diskname
Any keys other than the valid keys used by the cluster nodes that appear in the command output are spurious keys.
To obtain I/O fencing cluster information on the CP server, run the following command on one of the cluster nodes:
# cpsadm -s cp_server -a list_membership -c cluster_name
where cp server is the virtual IP address or virtual hostname on which the CP server is listening, and cluster name is the VCS name for the Storage Foundation High Availability.
Nodes which are not in GAB membership, but registered with CP server indicate a pre-existing network partition.
Note that when running this command on the Storage Foundation High Availability nodes, you need to first export the CPS_USERNAME and CPS_DOMAINTYPE variables.
The CPS_USERNAME value is the user name which is added for this node on the CP server.
To obtain the user name, run the following command on the CP server:
# cpsadm -s cp_server -a list_users
where cp server is the virtual IP address or virtual hostname on which the CP server is listening.
The CPS_DOMAINTYPE value is vx.
The following are export variable command examples:
# export CPS_USERNAME=_HA_VCS_test-system@HA_SERVICES@test-system.symantec.com
# export CPS_DOMAINTYPE=vx
Once a pre-existing network partition is detected using the above commands, all spurious keys on the coordinator disks or CP server must be removed by the administrator.
Troubleshooting mixed I/O fencing configuration (coordinator disks and CP server)
Review the current I/O fencing configuration by accessing and viewing the information in the vxfenmode file.
Enter the following command on one of the Storage Foundation High Availability nodes:
# cat /etc/vxfenmode
vxfen_mode=customized vxfen_mechanism=cps scsi3_disk_policy=dmp security=0 cps1=[10.140.94.101]:14250 vxfendg=vxfencoorddg
Review the I/O fencing cluster information.
Enter the vxfenadm -d command on one of the cluster nodes:
# vxfenadm -d
I/O Fencing Cluster Information: ================================ Fencing Protocol Version: 201 Fencing Mode: Customized Fencing Mechanism: cps Cluster Members: * 0 (galaxy) 1 (nebula) RFSM State Information: node 0 in state 8 (running) node 1 in state 8 (running)
Review the SCSI registration keys for the coordinator disks used in the I/O fencing configuration.
Enter the vxfenadm -s command on each of the Storage Foundation High Availability nodes.
# vxfenadm -s /dev/vx/rdmp/3pardata0_190
Device Name: /dev/vx/rdmp/3pardata0_190 Total Number Of Keys: 2 key[0]: [Numeric Format]: 86,70,66,69,65,68,48,48 [Character Format]: VFBEAD00 [Node Format]: Cluster ID: 57069 Node ID: 0 Node Name: galaxy key[1]: [Numeric Format]: 86,70,66,69,65,68,48,49 [Character Format]: VFBEAD01 * [Node Format]: Cluster ID: 57069 Node ID: 1 Node Name: nebula
# vxfenadm -s /dev/vx/rdmp/3pardata0_191
Device Name: /dev/vx/rdmp/3pardata0_191 Total Number Of Keys: 2 key[0]: [Numeric Format]: 86,70,66,69,65,68,48,48 [Character Format]: VFBEAD00 [Node Format]: Cluster ID: 57069 Node ID: 0 Node Name: galaxy key[1]: [Numeric Format]: 86,70,66,69,65,68,48,49 [Character Format]: VFBEAD01 * [Node Format]: Cluster ID: 57069 Node ID: 1 Node Name: nebula
Review the CP server information about the cluster nodes.
On the CPS server, run the cpsadm list nodes command to review a list of nodes in the cluster.
The command syntax is as follows:
# cpsadm -s cp_server -a list_nodes
where cp server is the virtual IP address or virtual hostname on which the CP server is listening.
# /opt/VRTS/bin/cpsadm -s 10.140.94.101 -a list_nodes ClusName UUID Hostname(Node ID) Registered gl-rh2 {25aeb8c6-1dd2-11b2-95b5-a82227078d73} node_101(0) 0 gl-rh2 {25aeb8c6-1dd2-11b2-95b5-a82227078d73} node_102(1) 0 cpstest {a0cf10e8-1dd1-11b2-87dc-080020c8fa36} node_220(0) 0 cpstest {a0cf10e8-1dd1-11b2-87dc-080020c8fa36} node_240(1) 0 ictwo {f766448a-1dd1-11b2-be46-5d1da09d0bb6} node_330(0) 0 ictwo {f766448a-1dd1-11b2-be46-5d1da09d0bb6} sassette(1) 0 fencing {e5288862-1dd1-11b2-bc59-0021281194de} CDC-SFLAB-CD-01(0) 0 fencing {e5288862-1dd1-11b2-bc59-0021281194de} CDC-SFLAB-CD-02(1) 0 gl-su2 {8f0a63f4-1dd2-11b2-8258-d1bcc1356043} gl-win03(0) 0 gl-su2 {8f0a63f4-1dd2-11b2-8258-d1bcc1356043} gl-win04(1) 0 gl-su1 {2d2d172e-1dd2-11b2-bc31-045b4f6a9562} gl-win01(0) 0 gl-su1 {2d2d172e-1dd2-11b2-bc31-045b4f6a9562} gl-win02(1) 0 gl-ax4 {c17cf9fa-1dd1-11b2-a6f5-6dbd1c4b5676} gl-ax06(0) 0 gl-ax4 {c17cf9fa-1dd1-11b2-a6f5-6dbd1c4b5676} gl-ax07(1) 0 gl-ss2 {da2be862-1dd1-11b2-9fb9-0003bac43ced} galaxy(0) 1 gl-ss2 {da2be862-1dd1-11b2-9fb9-0003bac43ced} nebula(1) 1
Review the CP server list membership.
On the CP server, run the following command to review the list membership. The command syntax is as follows:
# cpsadm -s cp_server -a list_membership -c cluster_name
where cp_server is the virtual IP address or virtual hostname on which the CP server is listening, and cluster_name is the VCS name for the Storage Foundation High Availability.
# cpsadm -s 10.140.94.101 -a list_membership -c gl-ss2 List of registered nodes: 0 1