Troubleshooting server-based I/O fencing in mixed mode


Troubleshooting server-based I/O fencing in mixed mode
Prev	Appendix G. Troubleshooting cluster installation	Next

The following procedure can be use to troubleshoot a mixed I/O fencing configuration (configuration using both coordinator disks and CP server for I/O fencing). This procedure involves using the following commands to obtain I/O fencing information:

To obtain I/O fencing cluster information on the coordinator disks, run the following command on one of the cluster nodes:
```
# vxfenadm -s diskname
```
Any keys other than the valid keys used by the cluster nodes that appear in the command output are spurious keys.
To obtain I/O fencing cluster information on the CP server, run the following command on one of the cluster nodes:
```
# cpsadm -s cp_server -a list_membership -c cluster_name
```
where cp server is the virtual IP address or virtual hostname on which the CP server is listening, and cluster name is the VCS name for the SFCFS cluster.

Nodes which are not in GAB membership, but registered with CP server indicate a pre-existing network partition.

Note that when running this command on the SFCFS cluster nodes, you need to first export the CPS_USERNAME and CPS_DOMAINTYPE variables.

The CPS_USERNAME value is the user name which is added for this node on the CP server.
To obtain the user name, run the following command on the CP server:
```
# cpsadm -s cp_server -a list_users
```
where cp server is the virtual IP address or virtual hostname on which the CP server is listening.

The CPS_DOMAINTYPE value is vx.

The following are export variable command examples:
```
# export CPS_USERNAME=_HA_VCS_test-system@HA_SERVICES@test-system.symantec.com
```
```
# export CPS_DOMAINTYPE=vx
```

Once a pre-existing network partition is detected using the above commands, all spurious keys on the coordinator disks or CP server must be removed by the administrator.

Troubleshooting mixed I/O fencing configuration (coordinator disks and CP server)

Review the current I/O fencing configuration by accessing and viewing the information in the vxfenmode file.

Enter the following command on one of the SFCFS cluster nodes:
```
# cat /etc/vxfenmode
```
```
vxfen_mode=customized
vxfen_mechanism=cps
scsi3_disk_policy=dmp
security=0
cps1=[10.140.94.101]:14250
vxfendg=vxfencoorddg
```

Review the I/O fencing cluster information.

Enter the vxfenadm -d command on one of the cluster nodes:

# vxfenadm -d

I/O Fencing Cluster Information:
================================

 Fencing Protocol Version: 201
 Fencing Mode: Customized
 Fencing Mechanism: cps
 Cluster Members:  

        * 0 (galaxy)
          1 (nebula)

 RFSM State Information:
        node   0 in state  8 (running)
        node   1 in state  8 (running)

Review the SCSI registration keys for the coordinator disks used in the I/O fencing configuration.

Enter the vxfenadm -s command on each of the SFCFS cluster nodes.

# vxfenadm -s /dev/vx/rdmp/3pardata0_190

Device Name: /dev/vx/rdmp/3pardata0_190
Total Number Of Keys: 2
key[0]:
	        [Numeric Format]:  86,70,66,69,65,68,48,48
	        [Character Format]: VFBEAD00
	        [Node Format]: Cluster ID: 57069 Node ID: 0   Node Name: galaxy
key[1]:
	        [Numeric Format]:  86,70,66,69,65,68,48,49
	        [Character Format]: VFBEAD01
*	       [Node Format]: Cluster ID: 57069 Node ID: 1   Node Name: nebula

# vxfenadm -s /dev/vx/rdmp/3pardata0_191

Device Name: /dev/vx/rdmp/3pardata0_191
Total Number Of Keys: 2
key[0]:
	        [Numeric Format]:  86,70,66,69,65,68,48,48
	        [Character Format]: VFBEAD00
	        [Node Format]: Cluster ID: 57069 Node ID: 0   Node Name: galaxy
key[1]:
	        [Numeric Format]:  86,70,66,69,65,68,48,49
	        [Character Format]: VFBEAD01
*	       [Node Format]: Cluster ID: 57069 Node ID: 1   Node Name: nebula

Review the CP server information about the cluster nodes.

On the CPS server, run the cpsadm list nodes command to review a list of nodes in the cluster.

The command syntax is as follows:

# cpsadm -s cp_server -a list_nodes

where cp server is the virtual IP address or virtual hostname on which the CP server is listening.

For example:

# /opt/VRTS/bin/cpsadm -s 10.140.94.101 -a list_nodes
        
ClusName        UUID                            Hostname(Node ID) Registered
gl-rh2  {25aeb8c6-1dd2-11b2-95b5-a82227078d73}  node_101(0)        0
gl-rh2  {25aeb8c6-1dd2-11b2-95b5-a82227078d73}  node_102(1)        0
cpstest {a0cf10e8-1dd1-11b2-87dc-080020c8fa36}  node_220(0)        0
cpstest {a0cf10e8-1dd1-11b2-87dc-080020c8fa36}  node_240(1)        0
ictwo   {f766448a-1dd1-11b2-be46-5d1da09d0bb6}  node_330(0)        0
ictwo   {f766448a-1dd1-11b2-be46-5d1da09d0bb6}  sassette(1)        0
fencing {e5288862-1dd1-11b2-bc59-0021281194de}  CDC-SFLAB-CD-01(0) 0
fencing {e5288862-1dd1-11b2-bc59-0021281194de}  CDC-SFLAB-CD-02(1) 0
gl-su2  {8f0a63f4-1dd2-11b2-8258-d1bcc1356043}  gl-win03(0)        0
gl-su2  {8f0a63f4-1dd2-11b2-8258-d1bcc1356043}  gl-win04(1)        0
gl-su1  {2d2d172e-1dd2-11b2-bc31-045b4f6a9562}  gl-win01(0)        0
gl-su1  {2d2d172e-1dd2-11b2-bc31-045b4f6a9562}  gl-win02(1)        0
gl-ax4  {c17cf9fa-1dd1-11b2-a6f5-6dbd1c4b5676}  gl-ax06(0)         0
gl-ax4  {c17cf9fa-1dd1-11b2-a6f5-6dbd1c4b5676}  gl-ax07(1)         0
gl-ss2  {da2be862-1dd1-11b2-9fb9-0003bac43ced}  galaxy(0)        1
gl-ss2  {da2be862-1dd1-11b2-9fb9-0003bac43ced}  nebula(1)        1

Review the CP server list membership.

On the CP server, run the following command to review the list membership. The command syntax is as follows:
```
# cpsadm -s cp_server -a list_membership -c cluster_name
```
where cp_server is the virtual IP address or virtual hostname on which the CP server is listening, and cluster_name is the VCS name for the SFCFS cluster.

For example:
```
# cpsadm -s 10.140.94.101 -a list_membership -c gl-ss2

List of registered nodes: 0 1
```

Prev	Up	Next
Vxfen service group activity after issuing the vxfenswap command	Home	Checking keys on coordination points when vxfen_mechanism value is set to cps