About the vxfenswap utility
The vxfenswap utility allows you to replace coordinator disks in a cluster that is online. The utility verifies that the serial number of the new disks are identical on all the nodes and the new disks can support I/O fencing.
Refer to the vxfenswap(1M) manual page.
See Veritas Cluster Server Installation Guide for details on the coordinator disk requirements.
You can replace the coordinator disks without stopping I/O fencing in the following cases:
If the vxfenswap operation is unsuccessful, then you can use the vxfenswap -a cancel command to manually roll back the changes that the vxfenswap utility does. You must run this command if a node fails during the process of disk replacement, or if you aborted the disk replacement.
Replacing I/O fencing coordinator disks when the cluster is online
Review the procedures to add, remove, or replace one or more coordinator disks in a cluster that is operational.
Caution
The cluster might panic if any node leaves the cluster membership before the vxfenswap script replaces the set of coordinator disks.
To replace a disk in an active coordinator diskgroup
-
Make sure system-to-system communication is functioning properly.
-
Make sure that the cluster is online.
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
-
Import the coordinator disk group.
The file /etc/vxfendg includes the name of the disk group (typically, vxfencoorddg) that contains the coordinator disks, so use the command:
# vxdg -tfC import 'cat /etc/vxfendg'
where:
-t specifies that the disk group is imported only until the node restarts.
-f specifies that the import is to be done forcibly, which is necessary if one or more disks is not accessible.
-C specifies that any import blocks are removed.
-
To remove disks from the disk group, use the VxVM disk administrator utility vxdiskadm.
You may also destroy the existing coordinator disk group.
For example:
# vxdg destroy vxfencoorddg
-
Add the new disk to the node, initialize it as a VxVM disk, and add it to the vxfencoorddg disk group.
Note that though the diskgroup content changes, the I/O fencing remains in the same state.
-
Make sure that the /etc/vxfenmode file is updated to specify the correct disk policy.
-
From one node, start the vxfenswap utility. You must specify the diskgroup to the utility:
Do one of the following:
-
If you use
ssh for communication:
# /opt/VRTSvcs/vxfen/bin/vxfenswap -g diskgroup
-
If you use rsh for communication:
# /opt/VRTSvcs/vxfen/bin/vxfenswap -g diskgroup -n
The utility performs the following tasks:
-
Backs up the existing /etc/vxfentab file.
-
Creates a test file /etc/vxfentab.test for the diskgroup that is modified on each node of the cluster.
-
Reads the diskgroup you specified in the vxfenwap command and adds the diskgroup to the /etc/vxfentab.test file on each node.
-
Verifies that the serial number of the new disks are identical on all the nodes. The script terminates if the check fails.
-
Verifies that the new disks can support I/O fencing on each node.
-
If the disk verification passes, the script reports success and asks if you want to commit the new set of coordinator disks.
Do you wish to commit this change? [y/n] (default: n) y
-
Review the message that the utility displays and confirm that you want to replace the diskgroup. Else skip to step 10.
If the utility successfully commits, the script moves the /etc/vxfentab.test file to the /etc/vxfentab file.
If you specified a diskgroup different to that in the /etc/vxfendg file, the utility also updates the /etc/vxfendg file.
-
If you do not want to replace the diskgroup, answer n.
The vxfenswap utility rolls back any changes to the coordinator diskgroup.
Replacing the coordinator diskgroup in a cluster that is online
You can also replace the coordinator diskgroup using the vxfenswap utility.
To replace the coordinator diskgroup
-
Make sure system-to-system communication is functioning properly.
-
Make sure that the cluster is online.
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
-
Find the name of the current coordinator diskgroup (typically vxfencoorddg) that is in the /etc/vxfendg file.
# cat /etc/vxfendg
vxfencoorddg
-
Find the alternative disk groups available to replace the current coordinator diskgroup.
# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
sda auto:cdsdisk - (vxfendg) online
sdb auto:cdsdisk - (vxfendg) online
sdc auto:cdsdisk - (vxfendg) online
sdx auto:cdsdisk - (vxfencoorddg) online
sdy auto:cdsdisk - (vxfencoorddg) online
sdz auto:cdsdisk - (vxfencoorddg) online
-
From any node, start the vxfenswap utility. For example, if vxfendg is the new diskgroup that you want to use as the coordinator diskgroup:
/opt/VRTSvcs/vxfen/bin/vxfenswap -g vxfendg [-n]
-
Verify that the coordinator disk group has changed.
# cat /etc/vxfendg
vxfendg
Changing the disk interaction policy in a cluster that is online
In a cluster that is online, you can change the disk interaction policy from raw to dmp using the vxfenswap utility.
To change the disk interaction policy
-
Make sure system-to-system communication is functioning properly.
-
Make sure that the cluster is online, and check the disk policy that is used.
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
-
On each node in the cluster, edit the /etc/vxfenmode file to change the disk policy.
cat /etc/vxfenmode
vxfen_mode=scsi3
scsi3_disk_policy=raw
-
From any node, start the vxfenswap utility:
/opt/VRTSvcs/vxfen/bin/vxfenswap -g vxfencoordg [-n]
-
Verify the change in the disk policy.
vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: raw
Adding disks from a recovered site to the coordinator diskgroup
In a campus cluster environment, consider a case where the primary site goes down and the secondary site comes online with a limited set of disks. When the primary site restores, the primary site's disks are also available to act as coordinator disks. You can use the vxfenswap utility to add these disks to the coordinator diskgroup.
See About I/O fencing in campus clusters
To add new disks from a recovered site to the coordinator diskgroup
-
Make sure system-to-system communication is functioning properly.
-
Make sure that the cluster is online.
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
-
Verify the name of the coordinator diskgroup.
cat /etc/vxfendg
vxfencoorddg
-
Run the following command:
vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
sdx auto:cdsdisk - (vxfencoorddg) online
sdy auto - - offline
sdz auto - - offline
-
Verify the number of disks used in the coordinator diskgroup.
vxfenconfig -l
I/O Fencing Configuration Information:
======================================
Count : 1
Disk List
Disk Name Major Minor Serial Number Policy
/dev/vx/rdmp/sdx 32 48 R450 00013154 0312 dmp
-
When the primary site comes online, start the vxfenswap utility on any node in the cluster:
/opt/VRTSvcs/vxfen/bin/vxfenswap -g vxfencoorddg [-n]
-
Verify the count of the coordinator disks.
vxfenconfig -l
I/O Fencing Configuration Information:
======================================
Single Disk Flag : 0
Count : 3
Disk List
Disk Name Major Minor Serial Number Policy
/dev/vx/rdmp/sdx 32 48 R450 00013154 0312 dmp
/dev/vx/rdmp/sdy 32 32 R450 00013154 0313 dmp
/dev/vx/rdmp/sdz 32 16 R450 00013154 0314 dmp
Refreshing lost keys on coordinator disks
If the coordinator disks lose the keys that are registered, the cluster might panic when a split-brain occurs.
You can use the vxfenswap utility to replace the coordinator disks with the same disks. The vxfenswap utility registers the missing keys during the disk replacement.
To refresh lost keys on coordinator disks
-
Make sure system-to-system communication is functioning properly.
-
Make sure that the cluster is online.
# vxfenadm -d
I/O Fencing Cluster Information:
================================
Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:
* 0 (galaxy)
1 (nebula)
RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)
-
Run the following command to view the coordinator disks that do not have keys:
vxfenadm -g all -f /etc/vxfentab
Device Name: /dev/vx/rdmp/sdx
Total Number of Keys: 0
No keys...
...
-
On any node, run the following command to start the vxfenswap utility:
/opt/VRTSvcs/vxfen/bin/vxfenswap -g vxfencoorddg [-n]
-
Verify that the keys are atomically placed on the coordinator disks.
vxfenadm -g all -f /etc/vxfentab
Device Name: /dev/vx/rdmp/sdx
Total Number of Keys: 4
...
