I/O fencing scenarios describes how I/O fencing works to prevent data corruption in different failure event scenarios. For each event, corrective operator actions are indicated.
Node A races for majority of coordinator disks. If Node A wins race for coordinator disks, Node A ejects Node B from the shared disks and continues. |
Node B races for majority of coordinator disks. If Node B loses the race for the coordinator disks, Node B removes itself from the cluster. |
When Node B is ejected from cluster, repair the private networks before attempting to bring Node B back. |
|
Node B has crashed. It cannot start the database since it is unable to write to the data disks. |
|||
Node A prints message about an IOFENCE on the console but continues. |
Node B prints message about an IOFENCE on the console but continues. |
Repair private network. After network is repaired, both nodes automatically use it. |
|
Node A is extremely busy for some reason or is in the kernel debugger. When Node A is no longer hung or in the kernel debugger, any queued writes to the data disks fail because Node A is ejected. When Node A receives message from GAB about being ejected, it removes itself from the cluster. |
Node B loses heartbeats with Node A, and races for a majority of coordinator disks. Node B wins race for coordinator disks and ejects Node A from shared data disks. |
||
Nodes A and B and private networks lose power. Coordinator and data disks retain power. Power returns to nodes and they restart, but private networks still have no power. |
Node A restarts and I/O fencing driver (vxfen) detects Node B is registered with coordinator disks. The driver does not see Node B listed as member of cluster because private networks are down. This causes the I/O fencing device driver to prevent Node A from joining the cluster. Node A console displays: Potentially a preexisting split brain. Dropping out of the cluster. Refer to the user documentation for steps required to clear preexisting split brain. |
Node B restarts and I/O fencing driver (vxfen) detects Node A is registered with coordinator disks. The driver does not see Node A listed as member of cluster because private networks are down. This causes the I/O fencing device driver to prevent Node B from joining the cluster. Node B console displays: Potentially a preexisting split brain. Dropping out of the cluster. Refer to the user documentation for steps required to clear preexisting split brain. |
Resolve preexisting split brain condition. See "System panics to prevent potential data corruption" on page 151. |
Node A crashes while Node B is down. Node B comes up and Node A is still down. |
Node B restarts and detects Node A is registered with the coordinator disks. The driver does not see Node A listed as member of the cluster. The I/O fencing device driver prints message on console: Potentially a preexisting split brain. Dropping out of the cluster. Refer to the user documentation for steps required to clear preexisting split brain. |
Resolve preexisting split brain condition. See "System panics to prevent potential data corruption" on page 151. |
|
The disk array containing two of the three coordinator disks is powered off. Node B leaves the cluster and the disk array is still powered off. |
Node A continues to operate as long as no nodes leave the cluster. Node A races for a majority of coordinator disks. Node A fails because only one of three coordinator disks is available. Node A removes itself from the cluster. |
Node B continues to operate as long as no nodes leave the cluster. |
Power on failed disk array and restart I/O fencing driver to enable Node A to register with all coordinator disks. |