How VCS campus clusters work

This topic describes how VCS works with VxVM to provide high availability in a campus cluster environment.

In a campus cluster setup, VxVM automatically mirrors volumes across sites. To enhance read performance, VxVM reads from the plexes at the local site where the application is running. VxVM writes to plexes at both the sites.

In the event of a storage failure at a site, VxVM detaches all the disks at the failed site from the disk group to maintain data consistency. When the failed storage comes back online, VxVM automatically reattaches the site to the disk group and recovers the plexes.

See the Storage Foundation Cluster File System High Availability Administrator's Guide for more information.

When service group or system faults occur, VCS fails over service groups based on the values you set for the cluster attribute SiteAware and the service group attribute AutoFailOver.

See Cluster attributes.

For campus cluster setup, you must define sites and add systems to the sites that you defined. A system can belong to only one site. Sit e definitions are uniform across VCS, You can define sites Veritas InfoScale Operations Manager, and VxVM. You can define site dependencies to restrict connected applications to fail over within the same site.

You can define sites by using:

Depending on the value of the AutoFailOver attribute, VCS failover behavior is as follows:

0

VCS does not fail over the service group.

1

VCS fails over the service group to another suitable node.

By default, the AutoFailOver attribute value is set to 1.

2

VCS fails over the service group if another suitable node exists in the same site. Otherwise, VCS waits for administrator intervention to initiate the service group failover to a suitable node in the other site.

This configuration requires the HA/DR license enabled.

Veritas recommends that you set the value of AutoFailOver attribute to 2.

Sample definition for these service group attributes in the VCS main.cf is as follows:

cluster VCS_CLUS (
        PreferredFencingPolicy = Site
        SiteAware = 1
        )
site MTV (
        SystemList = { sys1, sys2 }
        )
site SFO (
        Preference = 2
        SystemList = { sys3, sys4 }
        )

The sample configuration for hybrid_group with AutoFailover = 1 and failover_group with AutoFailover = 2 is as following:

hybrid_group (
    Parallel = 2
    SystemList = { sys1 = 0, sys2 = 1, sys3 = 2, sys4 = 3 }
)

failover_group (
    AutoFailover = 2
    SystemList = { sys1 = 0, sys2 = 1, sys3 = 2, sys4 = 3 }
)

Table: Failure scenarios in campus cluster lists the possible failure scenarios and how VCS campus cluster recovers from these failures.

Table: Failure scenarios in campus cluster

Failure

Description and recovery

Node failure

  • A node in a site fails.

    If the value of the AutoFailOver attribute is set to 1, VCS fails over the service group to another system within the same site defined for cluster or SystemZone defined by SystemZones attribute for the service group or defined by Veritas InfoScale Operations Manager.

  • All nodes in a site fail.

If the value of the AutoFailOver attribute is set to 0, VCS requires administrator intervention to initiate a fail over in both the cases of node failure.

Application failure

The behavior is similar to the node failure.

Storage failure - one or more disks at a site fails

VCS does not fail over the service group when such a storage failure occurs.

VxVM detaches the site from the disk group if any volume in that disk group does not have at least one valid plex at the site where the disks failed.

VxVM does not detach the site from the disk group in the following cases:

  • None of the plexes are configured on the failed disks.

  • Some of the plexes are configured on the failed disks, and at least one plex for a volume survives at each site.

If only some of the disks that failed come online and if the vxrelocd daemon is running, VxVM relocates the remaining failed disks to any available disks. Then, VxVM automatically reattaches the site to the disk group and resynchronizes the plexes to recover the volumes.

If all the disks that failed come online, VxVM automatically reattaches the site to the disk group and resynchronizes the plexes to recover the volumes.

Storage failure - all disks at both sites fail

VCS acts based on the DiskGroup agent's PanicSystemOnDGLoss attribute value.

See the Cluster Server Bundled Agents Reference Guide for more information.

Site failure

All nodes and storage at a site fail.

Depending on the value of the AutoFailOver attribute, VCS fails over the service group as follows:

  • If the value is set to 1, VCS fails over the service group to a system.

  • If the value is set to 2, VCS requires administrator intervention to initiate the service group failover to a system in the other site.

Because the storage at the failed site is inaccessible, VCS imports the disk group in the application service group with all devices at the failed site marked as NODEVICE.

When the storage at the failed site comes online, VxVM automatically reattaches the site to the disk group and resynchronizes the plexes to recover the volumes.

Network failure (LLT interconnect failure)

Nodes at each site lose connectivity to the nodes at the other site

The failure of all private interconnects between the nodes can result in split brain scenario and cause data corruption.

Review the details on other possible causes of split brain and how I/O fencing protects shared data from corruption.

Veritas recommends that you configure I/O fencing to prevent data corruption in campus clusters.

When the cluster attribute PreferredFencingPolicy is set as Site, the fencing driver gives preference to the node with higher site priority during the race for coordination points. VCS uses the site-level attribute Preference to determine the node weight.

Network failure (LLT and storage interconnect failure)

Nodes at each site lose connectivity to the storage and the nodes at the other site

Veritas recommends that you configure I/O fencing to prevent split brain and serial split brain conditions.

  • If I/O fencing is configured:

    The site that do not win the race triggers a system panic.

    When you restore the network connectivity, VxVM detects the storage at the failed site, reattaches the site to the disk group, and resynchronizes the plexes to recover the volumes.

  • If I/O fencing is not configured:

    If the application service group was online at site A during such failure, the application service group remains online at the same site. Because the storage is inaccessible, VxVM detaches the disks at the failed site from the disk group. At site B where the application service group is offline, VCS brings the application service group online and imports the disk group with all devices at site A marked as NODEVICE. So, the application service group is online at both the sites and each site uses the local storage. This causes inconsistent data copies and leads to a site-wide split brain.

    When you restore the network connectivity between sites, a serial split brain may exist.

    See the Storage Foundation Administrator's Guide for details to recover from a serial split brain condition.

More Information

Service group attributes

About data protection

About I/O fencing in campus clusters

About I/O fencing in campus clusters