To configure VxVM disk groups for Oracle RAC on an SF for Oracle RAC campus cluster

1. Initialize the disks as CDS disks

   # vxdisksetup -i disk01 format=cdsdisk
   # vxdisksetup -i disk02 format=cdsdisk
   # vxdisksetup -i disk03 format=cdsdisk
   # vxdisksetup -i disk05 format=cdsdisk
   # vxdisksetup -i disk06 format=cdsdisk
   # vxdisksetup -i disk07 format=cdsdisk
   # vxdisksetup -i disk08 format=cdsdisk

2. Set the site name for each host:

   # vxdctl set site=sitename

   The site name is stored in the /etc/vx/volboot file. To display the site names:

   # vxdctl list | grep siteid

   For example, for a four node cluster with two nodes at each site, mark the sites as follows:

   On the nodes at first site:

   # vxdctl set site=site1

   On the nodes at second site:

   # vxdctl set site=site2

3. Obtain the enclosure name using the following command:

   # vxdmpadm listenclosure
   ENCLR_NAME     ENCLR_TYPE   ENCLR_SNO STATUS    ARRAY_TYPE LUN_COUNT FIRMWARE
   =============================================================================
   ams_wms0       AMS_WMS      75040638  CONNECTED A/A-A      35        -
   hds9500-alua0  HDS9500-ALUA D600145E  CONNECTED A/A-A       9        -
   hds9500-alua1  HDS9500-ALUA D6001FD3  CONNECTED A/A-A       6        -
   disk           Disk         DISKS     CONNECTED Disk        2        -

4. Set the site name for all the disks in an enclosure.

   # vxdisk settag site=sitename encl:ENCLR_NAME

5. Run the following command if you want to tag only the specific disks:

   # vxdisk settag site=sitename disk

   For example:

   # vxdisk settag site=site1 disk01
   # vxdisk settag site=site1 disk02
   # vxdisk settag site=site1 disk03
   # vxdisk settag site=site2 disk06
   # vxdisk settag site=site2 disk08

6. Verify that the disks are registered to a site.

   # vxdisk listtag

   For example:

   # vxdisk listtag
   DEVICE          NAME                         VALUE
   disk01 										site                         site1
   disk02 										site                         site1
   disk03 										site                         site1
   disk04 										site                         site1
   disk05 										site                         site1
   disk06 										site                         site2
   disk07 										site                         site2
   disk08 										site                         site2
   disk09 										site                         site2

7. Create a disk group for OCR and Vote Disks and another for Oracle data, with disks picked from both the sites. While the example below shows a single disk group, you can create as many as you need.

   # vxdg -s init ocrvotedg disk05 disk07

   # vxdg -s init oradatadg disk01 disk06

8. Enable site-based allocation on the disk groups for each site.

   # vxdg -g ocrvotedg addsite site1

   # vxdg -g ocrvotedg addsite site2

   # vxdg -g oradatadg addsite site1

   # vxdg -g oradatadg addsite site2

9. If you are using an enclosure, set the tag on the enclosure for both sites.

   # vxdg -o retain -g ocrvotedg settag encl:3pardata0 site=site1

   # vxdg -o retain -g ocrvotedg settag encl:3pardata1 site=site2

   # vxdg -o retain -g oradatadg settag encl:3pardata0 site=site1

   # vxdg -o retain -g oradatadg settag encl:3pardata1 site=site2
   				

10. Configure site consistency for the disk groups.

    # vxdg -g ocrvotedg set siteconsistent=on

    # vxdg -g oradatadg set siteconsistent=on

11. Create one or more mirrored volumes in the disk group.

    # vxassist -g ocrvotedg make ocrvotevol 2048m nmirror=2

    # vxassist -g oradatadg make oradatavol 10200m nmirror=2

    # vxassist -g ocrvotedg make ocrvotevol 2048m nmirror=2

    # vxassist -g oradatadg make oradatavol 10200m nmirror=2

12. To verify the site awareness license, use the vxlicrep command. The Veritas Volume Manager product section should indicate: Site Awareness = Enabled

    With the Site Awareness license installed on all hosts, the volume created has the following characteristics by default.

    • The all sites attribute is set to ON; the volumes have at least one mirror at each site.

    • The volumes are automatically mirrored across sites.

    • The read policy (rdpol) is set to siteread.

      The read policy can be displayed using the vxprint -ht command.

    • The volumes inherit the site consistency value that is set on the disk group.

13. From the CVM master, start the volumes for all the disk groups.

    # vxvol -g ocrvotedg startall
    # vxvol -g oradatadg startall

14. Create a file system on each volume and mount the same.

    # mkfs -t vxfs /dev/vx/rdsk/ocrvotedg/ocrvotevol
    					

    # mkfs -t vxfs /dev/vx/rdsk/oradatadg/oradatavol
    					

    # mount -t vxfs -o cluster /dev/vx/dsk/ocrvotedg/ocrvotevol /ocrvote
    					

    # mount -t vxfs -o cluster /dev/vx/dsk/oradatadg/oradatavol /oradata
    					

15. Create seperate directories for OCR and Vote file as follows:

    # mkdir -p /ocrvote/ocr

    # mkdir -p /ocrvote/vote

16. After creating directories, change the ownership of these directories to Oracle or Grid user:

    # chown -R user:group /ocrvote

    Also change the ownership of /oradata to Oracle user:

    # chown user:group /oradata

    Note:

    One Vote Disk is sufficient since it is already mirrored by VxVM.

17. Install your database software.

    For Oracle RAC:

    • Insall Oracle Clusterware/GRID

    • Install Oracle RAC binaries

    • Perform library linking of Oracle binaries

    • Create the database on /oradata. For detailed steps,

      See the Storage Foundation for Oracle RAC Configuration and Upgrade Guide.