< Previous  | TOC | Index | Next  >

Oracle troubleshooting topics

Topics indicate symptoms and likely procedures required for a solution.

Oracle user must be able to read /etc/llttab File

Check the permissions of the file /etc/llttab. Oracle must be allowed to read it.

Error when starting an Oracle instance

If the VCSMM driver (the membership module) is not configured, an error displays on starting the Oracle instance that resembles:

ORA-29702: error occurred in Cluster Group Operation

 To start the VCSMM driver

Enter the following command:

# /etc/init.d/vcsmm start

The command included in the /etc/vcsmmtab file enables the VCSMM driver to be started at system boot.

Missing dialog box during installation of Oracle9i

During installation of Oracle9i Release 2 using the runInstaller utility, if you choose the Enterprise Edition or Custom Install (with RAC option), a dialog box prompting you about the installation nodes appears.

  Note   Symantec recommends using the installsfrac utility to install Oracle9i RAC. The following troubleshooting steps only apply if the runInstaller utility is called directly.

 To check if the dialog box fails to appear

1. Exit the runInstaller utility.
2. On the first system, copy the Veritas CM library:
   1. Enter:

      # cd /usr/lib

   2. Enter:

      # cp libvcsmm.so $ORACLE_HOME/lib/libcmdll.so

3. Start the VCSMM driver on both the nodes by entering:

   # /etc/init.d/vcsmm start

4. Restart the runInstaller utility.

Oracle log files show shutdown

The Oracle log files may show that a shutdown was called even when not shutdown manually.

The Oracle enterprise agent calls shutdown if monitoring of the Oracle/Netlsnr resources fails for some reason. On all cluster nodes, look at the following VCS and Oracle agent log files for any errors or status:

/var/VRTSvcs/log/engine_A.log

/var/VRTSvcs/log/Oracle_A.log

DBCA fails while creating database

Verify that the hostname -i command returns the public IP address of the current node. This command is used by the installer and the output is stored in the OCR. If hostname -i returns 127.0.0.1, it causes the DBCA to fail.

CRS processes fail to startup

Verify that the correct private IP address is configured on the private link using the PrivNIC agent. Check the CSS log files to learn more.

CRS fails after restart

If the CRS fails to start up after boot up, check for the occurrence of the following strings in the /var/log/ messages file:

"Oracle CSSD failure. Rebooting for cluster integrity"

• Communication failure occurred and CRS fenced out the node.
• OCR and Vote disk became unavailable.
• ocssd was killed manually and on restarting from inittab it rebooted the cluster.
• Killing the init.cssd script.

"Waiting for file system containing"

The CRS installation is on a shared disk and the init script is waiting for that file system to be made available.

"Oracle Cluster Ready Services disabled by corrupt install"

The following file is not available or has corrupt entries: /etc/oracle/scls_scr/hostname/root/crsstart.

"OCR initialization failed accessing OCR device"

The shared file system containing the OCR is not available and CRS is waiting for it to become available.

Removing Oracle CRS if installation fails

Use the following procedure to remove Oracle CRS.

 To reomve Oracle CRS

1. Run the rootdelete.sh script:

   # cd /crshome/install

   # ./rootdelete.sh

   1. Run the rootdeinstall.sh script:

      # cd /crshome/install

      # ./rootdeinstall.sh

2. Stop the applications on all nodes:

   # srvctl stop nodeapps -n node_name

3. Copy the file inittab.orig back to the name and remove other init files:

   # cd /etc

   # cp inittab.orig inittab

   # rm init.c* init.evmd

   # rm /etc/rc.d/rc2.d/K96init.crs

   # rm /etc/rc.d/rc2.d/S96init.crs

4. Remove ora* files from the /etc directory:

   # cd /etc

   # rm -r ora*

5. Remove file from $CRS_HOME and Oracle Inventory:

   # rm -r $CRS_HOME/*

6. Remove files from the OCR and Voting disk directories. For our example:

   # rm /ocrvote/ocr

   # rm /ocrvote/vote-disk

   If OCR and Voting disk storage are on raw volumes, use command resembling:

   # dd if=/dev/zero of=/dev/vx/rdsk/ocrvotedg/ocrvol bs=8192 \

   count=18000

   # dd if=/dev/zero of=/dev/vx/rdsk/ocrvotedg/votvol bs=8192 \

   count=3000

7. Reboot the systems to make sure no CRS daemons are running.

Troubleshooting the VIP Configuration

When encountering issues with the VIP configuration, you can use the following commands and files:

• Use the /etc/ifconfig -a command on all nodes to check for network problems.
• Use the command: /usr/bin/nslookup virtual_host_name to make sure the virtual host name is registered with the DNS server.
• Verify the /etc/hosts file on each node.
• Check the output from the command $CRS_HOME/bin/crs_stat.
• On the problem node, use the command: srvctl start nodeapps -n node_name. This command works only if the virtual IP address is plumbed.

OCR and Vote disk related issues

Verify that the permissions are set appropriately as given in the Oracle installation guide. If these files are present from a previous configuration, remove them. See CRS fails after restart.