Date: 2008-11-20
OS: HP-UX
OS Version: 11iv2
Packages: VRTSdbac
Etrack Incidents: 1380473, 1380575, 1380580, 1380628, 1380642, 1380714, 1380724, 1387835, 1413790

Fixes Applied for Products:
    Storage Foundation for Oracle RAC 4.1 MP2 for HP-UX 11iv2

Additional Notes: 

Patch Name: PVKL_03843

Patch Description: VRTSdbac 4.1MP2RP1 Kernel Patch

Creation Date: 11/20/2008-09:34:17

Hardware Platforms - OS Releases: 11iv2 (B11.23)

Products: VRTSdbac 4.1

Filesets:
	PVKL_03843.DBAC-RUN, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_IA/PA
	PVKL_03843.LMX-KRN-64, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_PA
	PVKL_03843.LMX-KRN-64, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_IA
	PVKL_03843.LMX-RUN, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_PA
	PVKL_03843.LMX-RUN, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_IA
	PVKL_03843.VCSMM-KRN-64, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_PA
	PVKL_03843.VCSMM-KRN-64, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_IA
	PVKL_03843.VCSMM-RUN, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_PA
	PVKL_03843.VCSMM-RUN, r=2.1, a=HP-UX_B.11.23_IA/PA, v=VERITAS, fr=2.1, fa=HP-UX_B.11.23_IA

Automatic Reboot?: No

Status: General Release

Critical: Yes

Category Tags:

Symptoms:
(VERITAS Incident Number: 1380473)
When IP was brought down using "ifconfig <if> down", 
the PrivNIC agent did not bring the IP back. The IP 
was brought up only when IP was brought down using
"ifconfig <if> unplumb".

(VERITAS Incident Number: 1380575)
VCSMM debug messages prints full file path.

(VERITAS Incident Number: 1380580)
VCSMM panics with panic string 
	"spinlock: locker forgot to unlock"

(VERITAS Incident Number: 1380628)
By default the values mm_slave_max is 4096. Hence requiring 
the customer to incur a downtime to change this.

(VERITAS Incident Number: 1380642)
For vcsmm tunables the min and max values are not defined.

(VERITAS Incident Number: 1380714)
VCSMM tunables mm_deblog_sz is set to 64k, which is the 
max value. Hence no room for change.

(VERITAS Incident Number: 1380724)
While unloading vcsmm driver, one may encounter a system panic.

(VERITAS Incident Number: 1387835)
After installing or upgrading the Oracle Clusterware and 
its binaries to version 10.2.0.4, the command "linkrac 10gR2" 
fails with the following error message:
   "ERROR: Oracle binary not linked with Veritas IPC library."

(VERITAS Incident Number: 1413790)
lmxpollport calls timeout with negative timeout value 
after the server is up for 410 days.

Defect Description:
(VERITAS Incident Number: 1380473)
PrivNIC agent parsed "netstat -in" output to determine 
whether IP is plumbed on the interface. When IP is brought 
down using "ifconfig <if> unplumb" the entry is removed 
from the "netstat -in" output, whereas when IP is brought 
down using "ifconfig <if> down", entry is not removed 
from the "netstat -in" output, instead only "*" is appended 
to the interface in the "netstat -in" output in that entry. 
The netstat output parsing does not handle this properly.

Resolution:
While parsing "netstat -in" output to determine whether IP 
is up on a interface, handle the entries with interface 
marked as down with "*" similar to non-existing entries 
for an interface. This would result into plumbing the IP
back if the IP is brought down with any of 
the above mechanism.

(VERITAS Incident Number: 1380575)
VCSMM debug message uses a macro __FILE__ in VCSMM_DBG 
macro. Depending on how it is built, __FILE__ can be 
expanded to very long path name. It wastes memory 
and makes it difficult to read.

Resolution:
A function "vcsmm_basename" is added which prints only 
the file name and used in VCSMM_DBG macro.

(VERITAS Incident Number: 1380580)
vcsmm_conf() calls vcsmm_add_deblog() with a spinlock 
held (vcsmm.mi_cvlock). vcsmm_add_deblog() could sleep 
while waiting to acquire the dlog->dl_lock. This 
results in panic.

Resolution:
The vcsmmdebug process should just take the spinlock, 
get pointers, release spinlock and copy the log.  
Since the pointers are always in the range of log 
buffer, there is not risk of panic even those pointers 
got changed during copying.  It is possible to get some 
garbage at the beginning or end of the log which should 
be okay because it is a circular buffer which will always 
lose some log if it wraps around. The only thing to worry 
about is that one of the "" may be overwritten before 
the log is copied out.  If the destination buffer is 
cleared before calling into vcsmm, it should be fine.

(VERITAS Incident Number: 1380628)
Currently the mm_slave_max is set to 4k, which is much 
less value for a running system. To update the mm_slave_max 
the system seems to require a downtime, which is 
not admired. Hence can bump up the default value 
so that there will be no need for a downtime.

Resolution:
The tunebale mm_slave_max default value 
has been changed to 32k.

(VERITAS Incident Number: 1380642)
The range of values that can be assigned to a 
vcsmm tunables is not defined. This leads user to be 
confused as in what max and min value can be set for 
the tunables to work fine.

Resolution:
The min and max values for vcsmm tunables are defined.

(VERITAS Incident Number: 1380714)
VCSMM tunable mm_deblog_sz is now already set to its 
maximum limit. So its not possible to increase any more.

Resolution:
Tunable is upgraded to have a maximum value of 1M and 
default value is set as 64K which is decent enough.

(VERITAS Incident Number: 1380724)
VCSMM merely checks if vcsmm.mi_conf is 1. While the driver 
has been unconfigured (vcsmm.mi_conf = 0), a process can 
open vcsmm device. Since the driver is unloaded, without 
checking if there are any processes having opened 
vcsmm, a close() call can cause panic.

Resolution:
VCSMM driver should check if vcsmm.mi_open_fds 
is zero before unloading driver.

(VERITAS Incident Number: 1387835)
The library format for the shared library, libskgxp, 
has changed from static to dynamic format in 
Oracle 10.2.0.4. As a result, the linkrac script is 
unable to locate the shared library and fails with 
the error message.

Resolution:
The linkrac script is now modified to determine 
the format of the shared library that must be used 
with the installed version of Oracle.

(VERITAS Incident Number: 1413790)
The time difference calculation are done using variable 
"lbolt" which wraps around after 410 days. If the lbolt 
wraps around between start time and end time, the
timeout gets called with the negative time.

Resolution:
Instead of using lbolt which wraps around after 410 days, 
use ticks_since_boot_whole, a 64 bit variable, 
which will not wrap for years.

Enhancement: No

Patch Files:
	/etc/VRTSvcs/conf/sample_rac/10g_main.cf.hp
	/opt/VRTSvcs/bin/PrivNIC/hpux_netutil.pm
	/opt/VRTSvcs/bin/PrivNIC/monitor
	/opt/VRTSvcs/ops/bin/clsinfo
	/opt/VRTSvcs/rac/bin/cssd-clean
	/opt/VRTSvcs/rac/bin/cssd-monitor
	/opt/VRTSvcs/rac/bin/cssd-offline
	/opt/VRTSvcs/rac/bin/cssd-online
	/opt/VRTSvcs/rac/lib/linkrac
	/opt/VRTSvcs/rac/lib/oramake
	/opt/nmapi/nmapi2/bin/clsinfo
	/usr/conf/mod/lmx
	/usr/conf/mod/lmx
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver23_64.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver23_64.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver23_64_dbg.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver23_64_dbg.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver25_64.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver25_64.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver25_64_dbg.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp10_ver25_64_dbg.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver21_64.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver21_64.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver21_64_dbg.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver21_64_dbg.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver22_64.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver22_64.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver22_64_dbg.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver22_64_dbg.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver24_64.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver24_64.sl
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver24_64_dbg.a
	/opt/VRTSvcs/rac/lib/pa20_64/libskgxp9_ver24_64_dbg.sl
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver23_64.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver23_64.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver23_64_dbg.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver23_64_dbg.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver25_64.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver25_64.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver25_64_dbg.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp10_ver25_64_dbg.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver21_64.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver21_64.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver21_64_dbg.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver21_64_dbg.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver22_64.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver22_64.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver22_64_dbg.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver22_64_dbg.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver24_64.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver24_64.so
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver24_64_dbg.a
	/opt/VRTSvcs/rac/lib/hpux64/libskgxp9_ver24_64_dbg.so
	/usr/conf/mod/vcsmm
	/usr/conf/mod/vcsmm
	/usr/lib/libvcsmm.1
	/usr/lib/pa20_64/libvcsmm.1
	/usr/lib/hpux32/libvcsmm.1
	/usr/lib/hpux64/libvcsmm.1

Patch Conflicts: None

Patch Dependencies: None

Hardware Dependencies: None

Other Dependencies: None

Supersedes: None

Equivalent Patches: None

Installation Instructions:
=========================
	Please review all instructions and the VERITAS support terms
	and conditions for precautions, scope of license,
	restrictions, and, limitation of liability and warranties,
	before installing this patch.

	Please go through the instructions below to install/remove 
	the patch from the system.

Patch requirements:
~~~~~~~~~~~~~~~~~~
You must have Veritas Storage Foundation for Oracle RAC (SF Oracle RAC)
4.1 MP2 installed on the HP-UX 11iv2 server.

For instructions on installing SF Oracle RAC 4.1 MP2, see 
the following documents:
	* VERITAS Storage Foundation 4.1 for Oracle RAC 
	    Installation and Configuration Guide for HP-UX
	* VERITAS Storage Foundation 4.1 for Oracle RAC 
	    Release Notes for HP-UX 11i v2 Maintenance Pack 2

Installing SF Oracle RAC 4.1 MP2 RP1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install SF Oracle RAC 4.1 MP2 RP1 follow the steps: 

1. Stop VCS on all nodes.
	# /opt/VRTS/bin/hastop -all

NOTE: If Oracle Database and Oracle Clusterware are not under VCS control, 
stop Oracle Database and Oracle Clusterware manually before stopping VCS.

2. Stop the Veritas Cluster Server Membership Module (VCSMM) and the 
   LLT Multiplexer (LMX) module on all nodes.
	# /sbin/vcsmmconfig -U
	# /sbin/lmxconfig -U

3. Use the swinstall command to install the SF Oracle RAC 4.1 MP2 RP1 on 
   each node in the cluster.
	# cd <patch_location>
	# swinstall -s `pwd` -x autoreboot=true PVKL_03843

4. Verify that SF Oracle RAC 4.1 MP2 RP1 is successfully installed.
	# swlist PVKL_03843

  The output of a successful installation is as follows: 
# PVKL_03843   			2.1	VRTSdbac 4.1MP2RP1 Kernel Patch
  PVKL_03843.DBAC-RUN   	2.1	Veritas DBAC generic files
  PVKL_03843.LMX-KRN-64		2.1	VERITAS LMX kernel files
  PVKL_03843.LMX-RUN		2.1	Veritas LMX userland files
  PVKL_03843.VCSMM-KRN-64	2.1	VERITAS VCSMM kernel files
  PVKL_03843.VCSMM-RUN		2.1	Veritas VCSMM userland files

5. Use kcmodule command to re-load the VCSMM and LMX modules.
	# kcmodule vcsmm=loaded
	# kcmodule lmx=loaded

6. Start VCSMM and LMX on each node of the cluster.
	# /sbin/init.d/vcsmm start
	# /sbin/init.d/lmx start

7. Verify the VCSMM module version. Make sure that the module version of 
   VCSMM is "1.401.1" in the command output.
	# kcmodule -v vcsmm

8. Verify the LMX module version. Make sure that the module version of 
   LMX is "1.401.2" in the command output.
	# kcmodule -v lmx

9. Start VCS on each node of the cluster
	# /opt/VRTS/bin/hastart 

NOTE: If Oracle Database and Oracle Clusterware are not under VCS control, 
start Oracle Database and Oracle Clusterware manually after starting VCS.

Removing SF Oracle RAC 4.1 MP2 RP1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Removing SF Oracle RAC 4.1 MP2 RP1 rolls back the installation to 
SF Oracle RAC 4.1 MP2.

To remove SF Oracle RAC 4.1 MP2 RP1:

1. Stop VCS on all nodes. 
	# /opt/VRTS/bin/hastop -all

NOTE: If Oracle Database and Oracle Clusterware are not under VCS control, 
stop Oracle Database and Oracle Clusterware manually before stopping VCS. 

2. Stop the Veritas Cluster Server Membership Module (VCSMM) and 
   the LLT Multiplexer (LMX) module on all nodes.
	# /sbin/vcsmmconfig -U
	# /sbin/lmxconfig -U

3. Use swremove to remove SF Oracle RAC 4.1 MP2 RP1 from each node.
	# swremove -x autoreboot=true PVKL_03843

4. Verify that SF Oracle RAC 4.1 MP2 RP1 is successfully removed.
	# swlist PVKL_03843

If SF Oracle RAC 4.1 MP2 RP1 is successfully removed, 
the following message is displayed:
	ERROR:   Software "PVKL_03843" was not found on host

5. Use kcmodule command to re-load the VCSMM and LMX modules.
	# kcmodule vcsmm=loaded
	# kcmodule lmx=loaded

6. Start VCSMM and LMX on all the nodes.
	# /sbin/init.d/vcsmm start
	# /sbin/init.d/lmx start

7. Verify the VCSMM module version. Make sure that the module version of
   VCSMM is "1.0.0" in the command output.
	# kcmodule -v vcsmm

8. Verify the LMX module version. Make sure that the module version of
   LMX is "1.401.1" in the command output.
	# kcmodule -v lmx

9. Start VCS on all nodes in the cluster.
	# /opt/VRTS/bin/hastart

NOTE: If Oracle Database and Oracle Clusterware are not under VCS control,
start Oracle Database and Oracle Clusterware manually after starting VCS.