* * * READ ME * * *
            * * * Veritas Storage Foundation HA 6.0.5 * * *
                      * * * Patch 6.0.5.300 * * *
                         Patch Date: 2015-08-21


This document provides the following information:

   * PATCH NAME
   * OPERATING SYSTEMS SUPPORTED BY THE PATCH
   * PACKAGES AFFECTED BY THE PATCH
   * BASE PRODUCT VERSIONS FOR THE PATCH
   * SUMMARY OF INCIDENTS FIXED BY THE PATCH
   * DETAILS OF INCIDENTS FIXED BY THE PATCH
   * INSTALLATION PRE-REQUISITES
   * INSTALLING THE PATCH
   * REMOVING THE PATCH
   * KNOWN ISSUES


PATCH NAME
----------
Veritas Storage Foundation HA 6.0.5 Patch 6.0.5.300


OPERATING SYSTEMS SUPPORTED BY THE PATCH
----------------------------------------
RHEL6 x86-64


PACKAGES AFFECTED BY THE PATCH
------------------------------
VRTSaslapm
VRTSvxfs
VRTSvxvm


BASE PRODUCT VERSIONS FOR THE PATCH
-----------------------------------
   * Symantec VirtualStore 6.0.1
   * Veritas Dynamic Multi-Pathing 6.0.1
   * Veritas Storage Foundation 6.0.1
   * Veritas Storage Foundation Cluster File System HA 6.0.1
   * Veritas Storage Foundation for Oracle RAC 6.0.1
   * Veritas Storage Foundation HA 6.0.1


SUMMARY OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
Patch ID: VRTSvxfs-6.0.500.200-RHEL6
* 3622423 (3250239) Panic in vx_naio_worker
* 3673958 (3660422) On RHEL 6.6, umount(8) system call hangs if an application is watching for inode
events using inotify(7) APIs.
* 3678626 (3613048) Support vectored AIO on Linux
Patch ID: VRTSvxfs-6.0.500.100-RHEL6
* 3409692 (3402618) The mmap read performance on VxFS is slow
* 3426009 (3412667) The RHEL 6 system panics with Stack Overflow.
* 3469683 (3469681) File system is disabled while free space defragmentation is going on.
* 3498950 (3356947) When there are multi-threaded writes with fsync calls between them, VxFS becomes slow.
* 3498954 (3352883) During the rename operation, lots of nfsd threads hang.
* 3498963 (3396959) RHEL 6.4 system panics with stack overflow errors due to memory pressure.
* 3498976 (3434811) The vxfsconvert(1M) in VxFS 6.1 hangs.
* 3498978 (3424564) fsppadm fails with ENODEV and "file is encrypted or is not a 
database" errors
* 3498998 (3466020) File System is corrupted with error message "vx_direrr: vx_dexh_keycheck_1"
* 3499005 (3469644) System panics in the vx_logbuf_clean() function.
* 3499008 (3484336) The fidtovp() system call can panic in the vx_itryhold_locked () function.
* 3499011 (3486726) VFR logs too much data on the target node.
* 3499014 (3471245) The mongodb fails to insert any record.
* 3499030 (3484353) The file system may hang with a partitioned directory feature enabled.
* 3514824 (3443430) Fsck allocates too much memory.
* 3515559 (3498048) while the system is making backup, the Als AlA command on the same file system may hang.
* 3515569 (3430461) The nested unmounts fail if the parent file system is disabled.
* 3515588 (3294074) System call fsetxattr() is slower on Veritas File System (VxFS) than ext3 file system.
* 3515737 (3511232) Stack overflow causes kernel panic in the vx_write_alloc() function.
* 3515739 (3510796) System panics when VxFS cleans the inode chunks.
* 3517702 (3517699) Return code 240 for command fsfreeze(1M) is not documented in man page for fsfreeze.
* 3517707 (3093821) The system panics due to referring freed super block after the vx_unmount() function errors.
* 3557193 (3473390) Multiple stack overflows with VxFS on RHEL6 lead to panics/system crashes.
* 3567855 (3567854) On VxFS 6.0.5, the vxedquota(1M) and vxrepquota(1M) commands fail in certain scenarios.
* 3579957 (3233315) "fsck" utility dumps core, with full scan.
* 3581566 (3560968) The delicache_enable tunable is not persistent in the Cluster File System (CFS) environment.
* 3584297 (3583930) While external quota file is restored or over-written, old quota records are preserved.
* 3588236 (3604007) Stack overflow on SLES11
* 3590573 (3331010) Command fsck(1M) dumped core with segmentation fault
* 3595894 (3595896) While creating the OracleRAC 12.1.0.2 database, the node panics.
* 3597454 (3602386) vfradmin man page shows the incorrect info about default
behavior of -d option
* 3597560 (3597482) The pwrite(2) function fails with the EOPNOTSUPP error.
Patch ID: VRTSvxvm-6.0.500.200-RHEL6
* 3648603 (3564260) VVR commands are unresponsive when replication is paused and resumed in a loop.
* 3654163 (2916877) vxconfigd hangs on a node leaving the cluster.
* 3690795 (2573229) On RHEL6, the server panics when Dynamic Multi-Pathing (DMP) executes 
PERSISTENT RESERVE IN command with REPORT CAPABILITIES service action on 
powerpath controlled device.
* 3774137 (3565212) IO failure is seen during controller giveback operations 
on Netapp Arrays in ALUA mode.
* 3780978 (3762580) In Linux kernels greater than or equal to RHEL6.6 like RHEL7 and SLES11SP3, the 
vxfen module fails to register the SCSI-3 PR keys to EMC devices when powerpath
exists in coexistence with DMP (Dynamic Multipathing).
* 3788751 (3788644) Reuse raw device number when checking for available raw devices.
* 3796596 (3433503) Due to an incorrect memory access, Vxconfigd(1M) daemon cores 
with a stack trace.
* 3796666 (3573262) System panic during space optimized snapshot operations  
on recent UltraSPARC architectures.
* 3832703 (3488071) The command "vxdmpadm settune dmp_native_support=on" fails to enable Dynamic 
Multipathing (DMP) Native Support
* 3832705 (3776520) Filters are not updated properly in lvm.conf file in VxDMP initrd(initial 
ramdisk) while enabling Dynamic Multipathing (DMP) Native Support.
* 3835367 (3137543) Issue with Root disk encapsulation(RDE) due to changes in
Linux trusted boot grub.
* 3835662 (3581646) Logical Volumes fail to migrate back to OS devices Dynamic Multipathing (DMP) when DMP native support is disabled while root("/") is mounted on 
LVM.
* 3836923 (3133322) When dmp native support is enabled, This command "vxdmpadm native ls" does not
display root devices under linux
logical volume manager(LVM) in any volume group(VG) .
Patch ID: VRTSvxvm-6.0.500.100-RHEL6
* 3496715 (3281004) For DMP minimum queue I/O policy with large number of CPUs a couple of issues 
are observed.
* 3501358 (3399323) The reconfiguration of Dynamic Multipathing (DMP) database fails.
* 3502662 (3380481) When you select a removed disk during the "5 Replace a failed or removed disk" operation, the vxdiskadm(1M) command displays an error message.
* 3521727 (3521726) System panicked for double freeing IOHINT.
* 3526501 (3526500) Disk IO failures occur with DMP IO timeout error messages when DMP (Dynamic Multi-pathing) IO statistics demon is not running.
* 3531332 (3077582) A Veritas Volume Manager (VxVM) volume may become inaccessible causing the read/write operations to fail.
* 3552411 (3482026) The vxattachd(1M) daemon reattaches plexes of manually detached site.
* 3600161 (3599977) During replica connection, referencing a port which is already deleted in 
another thread caused system panic.
* 3612801 (3596330) 'vxsnap refresh' operation fails with `Transaction aborted waiting for IO 
drain` error
* 3621240 (3621232) vradmin ibc command cannot be started/executed on VVR (Veritas Volume 
Replicator) Secondary.
* 3622069 (3513392) Reference to replication port that is already deleted caused panic.
* 3632969 (3631230) VRTSvxvm patch version 6.0.5 and 6.1.1 or previous will not work with RHEL6.6
update.
* 3638039 (3625890) Vxdisk resize a CDS disk failing with "invalid attribute specification"
Patch ID: VRTSaslapm-6.0.500.300-GA_RHEL6
* 3665733 (3665727) Array I/O POLICY is set to Single-active for SF6.1.1 with RHEL6.6


DETAILS OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
This patch fixes the following Symantec incidents:

Patch ID: VRTSvxfs-6.0.500.200-RHEL6

* 3622423 (Tracking ID: 3250239)

SYMPTOM:
Panic in vx_naio_worker, trace back:-
crash_kexec()
__die ()
do_page_fault()
error_exit()
vx_naio_worker()
vx_kthread_init()
kernel_thread()

DESCRIPTION:
If a thread submitting linux native AIO has posix sibling threads, and it 
simply exits after the submit, then kernel will not wait for the aio to finish 
in the exit processing. When aio complete, it may dereference the task_struct 
of that exited thread, this cause the panic.

RESOLUTION:
Install vxfs hook to task_struct, so when the thread exits, it will wait for 
its aio to complete no matter it's cloned or not. Set module load-time tunable 
vx_naio_wait to non-zero value to turn on this fix.

* 3673958 (Tracking ID: 3660422)

SYMPTOM:
On RHEL 6.6, umount(8) system call hangs if an application is watching for inode
events using inotify(7) APIs.

DESCRIPTION:
On RHEL 6.6, additional counters were added in the super block to track inotify
watches, these new counters were not implemented in VxFS.
Hence while doing umount, the operation hangs until the counter in the
superblock drops to zero, which would never happen since they are not handled in
VXFS.

RESOLUTION:
Code is modified to handle additional counters added in RHEL6.6.

* 3678626 (Tracking ID: 3613048)

SYMPTOM:
System can panic with the following stack::
 
machine_kexec
crash_kexec
oops_end
die
do_invalid_op
invalid_op
aio_complete
vx_naio_worker
vx_kthread_init

DESCRIPTION:
VxFS does not correctly support IOCB_CMD_PREADV and IOCB_CMD_PREADV, which 
causes a BUG to fire in the kernel code (in fs/aio.c:__aio_put_req()).

RESOLUTION:
Add support for the vectored AIO commands and fixed the increment of ->ki_users 
so it is guarded by the required spinlock.

Patch ID: VRTSvxfs-6.0.500.100-RHEL6

* 3409692 (Tracking ID: 3402618)

SYMPTOM:
The mmap read performance on VxFS is slow.

DESCRIPTION:
The mmap read performance on VxFS is not good, because the read ahead operation is not triggered while the mmap reads is executed.

RESOLUTION:
An enhancement has been made to the read ahead operation. It helps improve the mmap read performance.

* 3426009 (Tracking ID: 3412667)

SYMPTOM:
On RHEL 6, the inode update operation may create deep stack and cause system panic  due to stack overflow. Below is the stack trace:
dequeue_entity()
dequeue_task_fair()
dequeue_task()
deactivate_task()
thread_return()
io_schedule()
get_request_wait()
blk_queue_bio()
generic_make_request()
submit_bio()
vx_dev_strategy()
vx_bc_bwrite()
vx_bc_do_bawrite()
vx_bc_bawrite()
 vx_bwrite()
vx_async_iupdat()
vx_iupdat_local()
vx_iupdat_clustblks()
vx_iupdat_local()
vx_iupdat()
vx_iupdat_tran()
vx_tflush_inode()
vx_fsq_flush()
vx_tranflush()
vx_traninit()
vx_get_alloc()
vx_tran_get_alloc()
vx_alloc_getpage()
vx_do_getpage()
vx_internal_alloc()
vx_write_alloc()
vx_write1()
vx_write_common_slow()
vx_write_common()
vx_vop_write()
vx_writev()
vx_naio_write_v2()
do_sync_readv_writev()
do_readv_writev()
vfs_writev()
nfsd_vfs_write()
nfsd_write()
nfsd3_proc_write()
nfsd_dispatch()
svc_process_common()
svc_process()
nfsd()
kthread()
kernel_thread()

DESCRIPTION:
Some VxFS operation may need inode update. This may create very deep stack and cause system panic due to stack overflow.

RESOLUTION:
The code is modified to add a handoff point in the inode update function. If the stack usage reaches a threshold, it will start a separate thread to do the work to limit stack usage.

* 3469683 (Tracking ID: 3469681)

SYMPTOM:
Free space defragmentation results into EBUSY error and file system is disabled.

DESCRIPTION:
While remounting the file system, the re-initialization gives EBUSY error if the  in-core and on-disk version numbers of an inode does not match. When pushing data blocks to the clone, the inode version of the immediate clone inode is bumped. But if there is another clone in the chain, then the ILIST extent of this immediate clone inode is not pushed onto that clone. This is not right because the inode has been modified.

RESOLUTION:
The code is modified so that the ILIST extents of the immediate clone inode is pushed onto the next clone in chain.

* 3498950 (Tracking ID: 3356947)

SYMPTOM:
VxFS doesnAt work as fast as expected when multi-threaded writes are
issued onto a file, which is interspersed by fsync calls.

DESCRIPTION:
When multi-threaded writes are issued with fsync calls in between the writes, fsync can serialise the writes by taking IRWLOCK on the inode and doing the whole file putpages. Therefore out-of-the box performance is relatively slow in terms of throughput.

RESOLUTION:
The code is fixed to remove the fsync's serialisation with IRWLOCK and make it conditional only for some cases.

* 3498954 (Tracking ID: 3352883)

SYMPTOM:
During the rename operation, lots of nfsd threads waiting for mutex operation hang with the following stack trace :
vxg_svar_sleep_unlock 
vxg_get_block
vxg_api_initlock  
vx_glm_init_blocklock
vx_cbuf_lookup  
vx_getblk_clust 
vx_getblk_cmn 
vx_getblk
vx_fshdchange
vx_unlinkclones
vx_clone_dispose
vx_workitem_process
vx_worklist_process
vx_worklist_thread
vx_kthread_init


vxg_svar_sleep_unlock 
vxg_grant_sleep 
vxg_cmn_lock 
vxg_api_trylock 
vx_glm_trylock 
vx_glmref_trylock 
vx_mayfrzlock_try 
vx_walk_fslist 
vx_log_sync 
vx_workitem_process 
vx_worklist_process 
vx_worklist_thread 
vx_kthread_init

DESCRIPTION:
A race condition is observed between the NFS rename and additional dentry alias created by the current vx_splice_alias()function. 
This race condition causes two different directory dentries pointing to the same inode, which results in mutex deadlock in lock_rename()function.

RESOLUTION:
The code is modified to change the vx_splice_alias()function to prevent the creation of additional dentry alias.

* 3498963 (Tracking ID: 3396959)

SYMPTOM:
On RHEL 6.4 with insufficient free memory, creating a file may panic the system with the following stack trace: 

shrink_mem_cgroup_zone()
shrink_zone()
do_try_to_free_pages()
try_to_free_pages()
__alloc_pages_nodemask()
alloc_pages_current()
__get_free_pages()
vx_getpages()
vx_alloc()
vx_bc_getfreebufs()
vx_bc_getblk()
vx_getblk_bp()
vx_getblk_cmn()
vx_getblk()
vx_iread()
vx_local_iread()
vx_iget()
vx_ialloc()
vx_dirmakeinode()
vx_dircreate()
vx_dircreate_tran()
vx_pd_create()
vx_create1_pd()
vx_do_create()
vx_create1()
vx_create_vp()
vx_create()
vfs_create()
do_filp_open()
do_sys_open() 
sys_open()
system_call_fastpath()

DESCRIPTION:
VxFS estimates the stack that is required to perform various kernel operations and creates hand-off threads if the estimated stack usage goes above the allowed kernel limit. However, the estimation may go wrong when the system is under heavy memory pressure, as some Linux kernel changes in RHEL 6.4 increase the depth of stack. There might be additional functions that are called in case of getpage to alleviate the situation, which leads to increased stack usage.

RESOLUTION:
The code is modified to take stack depth calculations to
correctly estimate the stack usage under memory pressure conditions.

* 3498976 (Tracking ID: 3434811)

SYMPTOM:
In VxFS 6.1, the vxfsconvert(1M) command hangs within the vxfsl3_getext()
Function with following stack trace:

search_type()
bmap_typ()
vxfsl3_typext()
vxfsl3_getext()
ext_convert()
fset_convert()
convert()

DESCRIPTION:
There is a type casting problem for extent size. It may cause a non-zero value to overflow and turn into zero by mistake. This further leads to infinite looping inside the function.

RESOLUTION:
The code is modified to remove the intermediate variable and avoid type casting.

* 3498978 (Tracking ID: 3424564)

SYMPTOM:
fsppadm fails with ENODEV and "file is encrypted or is not a database" 
errors

DESCRIPTION:
The error handler was missing for ENODEV, while we process only the 
directory inodes and the database got corrupted for 2nd error.

RESOLUTION:
Added a error handler to ignore the ENODEV while processing 
directory inode only and for database corruption: we added a log message to 
capture all the db logs to understand/know why corruption happened.

* 3498998 (Tracking ID: 3466020)

SYMPTOM:
File System is corrupted with the following error message in the log:

WARNING: msgcnt 28 mesg 008: V-2-8: vx_direrr: vx_dexh_keycheck_1 - /TraceFile
file system dir inode 3277090 dev/block 0/0 diren
 WARNING: msgcnt 27 mesg 008: V-2-8: vx_direrr: vx_dexh_keycheck_1 - /TraceFile
file system dir inode 3277090 dev/block 0/0 diren
 WARNING: msgcnt 26 mesg 008: V-2-8: vx_direrr: vx_dexh_keycheck_1 - /TraceFile
file system dir inode 3277090 dev/block 0/0 diren
 WARNING: msgcnt 25 mesg 096: V-2-96: vx_setfsflags -
 /dev/vx/dsk/a2fdc_cfs01/trace_lv01 file system fullfsck flag set - vx_direr
 WARNING: msgcnt 24 mesg 008: V-2-8: vx_direrr: vx_dexh_keycheck_1 - /TraceFile
file system dir inode 3277090 dev/block 0/0 diren

DESCRIPTION:
In case error is returned from function vx_dirbread() via function vx_dexh_keycheck1(), the FULLFSCK flag is set on the FS unconditionally. A corrupted LDH can lead to the reading of the wrong block, which results in the setting of FULLFSCK flag. The system doesnAt verify whether it is reading the wrong value due to a corrupted LDH, so that the FULLFSCK flag is set unnecessarily because a corrupted LDH can be fixed online by recreating the hash.

RESOLUTION:
The code is modified such that when a corruption of the LDH is detected, the system removes the Large Directory hash instead of setting FULLFSCK. The Large Directory Hash will then be recreated the next time the directory is modified.

* 3499005 (Tracking ID: 3469644)

SYMPTOM:
System panics in the vx_logbuf_clean() function while traversing chain of transactions off the intent log buffer. The stack trace is as follows:


vx_logbuf_clean ()
vx_logadd ()
vx_log()
vx_trancommit()
vx_exh_hashinit ()
vx_dexh_create ()
vx_dexh_init ()
vx_pd_rename ()
vx_rename1_pd()
vx_do_rename ()
vx_rename1 ()
vx_rename ()
vx_rename_skey ()

DESCRIPTION:
The system panics as the vx_logbug_clean() function tries to access an already freed transaction from transaction chain to flush it to log.

RESOLUTION:
The code is modified to make sure that transaction gets flushed to the log before it is freed.

* 3499008 (Tracking ID: 3484336)

SYMPTOM:
The fidtovp() system call can panic in the vx_itryhold_locked() function with the following stack trace:

vx_itryhold_locked
vx_iget
vx_common_vget
vx_do_vget
vx_vget_skey
vfs_vget
fidtovp
kernel_add_gate_cstack
nfs3_fhtovp
rfs3_getattr
rfs_dispatch
svc_getreq
threadentry
[kdb_read_mem]

DESCRIPTION:
Some VxFS operations like the vx_vget() function try to get a hold on an in-core inode using the vx_itryhold_locked() function, but it doesnAt take the lock on the corresponding directory inode. This may lead to a race condition when this inode is present on the delicache list and is inactivated. Thereby this results in a panic when the vx_itryhold_locked() function tries to remove it from a free list. This is actually a known issue, but the last fix was not complete. It missed some functions which may also cause the race condition.

RESOLUTION:
The code is modified to take inode list lock inside the vx_inactive_tran(), vx_tranimdone() and vx_tranuninode() functions to avoid race condition.

* 3499011 (Tracking ID: 3486726)

SYMPTOM:
VFR logs too much data on the target node.

DESCRIPTION:
At the target node, it logs debug level messages evenif the debug mode was off. Also it doesnAt consider the debug mode specified at the time of job creation.

RESOLUTION:
The code is modified to not log the debug level messages on the target node if the specified debug mode is set off.

* 3499014 (Tracking ID: 3471245)

SYMPTOM:
The Mongodb fails to insert any record because lseek fails to seek to the EOF.

DESCRIPTION:
Fallocate doesn't update the inode's i_size on linux, which causes lseek unable to seek to the EOF.

RESOLUTION:
Before returning from the vx_fallocate() function, call the vx_getattr()function to update the Linux inode with the VxFS inode.

* 3499030 (Tracking ID: 3484353)

SYMPTOM:
It is a self-deadlock caused by a missing unlock of DIRLOCK. Its typical stack trace is like the following:
 
slpq_swtch_core()
real_sleep()
sleep_one()
vx_smp_lock()
vx_dirlock()
vx_do_rename()
vx_rename1()
vx_rename()
vn_rename()
rename()
syscall()

DESCRIPTION:
When a partitioned directory feature (PD) of Veritas File System (VxFS) is enabled, there is a possibility of self-deadlock when there are multiple renaming threads operating on the same target directory.
The issue is due to the fact that there is a missing unlock of DIRLOCK in the vx_int_rename() function.

RESOLUTION:
The code is modified by adding missing unlock for directory lock in the vx_int_rename()function..

* 3514824 (Tracking ID: 3443430)

SYMPTOM:
Fsck allocates too much memory.

DESCRIPTION:
Since Storage Foundation 6.0, parallel inode list processing with multiple threads is introduced to help reduce the fsck time. However, the parallel threads have to allocate redundant memory instead of reusing buffers in the buffer cache efficiently when inode list has many holes.

RESOLUTION:
The code is fixed to make each thread to maintain its own buffer cache from which it can reuse free memory.

* 3515559 (Tracking ID: 3498048)

SYMPTOM:
while the system is making backup, the Als AlA command on the same file system may hang.

DESCRIPTION:
When the dalloc (delayed allocation) feature is turned on, flushing takes quite a lot of time which keeps hold on getpage lock, as this lock is needed by writers which keep read write lock held on inodes. The Als AlA command needs ACLs(access control lists) to display information. But in Veritas File System (VxFS), ACLS are accessed only under protection of inode read write lock, which results in the hang.

RESOLUTION:
The code is modified to turn dalloc off and improve write throttling by restricting the kernel flusher from updating Intenal counter for write page flush..

* 3515569 (Tracking ID: 3430461)

SYMPTOM:
The nested unmounts as well as the force unmounts fail if, the parent file system is disabled which further inhibits the unmounting of the child file system.

DESCRIPTION:
If a file system is mounted inside another vxfs mount, and if the parent file system gets disabled, then it is not possible to sanely unmount the child even with the force unmounts. This issue is observed because a disabled file system does not allow directory look up on it. On Linux, a file system can be unmounted only by providing the path of the mount point.

RESOLUTION:
The code is modified to allow the exceptional path look for unmounts. These are read only operations and hence are safer. This makes it possible for the unmount of child file system to proceed.

* 3515588 (Tracking ID: 3294074)

SYMPTOM:
System call fsetxattr() is slower on Veritas File System (VxFS) than ext3 file system.

DESCRIPTION:
VxFS implements the fsetxattr() system call in a synchronized sync way.  Hence, it will take some time to flush the data to the disk before returning to the system call to guarantee file system consistency in case of file system crash.

RESOLUTION:
The code is modified to allow the transaction to flush the data in a delayed way.

* 3515737 (Tracking ID: 3511232)

SYMPTOM:
The kernel panics in the vx_write_alloc() function with the following stack trace:

__schedule_bug
thread_return
schedule_timeout
wait_for_common
wait_for_completion
vx_getpages_handoff
vx_getpages
vx_alloc
vx_mklbtran
vx_te_bufalloc
vx_te_bmap_split
vx_te_bmap_alloc
vx_bmap_alloc_typed
vx_bmap_alloc
vx_get_alloc
vx_tran_get_alloc
vx_alloc_getpage
vx_do_getpage
vx_internal_alloc
vx_write_alloc
vx_write1
vx_write_common_slow
vx_write_common
vx_write
vx_naio_write
vx_naio_write_v2
aio_rw_vect_retry
aio_run_iocb
do_io_submit
sys_io_submit
system_call_fastpath

DESCRIPTION:
During a memory allocation, the stack overflows  and corrupts the
thread_info structure of the thread. Then when the system sleeps for a handed-off result, the corruption caused by the overflow is detected and the system panics.

RESOLUTION:
The code is fixed to detect thread_info corruptions, as well as to change some stack allocations to dynamic allocations to save stack usage.

* 3515739 (Tracking ID: 3510796)

SYMPTOM:
System panics in Linux 3.0.x kernel when VxFS cleans the inode chunks.

DESCRIPTION:
On Linux, the kernel swap daemon (kswapd) takes a reference hold on a page but not the owning inode. In vx_softcnt_flush(), when the inode's final softcnt drops, the kernel calls the vx_real_destroy_inode() function. On 3.x.x kernels, vx_real_destroy_inode() temporarily clears the address-space operations. Therefore a window is created where kswapd works on a page without VxFS's address-space operations set, and results in a panic.

RESOLUTION:
The code is fixed to flush all pages for an inode before it calls the vx_softcnt_flush() function.

* 3517702 (Tracking ID: 3517699)

SYMPTOM:
Return code 240 for command fsfreeze(1M) is not documented in man page for fsfreeze.

DESCRIPTION:
Return code 240 for command fsfreeze(1M) is not documented in man page for fsfreeze.

RESOLUTION:
The man page for fsfreeze(1M) is modified to document return code 240.

* 3517707 (Tracking ID: 3093821)

SYMPTOM:
The system panics due to referring freed super block after the vx_unmount() function errors.

DESCRIPTION:
In the file system unmount process, once Linux VFS calls in to VxFS specific unmount processing vx_unmount(), it doesn't expect error from this call. So, once the vx_unmount()function returns, Linux frees the file systems corresponding super_block object. But if any error is observed during vx_unmount(), it may leave file system Inodes in vxfs inode cache as it is. Otherwise, when there is no error, the file system inodes are processed and dropped on vx_unmount().

As file systems inodes left in VxFS inode cache would still point to freed super block object, so now when these inodes on Inode cache are cleaned up to free or reuse, they may refer freed super block in certain cases, which might lead to Panic due to NULL pointer de-reference.

RESOLUTION:
Do not return EIO or ENXIO error from vx_detach_fset() when you unmounts the filesystem. Insted of returning error process, drop inode from the inode cache.

* 3557193 (Tracking ID: 3473390)

SYMPTOM:
In memory pressure scenarios, we see panics/system crashes due to stack overflows.

DESCRIPTION:
Specifically on RHEL6, the memory allocation routines consume much higher memory than other distributions like SLES, or even RHEL5.  Due to this, multiple overflows are reported for the RHEL6 platform. Most of these overflows occur when Veritas File System (VxFS) tries to allocate memory under memory pressure.

RESOLUTION:
The code is modified to fix multiple overflows by adding handoff codepaths, adjusting handoff limits, removing on-stack structures and reducing the number of function frames on stack wherever possible.

* 3567855 (Tracking ID: 3567854)

SYMPTOM:
On Veritas File System (VxFS) 6.0.5, 
A If the file system has no external quota file, the vxedquota(1M) command gives error. 
A If the first mounted VxFS file system in the mnttab file contains no quota file, the vxrepquota(1M) command fails.

DESCRIPTION:
A The vxedquota(1M) issue:
When the vxedquota(1M) command is executed, it expects the 32-bit quota file to be present in the mount-point, even if it does not contain any external quota file.
This results in the error message while you are editing the quotas.
This issue is fixed by only processing the file systems on which quota files are present.
A The vxrepquota(1M) issue:
On the other hand, when vxrepquota(1M) is executed, the command vxrepquota(1M) scans the mnttab file [0]to look for the VxFS file system which has external quota files. But, if the first VxFS file system doesn't contain any quota file, the command gives error and does not report any quota information.
This issue is fixed by looping through the mnttab file till we get mount-point in which quota files exist.

RESOLUTION:
A For the vxedquota(1M) issue, the code is modified by skipping the file systems  which don't contain quota files.
A For the vxrepquota(1M) issue, the code is modified by obtaining a VxFS mount-point (with quota files) from the mnttab file.

* 3579957 (Tracking ID: 3233315)

SYMPTOM:
"fsck" utility dumps core, while checking the RCT file.

DESCRIPTION:
"fsck" utility dumps core, while checking the RCT file. "bmap_search_typed()"
function is passed with wrong parameter, and results in the core dump with the
following stack trace:

bmap_get_typeparms ()
bmap_search_typed_raw()
bmap_search_typed()
rct_walk()
bmap_check_typed_raw()
rct_check()
main()

RESOLUTION:
Fixed the code to pass the correct parameters to "bmap_search_typed()" function.

* 3581566 (Tracking ID: 3560968)

SYMPTOM:
The delicache_enable tunable is inconsistent in the CFS environment.

DESCRIPTION:
On the secondary nodes, the tunable values are exported from the primary mount, while the delicache_enable tunable value comes from the AtunefstabA file. Therefore the  tunable values are not persistent.

RESOLUTION:
The code is fixed to read the "tunefstab" file only for the delicache_enable tunable during mount and set the value accordingly.

* 3584297 (Tracking ID: 3583930)

SYMPTOM:
When external quota file is over-written or restored from backup, new settings which were added after the backup still remain.

DESCRIPTION:
The purpose of the quotaon operation is to copy the quota limits from external to internal quota file, because internal quota file is not always updated with correct limits. To complete the copy operation, the extent of external file is compared to the extent of internal file at the corresponding offset.
     Now, if external quota file is overwritten (or restored to its original copy) and the size of internal file is more than that of external, the quotaon operation does not clear the additional (stale) quota records in the internal file. Later, the sync operation (part of quotaon) copies these stale records from internal to external file. Hence, both internal and external files contain stale records.

RESOLUTION:
The code is modified to get rid of the stale records in the internal file at the time of quotaon.

* 3588236 (Tracking ID: 3604007)

SYMPTOM:
Stack overflow was observed during extent allocation code path.

DESCRIPTION:
In extent allocation code path during write we are seeing stack 
overflow on sles11. We already have hand-off point in the code path but hand-off 
was not happening due to the reason that we were having some bytes more than 
necessary to trigger the hand-off.

RESOLUTION:
Now changing the value of trigger point so that hand-off can takes 
place.

* 3590573 (Tracking ID: 3331010)

SYMPTOM:
Command fsck(1M) dumped core with segmentation fault.
Following stack is observed.

fakebmap()
rcq_apply_op()
rct_process_pending_tasklist()
process_device()
main()

DESCRIPTION:
While working on the device in function precess_device(), command fsck tries to
access already freed device related structures available in pending task list
during retry code path.

RESOLUTION:
Code is modified to free up the pending task list before retrying in function
precess_device().

* 3595894 (Tracking ID: 3595896)

SYMPTOM:
While creating the OracleRAC 12.1.0.2 database, the node panics with the following stack:
aio_complete()
vx_naio_do_work()
vx_naio_worker()
vx_kthread_init()

DESCRIPTION:
For a zero size request (with a correctly aligned buffer), Veritas File System (VxFS) erroneously queues the work internally and returns -EIOCBQUEUED. The kernel calls function aio_complete() for this zero size request. However, while VxFS is performing the queued work internally,  function aio_complete()gets called again. The double call of function aio_complete() results in the panic.

RESOLUTION:
The code is modified such that zero size requests will not queue elements inside VxFS work queue.

* 3597454 (Tracking ID: 3602386)

SYMPTOM:
vfradmin man page shows the incorrect info about default behavior of -d
option

DESCRIPTION:
When we run vfradmin command without -d option then by default the
debugging is in ENABLED mode but man page indicates that the default debugging
should be in DISABLED mode.

RESOLUTION:
Changes has been done in man page of vfradmin to reflect the correct
default behavior.

* 3597560 (Tracking ID: 3597482)

SYMPTOM:
The pwrite(2) function fails with EOPNOTSUPP when the write range is in two indirect extents.

DESCRIPTION:
When the range of pwrite() falls in two indirect extents, one ZFOD extent belonging to DB2 pre-allocated files created with setext( , VX_GROWFILE, ) ioctl and another DATA extent belonging to adjacent INDIR, write fails with EOPNOTSUPP. 
The reason is that Veritas File System (VxFS) is trying to coalesce extents which belong to different indirect address extents as part of this transaction A such a meta-data change consumes more transaction resources which VxFS transaction engine is unable to support in the current implementation.

RESOLUTION:
The code is modified to retry the write transaction without combining the extents.

Patch ID: VRTSvxvm-6.0.500.200-RHEL6

* 3648603 (Tracking ID: 3564260)

SYMPTOM:
VVR commands are unresponsive when replication is paused and resumed in a loop.

DESCRIPTION:
While Veritas Volume Replicator (VVR) is in the process of sending updates then pausing a replication is deferred until acknowledgements of updates are received or until an error occurs. For some reason, if the acknowledgements get delayed or the delivery fails, the pause operation continues to get deferred resulting in unresponsiveness.

RESOLUTION:
The code is modified to resolve the issue that caused unresponsiveness.

* 3654163 (Tracking ID: 2916877)

SYMPTOM:
vxconfigd hangs, if a node leaves the cluster, while I/O error handling is in
progress. Stack observed is as follows:
volcvm_iodrain_dg 
volcvmdg_abort_complete
volcvm_abort_sio_start 
voliod_loop
vol_kernel_thread_init

DESCRIPTION:
A bug in DCO error handling code can lead to an infinite loop if a node leaves
cluster while I/O error handling is in progress. This causes vxconfigd to hang
and stop responding to VxVM commands like vxprint, vxdisk 
etc.

RESOLUTION:
DCO error handling code has been changed so that I/O errors are handled
correctly. Hence, hang is avoided.

* 3690795 (Tracking ID: 2573229)

SYMPTOM:
On RHEL6, the server panics when Dynamic Multi-Pathing (DMP) executes 
PERSISTENT RESERVE IN command with REPORT CAPABILITIES service action on 
powerpath controlled device. The following stack trace is displayed:

enqueue_entity at ffffffff81068f09
enqueue_task_fair at ffffffff81069384
enqueue_task at ffffffff81059216
activate_task at ffffffff81059253
pull_task at ffffffff81065401
load_balance_fair at ffffffff810657b7
thread_return at ffffffff81527d30
schedule_timeout at ffffffff815287b5
wait_for_common at ffffffff81528433
wait_for_completion at ffffffff8152854d
blk_execute_rq at ffffffff8126d9dc
emcp_scsi_cmd_ioctl at ffffffffa04920a2 [emcp]
PowerPlatformBottomDispatch at ffffffffa0492eb8 [emcp]
PowerSyncIoBottomDispatch at ffffffffa04930b8 [emcp]
PowerBottomDispatchPirp at ffffffffa049348c [emcp]
PowerDispatchX at ffffffffa049390d [emcp]
MpxSendScsiCmd at ffffffffa061853e [emcpmpx]
ClariionKLam_groupReserveRelease at ffffffffa061e495 [emcpmpx]
MpxDefaultRegister at ffffffffa061df0a [emcpmpx]
MpxTestPath at ffffffffa06227b5 [emcpmpx]
MpxExtraTry at ffffffffa06234ab [emcpmpx]
MpxTestDaemonCalloutGuts at ffffffffa062402f [emcpmpx]
MpxIodone at ffffffffa0624621 [emcpmpx]
MpxDispatchGuts at ffffffffa0625534 [emcpmpx]
MpxDispatch at ffffffffa06256a8 [emcpmpx]
PowerDispatchX at ffffffffa0493921 [emcp]
GpxDispatch at ffffffffa0644775 [emcpgpx]
PowerDispatchX at ffffffffa0493921 [emcp]
GpxDispatchDown at ffffffffa06447ae [emcpgpx]
VluDispatch at ffffffffa068b025 [emcpvlumd]
GpxDispatch at ffffffffa0644752 [emcpgpx]
PowerDispatchX at ffffffffa0493921 [emcp]
GpxDispatchDown at ffffffffa06447ae [emcpgpx]
XcryptDispatchGuts at ffffffffa0660b45 [emcpxcrypt]
XcryptDispatch at ffffffffa0660c09 [emcpxcrypt]
GpxDispatch at ffffffffa0644752 [emcpgpx]
PowerDispatchX at ffffffffa0493921 [emcp]
GpxDispatch at ffffffffa0644775 [emcpgpx]
PowerDispatchX at ffffffffa0493921 [emcp]
PowerSyncIoTopDispatch at ffffffffa04978b9 [emcp]
emcp_send_pirp at ffffffffa04979b9 [emcp]
emcp_pseudo_blk_ioctl at ffffffffa04982dc [emcp]
__blkdev_driver_ioctl at ffffffff8126f627
blkdev_ioctl at ffffffff8126faad
block_ioctl at ffffffff811c46cc
dmp_ioctl_by_bdev at ffffffffa074767b [vxdmp]
dmp_kernel_scsi_ioctl at ffffffffa0747982 [vxdmp]
dmp_scsi_ioctl at ffffffffa0786d42 [vxdmp]
dmp_send_scsireq at ffffffffa078770f [vxdmp]
dmp_do_scsi_gen at ffffffffa077d46b [vxdmp]
dmp_pr_check_aptpl at ffffffffa07834dd [vxdmp]
dmp_make_mp_node at ffffffffa0782c89 [vxdmp]
dmp_decode_add_disk at ffffffffa075164e [vxdmp]
dmp_decipher_instructions at ffffffffa07521c7 [vxdmp]
dmp_process_instruction_buffer at ffffffffa075244e [vxdmp]
dmp_reconfigure_db at ffffffffa076f40e [vxdmp]
gendmpioctl at ffffffffa0752a12 [vxdmp]
dmpioctl at ffffffffa0754615 [vxdmp]
dmp_ioctl at ffffffffa07784eb [vxdmp]
dmp_compat_ioctl at ffffffffa0778566 [vxdmp]
compat_blkdev_ioctl at ffffffff8128031d
compat_sys_ioctl at ffffffff811e0bfd
sysenter_dispatch at ffffffff81050c20

DESCRIPTION:
Dynamic Multi-Pathing (DMP) uses PERSISTENT RESERVE IN command with the REPORT
CAPABILITIES service action to discover target capabilities. On RHEL6, system
panics unexpectedly when Dynamic Multi-Pathing (DMP) executes PERSISTENT 
RESERVE IN command with REPORT CAPABILITIES service action on powerpath 
controlled device coming from EMC Clarion/VNX array. This bug has been reported 
to EMC powperpath engineering.

RESOLUTION:
The Dynamic Multi-Pathing (DMP) code is modified to execute PERSISTENT RESERVE 
IN command with the REPORT CAPABILITIES service action to discover target 
capabilities only on non-third party controlled devices.

* 3774137 (Tracking ID: 3565212)

SYMPTOM:
While performing controller giveback operations on NetApp ALUA arrays, the 
below messages are observed in /etc/vx/dmpevents.log

[Date]: I/O error occured on Path <path> belonging to Dmpnode <dmpnode>
[Date]: I/O analysis done as DMP_PATH_BUSY on Path <path> belonging to 
Dmpnode 
<dmpnode>
[Date]: I/O analysis done as DMP_IOTIMEOUT on Path <path> belonging to 
Dmpnode 
<dmpnode>

DESCRIPTION:
During the asymmetric access state transition, DMP puts the buffer pointer 
in the delay queue based on the flags observed in the logs. This delay 
resulted in timeout and thereby filesystem went into disabled state.

RESOLUTION:
DMP code is modified to perform immediate retries instead of putting the 
buffer pointer in the delay queue for transition in progress case.

* 3780978 (Tracking ID: 3762580)

SYMPTOM:
Below logs were seen while setting up fencing for the cluster.

VXFEN: vxfen_reg_coord_pt: end ret = -1
vxfen_handle_local_config_done: Could not register with a majority of the
coordination points.

DESCRIPTION:
It is observed that in Linux kernels greater than or equal to RHEL6.6, RHEL7 and 
SLES11SP3, the interface used by DMP to send the SCSI commands to block devices
does not transfer the data to/from the device. Therefore the SCSI-3 PR keys does
not get registered.

RESOLUTION:
Code change is done to use scsi request_queue to send the SCSI commands to the 
underlying block device.
Additional patch is required from EMC to support processing SCSI commands via the 
request_queue mechanism on EMC PowerPath devices. Please contact EMC for 
patch details for a specific kernel version.

* 3788751 (Tracking ID: 3788644)

SYMPTOM:
When DMP (Dynamic Multi-Pathing) native support enabled for Oracle ASM 
environment, if we constantly adding and removing DMP devices, it will cause error 
like:
/etc/vx/bin/vxdmpraw enable oracle dba 775 emc0_3f84
VxVM vxdmpraw INFO V-5-2-6157
Device enabled : emc0_3f84
Error setting raw device (Invalid argument)

DESCRIPTION:
There is a limitation (8192) of maximum raw device number N (exclusive) of 
/dev/raw/rawN. This limitation is defined in boot configuration file. When binding a raw 
device to a dmpnode, it uses /dev/raw/rawN to bind the dmpnode. The rawN is 
calculated by one-way incremental process. So even if we unbind the device later on, 
the "released" rawN number will not be reused in the next binding. When the rawN 
number is increased to exceed the maximum limitation, the error will be reported.

RESOLUTION:
Code has been changed to always use the smallest available rawN number instead of 
calculating by one-way incremental process.

* 3796596 (Tracking ID: 3433503)

SYMPTOM:
The Vxconfigd(1M) daemon dumps cores with a following stack trace 
while bringing a disk online: 

kernel_vsyscall()
raise() 
abort() 
libc_message() 
int_free() 
free() 
get_geometry_info()
devintf_disk_geom_raw()
cds_get_geometry()
cds_check_for_cds_fmt()
auto_determine_format()
auto_sys_online() auto_online()
da_online()
da_thread_online_disk()
vold_thread_exec()
start_thread() 
clone()

DESCRIPTION:
When a disk label, such as DOS label magic is corrupt and is 
written to a Sun label disk, vxconfigd(1M) daemon reads DOS partition table from 
the Sun label disk causing a core dump.The core occurs due to an error in the 
boundary memory access in the DOS partition read code.

RESOLUTION:
The code is modified to avoid the incorrect memory access.

* 3796666 (Tracking ID: 3573262)

SYMPTOM:
On recent UltraSPARC-T4 architectures, Panic is observed with the 
topmost stack frame pointing to bcopy during snapshot operations involving 
space optimized snapshots.

<trap>SPARC-T4:bcopy_more()
SPARC-T4:bcopy() 
vxio:vol_cvol_bplus_delete()
vxio:vol_cvol_dshadow1_done()
vxio:voliod_iohandle()
vxio:voliod_loop()

DESCRIPTION:
The bcopy kernel library routine on Solaris was optimized to take advantage 
of recent Ultrasparc-T4 architectures. But it has some known issues for large 
size copy in some patch versions of Solaris 10. The use of bcopy was causing 
in-core corruption of cache object metadata. The corruption later lead to 
system panic.

RESOLUTION:
The code is modified to use word by word copy of the buffer 
instead of bcopy kernel library routine.

* 3832703 (Tracking ID: 3488071)

SYMPTOM:
The command "vxdmpadm settune dmp_native_support=on" fails to enable Dynamic 
Multipathing (DMP) Native Support and fails 
with the below error:
VxVM vxdmpadm ERROR V-5-1-15690 Operation failed for one or more volume groups

DESCRIPTION:
From LVM version 105, global_filters were introduced as part of lvm.conf file. 
RHEL 6.6 and RHEL 6.6 onwards has LVM version 111 and hence 
supports global_filters. The code changes were not done to handle 
global_filter 
in lvm.conf file while 6.1.1.100 was 
released.

RESOLUTION:
Code changes have been done to handle global_filter in lvm.conf file and allow 
DMP Native Support to work.

* 3832705 (Tracking ID: 3776520)

SYMPTOM:
Filters are not updated properly in lvm.conf file in VxDMP initrd while enabling 
DMP Native Support leading to root Logical Volume (LV) mounted on OS 
device upon reboot.

DESCRIPTION:
From LVM version 105, global_filter were introduced as part of lvm.conf file. 
VxDMP updates initird lvm.conf file with the filters required for DMP 
Native Support to function. While updating the lvm.conf, VxDMP checks for the 
filter field to  be updated whereas ideally we should check for 
global_filter field to be updated in the latest LVM version. This leads to 
lvm.conf file not having the proper filters leading to the issue.

RESOLUTION:
Code changes have been made to properly update global_filter field in lvm.conf 
file in VxDMP initrd.

* 3835367 (Tracking ID: 3137543)

SYMPTOM:
After reboot, OS boots in maintenance mode as it fails to load
initrd image and the kernel modules properly.

DESCRIPTION:
Due to some changes in grub from OS, kernel modules and
initrd image entries, corresponding to VxVM root, are not populated properly in
the grub configuration file. Hence, system fails to load kernel properly and
boots in maintenance mode after the reboot.

RESOLUTION:
Code is modified to work with RDE.

* 3835662 (Tracking ID: 3581646)

SYMPTOM:
Sometimes Logical Volumes may fail to migrate back to OS devices when Dynamic Multipathing (DMP) Native Support is disabled when the root is 
mounted on LVM.

DESCRIPTION:
lvmetad caches open count on devices which are present in accept section of filter in lvm.conf file. When DMP Native Support is enabled, all 
non-VxVM devices are put in reject section of filter so that only "/dev/vx/dmp" devices remain in accept section of filter in lvm.conf file.
So lvmetad caches open count on "/dev/vx/dmp" devices. When DMP Native Support is disabled "/dev/vx/dmp" devices are not put in reject 
section of filter causing a stale open count for lvmetad which is causing physical volumes to point to stale devices even when DMP Native 
Support is disabled.

RESOLUTION:
Code changes have been made to add "/dev/vx/dmp" devices in reject section of filter in lvm.conf file so lvmetad releases open count on 
these devices.

* 3836923 (Tracking ID: 3133322)

SYMPTOM:
"vxdmpadm native ls" is not showing the rootvg when dmp native support is 
enabled.
vxdmpadm native ls
DMPNODENAME                      VOLUME GROUP
=============================================
sda                              -
sdb                              -

DESCRIPTION:
When dmp native support is enabled, there was a bug in pattern match for root
device name with the root device 
name corresponding to the volume group, therefore it is skipping display of
volume group of root devices under LVM.

RESOLUTION:
Appropriate code modification are done to resolve the incorrect pattern match 
of
root device name.

Patch ID: VRTSvxvm-6.0.500.100-RHEL6

* 3496715 (Tracking ID: 3281004)

SYMPTOM:
For DMP minimum queue I/O policy with large number of CPUs, the following 
issues are observed since the VxVM 5.1 SP1 release: 
1. CPU usage is high. 
2. I/O throughput is down if there are many concurrent I/Os.

DESCRIPTION:
The earlier minimum queue I/O policy is used to consider the host controller 
I/O load to select the least loaded path. For VxVM 5.1 SP1 version, an addition 
was made to consider the I/O load of the underlying paths of the selected host 
based controllers. However, this resulted in the performance issues, as there 
were lock contentions with the I/O processing functions and the DMP statistics 
daemon.

RESOLUTION:
The code is modified such that the host controller paths I/O load is not 
considered to avoid the lock contention.

* 3501358 (Tracking ID: 3399323)

SYMPTOM:
The reconfiguration of Dynamic Multipathing (DMP) database fails with the below error: VxVM vxconfigd DEBUG  V-5-1-0 dmp_do_reconfig: DMP_RECONFIGURE_DB failed: 2

DESCRIPTION:
As part of the DMP database reconfiguration process, controller information from DMP user-land database is not removed even though it is removed from DMP kernel database. This creates inconsistency between the user-land and kernel-land DMP database. Because of this, subsequent DMP reconfiguration fails with above error.

RESOLUTION:
The code changes have been made to properly remove the controller information from the user-land DMP database.

* 3502662 (Tracking ID: 3380481)

SYMPTOM:
When the vxdiskadm(1M) command selects a removed disk during the "5 Replace a failed or removed disk" operation, the vxdiskadm(1M) command displays the following error message:
"/usr/lib/vxvm/voladm.d/lib/vxadm_syslib.sh: line 2091:return: -1: invalid option".

DESCRIPTION:
From bash version 4.0, bash doesnt accept negative error values. If VxVM scripts return negative values to bash, the error message is displayed.

RESOLUTION:
The code is modified so that VxVM scripts dont return negative values to bash.

* 3521727 (Tracking ID: 3521726)

SYMPTOM:
When using Symantec Replication Option, system panic happens while freeing
memory with the following stack trace on AIX,

pvthread+011500 STACK:
[0001BF60]abend_trap+000000 ()
[000C9F78]xmfree+000098 ()
[04FC2120]vol_tbmemfree+0000B0 ()
[04FC2214]vol_memfreesio_start+00001C ()
[04FCEC64]voliod_iohandle+000050 ()
[04FCF080]voliod_loop+0002D0 ()
[04FC629C]vol_kernel_thread_init+000024 ()
[0025783C]threadentry+00005C ()

DESCRIPTION:
In certain scenarios, when a write IO gets throttled or un-winded in VVR, we
free the memory related to one of our data structures. When we restart this IO,
the same memory gets illegally accessed and freed again even though it was
freed.It causes system panic.

RESOLUTION:
Code changes have been done to fix the illegal memory access issue.

* 3526501 (Tracking ID: 3526500)

SYMPTOM:
Disk IO failures occur with DMP IO timeout error messages when DMP (Dynamic Multi-pathing) IO statistics daemon is not running. Following are the timeout error messages:

VxVM vxdmp V-5-3-0 I/O failed on path 65/0x40 after 1 retries for disk 201/0x70
VxVM vxdmp V-5-3-0 Reached DMP Threshold IO TimeOut (100 secs) I/O with start 
3e861909fa0 and end 3e86190a388 time
VxVM vxdmp V-5-0-0 [Error] i/o error occurred (errno=0x206) on dmpnode 201/0x70

DESCRIPTION:
When IO is submitted to DMP, it sets the start time on the IO buffer. The value of the start time depends on whether the DMP IO statistics daemon is running or not. When the IO is returned as error from SCSI to DMP, instead of retrying the IO on alternate paths, DMP failed that IO with 300 seconds timeout error, but the IO has elapsed only few milliseconds in its execution. The miscalculation of DMP timeout happens only when DMP IO statistics daemon is not running.

RESOLUTION:
The code is modified to calculate appropriate DMP IO timeout value when DMP IO statistics demon is not running.

* 3531332 (Tracking ID: 3077582)

SYMPTOM:
A Veritas Volume Manager (VxVM) volume may become inaccessible causing the read/write operations to fail with the following error:
# dd if=/dev/vx/dsk/<dg>/<volume> of=/dev/null count=10
dd read error: No such device
0+0 records in
0+0 records out

DESCRIPTION:
If I/Os to the disks timeout due to some hardware failures like weak Storage Area Network (SAN) cable link or Host Bus Adapter (HBA) failure, VxVM assumes that the disk is faulty or slow and it sets the failio flag on the disk. Due to this flag, all the subsequent I/Os fail with the No such device error.

RESOLUTION:
The code is modified such that vxdisk now provides a way to clear the failio flag. To check whether the failio flag is set on the disks, use the vxkprint(1M) utility (under /etc/vx/diag.d). To reset the failio flag, execute the vxdisk set <disk_name> failio=off command, or deport and import the disk group that holds these disks.

* 3552411 (Tracking ID: 3482026)

SYMPTOM:
The vxattachd(1M) daemon reattaches plexes of manually detached site.

DESCRIPTION:
The vxattachd daemon reattaches plexes for a manually detached site that is the site with state as OFFLINE. As there was no check to differentiate between a manually detach site and the site that was detached due to IO failure. Hence, the vxattachd(1M) daemon brings the plexes online for manually detached site also.

RESOLUTION:
The code is modified to differentiate between manually detached site and the site detached due to IO failure.

* 3600161 (Tracking ID: 3599977)

SYMPTOM:
system panics with following stack trace:
[00009514].simple_lock+000014
[004E4C48]soereceive+001AA8
[00504770]soreceive+000010
[00014F50].kernel_add_gate_cstack+000030
[04F6992C]kmsg_sys_rcv+000070
[04F89448]nmcom_get_next_mblk+0000A4
[04F63618]nmcom_get_next_msg+0000F0
[04F64434]nmcom_wait_msg_tcp+00015C
[04F7A71C]nmcom_server_proc_tcp+00006C
[04F7C7AC]nmcom_server_proc_enter+0000A4
[04D88AF4]vxvm_start_thread_enter+00005C

DESCRIPTION:
During replica connection, after created the port before increasing the count 
to protect the port from deleting, another thread deletes the port. While the 
replica connection thread proceeds, it refers to the port that is already 
deleted, hence causes NULL pointer reference and panic.

RESOLUTION:
Code changes have been done to add locking to the checking and modification of 
the count associated with the port.

* 3612801 (Tracking ID: 3596330)

SYMPTOM:
'vxsnap refresh' operation fails with following indicants:

Errors occur from DR (Disaster Recovery) Site of VVR (Veritas 
Volume Replicator):

o	vxio: [ID 160489 kern.notice] NOTICE: VxVM vxio V-5-3-1576 commit: 
Timedout waiting for rvg [RVG] to quiesce, iocount [PENDING_COUNT] msg 0
o	vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-8011 Internal 
transaction failed: Transaction aborted waiting for io drain

At the same time, following errors occur from Primary Site of VVR:

vxio: [ID 218356 kern.warning] WARNING: VxVM VVR vxio V-5-0-267 Rlink 
[RLINK] disconnecting due to ack timeout on update message

DESCRIPTION:
VM (Volume Manager) Transactions on DR site get aborted as pending IOs could 
not be drained in stipulated time leading to failure of FMR (Fast-Mirror 
Resync) 'snap' operations. These IOs could not be drained because of IO 
throttling. A bug/race in conjunction with timing in VVR causes a miss in 
clearing this throttling condition/state.

RESOLUTION:
Code changes have been done to fix the race condition which ensures clearance 
of throttling state at appropriate time.

* 3621240 (Tracking ID: 3621232)

SYMPTOM:
When run IBC(In-band Control) procedure by vradmin ibc command, vradmind (VVR 
daemon) on VVR secondary may goes into disconnected state, then the following 
IBC procedure or vradmin ibc commands cannot be started/executed on VVR 
secondary and below message will be outputted on VVR primary.
    
VxVM VVR vradmin ERROR V-5-52-532 Secondary is undergoing a state transition. 
Please re-try the command after some time.
VxVM VVR vradmin ERROR V-5-52-802 Cannot start command execution on Secondary.

DESCRIPTION:
When IBC procedure run into command finish state, the vradmind on VVR secondary 
may goes into disconnected state, but it was not sensed by vradmind on primary. 
In such situation, the vradmind on primary will not send handshake request to 
the secondary which can take it out of disconnected state to running state. 
Then the vradmind will stall in disconnected state and the following vradmin 
ibc command cannot be executed on VVR secondary, although vradmind looks 
normal and in running state on VVR primary.

RESOLUTION:
Code changes have been done to make sure the vradmind on VVR primary will be 
notified while it goes into disconnected state on VVR secondary, hence it can 
send out handshake request to take the secondary out of disconnected state.

* 3622069 (Tracking ID: 3513392)

SYMPTOM:
secondary panics when rebooted while heavy IOs are going on primary

PID: 18862  TASK: ffff8810275ff500  CPU: 0   COMMAND: "vxiod"
#0 [ffff880ff3de3960] machine_kexec at ffffffff81035b7b
#1 [ffff880ff3de39c0] crash_kexec at ffffffff810c0db2
#2 [ffff880ff3de3a90] oops_end at ffffffff815111d0
#3 [ffff880ff3de3ac0] no_context at ffffffff81046bfb
#4 [ffff880ff3de3b10] __bad_area_nosemaphore at ffffffff81046e85
#5 [ffff880ff3de3b60] bad_area_nosemaphore at ffffffff81046f53
#6 [ffff880ff3de3b70] __do_page_fault at ffffffff810476b1
#7 [ffff880ff3de3c90] do_page_fault at ffffffff8151311e
#8 [ffff880ff3de3cc0] page_fault at ffffffff815104d5
#9 [ffff880ff3de3d78] volrp_sendsio_start at ffffffffa0af07e3 [vxio]
#10 [ffff880ff3de3e08] voliod_iohandle at ffffffffa09991be [vxio]
#11 [ffff880ff3de3e38] voliod_loop at ffffffffa0999419 [vxio]
#12 [ffff880ff3de3f48] kernel_thread at ffffffff8100c0ca

DESCRIPTION:
If the replication stage IOs are started after serialization of the replica volume, 
replication port  could be deleted  and set to NULL during handling the replica 
connection changes,  this will cause the panic since we have not checked if the 
replication port is still valid before referencing to it.

RESOLUTION:
Code changes have been done to abort the stage IO if replication port is NULL.

* 3632969 (Tracking ID: 3631230)

SYMPTOM:
VRTSvxvm patch version 6.0.5 and 6.1.1 will not work with RHEL6.6 update.

# rpm -ivh VRTSvxvm-6.1.1.000-GA_RHEL6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:VRTSvxvm               ########################################### [100%]
Installing file /etc/init.d/vxvm-boot
creating VxVM device nodes under /dev
WARNING: No modules found for 2.6.32-494.el6.x86_64, using compatible modules 
for 
2.6.32-71.el6.x86_64. 
FATAL: Error inserting vxio (/lib/modules/2.6.32-
494.el6.x86_64/veritas/vxvm/vxio.ko): Unknown symbol in module, or unknown 
parameter 
(see dmesg) ERROR: modprobe error for vxio. See documentation.
warning: %post(VRTSvxvm-6.1.1.000-GA_RHEL6.x86_64) scriptlet failed, exit 
status 1

#

Or after OS update, the system log file will have the following messages logged.

vxio: disagrees about version of symbol poll_freewait
vxio: Unknown symbol poll_freewait
vxio: disagrees about version of symbol poll_initwait
vxio: Unknown symbol poll_initwait

DESCRIPTION:
Installation of VRTSvxvm patch version 6.0.5 and 6.1.1 fails on RHEL6.6 due to 
the changes in poll_initwait() and poll_freewait() interfaces.

RESOLUTION:
The VxVM package has re-compiled with RHEL6.6 build environment.

* 3638039 (Tracking ID: 3625890)

SYMPTOM:
Error message is reported as bellow after running vxdisk resize command:
"VxVM vxdisk ERROR V-5-1-8643 Device ibm_shark0_9: resize failed: Invalid 
attribute specification"

DESCRIPTION:
For CDS(cross platform data sharing) VTOC(volume table of contents) disks, 
there are 2 reserved cylinders for special usage. In case of expanding a disk 
with particular disk size on storage side, VxVM(veritas volume manager) may calculate the cylinder number as 2, which will cause the vxdisk 
resize failed with "Invalid attribute specification".

RESOLUTION:
Code changes have been made to avoid failure of resizing a CDS VTOC disk.

Patch ID: VRTSaslapm-6.0.500.300-GA_RHEL6

* 3665733 (Tracking ID: 3665727)

SYMPTOM:
vxdmpadm listapm output does not list any APM except default ones

[root@rpms]# vxdmpadm listapm
Filename           APM Name           APM Version  Array Types       State
================================================================================
dmpjbod.ko         dmpjbod                1        Disk              Active
dmpjbod.ko         dmpjbod                1        APdisk            Active
dmpalua.ko         dmpalua                1        ALUA              Not-Active
dmpaaa.ko          dmpaaa                 1        A/A-A             Not-Active
dmpapg.ko          dmpapg                 1        A/PG              Not-Active
dmpapg.ko          dmpapg                 1        A/PG-C            Not-Active
dmpaa.ko           dmpaa                  1        A/A               Active
dmpap.ko           dmpap                  1        A/P               Active
dmpap.ko           dmpap                  1        A/P-C             Active
dmpapf.ko          dmpapf                 1        A/PF-VERITAS      Not-Active
dmpapf.ko          dmpapf                 1        A/PF-T3PLUS       Not-Active
[root@rpms]#

DESCRIPTION:
For supporting RHEL6.6 update, dmp module is recomipled with latest RHEL6.6 
kernel version. During post install of the package
the APM modules fails to load due to mismatch in DMP and additional APM module 
kernel version.

RESOLUTION:
ASLAPM package is recompiled with RHEL6.6 kernel.



INSTALLING THE PATCH
--------------------
Run the Installer script to automatically install the patch:
-----------------------------------------------------------
To install the patch perform the following steps on at least one node in the cluster:
1. Copy the patch sfha-rhel6_x86_64-Patch-6.0.5.300.tar.gz to /tmp
2. Untar sfha-rhel6_x86_64-Patch-6.0.5.300.tar.gz to /tmp/hf
    # mkdir /tmp/hf
    # cd /tmp/hf
    # gunzip /tmp/sfha-rhel6_x86_64-Patch-6.0.5.300.tar.gz
    # tar xf /tmp/sfha-rhel6_x86_64-Patch-6.0.5.300.tar
3. Install the hotfix
    # pwd 
    /tmp/hf
    # ./installSFHA605P3 [<host1> <host2>...]

You can also install this patch together with 6.0.1 GA release and 6.0.5 Patch release
    # ./installSFHA605P3 -base_path [<601 path>] -mr_path [<605 path>] [<host1> <host2>...]
where the -mr_path should point to the 6.0.5 image directory, while -base_path to the 6.0.1 image.

Install the patch manually:
--------------------------
o Before-the-upgrade :-
  (a) Stop I/Os to all the VxVM volumes.
  (b) Umount any filesystems with VxVM volumes.
  (c) Stop applications using any VxVM volumes.
o Select the appropriate RPMs for your system, and upgrade to the new patch.
   # rpm -Uhv VRTSvxvm-6.0.500.200-RHEL6.x86_64.rpm


REMOVING THE PATCH
------------------
# rpm -e  <rpm-name>


KNOWN ISSUES
------------
* Tracking ID: 3688085

SYMPTOM: The 'delayed allocation' (ie 'dalloc') feature on VxFS 6.0.5.100
p-patch can cause data loss or stale data.

 Dalloc feature is enabled by default for local mounted file system
and is not supported for cluster mounted file systems. Dalloc with sequential
extending buffer writes can possibly cause data loss or stale data. This issue
is seen only with 6.0.5.100 p patch.

WORKAROUND: disable the 'delayed allocation' ('dalloc') feature on the VxFS filesystems.

Following commands are used to disable dalloc.

1)For a filesystem which is already mounted
# vxtunefs -s -o dalloc_enable=0 $MOUNT_POINT

2) To make the value persistent across system reboot, add an entry to
/etc/vx/tunefstab
/dev/vx/dsk/$DISKGROUP/$VOLUME      dalloc_enable=0



SPECIAL INSTRUCTIONS
--------------------
NONE


OTHERS
------
NONE