* * * READ ME * * * * * * Veritas File System 5.0 MP2 * * * * * * Rolling Patch 10 * * * Patch Date: 2014-06-13 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Veritas File System 5.0 MP2 Rolling Patch 10 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- HP-UX 11i v2 (11.23) PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxfs VRTSvxfs BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Veritas File System 5.0 MP2 * Veritas Storage Foundation 5.0 MP2 * Veritas Storage Foundation Cluster File System 5.0 MP2 * Veritas Storage Foundation for Oracle 5.0 MP2 * Veritas Storage Foundation for Oracle RAC 5.0 MP2 * Veritas Storage Foundation HA 5.0 MP2 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: PHKL_44051, PHCO_44050 * 2810111 (2597347) fsck segmentation fault bc_rgetblk () * 3023997 (2768505) While reading the Dynamic Name Lookup Cache (DNLC) entries using the pstat_getmpathname(2) function, the system might panic. * 3131944 (3099638) The vxfs_ifree_timelag(5) tunable when tuned, displays incorrect minimum value. * 3338084 (2175113) Internal noise test on Cluster File System hits the debug assert, when the snapshot related message is sent. * 3428222 (1244756) A lookup operation on a VxFS file system may fail with a stack trace. * 3502809 (2874172) Network File System (NFS) file creation thread might loop continuously with the stack trace. DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following Symantec incidents: Patch ID: PHKL_44051, PHCO_44050 * 2810111 (Tracking ID: 2597347) SYMPTOM: The fsck(1M) command dumps core with the following stack trace: _lwp_kill abort iget_i iget ag_ivalidate ag_validate_repino ag_olt_init ag_init process_device main _start DESCRIPTION: The fsck(1M) command dumps core when one of the device record gets corrupted, while the replica-device record is valid. The fsck(1M) command is unable to reconstruct the corrupted record with the valid record. RESOLUTION: The code is modified to reconstruct the corrupted-device record using the valid replica-device record. * 3023997 (Tracking ID: 2768505) SYMPTOM: While reading the Dynamic Name Lookup Cache (DNLC) entries using the pstat_getmpathname(2) function, the system might panic and display the following stack trace: vx_dnlc_appened_dnlcnodes vx_dnlc_getentries pstat_mpathname $cold_pstat syscall DESCRIPTION: While reading all the DNLC entries, the VxFS pointer is derived from the VNODE pointer. The VNODE pointer can be reused and de-referencing the VxFS pointer might panic the system. RESOLUTION: The code is modified to pass the VxFS pointer directly to the function instead of deriving it from the VNODE pointer. * 3131944 (Tracking ID: 3099638) SYMPTOM: When the vxfs_ifree_timelag(5) tunable is tuned the following error message is displayed: # kctune vxfs_ifree_timelag=400 ERROR: mesg 095: V-2-95: Setting vxfs_ifree_timelag to 450 since the specified value for vxfs_ifree_timelag is less than the recommended minimum value of 1035 DESCRIPTION: In the vxfs_ifree_timelag(5) tunable man page, the minimum value is set to "None". The error message is displayed when the vxfs_ifree_timelag(5) tunable is set to a value which is less than 450. In the error message, a garbage value is displayed as the recommended minimum value. The error occurs because a single argument is passed for the error message that has two format specifier's. RESOLUTION: The code is modified to set the correct minimum value of the vxfs_ifree_timelag (5) tunable, and display the correct error message. * 3338084 (Tracking ID: 2175113) SYMPTOM: Internal noise test on Cluster File System (CFS) hits the debug assert, when the snapshot related message is sent. DESCRIPTION: The priority level of the message related to the snapshot conflicts with the recovery message.Since both the messages are processed simultaneously and with same priority level, the internal debug assert is triggered. RESOLUTION: The code is modified to change the priority level of the message related to the snapshot. * 3428222 (Tracking ID: 1244756) SYMPTOM: A lookup operation on a VxFS file system may fail with the following stack trace: vx_cbdnlc_purge_iplist vx_inode_free_list vx_ifree_scan_list vx_workitem_process vx_worklist_process DESCRIPTION: Due to a race condition between the Directory Name Lookup Cache (DNLC) lookup and the DNLC get functions, there is an attempt to move a DNLC entry to the tail of the freelist in the lookup function when it has already been removed from the freelist by the DNLC get function. This leads to a null pointer de- reference. RESOLUTION: The code is modified to verify that the DNLC entry is present on the freelist before it is moved to the tail by the DNLC get function. * 3502809 (Tracking ID: 2874172) SYMPTOM: Network File System (NFS) file creation thread might loop continuously with the following stack trace: vx_getblk_cmn(inlined) vx_getblk+0x3a0 vx_exh_allocblk+0x3c0 vx_exh_hashinit+0xa50 vx_dexh_create+0x240 vx_dexh_init+0x8b0 vx_do_create+0x1e0 vx_create1+0x1d0 vx_create0+0x270 vx_create+0x40 rfs3_create+0x420 common_dispatch+0xb40 rfs_dispatch+0x40 svc_getreq+0x250 svc_run+0x310 svc_do_run+0xd0 nfssys+0x6a0 hpnfs_nfssys+0x60 coerce_scall_args+0x130 syscall+0x590 DESCRIPTION: The Veritas File System (VxFS) file creation vnode operation (VOP) routine expects the parent vnode to be a directory vnode pointer. But, the NFS layer passes a stale file vnode pointer by default. This might cause unexpected results such as hang during VOP handling. RESOLUTION: The code is modified to check for the vnode type of the parent vnode pointer at the beginning of the create VOP call and return an error if it is not a directory vnode pointer. INSTALLING THE PATCH -------------------- To install the VxFS 5.0-MP2RP10 patch: a) To install this patch on a CVM cluster, install it one system at a time so that all the nodes are not brought down simultaneously. b) The VxFS 5.0(GA) must be installed before applying these patches. c) To verify the VERITAS file system level, execute: # swlist -l product | egrep -i 'VRTSvxfs' VRTSvxfs 5.0.01.04 VERITAS File System Note: VRTSfsman is a corequisite for VRTSvxfs. So, VRTSfsman also needs to be installed with VRTSvxfs. # swlist -l product | egrep -i 'VRTS' VRTSvxfs 5.0.01.04 Veritas File System VRTSfsman 5.0.01.02 Veritas File System Manuals d) All prerequisite/corequisite patches must be installed. The Kernel patch requires a system reboot for both inst allation and removal. e) To install the patch, execute the following command: # swinstall -x autoreboot=true -s PHCO_44050 PHKL_44051 If the patch is not registered, you can register it using the following command: # swreg -l depot The is the absolute path where the patch resides. REMOVING THE PATCH ------------------ To remove the VxFS 5.0-MP2RP10 patch: a) Execute the following command: # swremove -x autoreboot=true PHCO_44050 PHKL_44051 SPECIAL INSTRUCTIONS -------------------- Before installing 5.0MP2RP10 patch , it requires PHKL_33535 to be installed on the machine. OTHERS ------ PHKL_42729: SUMMARY OF FIXED ISSUES: ----------------------------------------- 2244365 turn DRB lock into a no-op on non-AIX platforms 2405530 Unable to tune vxfs_bc_bufhwm on systems > 227 GB memory 2494597 Memory leak in internal buffercache after 497 days (lbolt wrap-over) 2556095 ff_vxfs ERROR: V-3-24347: program limit of 30701385 exceeded 2558844 FTP is degraded due to buffer cache issue. 2607352 One second delays introduced with tranflush mountoption 2647802 Full Fsck enhancement - Memory Reduction 2654474 Performance degradation of 'll' and high SYS% CPU in vx_ireuse() 2669100 Spinlock contention on vx_memlist_lk on 128-way Superdome 2684626 panic: deadlock: terminated process associated with filelock -- vx_lockctl not allowing release of locks on filesystem disable * INCIDENT NO:2244365 TRACKING ID:1143552 SYMPTOM: During re-tune operations on vx_ninode, memory hangs and the system stops responding. DESCRIPTION: The dynamic reconfiguration added several locks to serialize re-tuning operations on a different platform. This was not suitable on the HP-UX platform. Among the several locks added, one of the dynamic reconfiguration locks was acquired in ways that caused performance degradation on large systems. RESOLUTION:A re-tune lock has now been added to correct the memory hang. * INCIDENT NO:2405530 TRACKING ID:2274267 SYMPTOM: Unable to tune vxfs_bc_bufhwm on systems > 227 GB.When they try to tune vxfs_bc_bufhwm, following error is displayed: root@zu8003hp:/#kctune vxfs_bc_bufhwm=65536 ERROR: mesg 113: V-2-113: The specified value for vx_bc_bufhwm is greater than 90% of the total kernel memory 267444224 Kbytes. DESCRIPTION: Error is caused by integer overflow in calculation of upper bound for the tunable value. RESOLUTION:Fixed integer overflow in the calculation of upper bound of the tunable. * INCIDENT NO:2494597 TRACKING ID:2429566 SYMPTOM: Memory used for VxFS internal buffer cache may significantly grow after 497 days uptime when LBOLT(global which gives current system time) wraps over. DESCRIPTION: We calculate age of buffers based on LBOLT value. Like age = (current LBOLT - LBOLT when buffer added to list). Buffer is reused when age becomes greater than threshold. When LBOLT wraps, current LBOLT becomes very small value and age becomes negative. VxFS thinks that this is not old buffer and never reuses it. Buffer cache memory usage increases as buffers are not reused. RESOLUTION:Now we check if the the LBOLT has wrapped around. If it is, we reassign the buffer time with current LBOLT so that it gets reused after some time. * INCIDENT NO:2556095 TRACKING ID:2515380 SYMPTOM: The ff command hangs and later it exits after program exceeds memory limit with following error. # ff -F vxfs /dev/vx/dsk/bernddg/testvol UX:vxfs ff: ERROR: V-3-24347: program limit of 30701385 exceeded for directory data block list UX:vxfs ff: ERROR: V-3-20177: /dev/vx/dsk/bernddg/testvol DESCRIPTION: 'ff' command lists all files on device of vxfs file system. In 'ff' command we do directory lookup. In a function we save the block addresses for a directory. For that we traverse all the directory blocks. Then we have function which keeps track of buffer in which we read directory blocks and the extent up to which we have read directory blocks. This function is called with offset and it return the offset up to which we have read the directory blocks. The offset passed to this function has to be the offset within the extent. But, we were wrongly passing logical offset which can be greater than extent size. As a effect the offset returned gets wrapped to 0. The caller thinks that we have not read anything and hence the loop. RESOLUTION:Remove call to function which maintains buffer offsets for reading data. That call was incorrect and redundant. We actually call that function correctly from one of the functions above. * INCIDENT NO:2558844 TRACKING ID:2532934 SYMPTOM: Performance of ftp transfer which use 'VOP_BREAD()' vnode operation for reads can be slow when max_buf_data_size is set to 64K. DESCRIPTION: Read done using 'VOP_BREAD()' was not doing read ahead even for sequential I/Os when max_buf_data_size is set to 64K. Therefore performance degradation was seen. read ahead was not performed because of wrong read length passed to read ahead detection function. It detects this by storing the last read offset plus the amount of read requested into the inode. It then compares the next read's start offset with this value. If it is sequential they should match. In this case we were not supplying the correct read length field to the read ahead detection function. We should have provided max_buf_data_size as the length but instead we provided VX_MAXBSIZE(8192). Therefore detection function did not detect a valid read ahead when using max_buf_data_size of 65536 and hence no read ahead was performed. This in turn can cause cache miss which can affect performance. RESOLUTION:correct read length is now passed to read ahead detection function so that read ahead on sequential I/O can get triggered correctly. * INCIDENT NO:2607352 TRACKING ID:2334061 SYMPTOM: When file system is mounted with tranflush option, operations requiring metadata update take comparatively more time. DESCRIPTION: When VxFS file system is mounted with tranflush option, we flush transaction metadata on the disk and wait for 100 milliseconds before flushing next transaction. This delay is affecting severely operation of various commands on the VxFS file system. RESOLUTION:Since the flushing is synchronous and is performed in loop, 100 milliseconds delay is too much. To solve the problem, the delay is reduced to a more reasonable 2 milliseconds value from 100 milliseconds. * INCIDENT NO:2647802 TRACKING ID:1067468 SYMPTOM: fsck(1M) command for full file system check enhancement for Memory Reduction. DESCRIPTION: 1. Currently in fsck code path a flat bitmap is allocated per device. This can be optimized by allocating compressed bit map instead of a flat bitmap. 2. Currently in fsck code path 3 arrays are used to track inode's dotdot linkage. This can be optimized by eliminating 2 of these 3 redundant arrays. 3. Currently in fsck code path memory is allocated for inode tables based on the value of fsh_ninode. In MTS the ilist file can become sparse , increasing the value of fsh_ninode. This memory allocation can be optimized by allocating the memory based on the actual number of the inodes present in the ilist file. RESOLUTION:1. Changed the code to allocate a compressed bit map instead a flat bitmap. 2. Changed the code to eliminate the 2 of the 3 redundant arrays used to track inodes dotdot linkages. 3. Changed the code to allocate the memory to inode tables based on the actual number of inodes present in the ilist file. * INCIDENT NO:2654474 TRACKING ID:2651922 SYMPTOM: "ls -l" command on Local VxFS file system is running slow and high CPU usage is seen on HP platform. DESCRIPTION: This issue occurs when system is under inode pressure, and the most of inodes on inode free list are CFS inodes. On HP-UX platform, currently, CFS inodes are not allowed to be reused as local inodes to avoid GLM deadlock issue when vxFS reconfig is in process. So if system needs a VxFS local inode, it has to take a amount of time to loop through all the inode free lists to find a local inode, if the free lists are almost filled up with CFS inodes. RESOLUTION:1. added a global "vxi_icache_cfsinodes" to count cfs inodes in inode cache. 2. relaxed the condition for converting cluster inode to local inode when the number of in-core cfs inodes is greater than the threshold"vx_clreuse_threshold" and reconfig is not in progress. * INCIDENT NO:2669100 TRACKING ID:848619 SYMPTOM: High CPU utilization occurs when several processes are performing direct I/Os on a VxFS file. DESCRIPTION: The high CPU utilization occurs because of high contention for a global spinlock that protects the linked list containing the pages that are participating in the direct I/Os. RESOLUTION:The hash list function used by VxFS code has been modified to ensure that the lock contention is reduced. * INCIDENT NO:2684626 TRACKING ID:1180759 SYMPTOM: System panics in case of the deadlock because of improper lock handling. DESCRIPTION: This issue could be due to vx_lockctl not allowing the release of locks if the filesystem has been disabled.Due to this the process would have exited without releasing the lock and another process that is trying to get a lock would hit a deadlock causing the panic. RESOLUTION:Unlock all the file locks of the vnode in case of disabled file system to satisfy an assertion in the HP-UX. The incidents fixed as a part of MP2RP5 Patch: =================================================== 2275532 write() system call hangs for over 10 seconds on VxFS 3.5 on 11.23 2277468 CFS panic in vx_irtyhold_locked() due to corrupt freelist 2371928 HP CFS conformance misc> > crashconfig test fails in 5.0_A44build using 11.31 sfqapa17-18-19-21 cluster. 2405532 Access denied on files inheriting default group ACL from parent directory. 2405549 Hit ted assertion "f:fdd_getdev:1" when did verification for e1209803 2423690 many threads were stuck on vx_ireuse+0x60c. MP2RP4 ====== 2026602 Adding quota support for user 'nobody' on HP-UX 2080396 vxfsstat does not reflect the change of vx_ninode 2167257 O_SYNC isn't sync enough. also, missing timestamp updates. 2194641 System panic in vx_iupdat_clustblks() 2196887 VxFS5.0 quota information gets messed up on large files 2196895 fsck fails to repair corrupt directory blocks having duplicate directory entries. 2198646 fdd_enospc can come accross a buffer with B_DONE set. 2220464 Metadata was not updated correctly after write() with O_SYNC flag. 2244375 1009: HART Vpars:SYM5.1 6/8:non-telnetable hang after 14+ CHO 2244381 odmstat with vol_vxtrace 0 shows high WTIME values 2244385 CFS noise.replay hung due to busy loop in vx_statvfs 2252145 Panic at getblk() when growing a full filesystem with fsadm 2272980 Threads stuck in vx_rwsleep_rec_lock_em 2288807 LM stress aborted due to "run_fsck : Failed to full fsck cleanly". 2289614 Panic in vx_sum_upd_efree1() 2289630 Panic in vx_inull_list after forced unmount of writable clone and unmount of prinmary fileset. MP2RP3 ======= 1891139 secondaries ias_ilist not updated fully. 1894113 fsck coredump in cfcomp() 1945169 VxFS4.1: VxFS doesnt specify limits on tuning on vxfs_bc_bufhwm 1946129 fsadm keeps relocating and copying already relocated and copied (reorg-ed) regions of a file in subsequent passes. 1959376 CFS: rm performance could be affected due to VX_DIRLOCK() contention 2026515 Data key miss fault during hazard testing. 2026557 System hang with many processes sleeping in vx_recsmp_range_lock 2026575 VxFS 5.0GA & MP1 - fsadm -b - ioctl(VX_IOCTL | 128) returns EBUSY 2026624 sar(1M) -v does not report VxFS inode table overflows 2080387 RFE fsadm to move any structures in shrinking a VxFS 2082342 QXCR1001054154 VxFS 11.23/5.0:Panic at vx_inactive 2090241 getacl shows no permission for user but user can still access file 2114136 QXCR1000550182 issue, not ported to 4.1/11.23 2117230 Vxfs log replay fails at double fault for 2TB lun. 2117234 vxfsckd may loop uninterruptible on system using default values for max_thread_proc 3000 and nkthread 8416 2125456 vx_clone_setup can call vx_ierror when clone inodes are free 2159494 HPUX11.23 5.0MP2RP5 setup running test LM stress.enterprise hit an assert f:vx_getimap:1a 2159499 5.0v2MP2RP5 setup running LM noise.replay hit an assert vx_msgprint:ndebug via vx_fset_markbad 2161555 HXRT50MP2RP5: fsmap return incorrect volume name when nest mount. MP2RP1 => b_5_0_hp_mp1_rp_maint ========= 1093209 : filesystem hangs during vxupgrade -n 6 1910932 : HPUX11.23 50MP2RP1 LM conformance -DMAPI test hit an assert :f:vx_detach_fset:10e 1500767 : HP-UX 11.11; JFS 3.3; poor performance writing to mmap'ed sparse file. 1503701 : DPF in vx_dnlc_pathname_realloc 1521633 : Data page fault panic at vx_readnomap- > uiomove 1541759 : LxRT 5.0MP3 CFS fsqa fcl/miscfcl_write.2 test hits assert f:vx_getmap:4 1542126 : vxresize fails to resize the volume but does resize the filesystem. 1542500 : GLM hang on VxFS/CFS 5.0/11.23 1542503 : VxFS does not release vfs in the presence of nested mounts. 1542510 : f:vx_extentalloc:1d during policy enforcement on CFS secondary 1593907 : CFS filesystem needs fsck but is mounted on one node 1593911 : CFS looping in vx_dele_get_freespace() due to smap marked bad 1601442 : Panic in vx_unlockmap due to null ml_tranp pointer 1664321 : bdf commands hung in VX_CFS_GLOCK_GRANT_WAIT when CVM master switched over. 1665419 : CFS 5.x primary umount hang when full fsck is set 1665426 : invalid ACLs can be set on HP 1665429 : vxupgrade 5-> 6 still fails with ENFILE 1918538 : LM noise replay(UxN2) on build5.0MP3debug-2007-11-7b on SLES9 sp3_rc4is not progressing 1879460 : HPUX11.31 LM-conformance-> fcl -> fcl_inode is hit an assert f:vx_msgprint:ndebug on machine with Build :: 501A39 setup. 1709583 : AIX5.3 VxFS 5.0MP1 - CIO returned ENOTSUP (124) and caused DB2 to crash 1782643 : HP-UX 11.31 cfs Test cfs-stress-enterprise hit the same assert :f:vx_cwfrz_wait:2 is with 501A38 build s etup. 1782708 : filesystem performance degradation 1787880 : VX_SETEXT with VX_GROWFILE could fail to grow the file or expose uninitialized data. 1798045 : In some cases we miss calling vx_metaioerr() when fm_badwrite is set. 1820190 : vxfs spends too much CPU looking for odd-sized extents (vxi_alloc_fail) 1820258 : State Map corruption reported, followed by a CFS hang. 1828190 : System panic in inctext: VTEXT not set and tcount > 0 1839263 : HPUX11.23 SFCFS 5.0MP1RP1HF6 - Request fix Etrack 1122868 - vxupgrade to 7 fields uninitialised 1840015 : file system disabled. 1840289 : backport large directory feature enhancements 1842253 : vx_vn_brelse() incorrectly calls vn_rele() 1865167 : Data page fault at vx_qiostats_update() due to fiostats structure already free'd 1865181 : 'mv' hung on CFS 1870495 : CFS hang while expanding AUs 1877255 : HPUX11.31 test LM cmds-> fsvoladm-> devreorg-> fsvoladm.devreorg.26 is failing with build setup 501A3 2 1884806 : HPUX11.31 LM-conformance-> fcl -> fcl_inode is hit an assert f:vx_msgprint:ndebug on machine with Build :: 501A39 setup. 1404894 : quotacheck coredumps with more than 30 quota- enabled filesystems in /etc/fstab 1596423 : ncheck command dumps core 1705439 : subtype command not returning correct value when run as non root user 1781092 : VX_SETEXT ioctl doesn't require full license on 11.31 1783166 : mkfs fails on EVFS volume 1839263 : HPUX11.23 SFCFS 5.0MP1RP1HF6 - Request fix Etrack 1122868 - vxupgrade to 7 fields uninitialised 1910440 : AIX-6.0A4-CFS-Cmds-> fsck/basic test failed with "mountit.fatalerror". 1840289 : backport large directory feature enhancements MP2 => b_5_0_hp_mp1_rp_maint ======= 1029522 : Need to change VX_GET_NINODE value back to 57 in 4.x and later 1040634 : vxupgrade from Layout5 to Layout6 getting inode table overflow 1274315 : AIX-5.1 cfs-stress+mds hit assert 'f:vx_clone_setup:1a, 1' 1274326 : VxFS 5.x got into live lock in vx_iupdat_msg() due to not inode owner error 1425413 : security exploit in vxfs commands due to lack of bounds checking 1445129 : bug in vx_ifree_scan_list() in vx_iflush.c 1449658 : [p1][hp-oem/Samsung] deadlock in VxFS-HP CFS code and need a patch urgently 1465699 : VxFS4.1/v2 MP2 patch PHKL_36114 introduced NFS performance problems. 1470927 : Need to release CPU in vx_multi_bufinval () for local mount large extent 1465693 : Performance degradation in vx_logbuf_clean seen on AIX even with e1097682 fix 1479245 : 5.0MP2/11.23: LM noise_fullfsck.ent hit an assert "f:vx_bmap:4" while running on build 5.0MP2 1507758 : Titan 311-850-669. Infinite loop in bmap/transaction processing during clone removal truncate/push opera tion. 1508116 : system reboot after pointed "restart database" of vea on windows client. Reporter: Tina 1510551 : AIX 5.0MP3A9 CFS-Stress(Reconfig) test hit assert " f:vx_cbdnlc_lookup:3" via "vx_dirlook" 1510791 : vx_lookup returning ENOENT due to stale negative DNLC entry 1512896 : stress test goes in "FS busy...sleeping 5" loop 1518229 : SDET performance regression on 128 cpu machines 1274317 : fsck log replay extremely slow 1417404 : JAGag10814: prevent accidental vxupgrade of / and /stand 1425405 : odmmkfile/qiomkfile should fail for non-root users 1425413 : security exploit in vxfs commands due to lack of bounds checking 1465699 : VxFS4.1/v2 MP2 patch PHKL_36114 introduced NFS performance problems.