* * * READ ME * * * * * * Veritas File System 5.0 MP1 * * * * * * P-patch 13 * * * Patch Date: 2014-10-24 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Veritas File System 5.0 MP1 P-patch 13 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- HP-UX 11i v3 (11.31) PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxfs BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Veritas File System 5.0 * Veritas Storage Foundation 5.0 * Veritas Storage Foundation Cluster File System 5.0 * Veritas Storage Foundation for Oracle 5.0 * Veritas Storage Foundation for Oracle RAC 5.0 * Veritas Storage Foundation HA 5.0 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: PHCO_44198, PHKL_44199 * 2829721 (2829708) On a locally mounted Veritas File System (VxFS) machine, the system hangs due to low memory during an internal test. * 3131960 (2899907) On CFS, some file-system operations like vxcompress utility and de-duplication fail to respond. * 3464622 (2134670) The extendfs(1M) utility fails when the file system is marked for the full-file- system check (fsck). * 3616610 (3259634) A Cluster File System (CFS) with blocks larger than 4GB may become corrupt. * 3616940 (2411970) The system panics at the vx_inode_trandone() function. Patch ID: PHKL_43684, PHCO_43685, PHCO_43686 * 2705245 (2555198) On VxFS sendfile() does not create DMAPI events for HSM. * 2894568 (2894560) While running internal stress test on Cluster File System, the fsadm(1M) command dumps core. * 3131811 (2966277) Systems with high file system activity like read/write/open/lookup may panic the system. * 3131881 (3010444) On a NFS filesystem cksum(1m) fails with the "cksum: read error on : Bad address" error. * 3235161 (3235160) Data Key Miss Fault is observed in the vx_getpage() function. * 3277010 (3226462) On a cluster mounted file-system with unequal CPUs, a node may panic while doing a lookup operation. * 3277016 (3244613) fsadm(1M) command hangs while the I/O load on regular vxfs filesystem and checkpoint. * 3277028 (1960815) VxFS read ahead can cause stalled IO on all the write operations during a re- tune operation. * 3277046 (3107628) The vxdump(1M) utility incorrectly estimates the default tapesize. Patch ID: PHKL_43474, PHCO_43496, PHCO_43501 * 2245466 (1934537) Panic in vx_free() due to NULL pointer * 2857609 (2750860) Performance issue due to CFS fragmentation in CFS cluster * 2904380 (2599590) Expanding or shrinking a DLV5 file system using the fsadm(1M)command causes a system panic. * 3024006 (2858683) Reserve extent attributes changed after vxrestore, only for files greater than 8192bytes * 3079813 (2964018) VxFS 11.31/5.0 : statfsdev causes heavy spinlock contention on snode_table_lock * 3095296 (3031869) "vxfsstat -b" does not print correct information on maximum buffer size * 3096591 (2565400) poor read performance with DSMC (TSM) backup on CFS filesystems * 3100058 (3099638) The minimum value of vxfs_ifree_timelag is different from man page. * 3107775 (3099638) The minimum value of vxfs_ifree_timelag is different from man page. Patch ID: PHKL_43128 * 2245427 (2178147) [VxFS]Link of IFSOC file does not call vx_dotdot_op resulting in a corrupted inode * 2800277 (2566875) A write(2) operation exceeding the quota limit fails with an EDQUOT error. * 2800335 (1092933) VxFS 4.1 reports a "FALSE" write success on ThP LUN * 2846939 (2768505) Data Key Miss Fault at vx_dnlc_appened_dnlcnodes due to null pointer dereference * 2857613 (2830513) ls hangs in vxglm:vxg_grant_sleep. * 2857619 (2845175) vx_do_getacl() panic. * 2876845 (2874172) Infinite looping in vx_exh_hashinit() Patch ID: PHCO_42918, PHKL_42919 * 2245447 (1537731) panic in vx_write whilst resize * 2370062 (2370046) [VxFS][413-594-866][HP] VxFS 11.31/5.0: readahead with read nstream<>1 misses early blocks * 2604150 (2559450) [VxFS][415-229-227][HP] QXCR1001161039 on VxFS 11.23/4.1 : fsck_vxfs(1m) coredumps SEGV_ACCERR * 2696070 (2696067) [VxFS][415-910-567][HP] QXCR1001185027 case 415-910-567 on VxFS 11.31/5.0 : vx_daccess() does not observe GROUP_OBJ permissions * 2722872 (1244756) VxFS 5.0MP1 panic in vx_dnlc_purge_ip() due to deref a NULL d_flhead pointer * 2723003 (2670022) Duplicate file names can be seen in a directory. * 2723005 (2651922) Performance degradation of 'll' and high SYS% CPU in vx_ireuse() * 2723127 (2680946) panic in vx_itryhold+0x40/spinlock() - due to NULL d_childp in dnlc Patch ID: PHCO_42617, PHKL_42618 DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following Symantec incidents: Patch ID: PHCO_44198, PHKL_44199 * 2829721 (Tracking ID: 2829708) SYMPTOM: On a locally mounted Veritas File System (VxFS) machine, the system hangs due to low memory during an internal test. DESCRIPTION: The system hangs because the memory allocated to the structure used to enqueue or dequeue work items in batches, is not freed. RESOLUTION: The code is modified to free the memory. * 3131960 (Tracking ID: 2899907) SYMPTOM: Some file-system operations on a Cluster File System (CFS) may hang with the following stack trace. vxg_svar_sleep_unlock vxg_grant_sleep vxg_cmn_lock vxg_api_lock vx_glm_lock vx_mdele_hold vx_extfree1 vx_exttrunc vx_trunc_ext4 vx_trunc_tran2 vx_trunc_tran vx_cfs_trunc vx_trunc vx_inactive_remove vx_inactive_tran vx_cinactive_list vx_workitem_process vx_worklist_process vx_worklist_thread vx_kthread_init kernel_thread DESCRIPTION: In CFS, a node can lock a mdelelock for an extent map while holding a mdelelock for a different extent map locked. This can result in a deadlock between different nodes in the cluster. RESOLUTION: The code is modified to prevent the deadlock between different nodes in the cluster. * 3464622 (Tracking ID: 2134670) SYMPTOM: The extendfs(1M) utility fails when the file system is marked for the full- file-system check (fsck). The following error message is displayed: # extendfs extendfs: /etc/default/fs is used for determining the file system type UX:vxfs fsck: ERROR: V-3-20858: cannot bmap stateino UX:vxfs extendfs: ERROR: V-3-20141: Invocation of the fsck program terminated abnormally. The file system is marked bad. Run full fsck manually. DESCRIPTION: The extendfs(1M) utility increases the file system size via the fsck. It calls the fsck with the extendfs option and the fsck in turn increases the size of the file system, and modifies the relevant data structures by calculating the new size of the various metadata. The blocks that are to be allocated for the state-inode file are calculated incorrectly. As a result, one less block is allocated and hence the fsck operation fails. The one extra block is allocated only when there is an extra byte that would not fit in the existing blocks. But there may be situations where there are two extra bits, or four to six extra bits that are less than 1 byte, for one AU. This may require an extra block. But since the extra space required in the state-inode file is less than 1 byte, it is ignored. As a result, the extendfs (1M) utility fails. RESOLUTION: The code is modified to allocate a block in the state-inode file, even when the extra space that is required is less than a byte. * 3616610 (Tracking ID: 3259634) SYMPTOM: In CFS, each node with mounted file system cluster has its own intent log in the file system. A CFS with more than 4, 294, 967, 296 file system blocks can zero out an incorrect location resulting from an incorrect typecasting. For example, that kind of CFS can incorrectly zero out 65536 file system blocks at the block offset of 1, 537, 474, 560 (file system blocks) with a 8-Kb file system block size and an intent log with the size of 65536 file system blocks. This issue can only occur if an intent log is located above an offset of 4, 294, 967, 296 file system blocks. This situation can occur when you add a new node to the cluster and mount an additional CFS secondary for the first time, which needs to create and zero a new intent log. This situation can also happen if you resize a file system or intent log and clear an intent log. The problem occurs only with the following file system size and the FS block size combinations: 1kb block size and FS size > 4TB 2kb block size and FS size > 8TB 4kb block size and FS size > 16TB 8kb block size and FS size > 32TB For example, the message log can contain the following messages: The full fsck flag is set on a file system with the following type of messages: 2013 Apr 17 14:52:22 sfsys kernel: vxfs: msgcnt 5 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol1 file system fullfsck flag set - vx_ierror 2013 Apr 17 14:52:22 sfsys kernel: vxfs: msgcnt 6 mesg 017: V-2-17: vx_attr_iget - /dev/vx/dsk/sfsdg/vol1 file system inode 13675215 marked bad incore 2013 Jul 17 07:41:22 sfsys kernel: vxfs: msgcnt 47 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol1 file system fullfsck flag set - vx_ierror 2013 Jul 17 07:41:22 sfsys kernel: vxfs: msgcnt 48 mesg 017: V-2-17: vx_dirbread - /dev/vx/dsk/sfsdg/vol1 file system inode 55010476 marked bad incore DESCRIPTION: In CFS, each node with mounted file system cluster has its own intent log in the file system. When an additional node mounts the file system as a CFS Secondary, the CFS creates an intent log. Note that intent logs are never removed, they are reused. When you clear an intent log, Veritas File System (VxFS) passes an incorrect block number to the log clearing routine, which zeros out an incorrect location. The incorrect location might point to the file data or file system metadata. Or, the incorrect location might be part of the file systems available free space. This is silent corruption. If the file system metadata corrupts, VxFS can detect the corruption when it subsequently accesses the corrupt metadata and marks the file system for full fsck. RESOLUTION: The code is modified so that VxFS can pass the correct block number to the log clearing routine. * 3616940 (Tracking ID: 2411970) SYMPTOM: The system panics at the vx_inode_trandone() function. The following stack trace is observed: vx_tranimdone() vx_trancommit2() vx_trancommit() vx_dircreate_tran() vx_do_create() vx_create1() vx_create0() vx_create() vn_create() unp_bind () bind() syscall() DESCRIPTION: A VxFS transaction is composed of multiple sub-transactions. All the sub- transactions for a specific inode are maintained in a double-linked list, that is also a part of the file-system-transaction list. The panic is due to the corruption of links between the sub-transactions. Meanwhile, the double-linked list is removed from the file-system-transaction. As a result, the system panics at the vx_inode_trandone() function. RESOLUTION: The code is modified so that VxFS handles the sub-transaction double-linked list to detect the possible race conditions and provide more diagnostic information. Patch ID: PHKL_43684, PHCO_43685, PHCO_43686 * 2705245 (Tracking ID: 2555198) SYMPTOM: On HPUX 11.31 binary mode, File Transfer Protocol (FTP) transfer uses the sendfile() interface, which does not create the DMAPI events for Hierarchical Storage Management (HSM). DESCRIPTION: The sendfile() interface does not call the Veritas File System (VxFS) read() function that creates the DMAPI events. It uses the HP Unified File Cache(UFC) interface. The UFC interface is not aware of the HSM application. As a result, the DMAPI events are not generated. RESOLUTION: The code is modified to set a flag in the vfs structure during the mount time, to indicate if the file system is configured under HSM. This flag information is used by the UFC interface to generate the DMAPI events. * 2894568 (Tracking ID: 2894560) SYMPTOM: When the internal stress test is run on CFS, the fsadm(1M) command dumps core when an invalid volume name is passed. The following stack trace is displayed: fs_dev_lookup() extchk_typed() inochk_typd() fset_inochk() reorg_inode() fset_ilist_process() do_reorg() ext_reorg() do_fsadm() main() DESCRIPTION: During a device lookup operation, the boundary check is not present on the volume ID field. When an invalid volume-ID is passed, this may result in a core dump. RESOLUTION: The code is modified to add the boundary check for volume ID field in the fs_dev_lookup() function. * 3131811 (Tracking ID: 2966277) SYMPTOM: Systems with high file-system activity like read/write/open/lookup may panic with the following stack trace due to a rare race condition: spinlock+0x21 ( ) -> vx_rwsleep_unlock() vx_ipunlock+0x40() vx_inactive_remove+0x530() vx_inactive_tran+0x450() vx_local_inactive_list+0x30() vx_inactive_list+0x420() -> vx_workitem_process() -> vx_worklist_process() vx_worklist_thread+0x2f0() kthread_daemon_startup+0x90() DESCRIPTION: ILOCK is released before doing a IPUNLOCK that causes a race condition. This results in a panic when an inode that has been set free is accessed. RESOLUTION: The code is modified so that the ILOCK is used to protect the inodes' memory from being set free, while the memory is being accessed. * 3131881 (Tracking ID: 3010444) SYMPTOM: On a Network File System (NFS) mounted file system, the operations which read the file via the cksum (1m) command may fail with the following error message: cksum: read error on : Bad address The following error messages would also be seen in the syslog vmunix: WARNING: Synchronous Page I/O error DESCRIPTION: When the read-vnode operation (VOP_RDWR) is performed, certain requests are converted to direct the I/O for optimisation. However, the NFS buffers passed during the read requests are not the user buffers. As a result, there is an error. RESOLUTION: The code is modified to convert the I/O requests to the direct I/O, only if the buffer passed during the I/O is the user buffer. * 3235161 (Tracking ID: 3235160) SYMPTOM: When a file system is unmounted when there are some file-system operations in progress, like the read/lookup operations, the system may panic with the following stack trace: panic bad_kern_reference $cold_pfault vm_hndlr bubbleup vx_getpage preg_vn_fault virtual_fault vfault vm_hndlr bubbleup DESCRIPTION: Due to the race condition between the umount thread and the page-fault handler thread, a local variable updated from the global-fs structure may contain a stale value. This leads to the panic. RESOLUTION: The code is modified to protect the local variable using the active levels. This ensures that the local variable is not changed once it is updated. * 3277010 (Tracking ID: 3226462) SYMPTOM: On a cluster mounted file-system with unequal CPUs, while doing a lookup operation, a node may panic with the stack trace: vx_dnlc_recent_cookie vx_dnlc_getpathname audit_get_pathname_from_dnlc audit_clean_path $cold_audit_build_full_dir_name inline change_p_cdir DESCRIPTION: The cause of the panic is of out-of-bounds access in the counters[] array whose size is defined by the vx_max_cpu variable. The value of vx_max_cpu can differ between the CFS nodes, if the nodes have different number of processors. However, the code assumes this value is the same across the cluster. When propagating inode cookies across the cluster, the counter[] array is allocated based on the vx_max_cpu of the current CFS node. If the cookie is populated via vx_cbdnlc_populate_cookie(), having a CPU ID from another CFS node exceeding the local vx_max_cpu, the function vx_dnlc_recent_cookie() would access locations beyond the counter[] array allocated. RESOLUTION: The code is modified to detect the out-of-bound access at vx_dnlc_recent_cookie () and return the ENOENT error. * 3277016 (Tracking ID: 3244613) SYMPTOM: A file-system extent operation by using the fsadm(1M) command may hang with the following stack trace: vx_event_wait(inlined) vx_delay2+0x2a0 cold_vx_active_common_flush+0x80 vx_close+0x70 vn_close(inlined) vno_close+0xe0 closef(inlined) DESCRIPTION: During a resize operation, the fsadm(1M) command freezes the file system. In an error case, the fsadm(1M) command exits without thawing the file system. This results in a hang. RESOLUTION: The code is modified to thaw the file system, before the fsadm(1M) command exits in the error case. * 3277028 (Tracking ID: 1960815) SYMPTOM: The vxtunefs (1M) command manpage does not mention the performance impact of tuning any of the Veritas File System (VxFS) parameters on the system with heavy load. DESCRIPTION: The performance impact that the system face on tuning any of the VxFS parameters needs to be mentioned in the vxtunefs (1M) command manpage. RESOLUTION: The manpage is updated to mention the performance impact of tuning any of the VxFS parameters on the system with heavy load. * 3277046 (Tracking ID: 3107628) SYMPTOM: The vxdump(1M) utility incorrectly estimates the number of tapes required to complete the backup and prematurely prompts for the next tape. DESCRIPTION: The vxdump(1M) utility prematurely prompts for next tape. RESOLUTION: The code is modified to fix the initialization for track and tape variables. Patch ID: PHKL_43474, PHCO_43496, PHCO_43501 * 2245466 (Tracking ID: 1934537) SYMPTOM: The reverse-name-lookup operation on an inode may panic the machine with the following stack trace: vx_free vx_traverse_tree+0x4a0 vx_dir_lookup+0x1e4 vx_rev_namelookup+0x294 vx_aioctl_common+0xac4 vx_aioctl+0x12c vx_ioctl+0xe0 DESCRIPTION: During the lookup operation if the memory allocation fails, the user still goes ahead and adds it to the used memory, and later tries to free that memory. This results in the panic. RESOLUTION: The code is modified to update the usage counters correctly, and skip updating the count during the error condition. An additional check is added to free only the non-null buffers. * 2857609 (Tracking ID: 2750860) SYMPTOM: On a large file system(4TB or greater), the performance of the write(1) operation with many small request sizes may degrade, and many threads may be found sleeping with the following stack trace: real_sleep sleep_one vx_sleep_lock vx_lockmap vx_getemap vx_extfind vx_searchau_downlevel vx_searchau_downlevel vx_searchau_downlevel vx_searchau_downlevel vx_searchau_uplevel vx_searchau vx_extentalloc_device vx_extentalloc vx_te_bmap_alloc vx_bmap_alloc_typed vx_bmap_alloc vx_write_alloc3 vx_recv_prealloc vx_recv_rpc vx_msg_recvreq vx_msg_process_thread kthread_daemon_startup DESCRIPTION: For a cluster-mounted file system, the free-extend-search algorithm is not optimized for a large file system (4TB or greater), and for instances where the number of free Allocation Units (AUs) available can be very large. RESOLUTION: The code is modified to optimize the free-extend-search algorithm by skipping certain AUs. This reduces the overall search time. * 2904380 (Tracking ID: 2599590) SYMPTOM: Expansion of a 100% full file system may panic the machine with the following stack trace. bad_kern_reference() $cold_vfault() vm_hndlr() bubbledown() vx_logflush() vx_log_sync1() vx_log_sync() vx_worklist_thread() kthread_daemon_startup() DESCRIPTION: When 100% full file system is expanded intent log of the file system is truncated and blocks freed up are used during the expansion. Due to a bug the block map of the replica intent log inode was not getting updated thereby causing the block maps of the two inodes to differ. This caused some of the in- core structures of the intent log to go NULL. The machine panics while de- referencing one of this structure. RESOLUTION: Updated the block map of the replica intent log inode correctly. 100% full file system now can be expanded only If the last extent in the intent log contains more than 32 blocks, otherwise fsadm will fail. To expand such a file-system, some of the files should be deleted manually and resize be retried. * 3024006 (Tracking ID: 2858683) SYMPTOM: The reserve-extent attributes are changed after the vxrestore(1M ) operation, for files that are greater than 8192 bytes. DESCRIPTION: A local variable is used to contain the number of the reserve bytes that are reused during the vxrestore(1M) operation, for further VX_SETEXT ioctl call for files that are greater than 8k. As a result, the attribute information is changed. RESOLUTION: The code is modified to preserve the original variable value till the end of the function. * 3079813 (Tracking ID: 2964018) SYMPTOM: On a high end machine with about 125 CPUs operations using the lstat64(2) function, may seem to be hung and the following stack trace is observed: spinlock+0xe0 rwspin_wrlock+0x30 specvp+0x510 vx_lookup+0x8a0 -> lookuppnvp(inlined) -> lookuppn(inlined) DESCRIPTION: The statvfsdev search calls the devnm() function to search the whole /dev/ directory for reverse- pathname RESOLUTION: The code is modified such that a new fs_load() function is implemented to make use of the incoming file descriptor, if it is already a character device. However, the devnm() function is still needed if the incoming file descriptor is a block device. * 3095296 (Tracking ID: 3031869) SYMPTOM: In a multi-CPU environment, the "vxfsstat -b" command does not print the correct information on the maximum-size buffer. DESCRIPTION: The "vx_bc_bufhwm" parameter represents the maximum amount of memory that can be used to cache the VxFS metadata. When the kctune(1M) command is used to tune the "vxfs_bc_bufhwm" parameter to a different value, the tunable is not set correctly due to the incorrect arithmetic. As a consequence, the "vxfsstat -b" command reports the maximum-size buffer to be increased, even though the "vxfs_bc_bufhwm" parameter is tuned to a lower value. RESOLUTION: The code is modified to correct the arithmetic for tuning the "vx_bc_bufhwm" parameter. * 3096591 (Tracking ID: 2565400) SYMPTOM: Sequential buffered I/O reads are slow in performance. DESCRIPTION: Read-Aheads are not happening because the file-system's read-ahead size gets incorrectly calculated. RESOLUTION: Fixed the incorrect typecast. * 3100058 (Tracking ID: 3099638) SYMPTOM: When the vxfs_ifree_timelag(5) tunable is tuned the following error message is displayed: # kctune vxfs_ifree_timelag=400 ERROR: mesg 095: V-2-95: Setting vxfs_ifree_timelag to 450 since the specified value for vxfs_ifree_timelag is less than the recommended minimum value of 1035 DESCRIPTION: In the vxfs_ifree_timelag(5) tunable man page, the minimum value is set to "None". The error message is displayed when the vxfs_ifree_timelag(5) tunable is set to a value which is less than 450. In the error message, a garbage value is displayed as the recommended minimum value. The error occurs because a single argument is passed for the error message that has two format specifier's. RESOLUTION: The code is modified to set the correct minimum value of the vxfs_ifree_timelag (5) tunable, and display the correct error message. * 3107775 (Tracking ID: 3099638) SYMPTOM: When the vxfs_ifree_timelag(5) tunable is tuned the following error message is displayed: # kctune vxfs_ifree_timelag=400 ERROR: mesg 095: V-2-95: Setting vxfs_ifree_timelag to 450 since the specified value for vxfs_ifree_timelag is less than the recommended minimum value of 1035 DESCRIPTION: In the vxfs_ifree_timelag(5) tunable man page, the minimum value is set to "None". The error message is displayed when the vxfs_ifree_timelag(5) tunable is set to a value which is less than 450. In the error message, a garbage value is displayed as the recommended minimum value. The error occurs because a single argument is passed for the error message that has two format specifier's. RESOLUTION: The code is modified to set the correct minimum value of the vxfs_ifree_timelag (5) tunable, and display the correct error message. Patch ID: PHKL_43128 * 2245427 (Tracking ID: 2178147) SYMPTOM: If a socket file is removed, the file system is marked for full fsck (1M) operation. The following error message is displayed in the system log: vmunix: vxfs: WARNING: msgcnt 1 mesg 087: V-2-87: vx_dotdot_manipulate: - / file system 2437 inode 541 dotdot inode error DESCRIPTION: During the socket file creation, the attribute inode for the parent directory is not updated. Hence, the error occurs when the socket file is removed. RESOLUTION: The code is modified to update the socket file linkage during creation, thus avoiding the error message. * 2800277 (Tracking ID: 2566875) SYMPTOM: The write(2) operation exceeding the quota limit fails with an EDQUOT error ("Disc quota exceeded") before the user quota limit is reached. DESCRIPTION: When a write request exceeds a quota limit, the EDQUOT error is handled so that Veritas File System (VxFS) can allocate space up to the hard quota limit to proceed with a partial write. However, VxFS does not handle this error and an erroris returned without performing a partial write. RESOLUTION: The code is modified to handle the EDQUOT error from the extent allocation routine. * 2800335 (Tracking ID: 1092933) SYMPTOM: The system may panic in the vx_fsync_chains() function when it tries to sleep in the interrupt context. The following stack trace is displayed: vx_event_wait vx_delay2 vx_fsync_chains vx_disable vx_dataioerr vx_pageio_done DESCRIPTION: While handling the external interrupt, the vx_pageio_done() function calls the function vx_fsync_chains() function. The vx_fsync_chains() function may sleep during the execution. The function vx_fsync_chains() is required in the case of Input/output (IO) errors. The function vx_fsync_chains() is called at a couple of places, when the I/O strategy fails. But, the error variable is overwritten improperly. RESOLUTION: The code is modified to save Input/Output errors so that the vx_fsync_chains() function can be called and the call to vx_fsync_chains() is removed from the vx_disable(). * 2846939 (Tracking ID: 2768505) SYMPTOM: While reading the Dynamic Name Lookup Cache (DNLC) entries using the pstat_getmpathname(2) function, the system might panic and display the following stack trace: vx_dnlc_appened_dnlcnodes vx_dnlc_getentries pstat_mpathname $cold_pstat syscall DESCRIPTION: While reading all the DNLC entries, the VxFS pointer is derived from the VNODE pointer. The VNODE pointer can be reused and de-referencing the VxFS pointer might panic the system. RESOLUTION: The code is modified to pass the VxFS pointer directly to the function instead of deriving it from the VNODE pointer. * 2857613 (Tracking ID: 2830513) SYMPTOM: The Cluster File System (CFS) hangs while performing file removal operations and the following stack trace is displayed: vxglm::vxg_grant_sleep+0x110 () vxglm::vxg_cmn_lock+0x5a0 () vxglm::vxg_api_lock+0x310 () vx_glm_lock+0x70 () vx_mdelelock+0x70 () vx_mdele_hold+0xe0 () vx_extfree+0x700 () DESCRIPTION: The CFS hangs due to a missing unlock call for the file removal operations. RESOLUTION: The code is modified to unlock the file removal operations. * 2857619 (Tracking ID: 2845175) SYMPTOM: When the Access Control List (ACL) feature is enabled, the system may panic with "Data Key Miss Fault in KERNEL mode" error message in the vx_do_getacl() function and the following stack trace is displayed: vx_do_getacl+0x840 () vx_getacl+0x70 () acl+0x480 () DESCRIPTION: In the vx_do_getacl() function, a local variable is accessed without being initializing as a result leading to a panic. RESOLUTION: The code is modified to initialize the local variable to NULL before using it. * 2876845 (Tracking ID: 2874172) SYMPTOM: Network File System (NFS) file creation thread might loop continuously with the following stack trace: vx_getblk_cmn(inlined) vx_getblk+0x3a0 vx_exh_allocblk+0x3c0 vx_exh_hashinit+0xa50 vx_dexh_create+0x240 vx_dexh_init+0x8b0 vx_do_create+0x1e0 vx_create1+0x1d0 vx_create0+0x270 vx_create+0x40 rfs3_create+0x420 common_dispatch+0xb40 rfs_dispatch+0x40 svc_getreq+0x250 svc_run+0x310 svc_do_run+0xd0 nfssys+0x6a0 hpnfs_nfssys+0x60 coerce_scall_args+0x130 syscall+0x590 DESCRIPTION: The Veritas File System (VxFS) file creation vnode operation (VOP) routine expects the parent vnode to be a directory vnode pointer. But, the NFS layer passes a stale file vnode pointer by default. This might cause unexpected results such as hang during VOP handling. RESOLUTION: The code is modified to check for the vnode type of the parent vnode pointer at the beginning of the create VOP call and return an error if it is not a directory vnode pointer. Patch ID: PHCO_42918, PHKL_42919 * 2245447 (Tracking ID: 1537731) SYMPTOM: A panic can occur when a write hits an ENOSPC whilst a re-size is in operation. DESCRIPTION: The cause of problem here is we get an ENOSPC during a write and re-size it at the same time. We drop the active level do some other task and then again raise it. The time gap between dropping and raising active level gives a window for the re-size to come in and do its work. Here, we do not update the related pointer and this results in an panic. RESOLUTION: Updated fs pointer after re-aquiring active level. * 2370062 (Tracking ID: 2370046) SYMPTOM: Readahead with read_nstream value <> 1 misses early blocks of data. DESCRIPTION: Readahead is not reading portion of the file which is expected to be read in advance. These blocks are read on demand rather than actual readahead. This is happening because the parameter determining next readahead offset is wrongly zeroed out. RESOLUTION: Changed the code which causes zeroing out of the parameter determining next readahead offset. * 2604150 (Tracking ID: 2559450) SYMPTOM: Command fsck_vxfs(1m) may core-dump with SEGV_ACCERR error. DESCRIPTION: Command fsck_vxfs(1M) is trying to allocate the memory for a structure, but memory allocation fails for the structure resulting in the segmentation fault. RESOLUTION: A check is added in the code for failed memory allocation to print an error message instead of a core dump. * 2696070 (Tracking ID: 2696067) SYMPTOM: A file created which inherits the default ACL from parent shows wrong permissions when getaccess command is issued. DESCRIPTION: vx_daccess() does not fabricate any GROUP_OBJ entry (as vx_do_getacl does). If a newly created file leverages its parent directory's ACL entries RESOLUTION: To address this, a fix is to fabricate a GROUP_OBJ entry. * 2722872 (Tracking ID: 1244756) SYMPTOM: A race condition corrupts data structures, causing a NULL pointer dereference. DESCRIPTION: There is a race condition between some DNLC functions. One of those functions is already holding a lock and wants another lock which results in a race condition. This causes the freelist corruption and a NULL pointer dereference. RESOLUTION: Updated DNLC_LOOKUP to make sure DNLC entry is not free and on freelist when moving it to tail, avoiding a race with DNLC_GET or DNLC_ENTER. * 2723003 (Tracking ID: 2670022) SYMPTOM: Duplicate file names can be seen in a directory. DESCRIPTION: VxFS maintains internal directory name lookup cache (DNLC) to improve the performance of directory lookups. A race condition is arising in DNLC lists manipulation code during lookup/creation of file names having >32 characters ( which will further affect other file creations). This is causing DNLC to have a stale entry for an existing file in the directory. A lookup of such a file through DNLC will say file as non-existent which will allow another duplicate file name in the directory. RESOLUTION: Fixed the race condition by protecting the DNLC lists through proper locks. * 2723005 (Tracking ID: 2651922) SYMPTOM: "ls -l" command on Local VxFS file system is running slow and high CPU usage is seen on HP platform. DESCRIPTION: This issue occurs when system is under inode pressure, and the most of inodes on inode free list are CFS inodes. On HP-UX platform, currently, CFS inodes are not allowed to be reused as local inodes to avoid GLM deadlock issue when vxFS reconfig is in process. So if system needs a VxFS local inode, it has to take a amount of time to loop through all the inode free lists to find a local inode, if the free lists are almost filled up with CFS inodes. RESOLUTION: 1. added a global "vxi_icache_cfsinodes" to count cfs inodes in inode cache. 2. relaxed the condition for converting cluster inode to local inode when the number of in-core cfs inodes is greater than the threshold"vx_clreuse_threshold" and reconfig is not in progress. * 2723127 (Tracking ID: 2680946) SYMPTOM: panic in vx_itryhold+0x40/spinlock() - due to NULL child pointer of dnlc entry. Data Access Rights Fault in KERNEL mode spinlock+0x40 vx_itryhold+0x40 vx_dnlc_lookup+0x1b0 vx_cbdnlc_lookup+0x130 vx_fast_lookup+0x120 vx_lookup+0x3c0 lookuppnvp+0x2d0 lookuppn+0x90 lookupname+0x60 vn_open+0xa0 copen+0x170 open+0x80 syscall+0x920 DESCRIPTION: The panic is because of NULL pointer de-reference. The reason for NULL pointer is not known for sure. One of the possibility for NULL child pointer is while inserting an entry in DNLC. RESOLUTION: We have global variable which we increment if we see the DNLC with NULL child pointer is getting inserted. So when next time issue occurs we can look value of the global and rule out this possibility. Patch ID: PHCO_42617, PHKL_42618 * 2289635 (Tracking ID: 1633670) SYMPTOM: Panic in vx_inull_list() / vx_inactive() / vx_inode_deinit() after forced unmount of writable clone and unmount of primary fileset. DESCRIPTION: This happens due to accessing of NULL pointer dereferencing. RESOLUTION: Do not iflush clone inodes which have already been force unmounted when running iflush on force unmount of a primary fileset. Beware of null vfsp pointers in partially processed force-unmounted inodes. Reference the vx_vfs struct through the fset instead of the vnode. * 2351018 (Tracking ID: 2253938) SYMPTOM: In a Cluster File System (CFS) environment , the file read performances gradually degrade up to 10% of the original read performance and the fsadm(1M) -F vxfs -D -E shows a large number (> 70%) of free blocks in extents smaller than 64k. For example, % Free blocks in extents smaller than 64 blks: 73.04 % Free blocks in extents smaller than 8 blks: 5.33 DESCRIPTION: In a CFS environment, the disk space is divided into Allocation Units (AUs).The delegation for these AUs is cached locally on the nodes. When an extending write operation is performed on a file, the file system tries to allocate the requested block from an AU whose delegation is locally cached, rather than finding the largest free extent available that matches the requested size in the other AUs. This leads to a fragmentation of the free space, thus leading to badly fragmented files. RESOLUTION: The code is modified such that the time for which the delegation of the AU is cached can be reduced using a tuneable, thus allowing allocations from other AUs with larger size free extents. Also, the fsadm(1M) command is enhanced to de-fragment free space using the -C option. * 2406458 (Tracking ID: 2283893) SYMPTOM: In a Cluster File System (CFS) environment , the file read performances gradually degrade up to 10% of the original read performance and the fsadm(1M) -F vxfs -D -E shows a large number (> 70%) of free blocks in extents smaller than 64k. For example, % Free blocks in extents smaller than 64 blks: 73.04 % Free blocks in extents smaller than 8 blks: 5.33 DESCRIPTION: In a CFS environment, the disk space is divided into Allocation Units (AUs).The delegation for these AUs is cached locally on the nodes. When an extending write operation is performed on a file, the file system tries to allocate the requested block from an AU whose delegation is locally cached, rather than finding the largest free extent available that matches the requested size in the other AUs. This leads to a fragmentation of the free space, thus leading to badly fragmented files. RESOLUTION: The code is modified such that the time for which the delegation of the AU is cached can be reduced using a tuneable, thus allowing allocations from other AUs with larger size free extents. Also, the fsadm(1M) command is enhanced to de-fragment free space using the -C option. * 2410789 (Tracking ID: 1466351) SYMPTOM: Mount hangs in vx_bc_binval_cookie like the following stack, delay vx_bc_binval_cookie vx_blkinval_cookie vx_freeze_flush_cookie vx_freeze_all vx_freeze vx_set_tunefs1 vx_set_tunefs vx_aioctl_full vx_aioctl_common vx_aioctl vx_ioctl genunix:ioctl unix:syscall_trap32 DESCRIPTION: The hanging process is waiting for a buffer to be unlocked. But that buffer can only be released if its associated cloned map writes get flushed. But a necessary flush is missed. RESOLUTION: Add code to synchronize cloned map writes so that all the cloned maps will be cleared and the buffers associated with them will be released. * 2551555 (Tracking ID: 2428964) SYMPTOM: Value of kernel tunable max_thread_proc gets incremented by 1 after every software maintenance related activity (install, remove etc.) of VRTSvxfs package. DESCRIPTION: In the postinstall script for VRTSvxfs package, value of kernel tunable max_thread_proc is wrongly increment by 1. RESOLUTION: From postinstall script increment operation of max_thread_proc tunable is removed. * 2551569 (Tracking ID: 2510903) SYMPTOM: Writing to clones loops permanently on HP-UX 11.31, there are some threads of the typical stack like following: vx_tranundo vx_logged_cwrite vx_write_clone vx_write1 vx_rdwr vno_rw inline rwuio write syscall DESCRIPTION: A VxFS write with small size can go to logged write which stores the data in intent log. The logged write can boost performance for small writes but requires the write size within logged write limit. However, When we write data to check points and the write length is greater than logged write limit, vxfs cannot proceed with logged write and retry forever. RESOLUTION: Skipped the logged write if the write size exceeds the specific limit. * 2556096 (Tracking ID: 2515380) SYMPTOM: The ff command hangs and later it exits after program exceeds memory limit with following error. # ff -F vxfs /dev/vx/dsk/bernddg/testvol UX:vxfs ff: ERROR: V-3-24347: program limit of 30701385 exceeded for directory data block list UX:vxfs ff: ERROR: V-3-20177: /dev/vx/dsk/bernddg/testvol DESCRIPTION: 'ff' command lists all files on device of vxfs file system. In 'ff' command we do directory lookup. In a function we save the block addresses for a directory. For that we traverse all the directory blocks. Then we have function which keeps track of buffer in which we read directory blocks and the extent up to which we have read directory blocks. This function is called with offset and it return the offset up to which we have read the directory blocks. The offset passed to this function has to be the offset within the extent. But, we were wrongly passing logical offset which can be greater than extent size. As a effect the offset returned gets wrapped to 0. The caller thinks that we have not read anything and hence the loop. RESOLUTION: Remove call to function which maintains buffer offsets for reading data. That call was incorrect and redundant. We actually call that function correctly from one of the functions above. * 2561752 (Tracking ID: 2561739) SYMPTOM: When the file is created and the if the parent has default ACL entry then that entry is not taken into account for calculating the class entry of that file. When a separate dummy entry added we take into account the default entry from parent as well. e.g. $ getacl . # file: . # owner: root # group: sys user::rwx group::rwx class:rwx other:rwx default:user:user_one:r-x $ touch file1 $ getacl file1 # file: try1 # owner: root # group: sys user::rw- user:user_one:r-x group::rw- class:rw- <------ other:rw- The class entry here should be rwx. DESCRIPTION: We were not taking into account the default entry of parent. We share the attribute inode with parent and do not create new attribute inode for newly created file. But when an ACL entry is explicitly made we create the separate attribute inode so the default entry also get copied in new inode and taken into consideration while returning the class entry of file. RESOLUTION: Now before returning the ACL entry buffer we calculate the class entry again and consider all the entries. * 2561757 (Tracking ID: 2492304) SYMPTOM: "find" command displays duplicate directory entries. DESCRIPTION: Whenever the directory entries can fit in the inode's immediate area VxFS doesn't allocate new directory blocks. As new directory entries get added to the directory this immediate area gets filled and all the directory entries are then moved to a newly allocated directory block. The directory blocks have space reserved at the start of the block to hold the block hash information which is used for fast lookup of entries in that block. Offset of the directory entry, which was at say x bytes in the inode's immediate area, when moved to the directory block, will be at (x + y) bytes. "y" is the size of the block hash. During this transition phase from immediate area to directory blocks, a readdir() can report a directory entry more than once. RESOLUTION: Directory entry offsets returned to the "readdir" call are adjusted so that when the entries move to a new block, they will be at the same offsets. * 2616625 (Tracking ID: 2616622) SYMPTOM: slow mmap() performace when the filesystem block size is 8k and pagesize is 4K. DESCRIPTION: When we have an 8k block size file system and 4k pages and mmap say a 8K file, the file as represented in memory ends up as two pages (0 and 1). When the memory at offset 0 into the mapping is modified, we get a page fault for page 0 in the file. However, we haven't had a page fault yet for page 1 and can't guarantee that we will in the future. When we allocate that disk block and mark it as valid, we trust that the page mentioned in the fault request will get flushed out to disk and thereby leave it uninitialized on disk by default. We clear just the page in memory and leave it dirty so we know the data in memory is more recent than the data on disk. However, the other half of the block (which could eventually be mapped to page 1) gets cleared with a synchronous write because we don't know if we will ever see a fault. This synchronous clearing of the other half of 8K block was causing performance degradation. RESOLUTION: We now expand the range of the fault to cover whole 8K block. In this case we would just ignore that the OS asked for only one page and give it two pages anyway to cover the whole file system block to save the separate synchronous clearing of the other half of 8K block. * 2619959 (Tracking ID: 1027438) SYMPTOM: Internal noise for VxFS hit "f:vx_statvfs_pri:2" assert on cluster file system DESCRIPTION: While updating summaries for Allocation units on node, summaries for all node of cluster need to be synchronized using broadcast message. The broadcast message is not sent as recovery for file system is in progress, resulting in wrong free space calculation on node causing the assert. RESOLUTION: Code is changed to send the broadcast message after file system recovery is complete. INSTALLING THE PATCH -------------------- Installing VxFS 5.0 MP1P13 patch: a)If you install this patch on a CVM cluster, install it one system at a time so that all the nodes are not brought down simultaneously. b)VxFS 5.0(GA) must be installed before applying these patches. c)To verify the VERITAS file system level, enter: # swlist -l product | egrep -i 'VRTSvxfs' VRTSvxfs 5.0.31.0 VERITAS File System Note : VRTSfsman is a corequisite for VRTSvxfs.Hence VRTSfsman also needs to be installed alongwith VRTSvxfs. # swlist -l product | egrep -i 'VRTS' VRTSvxfs 5.0.31.0 Veritas File System VRTSfsman 5.0.31.0 Veritas File System Manuals d)All prerequisite/corequisite patches have to be installed.The Kernel patch requires a system reboot for both installation and removal. e)To install the patch, enter the following command: # swinstall -x autoreboot=true -s PHCO_44198 PHKL_44199 PHCO_38913 PHNE_42470 Incase the patch is not registered, the patch can be registered using the following command: # swreg -l depot , where is the absolute path where the patch resides. REMOVING THE PATCH ------------------ Removing VxFS 5.0 MP1P13 patches: a)To remove the patch, enter the following command: # swremove -x autoreboot=true PHCO_44198 PHKL_44199 PHCO_38913 PHNE_42470 SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE