* * * READ ME * * * * * * Veritas File System 5.0 MP1 * * * * * * P-patch 15 * * * Patch Date: 2016-11-25 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH PATCH NAME ---------- Veritas File System 5.0 MP1 P-patch 15 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- HP-UX 11i v3 (11.31) PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxfs BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Veritas File System 5.0 * Veritas Storage Foundation 5.0 * Veritas Storage Foundation Cluster File System 5.0 * Veritas Storage Foundation for Oracle 5.0 * Veritas Storage Foundation for Oracle RAC 5.0 * Veritas Storage Foundation HA 5.0 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: PHCO_44556, PHKL_44555 * 3879386 (3660342) VxFS package does not set the MANPATH environment variable correctly. * 3879451 (3867995) Veritas File System (VxFS) worker threads hangs while flushing transaction log buffer to the disk. * 3897306 (3781085) The extendfs(1M) command cannot extend a file system beyond 4 TB. * 3898606 (3873624) In case of memory mapped I/O, synchronous buffer allocated from FC_BUF_DEFAULT_arena may not be freed, if error flag is set on them. Patch ID: PHCO_44458, PHKL_44459, PHCO_44462 * 2038745 (2038736) In case of a clone, the Bmap (block map) operation that allocates or truncates the file change log (FCL) file may panic the system. * 3638176 (2104575) When the file system is disabled due to I/O errors on the disk, the page flushing to the disk may hang. * 3703181 (3703176) The tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. * 3706658 (3066116) When adb t vx_inactive_throttling and vx_inactive_process_throttling tunables are set to 1, the system panics due to NULL pointer dereference. * 3706716 (3703176) The tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. * 3751308 (2439108) System crashes when the read_preferred_io tunable is set to a non-page aligned size. * 3770865 (3469644) The system panics in the vx_logbuf_clean() function. * 3816309 (3718924) On cluster file system (CFS), the file system I/O hangs for a few seconds and all the file pages are invalidated during the hang. * 3832397 (3784126) The mmap pages are invalidated during the file system freeze. * 3832588 (3832584) Files and directories on the VxFS file system are not able to inherit the default ACL entries of other objects resulting in incorrect inherited- permission for files and directories created under the parent directory. * 3859666 (3807129) The file system resize operation may result in panic on an almost full file system. * 3859667 (1482790) System panics when VxFS DMAPI is used. * 3860290 (3841894) The ncheck(1M) command fails on the file system that has a large number of directories. The error message is displayed as follows: ERROR: V-3-21550: no more room for inode info buffers. * 3867888 (3673599) The effective user permission is incorrectly displayed in the getacl(1M) command output. * 3867889 (3433777) A single CPU machine panics due to the safety-timer check when the inodes are re-tuned. * 3867894 (3150368) vx_writesuper() function causes the system to panic in evfsevol_strategy(). Patch ID: PHCO_44198, PHKL_44199 * 2829721 (2829708) On a locally mounted Veritas File System (VxFS) machine, the system hangs due to low memory during an internal test. * 3131960 (2899907) On CFS, some file-system operations like vxcompress utility and de-duplication fail to respond. * 3464622 (2134670) The extendfs(1M) utility fails when the file system is marked for the full-file- system check (fsck). * 3616610 (3259634) A Cluster File System (CFS) with blocks larger than 4GB may become corrupt. * 3616940 (2411970) The system panics at the vx_inode_trandone() function. Patch ID: PHKL_43684, PHCO_43685, PHCO_43686 * 2705245 (2555198) On VxFS sendfile() does not create DMAPI events for HSM. * 2894568 (2894560) While running internal stress test on Cluster File System, the fsadm(1M) command dumps core. * 3131811 (2966277) Systems with high file system activity like read/write/open/lookup may panic the system. * 3131881 (3010444) On a NFS filesystem cksum(1m) fails with the "cksum: read error on : Bad address" error. * 3235161 (3235160) Data Key Miss Fault is observed in the vx_getpage() function. * 3277010 (3226462) On a cluster mounted file-system with unequal CPUs, a node may panic while doing a lookup operation. * 3277016 (3244613) fsadm(1M) command hangs while the I/O load on regular vxfs filesystem and checkpoint. * 3277028 (1960815) VxFS read ahead can cause stalled IO on all the write operations during a re- tune operation. * 3277046 (3107628) The vxdump(1M) utility incorrectly estimates the default tapesize. Patch ID: PHKL_43474, PHCO_43496, PHCO_43501 * 2245466 (1934537) Panic in vx_free() due to NULL pointer * 2857609 (2750860) Performance issue due to CFS fragmentation in CFS cluster * 2904380 (2599590) Expanding or shrinking a DLV5 file system using the fsadm(1M)command causes a system panic. * 3024006 (2858683) Reserve extent attributes changed after vxrestore, only for files greater than 8192bytes * 3079813 (2964018) VxFS 11.31/5.0 : statfsdev causes heavy spinlock contention on snode_table_lock * 3095296 (3031869) "vxfsstat -b" does not print correct information on maximum buffer size * 3096591 (2565400) poor read performance with DSMC (TSM) backup on CFS filesystems * 3100058 (3099638) The minimum value of vxfs_ifree_timelag is different from man page. * 3107775 (3099638) The minimum value of vxfs_ifree_timelag is different from man page. Patch ID: PHKL_43128 * 2245427 (2178147) [VxFS]Link of IFSOC file does not call vx_dotdot_op resulting in a corrupted inode * 2800277 (2566875) A write(2) operation exceeding the quota limit fails with an EDQUOT error. * 2800335 (1092933) VxFS 4.1 reports a "FALSE" write success on ThP LUN * 2846939 (2768505) Data Key Miss Fault at vx_dnlc_appened_dnlcnodes due to null pointer dereference * 2857613 (2830513) ls hangs in vxglm:vxg_grant_sleep. * 2857619 (2845175) vx_do_getacl() panic. * 2876845 (2874172) Infinite looping in vx_exh_hashinit() Patch ID: PHCO_42918, PHKL_42919 * 2245447 (1537731) panic in vx_write whilst resize * 2370062 (2370046) [VxFS][413-594-866][HP] VxFS 11.31/5.0: readahead with read nstream<>1 misses early blocks * 2604150 (2559450) [VxFS][415-229-227][HP] QXCR1001161039 on VxFS 11.23/4.1 : fsck_vxfs(1m) coredumps SEGV_ACCERR * 2696070 (2696067) [VxFS][415-910-567][HP] QXCR1001185027 case 415-910-567 on VxFS 11.31/5.0 : vx_daccess() does not observe GROUP_OBJ permissions * 2722872 (1244756) VxFS 5.0MP1 panic in vx_dnlc_purge_ip() due to deref a NULL d_flhead pointer * 2723003 (2670022) Duplicate file names can be seen in a directory. * 2723005 (2651922) Performance degradation of 'll' and high SYS% CPU in vx_ireuse() * 2723127 (2680946) panic in vx_itryhold+0x40/spinlock() - due to NULL d_childp in dnlc Patch ID: PHCO_42617, PHKL_42618 DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following incidents: Patch ID: PHCO_44556, PHKL_44555 * 3879386 (Tracking ID: 3660342) SYMPTOM: VxFS package does not set the MANPATH environment variable correctly. DESCRIPTION: The VxFS install package does not set the man-page-search path for VxFS correctly in /etc/MANPATH. It inserts the search path for VxFS after /usr/share/man/. However, there are old versions of VxFS man pages in /usr/share/man/man1m.Z. As a result, the older version of the man page is displayed first and VxFS man pages are not displayed. RESOLUTION: The post-install script is modified to place the search path for VxFS man page before /usr/share/man. * 3879451 (Tracking ID: 3867995) SYMPTOM: VxFS file system appears hang, while vxfsd threads are waiting to flush log buffers with the following stack traces: vx_sleep_lock() vx_logbuf_clean() vx_map_delayflush() vx_tflush_map() vx_fsq_flush() vx_tranflush_threaded() vx_workitem_process() vx_worklist_process() Or vx_sleep_lock() vx_logbuf_clean() vx_logflush() vx_async_iupdat() vx_iupdat_local() vx_iupdat() vx_iflush_list() vx_iflush() vx_workitem_process() vx_worklist_process() DESCRIPTION: If an extent free operation is executed instantaneously before the completion of the previous extent free transaction, then the extent map is updated instantly and the freed extent can be picked up for some other inode. But, since the previous extent free transaction is incomplete and has not hit the disk due to a particular sequence of events, VxFS transaction log encounters an inconsistent state. Thus, transaction log buffer flush may hang. RESOLUTION: The code is modified to treat every extent free operation as delayed extent free implicitly. This means extent free operation will no longer be executed instantaneously and the particular hang should be avoided. * 3897306 (Tracking ID: 3781085) SYMPTOM: When you work on a file system, which is short of 4 TB, and attempt to extend the file system beyond 4 TB, the command fails with the following error: UX:vxfs extendfs: ERROR: V-3-23773: With existing block size 1K, cannot extend fs beyond 4296539903 sectors. The above error must not occur on disk layout versions (DLVs) higher than DLV 5. DESCRIPTION: For DLV 5, VxFS has a check for whether the file system size is greater than the limits set for DLV 5. However, the extendfs(1M) command uses the same check for all the DLVs, instead of DLV 5. This results in an error message. RESOLUTION: The code is modified to determine the maximum file system size, according to the corresponding DLVs. * 3898606 (Tracking ID: 3873624) SYMPTOM: Memory leaks occur in FC_BUF_DEFAULT_ arena, showing buffers corresponding to mmap I/Os that have returned errors. DESCRIPTION: In case of memory mapped I/O, currently synchronous buffer having error flag set will not be freed due to glitch in the code, leading to memory leak in FC_BUF_DEFAULT_arena. RESOLUTION: The code is modified to free all synchronous buffers, irrespective of error. Patch ID: PHCO_44458, PHKL_44459, PHCO_44462 * 2038745 (Tracking ID: 2038736) SYMPTOM: In case of a clone, the Bmap operation that allocates or truncates the file change log (FCL) file may panic the system. DESCRIPTION: The extent size can be truncated during the Bmap operation on a clone FCL file. This truncated buffer size is used to allocate buffer that does not match with the real buffer size of the extent. Use of this truncated buffer may result in a panic. RESOLUTION: The code is modified to use the real extent size instead of the truncated extent size during the buffer allocation of the FCL file. * 3638176 (Tracking ID: 2104575) SYMPTOM: When the file system is disabled due to I/O errors on the disk, the page flushing to the disk may hang with following stack trace: vx_fsync_chains() vx_flush_chain() vx_add_chain() vx_putpage_dirty() vx_putpage1() vx_putpage() DESCRIPTION: When the file system is disabled and the page flushing is in progress, VxFS ignores the status of the filesystem, which results in hang of the flushing thread. RESOLUTION: The code is modified to verify the file system status, and break the page flush, if the file system is disabled. * 3703181 (Tracking ID: 3703176) SYMPTOM: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. DESCRIPTION: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. RESOLUTION: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling have been added. * 3706658 (Tracking ID: 3066116) SYMPTOM: When adb vx_inactive_throttling and vx_inactive_process_throttling tunables are set to 1, the system panics due to NULL pointer dereference with the following stack trace: ... bubbleup vx_worklist_process vx_worklist_thread ... IMPORTANT: The "vx_inactive_throttling" and "vx_inactive_process_throttling" adb tunables are for internal use and should remain at 0. Only under very special circumstances will HP/Symantec advise these tunables to be set. DESCRIPTION: To prevent too many running inactive threads, two adb tunables; "vx_inactive_throttling" and "vx_inactive_process_throttling" are introduced to fix the issue, that is, vxfsd is taking lot of CPU time after deleting some large directories. A bug in the code increments a local counter from 0 to 1. This in turn affects inactive work item dispatch. As a result, the empty work items are added to the local batch of work items. The system panics while processing this empty work item. RESOLUTION: The code is modified so that the counter is not incremented. IMPORTANT: These adb tunables are for internal use and should remain at 0 only. Under special occasions, HP/Symantec advices the customer to set these tunables. IMPORTANT: The "vx_inactive_throttling" and "vx_inactive_process_throttling" adb tunables are for internal use and should remain at 0. Only under very special circumstances will HP/Symantec advise these tunables to be set. * 3706716 (Tracking ID: 3703176) SYMPTOM: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. DESCRIPTION: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling was not available. RESOLUTION: Man pages for the tunable to enable or disable VxFS inactive-thread throttling, and VxFS inactive-thread process throttling have been added. * 3751308 (Tracking ID: 2439108) SYMPTOM: Due to the page alignment issues in the VxFS code, the system panics when the read_preferred_io tunable is set to a non-page aligned size. The following stack trace is observed: fcache_buf_create() vx_fcache_buf_create() vx_io_setup() vx_io_ext() vx_alloc_getpage() vx_do_getpage() vx_getpage1() vx_getpage() preg_vn_fault() fcache_as_fault() vx_fcache_as_fault() vx_do_read_ahead() vx_read_ahead() vx_fcache_read() vx_read1() vx_rdwr() DESCRIPTION: VxFS ends up consuming an extra page, when the preferred read I/O size is not a multiple of the page size, and runs out of the allocated pages before the getpage() function call could finish. This results in the panic. RESOLUTION: The code is modified to use the read_preferred_io tunable size only after rounding it by the page size. * 3770865 (Tracking ID: 3469644) SYMPTOM: The system panics in the vx_logbuf_clean() function, when it traverses the chain of transactions off the intent-log buffer. The stack trace is as follows: vx_logbuf_clean () vx_logadd () vx_log() vx_trancommit() vx_exh_hashinit () vx_dexh_create () vx_dexh_init () vx_pd_rename () vx_rename1_pd() vx_do_rename () vx_rename1 () vx_rename () vx_rename_skey () DESCRIPTION: The system panics in the vx_logbug_clean() function, when it tries to access an already freed transaction from the transaction chain to flush-it-to log. RESOLUTION: The code is modified to ensure that the transaction gets flushed to the log before it is freed. * 3816309 (Tracking ID: 3718924) SYMPTOM: On CFS, the file system I/O hangs for a few seconds and all the file pages are invalidated during the hang. This could occur as a routine It may cause delay in running the application processes, if the text pages of these processes are invalidated. DESCRIPTION: When VxFS intent log-ID is overflowed, a log-ID reset is required. The reset triggers a file system freeze which subsequently invalidates all the pages unnecessarily. RESOLUTION: The code is modified to avoid the unnecessary page invalidation. * 3832397 (Tracking ID: 3784126) SYMPTOM: The application experienced delays showing high vfault counters, as process text pages in memory are invalidated and need to be reloaded. For a Serviceguard cluster, the heartbeat communication is slowed down considerably because cmclds TEXT pages need to be paged in again. This results in a SG INIT failure. DESCRIPTION: When a freeze operation is handled, VxFS flushes and invalidates dirty pages of a file system. Due to a bug in the code, even read-only mmap pages, which are typically process TEXT pages, also get invalidated unnecessarily. Consider a freeze on /usr or some other file systems that host the program executables, such page invalidation can cause delays to applications as TEXT pages need to be faulted in again. RESOLUTION: The code is modified to skip the read only mmap pages invalidation during the file system freeze. * 3832588 (Tracking ID: 3832584) SYMPTOM: Files and directories on the VxFS file system are not able to inherit properly the default USER_OBJ, CLASS_OBJ and OTHER_OBJ ACL entries. DESCRIPTION: The condition set to calculate the inheritance-permission mask of the parent directory is incorrect. This subsequently results in incorrect inherited-permission for files and directories created under the parent directory. RESOLUTION: The code is modified to correct the condition that calculates the inheritance- permission mask of the parent directory. * 3859666 (Tracking ID: 3807129) SYMPTOM: When a file system that is above 2 TB in size is re-sized, a panic may occur with the following stack trace if the file system is almost full: vx_multi_bufinval() vx_alloc.c() vx_dunemap() vx_demap() vx_trancommit2() vx_trancommit () vx_trunc_tran2() vx_trunc_tran() vx_trunc() vx_inactive_remove() vx_inactive_tran() vx_local_inactive_list() vx_inactive_list () vx_worklist_process() vx_worklist_thread() kthread_daemon_startup() DESCRIPTION: When resizing a file system that is almost full, VxFS may have to temporarily steal the last 32 blocks from the intent log to grow the extent allocation maps. The intent log files organization can be switched from IORG_EXT4 to IORG_TYPED during the process. But the truncation code still assumes IORG_EXT4, causing the truncated blocks to still appear in the intent logs extent map. Subsequently, the truncated blocks will be allocated for growing the extent map, resulting in corruption as the same blocks will appear allocated to both the intent log and another file structure. The panic occurs when such corruption is detected. RESOLUTION: The code is modified to switch the intent log file to IORG_TYPED at the start of the resize operation, to ensure that the last 32 blocks should be truncated properly. * 3859667 (Tracking ID: 1482790) SYMPTOM: System may panic when the mknod() operation is performed on the file system, when VxFS data management API (DMAPI) is used. The following stack trace is observed: vx_hsm_createandmkdir() vx_create() vns_create() vn_create() mknod() mknod() syscall() DESCRIPTION: DMAPI feature is always enabled in VxFS for hierarchical storage management. As part of VxFS DMAPI code, snode for the device are handled as VxFS inodes for the mknod () operation. This results in the inappropriate memory access and panic. RESOLUTION: The code is modified such that the inode type is checked and the further processing is done only if it is a VxFS inode. * 3860290 (Tracking ID: 3841894) SYMPTOM: The ncheck(1M) command fails on the file system that has a large number of directories and the following error message is displayed: ERROR: V-3-21550: no more room for inode info buffers. DESCRIPTION: In the ncheck utility, there is a usage limit for the buffer that stores the name of the directories. This limit is set to about 12 million directories. If the number of directories exceeds this limit, the ncheck utility fails. RESOLUTION: The code is modified to increase the buffer limit to store about 24 million directories. * 3867888 (Tracking ID: 3673599) SYMPTOM: For VxFS files inheriting some ACL entry from the parent directory having a default ACL entry, the initial class permission is not set correctly to align with the file mode creation mask and the umask setting at file creation time. DESCRIPTION: When a file is created, the ACL inheritance needs to take place before applying the file mode creation mask and the umask setting, so that the latter is honored . RESOLUTION: The code is modified to honour the file mode creation mask and the umask setting when creating files with ACL inheritance. * 3867889 (Tracking ID: 3433777) SYMPTOM: A single CPU machine panics due to the safety-timer check when the inodes are re-tuned. The following stack trace is observed: spinunlock() vx_ilist_chunkclean() vx_inode_free_list() vx_retune_ninode() vx_do_inode_kmcache_callback() vx_worklist_thread () kthread_daemon_startup ( ) DESCRIPTION: When the inode cache list is traversed, the vxfsd daemon schedules a "vx_do_inode_kmcache_callback which does not free the CPU between the iterations. Thereby, the other threads cannot get access to the CPU. This results in the panic. RESOLUTION: The code is modified to use the sched_yield() function for every iteration in vx_inode_free_list to free the CPU, so that the other threads get a chance to be scheduled. * 3867894 (Tracking ID: 3150368) SYMPTOM: A periodic sync operation on an Encrypted Volume and File System (EVFS) configuration may cause the system to panic with the following stack trace: evfsevol_strategy() io_invoke_devsw() vx_writesuper() vx_fsetupdate() vx_sync1() vx_sync0() $cold_vx_do_fsext() vx_workitem_process() vx_worklist_process() vx_walk_fslist_threaded() vx_walk_fslist() vx_sync_thread() vx_worklist_thread() kthread_daemon_startup() DESCRIPTION: In the EVFS environment, EVFS may get the STALE or garbage value of b_filevp, which is not initialized by Veritas File System (VxFS) causing the system to panic. RESOLUTION: The code is modified to initialize the b_filevp. Patch ID: PHCO_44198, PHKL_44199 * 2829721 (Tracking ID: 2829708) SYMPTOM: On a locally mounted Veritas File System (VxFS) machine, the system hangs due to low memory during an internal test. DESCRIPTION: The system hangs because the memory allocated to the structure used to enqueue or dequeue work items in batches, is not freed. RESOLUTION: The code is modified to free the memory. * 3131960 (Tracking ID: 2899907) SYMPTOM: Some file-system operations on a Cluster File System (CFS) may hang with the following stack trace. vxg_svar_sleep_unlock vxg_grant_sleep vxg_cmn_lock vxg_api_lock vx_glm_lock vx_mdele_hold vx_extfree1 vx_exttrunc vx_trunc_ext4 vx_trunc_tran2 vx_trunc_tran vx_cfs_trunc vx_trunc vx_inactive_remove vx_inactive_tran vx_cinactive_list vx_workitem_process vx_worklist_process vx_worklist_thread vx_kthread_init kernel_thread DESCRIPTION: In CFS, a node can lock a mdelelock for an extent map while holding a mdelelock for a different extent map locked. This can result in a deadlock between different nodes in the cluster. RESOLUTION: The code is modified to prevent the deadlock between different nodes in the cluster. * 3464622 (Tracking ID: 2134670) SYMPTOM: The extendfs(1M) utility fails when the file system is marked for the full- file-system check (fsck). The following error message is displayed: # extendfs extendfs: /etc/default/fs is used for determining the file system type UX:vxfs fsck: ERROR: V-3-20858: cannot bmap stateino UX:vxfs extendfs: ERROR: V-3-20141: Invocation of the fsck program terminated abnormally. The file system is marked bad. Run full fsck manually. DESCRIPTION: The extendfs(1M) utility increases the file system size via the fsck. It calls the fsck with the extendfs option and the fsck in turn increases the size of the file system, and modifies the relevant data structures by calculating the new size of the various metadata. The blocks that are to be allocated for the state-inode file are calculated incorrectly. As a result, one less block is allocated and hence the fsck operation fails. The one extra block is allocated only when there is an extra byte that would not fit in the existing blocks. But there may be situations where there are two extra bits, or four to six extra bits that are less than 1 byte, for one AU. This may require an extra block. But since the extra space required in the state-inode file is less than 1 byte, it is ignored. As a result, the extendfs (1M) utility fails. RESOLUTION: The code is modified to allocate a block in the state-inode file, even when the extra space that is required is less than a byte. * 3616610 (Tracking ID: 3259634) SYMPTOM: In CFS, each node with mounted file system cluster has its own intent log in the file system. A CFS with more than 4, 294, 967, 296 file system blocks can zero out an incorrect location resulting from an incorrect typecasting. For example, that kind of CFS can incorrectly zero out 65536 file system blocks at the block offset of 1, 537, 474, 560 (file system blocks) with a 8-Kb file system block size and an intent log with the size of 65536 file system blocks. This issue can only occur if an intent log is located above an offset of 4, 294, 967, 296 file system blocks. This situation can occur when you add a new node to the cluster and mount an additional CFS secondary for the first time, which needs to create and zero a new intent log. This situation can also happen if you resize a file system or intent log and clear an intent log. The problem occurs only with the following file system size and the FS block size combinations: 1kb block size and FS size > 4TB 2kb block size and FS size > 8TB 4kb block size and FS size > 16TB 8kb block size and FS size > 32TB For example, the message log can contain the following messages: The full fsck flag is set on a file system with the following type of messages: 2013 Apr 17 14:52:22 sfsys kernel: vxfs: msgcnt 5 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol1 file system fullfsck flag set - vx_ierror 2013 Apr 17 14:52:22 sfsys kernel: vxfs: msgcnt 6 mesg 017: V-2-17: vx_attr_iget - /dev/vx/dsk/sfsdg/vol1 file system inode 13675215 marked bad incore 2013 Jul 17 07:41:22 sfsys kernel: vxfs: msgcnt 47 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol1 file system fullfsck flag set - vx_ierror 2013 Jul 17 07:41:22 sfsys kernel: vxfs: msgcnt 48 mesg 017: V-2-17: vx_dirbread - /dev/vx/dsk/sfsdg/vol1 file system inode 55010476 marked bad incore DESCRIPTION: In CFS, each node with mounted file system cluster has its own intent log in the file system. When an additional node mounts the file system as a CFS Secondary, the CFS creates an intent log. Note that intent logs are never removed, they are reused. When you clear an intent log, Veritas File System (VxFS) passes an incorrect block number to the log clearing routine, which zeros out an incorrect location. The incorrect location might point to the file data or file system metadata. Or, the incorrect location might be part of the file systems available free space. This is silent corruption. If the file system metadata corrupts, VxFS can detect the corruption when it subsequently accesses the corrupt metadata and marks the file system for full fsck. RESOLUTION: The code is modified so that VxFS can pass the correct block number to the log clearing routine. * 3616940 (Tracking ID: 2411970) SYMPTOM: The system panics at the vx_inode_trandone() function. The following stack trace is observed: vx_tranimdone() vx_trancommit2() vx_trancommit() vx_dircreate_tran() vx_do_create() vx_create1() vx_create0() vx_create() vn_create() unp_bind () bind() syscall() DESCRIPTION: A VxFS transaction is composed of multiple sub-transactions. All the sub- transactions for a specific inode are maintained in a double-linked list, that is also a part of the file-system-transaction list. The panic is due to the corruption of links between the sub-transactions. Meanwhile, the double-linked list is removed from the file-system-transaction. As a result, the system panics at the vx_inode_trandone() function. RESOLUTION: The code is modified so that VxFS handles the sub-transaction double-linked list to detect the possible race conditions and provide more diagnostic information. Patch ID: PHKL_43684, PHCO_43685, PHCO_43686 * 2705245 (Tracking ID: 2555198) SYMPTOM: On HPUX 11.31 binary mode, File Transfer Protocol (FTP) transfer uses the sendfile() interface, which does not create the DMAPI events for Hierarchical Storage Management (HSM). DESCRIPTION: The sendfile() interface does not call the Veritas File System (VxFS) read() function that creates the DMAPI events. It uses the HP Unified File Cache(UFC) interface. The UFC interface is not aware of the HSM application. As a result, the DMAPI events are not generated. RESOLUTION: The code is modified to set a flag in the vfs structure during the mount time, to indicate if the file system is configured under HSM. This flag information is used by the UFC interface to generate the DMAPI events. * 2894568 (Tracking ID: 2894560) SYMPTOM: When the internal stress test is run on CFS, the fsadm(1M) command dumps core when an invalid volume name is passed. The following stack trace is displayed: fs_dev_lookup() extchk_typed() inochk_typd() fset_inochk() reorg_inode() fset_ilist_process() do_reorg() ext_reorg() do_fsadm() main() DESCRIPTION: During a device lookup operation, the boundary check is not present on the volume ID field. When an invalid volume-ID is passed, this may result in a core dump. RESOLUTION: The code is modified to add the boundary check for volume ID field in the fs_dev_lookup() function. * 3131811 (Tracking ID: 2966277) SYMPTOM: Systems with high file-system activity like read/write/open/lookup may panic with the following stack trace due to a rare race condition: spinlock+0x21 ( ) -> vx_rwsleep_unlock() vx_ipunlock+0x40() vx_inactive_remove+0x530() vx_inactive_tran+0x450() vx_local_inactive_list+0x30() vx_inactive_list+0x420() -> vx_workitem_process() -> vx_worklist_process() vx_worklist_thread+0x2f0() kthread_daemon_startup+0x90() DESCRIPTION: ILOCK is released before doing a IPUNLOCK that causes a race condition. This results in a panic when an inode that has been set free is accessed. RESOLUTION: The code is modified so that the ILOCK is used to protect the inodes' memory from being set free, while the memory is being accessed. * 3131881 (Tracking ID: 3010444) SYMPTOM: On a Network File System (NFS) mounted file system, the operations which read the file via the cksum (1m) command may fail with the following error message: cksum: read error on : Bad address The following error messages would also be seen in the syslog vmunix: WARNING: Synchronous Page I/O error DESCRIPTION: When the read-vnode operation (VOP_RDWR) is performed, certain requests are converted to direct the I/O for optimisation. However, the NFS buffers passed during the read requests are not the user buffers. As a result, there is an error. RESOLUTION: The code is modified to convert the I/O requests to the direct I/O, only if the buffer passed during the I/O is the user buffer. * 3235161 (Tracking ID: 3235160) SYMPTOM: When a file system is unmounted when there are some file-system operations in progress, like the read/lookup operations, the system may panic with the following stack trace: panic bad_kern_reference $cold_pfault vm_hndlr bubbleup vx_getpage preg_vn_fault virtual_fault vfault vm_hndlr bubbleup DESCRIPTION: Due to the race condition between the umount thread and the page-fault handler thread, a local variable updated from the global-fs structure may contain a stale value. This leads to the panic. RESOLUTION: The code is modified to protect the local variable using the active levels. This ensures that the local variable is not changed once it is updated. * 3277010 (Tracking ID: 3226462) SYMPTOM: On a cluster mounted file-system with unequal CPUs, while doing a lookup operation, a node may panic with the stack trace: vx_dnlc_recent_cookie vx_dnlc_getpathname audit_get_pathname_from_dnlc audit_clean_path $cold_audit_build_full_dir_name inline change_p_cdir DESCRIPTION: The cause of the panic is of out-of-bounds access in the counters[] array whose size is defined by the vx_max_cpu variable. The value of vx_max_cpu can differ between the CFS nodes, if the nodes have different number of processors. However, the code assumes this value is the same across the cluster. When propagating inode cookies across the cluster, the counter[] array is allocated based on the vx_max_cpu of the current CFS node. If the cookie is populated via vx_cbdnlc_populate_cookie(), having a CPU ID from another CFS node exceeding the local vx_max_cpu, the function vx_dnlc_recent_cookie() would access locations beyond the counter[] array allocated. RESOLUTION: The code is modified to detect the out-of-bound access at vx_dnlc_recent_cookie () and return the ENOENT error. * 3277016 (Tracking ID: 3244613) SYMPTOM: A file-system extent operation by using the fsadm(1M) command may hang with the following stack trace: vx_event_wait(inlined) vx_delay2+0x2a0 cold_vx_active_common_flush+0x80 vx_close+0x70 vn_close(inlined) vno_close+0xe0 closef(inlined) DESCRIPTION: During a resize operation, the fsadm(1M) command freezes the file system. In an error case, the fsadm(1M) command exits without thawing the file system. This results in a hang. RESOLUTION: The code is modified to thaw the file system, before the fsadm(1M) command exits in the error case. * 3277028 (Tracking ID: 1960815) SYMPTOM: The vxtunefs (1M) command manpage does not mention the performance impact of tuning any of the Veritas File System (VxFS) parameters on the system with heavy load. DESCRIPTION: The performance impact that the system face on tuning any of the VxFS parameters needs to be mentioned in the vxtunefs (1M) command manpage. RESOLUTION: The manpage is updated to mention the performance impact of tuning any of the VxFS parameters on the system with heavy load. * 3277046 (Tracking ID: 3107628) SYMPTOM: The vxdump(1M) utility incorrectly estimates the number of tapes required to complete the backup and prematurely prompts for the next tape. DESCRIPTION: The vxdump(1M) utility prematurely prompts for next tape. RESOLUTION: The code is modified to fix the initialization for track and tape variables. Patch ID: PHKL_43474, PHCO_43496, PHCO_43501 * 2245466 (Tracking ID: 1934537) SYMPTOM: The reverse-name-lookup operation on an inode may panic the machine with the following stack trace: vx_free vx_traverse_tree+0x4a0 vx_dir_lookup+0x1e4 vx_rev_namelookup+0x294 vx_aioctl_common+0xac4 vx_aioctl+0x12c vx_ioctl+0xe0 DESCRIPTION: During the lookup operation if the memory allocation fails, the user still goes ahead and adds it to the used memory, and later tries to free that memory. This results in the panic. RESOLUTION: The code is modified to update the usage counters correctly, and skip updating the count during the error condition. An additional check is added to free only the non-null buffers. * 2857609 (Tracking ID: 2750860) SYMPTOM: On a large file system(4TB or greater), the performance of the write(1) operation with many small request sizes may degrade, and many threads may be found sleeping with the following stack trace: real_sleep sleep_one vx_sleep_lock vx_lockmap vx_getemap vx_extfind vx_searchau_downlevel vx_searchau_downlevel vx_searchau_downlevel vx_searchau_downlevel vx_searchau_uplevel vx_searchau vx_extentalloc_device vx_extentalloc vx_te_bmap_alloc vx_bmap_alloc_typed vx_bmap_alloc vx_write_alloc3 vx_recv_prealloc vx_recv_rpc vx_msg_recvreq vx_msg_process_thread kthread_daemon_startup DESCRIPTION: For a cluster-mounted file system, the free-extend-search algorithm is not optimized for a large file system (4TB or greater), and for instances where the number of free Allocation Units (AUs) available can be very large. RESOLUTION: The code is modified to optimize the free-extend-search algorithm by skipping certain AUs. This reduces the overall search time. * 2904380 (Tracking ID: 2599590) SYMPTOM: Expansion of a 100% full file system may panic the machine with the following stack trace. bad_kern_reference() $cold_vfault() vm_hndlr() bubbledown() vx_logflush() vx_log_sync1() vx_log_sync() vx_worklist_thread() kthread_daemon_startup() DESCRIPTION: When 100% full file system is expanded intent log of the file system is truncated and blocks freed up are used during the expansion. Due to a bug the block map of the replica intent log inode was not getting updated thereby causing the block maps of the two inodes to differ. This caused some of the in- core structures of the intent log to go NULL. The machine panics while de- referencing one of this structure. RESOLUTION: Updated the block map of the replica intent log inode correctly. 100% full file system now can be expanded only If the last extent in the intent log contains more than 32 blocks, otherwise fsadm will fail. To expand such a file-system, some of the files should be deleted manually and resize be retried. * 3024006 (Tracking ID: 2858683) SYMPTOM: The reserve-extent attributes are changed after the vxrestore(1M ) operation, for files that are greater than 8192 bytes. DESCRIPTION: A local variable is used to contain the number of the reserve bytes that are reused during the vxrestore(1M) operation, for further VX_SETEXT ioctl call for files that are greater than 8k. As a result, the attribute information is changed. RESOLUTION: The code is modified to preserve the original variable value till the end of the function. * 3079813 (Tracking ID: 2964018) SYMPTOM: On a high end machine with about 125 CPUs operations using the lstat64(2) function, may seem to be hung and the following stack trace is observed: spinlock+0xe0 rwspin_wrlock+0x30 specvp+0x510 vx_lookup+0x8a0 -> lookuppnvp(inlined) -> lookuppn(inlined) DESCRIPTION: The statvfsdev search calls the devnm() function to search the whole /dev/ directory for reverse- pathname RESOLUTION: The code is modified such that a new fs_load() function is implemented to make use of the incoming file descriptor, if it is already a character device. However, the devnm() function is still needed if the incoming file descriptor is a block device. * 3095296 (Tracking ID: 3031869) SYMPTOM: In a multi-CPU environment, the "vxfsstat -b" command does not print the correct information on the maximum-size buffer. DESCRIPTION: The "vx_bc_bufhwm" parameter represents the maximum amount of memory that can be used to cache the VxFS metadata. When the kctune(1M) command is used to tune the "vxfs_bc_bufhwm" parameter to a different value, the tunable is not set correctly due to the incorrect arithmetic. As a consequence, the "vxfsstat -b" command reports the maximum-size buffer to be increased, even though the "vxfs_bc_bufhwm" parameter is tuned to a lower value. RESOLUTION: The code is modified to correct the arithmetic for tuning the "vx_bc_bufhwm" parameter. * 3096591 (Tracking ID: 2565400) SYMPTOM: Sequential buffered I/O reads are slow in performance. DESCRIPTION: Read-Aheads are not happening because the file-system's read-ahead size gets incorrectly calculated. RESOLUTION: Fixed the incorrect typecast. * 3100058 (Tracking ID: 3099638) SYMPTOM: When the vxfs_ifree_timelag(5) tunable is tuned the following error message is displayed: # kctune vxfs_ifree_timelag=400 ERROR: mesg 095: V-2-95: Setting vxfs_ifree_timelag to 450 since the specified value for vxfs_ifree_timelag is less than the recommended minimum value of 1035 DESCRIPTION: In the vxfs_ifree_timelag(5) tunable man page, the minimum value is set to "None". The error message is displayed when the vxfs_ifree_timelag(5) tunable is set to a value which is less than 450. In the error message, a garbage value is displayed as the recommended minimum value. The error occurs because a single argument is passed for the error message that has two format specifier's. RESOLUTION: The code is modified to set the correct minimum value of the vxfs_ifree_timelag (5) tunable, and display the correct error message. * 3107775 (Tracking ID: 3099638) SYMPTOM: When the vxfs_ifree_timelag(5) tunable is tuned the following error message is displayed: # kctune vxfs_ifree_timelag=400 ERROR: mesg 095: V-2-95: Setting vxfs_ifree_timelag to 450 since the specified value for vxfs_ifree_timelag is less than the recommended minimum value of 1035 DESCRIPTION: In the vxfs_ifree_timelag(5) tunable man page, the minimum value is set to "None". The error message is displayed when the vxfs_ifree_timelag(5) tunable is set to a value which is less than 450. In the error message, a garbage value is displayed as the recommended minimum value. The error occurs because a single argument is passed for the error message that has two format specifier's. RESOLUTION: The code is modified to set the correct minimum value of the vxfs_ifree_timelag (5) tunable, and display the correct error message. Patch ID: PHKL_43128 * 2245427 (Tracking ID: 2178147) SYMPTOM: If a socket file is removed, the file system is marked for full fsck (1M) operation. The following error message is displayed in the system log: vmunix: vxfs: WARNING: msgcnt 1 mesg 087: V-2-87: vx_dotdot_manipulate: - / file system 2437 inode 541 dotdot inode error DESCRIPTION: During the socket file creation, the attribute inode for the parent directory is not updated. Hence, the error occurs when the socket file is removed. RESOLUTION: The code is modified to update the socket file linkage during creation, thus avoiding the error message. * 2800277 (Tracking ID: 2566875) SYMPTOM: The write(2) operation exceeding the quota limit fails with an EDQUOT error ("Disc quota exceeded") before the user quota limit is reached. DESCRIPTION: When a write request exceeds a quota limit, the EDQUOT error is handled so that Veritas File System (VxFS) can allocate space up to the hard quota limit to proceed with a partial write. However, VxFS does not handle this error and an erroris returned without performing a partial write. RESOLUTION: The code is modified to handle the EDQUOT error from the extent allocation routine. * 2800335 (Tracking ID: 1092933) SYMPTOM: The system may panic in the vx_fsync_chains() function when it tries to sleep in the interrupt context. The following stack trace is displayed: vx_event_wait vx_delay2 vx_fsync_chains vx_disable vx_dataioerr vx_pageio_done DESCRIPTION: While handling the external interrupt, the vx_pageio_done() function calls the function vx_fsync_chains() function. The vx_fsync_chains() function may sleep during the execution. The function vx_fsync_chains() is required in the case of Input/output (IO) errors. The function vx_fsync_chains() is called at a couple of places, when the I/O strategy fails. But, the error variable is overwritten improperly. RESOLUTION: The code is modified to save Input/Output errors so that the vx_fsync_chains() function can be called and the call to vx_fsync_chains() is removed from the vx_disable(). * 2846939 (Tracking ID: 2768505) SYMPTOM: While reading the Dynamic Name Lookup Cache (DNLC) entries using the pstat_getmpathname(2) function, the system might panic and display the following stack trace: vx_dnlc_appened_dnlcnodes vx_dnlc_getentries pstat_mpathname $cold_pstat syscall DESCRIPTION: While reading all the DNLC entries, the VxFS pointer is derived from the VNODE pointer. The VNODE pointer can be reused and de-referencing the VxFS pointer might panic the system. RESOLUTION: The code is modified to pass the VxFS pointer directly to the function instead of deriving it from the VNODE pointer. * 2857613 (Tracking ID: 2830513) SYMPTOM: The Cluster File System (CFS) hangs while performing file removal operations and the following stack trace is displayed: vxglm::vxg_grant_sleep+0x110 () vxglm::vxg_cmn_lock+0x5a0 () vxglm::vxg_api_lock+0x310 () vx_glm_lock+0x70 () vx_mdelelock+0x70 () vx_mdele_hold+0xe0 () vx_extfree+0x700 () DESCRIPTION: The CFS hangs due to a missing unlock call for the file removal operations. RESOLUTION: The code is modified to unlock the file removal operations. * 2857619 (Tracking ID: 2845175) SYMPTOM: When the Access Control List (ACL) feature is enabled, the system may panic with "Data Key Miss Fault in KERNEL mode" error message in the vx_do_getacl() function and the following stack trace is displayed: vx_do_getacl+0x840 () vx_getacl+0x70 () acl+0x480 () DESCRIPTION: In the vx_do_getacl() function, a local variable is accessed without being initializing as a result leading to a panic. RESOLUTION: The code is modified to initialize the local variable to NULL before using it. * 2876845 (Tracking ID: 2874172) SYMPTOM: Network File System (NFS) file creation thread might loop continuously with the following stack trace: vx_getblk_cmn(inlined) vx_getblk+0x3a0 vx_exh_allocblk+0x3c0 vx_exh_hashinit+0xa50 vx_dexh_create+0x240 vx_dexh_init+0x8b0 vx_do_create+0x1e0 vx_create1+0x1d0 vx_create0+0x270 vx_create+0x40 rfs3_create+0x420 common_dispatch+0xb40 rfs_dispatch+0x40 svc_getreq+0x250 svc_run+0x310 svc_do_run+0xd0 nfssys+0x6a0 hpnfs_nfssys+0x60 coerce_scall_args+0x130 syscall+0x590 DESCRIPTION: The Veritas File System (VxFS) file creation vnode operation (VOP) routine expects the parent vnode to be a directory vnode pointer. But, the NFS layer passes a stale file vnode pointer by default. This might cause unexpected results such as hang during VOP handling. RESOLUTION: The code is modified to check for the vnode type of the parent vnode pointer at the beginning of the create VOP call and return an error if it is not a directory vnode pointer. Patch ID: PHCO_42918, PHKL_42919 * 2245447 (Tracking ID: 1537731) SYMPTOM: A panic can occur when a write hits an ENOSPC whilst a re-size is in operation. DESCRIPTION: The cause of problem here is we get an ENOSPC during a write and re-size it at the same time. We drop the active level do some other task and then again raise it. The time gap between dropping and raising active level gives a window for the re-size to come in and do its work. Here, we do not update the related pointer and this results in an panic. RESOLUTION: Updated fs pointer after re-aquiring active level. * 2370062 (Tracking ID: 2370046) SYMPTOM: Readahead with read_nstream value <> 1 misses early blocks of data. DESCRIPTION: Readahead is not reading portion of the file which is expected to be read in advance. These blocks are read on demand rather than actual readahead. This is happening because the parameter determining next readahead offset is wrongly zeroed out. RESOLUTION: Changed the code which causes zeroing out of the parameter determining next readahead offset. * 2604150 (Tracking ID: 2559450) SYMPTOM: Command fsck_vxfs(1m) may core-dump with SEGV_ACCERR error. DESCRIPTION: Command fsck_vxfs(1M) is trying to allocate the memory for a structure, but memory allocation fails for the structure resulting in the segmentation fault. RESOLUTION: A check is added in the code for failed memory allocation to print an error message instead of a core dump. * 2696070 (Tracking ID: 2696067) SYMPTOM: A file created which inherits the default ACL from parent shows wrong permissions when getaccess command is issued. DESCRIPTION: vx_daccess() does not fabricate any GROUP_OBJ entry (as vx_do_getacl does). If a newly created file leverages its parent directory's ACL entries RESOLUTION: To address this, a fix is to fabricate a GROUP_OBJ entry. * 2722872 (Tracking ID: 1244756) SYMPTOM: A race condition corrupts data structures, causing a NULL pointer dereference. DESCRIPTION: There is a race condition between some DNLC functions. One of those functions is already holding a lock and wants another lock which results in a race condition. This causes the freelist corruption and a NULL pointer dereference. RESOLUTION: Updated DNLC_LOOKUP to make sure DNLC entry is not free and on freelist when moving it to tail, avoiding a race with DNLC_GET or DNLC_ENTER. * 2723003 (Tracking ID: 2670022) SYMPTOM: Duplicate file names can be seen in a directory. DESCRIPTION: VxFS maintains internal directory name lookup cache (DNLC) to improve the performance of directory lookups. A race condition is arising in DNLC lists manipulation code during lookup/creation of file names having >32 characters ( which will further affect other file creations). This is causing DNLC to have a stale entry for an existing file in the directory. A lookup of such a file through DNLC will say file as non-existent which will allow another duplicate file name in the directory. RESOLUTION: Fixed the race condition by protecting the DNLC lists through proper locks. * 2723005 (Tracking ID: 2651922) SYMPTOM: "ls -l" command on Local VxFS file system is running slow and high CPU usage is seen on HP platform. DESCRIPTION: This issue occurs when system is under inode pressure, and the most of inodes on inode free list are CFS inodes. On HP-UX platform, currently, CFS inodes are not allowed to be reused as local inodes to avoid GLM deadlock issue when vxFS reconfig is in process. So if system needs a VxFS local inode, it has to take a amount of time to loop through all the inode free lists to find a local inode, if the free lists are almost filled up with CFS inodes. RESOLUTION: 1. added a global "vxi_icache_cfsinodes" to count cfs inodes in inode cache. 2. relaxed the condition for converting cluster inode to local inode when the number of in-core cfs inodes is greater than the threshold"vx_clreuse_threshold" and reconfig is not in progress. * 2723127 (Tracking ID: 2680946) SYMPTOM: panic in vx_itryhold+0x40/spinlock() - due to NULL child pointer of dnlc entry. Data Access Rights Fault in KERNEL mode spinlock+0x40 vx_itryhold+0x40 vx_dnlc_lookup+0x1b0 vx_cbdnlc_lookup+0x130 vx_fast_lookup+0x120 vx_lookup+0x3c0 lookuppnvp+0x2d0 lookuppn+0x90 lookupname+0x60 vn_open+0xa0 copen+0x170 open+0x80 syscall+0x920 DESCRIPTION: The panic is because of NULL pointer de-reference. The reason for NULL pointer is not known for sure. One of the possibility for NULL child pointer is while inserting an entry in DNLC. RESOLUTION: We have global variable which we increment if we see the DNLC with NULL child pointer is getting inserted. So when next time issue occurs we can look value of the global and rule out this possibility. Patch ID: PHCO_42617, PHKL_42618 * 2289635 (Tracking ID: 1633670) SYMPTOM: Panic in vx_inull_list() / vx_inactive() / vx_inode_deinit() after forced unmount of writable clone and unmount of primary fileset. DESCRIPTION: This happens due to accessing of NULL pointer dereferencing. RESOLUTION: Do not iflush clone inodes which have already been force unmounted when running iflush on force unmount of a primary fileset. Beware of null vfsp pointers in partially processed force-unmounted inodes. Reference the vx_vfs struct through the fset instead of the vnode. * 2351018 (Tracking ID: 2253938) SYMPTOM: In a Cluster File System (CFS) environment , the file read performances gradually degrade up to 10% of the original read performance and the fsadm(1M) -F vxfs -D -E shows a large number (> 70%) of free blocks in extents smaller than 64k. For example, % Free blocks in extents smaller than 64 blks: 73.04 % Free blocks in extents smaller than 8 blks: 5.33 DESCRIPTION: In a CFS environment, the disk space is divided into Allocation Units (AUs).The delegation for these AUs is cached locally on the nodes. When an extending write operation is performed on a file, the file system tries to allocate the requested block from an AU whose delegation is locally cached, rather than finding the largest free extent available that matches the requested size in the other AUs. This leads to a fragmentation of the free space, thus leading to badly fragmented files. RESOLUTION: The code is modified such that the time for which the delegation of the AU is cached can be reduced using a tuneable, thus allowing allocations from other AUs with larger size free extents. Also, the fsadm(1M) command is enhanced to de-fragment free space using the -C option. * 2406458 (Tracking ID: 2283893) SYMPTOM: In a Cluster File System (CFS) environment , the file read performances gradually degrade up to 10% of the original read performance and the fsadm(1M) -F vxfs -D -E shows a large number (> 70%) of free blocks in extents smaller than 64k. For example, % Free blocks in extents smaller than 64 blks: 73.04 % Free blocks in extents smaller than 8 blks: 5.33 DESCRIPTION: In a CFS environment, the disk space is divided into Allocation Units (AUs).The delegation for these AUs is cached locally on the nodes. When an extending write operation is performed on a file, the file system tries to allocate the requested block from an AU whose delegation is locally cached, rather than finding the largest free extent available that matches the requested size in the other AUs. This leads to a fragmentation of the free space, thus leading to badly fragmented files. RESOLUTION: The code is modified such that the time for which the delegation of the AU is cached can be reduced using a tuneable, thus allowing allocations from other AUs with larger size free extents. Also, the fsadm(1M) command is enhanced to de-fragment free space using the -C option. * 2410789 (Tracking ID: 1466351) SYMPTOM: Mount hangs in vx_bc_binval_cookie like the following stack, delay vx_bc_binval_cookie vx_blkinval_cookie vx_freeze_flush_cookie vx_freeze_all vx_freeze vx_set_tunefs1 vx_set_tunefs vx_aioctl_full vx_aioctl_common vx_aioctl vx_ioctl genunix:ioctl unix:syscall_trap32 DESCRIPTION: The hanging process is waiting for a buffer to be unlocked. But that buffer can only be released if its associated cloned map writes get flushed. But a necessary flush is missed. RESOLUTION: Add code to synchronize cloned map writes so that all the cloned maps will be cleared and the buffers associated with them will be released. * 2551555 (Tracking ID: 2428964) SYMPTOM: Value of kernel tunable max_thread_proc gets incremented by 1 after every software maintenance related activity (install, remove etc.) of VRTSvxfs package. DESCRIPTION: In the postinstall script for VRTSvxfs package, value of kernel tunable max_thread_proc is wrongly increment by 1. RESOLUTION: From postinstall script increment operation of max_thread_proc tunable is removed. * 2551569 (Tracking ID: 2510903) SYMPTOM: Writing to clones loops permanently on HP-UX 11.31, there are some threads of the typical stack like following: vx_tranundo vx_logged_cwrite vx_write_clone vx_write1 vx_rdwr vno_rw inline rwuio write syscall DESCRIPTION: A VxFS write with small size can go to logged write which stores the data in intent log. The logged write can boost performance for small writes but requires the write size within logged write limit. However, When we write data to check points and the write length is greater than logged write limit, vxfs cannot proceed with logged write and retry forever. RESOLUTION: Skipped the logged write if the write size exceeds the specific limit. * 2556096 (Tracking ID: 2515380) SYMPTOM: The ff command hangs and later it exits after program exceeds memory limit with following error. # ff -F vxfs /dev/vx/dsk/bernddg/testvol UX:vxfs ff: ERROR: V-3-24347: program limit of 30701385 exceeded for directory data block list UX:vxfs ff: ERROR: V-3-20177: /dev/vx/dsk/bernddg/testvol DESCRIPTION: 'ff' command lists all files on device of vxfs file system. In 'ff' command we do directory lookup. In a function we save the block addresses for a directory. For that we traverse all the directory blocks. Then we have function which keeps track of buffer in which we read directory blocks and the extent up to which we have read directory blocks. This function is called with offset and it return the offset up to which we have read the directory blocks. The offset passed to this function has to be the offset within the extent. But, we were wrongly passing logical offset which can be greater than extent size. As a effect the offset returned gets wrapped to 0. The caller thinks that we have not read anything and hence the loop. RESOLUTION: Remove call to function which maintains buffer offsets for reading data. That call was incorrect and redundant. We actually call that function correctly from one of the functions above. * 2561752 (Tracking ID: 2561739) SYMPTOM: When the file is created and the if the parent has default ACL entry then that entry is not taken into account for calculating the class entry of that file. When a separate dummy entry added we take into account the default entry from parent as well. e.g. $ getacl . # file: . # owner: root # group: sys user::rwx group::rwx class:rwx other:rwx default:user:user_one:r-x $ touch file1 $ getacl file1 # file: try1 # owner: root # group: sys user::rw- user:user_one:r-x group::rw- class:rw- <------ other:rw- The class entry here should be rwx. DESCRIPTION: We were not taking into account the default entry of parent. We share the attribute inode with parent and do not create new attribute inode for newly created file. But when an ACL entry is explicitly made we create the separate attribute inode so the default entry also get copied in new inode and taken into consideration while returning the class entry of file. RESOLUTION: Now before returning the ACL entry buffer we calculate the class entry again and consider all the entries. * 2561757 (Tracking ID: 2492304) SYMPTOM: "find" command displays duplicate directory entries. DESCRIPTION: Whenever the directory entries can fit in the inode's immediate area VxFS doesn't allocate new directory blocks. As new directory entries get added to the directory this immediate area gets filled and all the directory entries are then moved to a newly allocated directory block. The directory blocks have space reserved at the start of the block to hold the block hash information which is used for fast lookup of entries in that block. Offset of the directory entry, which was at say x bytes in the inode's immediate area, when moved to the directory block, will be at (x + y) bytes. "y" is the size of the block hash. During this transition phase from immediate area to directory blocks, a readdir() can report a directory entry more than once. RESOLUTION: Directory entry offsets returned to the "readdir" call are adjusted so that when the entries move to a new block, they will be at the same offsets. * 2616625 (Tracking ID: 2616622) SYMPTOM: slow mmap() performace when the filesystem block size is 8k and pagesize is 4K. DESCRIPTION: When we have an 8k block size file system and 4k pages and mmap say a 8K file, the file as represented in memory ends up as two pages (0 and 1). When the memory at offset 0 into the mapping is modified, we get a page fault for page 0 in the file. However, we haven't had a page fault yet for page 1 and can't guarantee that we will in the future. When we allocate that disk block and mark it as valid, we trust that the page mentioned in the fault request will get flushed out to disk and thereby leave it uninitialized on disk by default. We clear just the page in memory and leave it dirty so we know the data in memory is more recent than the data on disk. However, the other half of the block (which could eventually be mapped to page 1) gets cleared with a synchronous write because we don't know if we will ever see a fault. This synchronous clearing of the other half of 8K block was causing performance degradation. RESOLUTION: We now expand the range of the fault to cover whole 8K block. In this case we would just ignore that the OS asked for only one page and give it two pages anyway to cover the whole file system block to save the separate synchronous clearing of the other half of 8K block. * 2619959 (Tracking ID: 1027438) SYMPTOM: Internal noise for VxFS hit "f:vx_statvfs_pri:2" assert on cluster file system DESCRIPTION: While updating summaries for Allocation units on node, summaries for all node of cluster need to be synchronized using broadcast message. The broadcast message is not sent as recovery for file system is in progress, resulting in wrong free space calculation on node causing the assert. RESOLUTION: Code is changed to send the broadcast message after file system recovery is complete. INSTALLING THE PATCH -------------------- Installing VxFS 5.0 MP1P15 patch: a)If you install this patch on a CVM cluster, install it one system at a time so that all the nodes are not brought down simultaneously. b)VxFS 5.0(GA) must be installed before applying these patches. c)To verify the VERITAS file system level, enter: # swlist -l product | egrep -i 'VRTSvxfs' VRTSvxfs 5.0.31.0 VERITAS File System Note : VRTSfsman is a corequisite for VRTSvxfs.Hence VRTSfsman also needs to be installed alongwith VRTSvxfs. # swlist -l product | egrep -i 'VRTS' VRTSvxfs 5.0.31.0 Veritas File System VRTSfsman 5.0.31.0 Veritas File System Manuals d)All prerequisite/corequisite patches have to be installed.The Kernel patch requires a system reboot for both installation and removal. e)To install the patch, enter the following command: # swinstall -x autoreboot=true -s <patch_directory> PHCO_44556 PHKL_44555 PHNE_42470 PHCO_38913 Incase the patch is not registered, the patch can be registered using the following command: # swreg -l depot <patch_directory> , where <patch_directory> is the absolute path where the patch resides. REMOVING THE PATCH ------------------ Removing VxFS 5.0 MP1P15 patches: a)To remove the patch, enter the following command: # swremove -x autoreboot=true PHCO_44556 PHKL_44555 PHCO_38913 PHNE_42470 SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE