* * * READ ME * * * * * * Symantec File System 6.2.1 * * * * * * Patch 300 * * * Patch Date: 2016-12-28 This document provides the following information: * PATCH NAME * OPERATING SYSTEMS SUPPORTED BY THE PATCH * PACKAGES AFFECTED BY THE PATCH * BASE PRODUCT VERSIONS FOR THE PATCH * SUMMARY OF INCIDENTS FIXED BY THE PATCH * DETAILS OF INCIDENTS FIXED BY THE PATCH * INSTALLATION PRE-REQUISITES * INSTALLING THE PATCH * REMOVING THE PATCH * KNOWN ISSUES PATCH NAME ---------- Symantec File System 6.2.1 Patch 300 OPERATING SYSTEMS SUPPORTED BY THE PATCH ---------------------------------------- SLES12 x86-64 PACKAGES AFFECTED BY THE PATCH ------------------------------ VRTSvxfs BASE PRODUCT VERSIONS FOR THE PATCH ----------------------------------- * Symantec File System 6.2.1 * Symantec Storage Foundation 6.2.1 * Symantec Storage Foundation Cluster File System HA 6.2.1 * Symantec Storage Foundation for Oracle RAC 6.2.1 * Symantec Storage Foundation HA 6.2.1 SUMMARY OF INCIDENTS FIXED BY THE PATCH --------------------------------------- Patch ID: 6.2.1.300 * 3817229 (3762174) fsfreeze and vxdump commands may not work together. * 3896150 (3833816) Read returns stale data on one node of the CFS. * 3896151 (3827491) Data relocation is not executed correctly if the IOTEMP policy is set to AVERAGE. * 3896154 (1428611) 'vxcompress' can spew many GLM block lock messages over the LLT network. * 3896156 (3633683) vxfs thread consumes high CPU while running an application that makes excessive sync() calls. * 3896160 (3808033) When using 6.2.1 ODM on RHEL7, Oracle resource cannot be killed after forced umount via VCS. * 3896223 (3735697) vxrepquota reports error * 3896231 (3708836) fallocate causes data corruption * 3896261 (3855726) Panic in vx_prot_unregister_all(). * 3896267 (3861271) Missing an inode clear operation when a Linux inode is being de-initialized on SLES11. * 3896269 (3879310) File System may get corrupted after a failed vxupgrade. * 3896270 (3707662) Race between reorg processing and fsadm timer thread (alarm expiry) leads to panic in vx_reorg_emap. * 3896273 (3558087) The ls -l and other commands which uses stat system call may take long time to complete. * 3896277 (3691633) Remove RCQ Full messages * 3896281 (3830300) Degraded CPU performance during backup of Oracle archive logs on CFS vs local filesystem * 3896285 (3757609) CPU usage going high because of contention over ODM_IO_LOCK * 3896303 (3762125) Directory size increases abnormally. * 3896304 (3846521) "cp -p" fails if modification time in nano seconds have 10 digits. * 3896306 (3790721) High cpu usage caused by vx_send_bcastgetemapmsg_remaus * 3896308 (3695367) Unable to remove volume from multi-volume VxFS using "fsvoladm" command. * 3896310 (3859032) System panics in vx_tflush_map() due to NULL pointer de-reference. * 3896311 (3779916) vxfsconvert fails to upgrade layout verison for a vxfs file system with large number of inodes. * 3896312 (3811849) On cluster file system (CFS), while executing lookup() function in a directory with Large Directory Hash (LDH), the system panics and displays an error. * 3896313 (3817734) Direct command to run fsck with -y|Y option was mentioned in the message displayed to user when file system mount fails. * 3896314 (3856363) Filesystem inodes have incorrect blocks. * 3901379 (3897793) Panic happens because of race where the mntlock ID is cleared while mntlock flag still set. * 3903657 (3857254) Assert failure because of missed flush before taking filesnap of the file. * 3904841 (3901318) VxFS module failed to load on RHEL7.3. * 3905056 (3879761) Performance issue observed due to contention on vxfs spin lock vx_worklist_lk. * 3906148 (3894712) ACL permissions are not inherited correctly on cluster file system. * 3906846 (3872202) VxFS internal test hits an assert. * 3906961 (3891801) Internal test hit debug assert. * 3907350 (3817734) Direct command to run fsck with -y|Y option was mentioned in the message displayed to user when file system mount fails. * 3907359 (3907722) kernel BUG at fs/dcache.c:964. Patch ID: 6.2.1.200 * 3864963 (3853338) Files on VxFS are corrupted while running the sequential asynchronous write workload under high memory pressure. * 3865600 (3865599) VxFS module failed to load on SLES12 SP1. Patch ID: 6.2.1.100 * 3753724 (3731844) umount -r option fails for vxfs 6.2. * 3754492 (3761603) Internal assert failure because of invalid extop processing at the mount time. * 3756002 (3764824) Internal cluster file system(CFS) testing hit debug assert * 3765324 (3736398) NULL pointer dereference panic in lazy unmount. * 3765998 (3759886) In case of nested mount, force umount of parent leaves stale child entry in /etc/mtab even after subsequent umount of child. * 3769992 (3729158) Deadlock due to incorrect locking order between write advise and dalloc flusher thread. * 3793241 (3793240) Vxrestore command dumps core file because of invalid japanese strings. * 3798437 (3812914) On RHEL 6.5 and RHEL 6.4 latest kernel patch, umount(8) system call hangs if an application watches for inode events using inotify(7) APIs. * 3808285 (3808284) fsdedupadm status Japanese text includes strange character. * 3817120 (3804400) VRTS/bin/cp does not return any error when quota hard limit is reached and partial write is encountered. * 3821688 (3821686) VxFS module failed to load on SLES11 SP4. Patch ID: 6.2.1.000 * 3093833 (3093821) The system panics due to referring freed super block after the vx_unmount() function errors. * 3657150 (3604071) High CPU usage consumed by the vxfs thread process. * 3657151 (3513507) Filesystem umount() may hang due to deadlock in kernel * 3657152 (3602322) System panics while flushing the dirty pages of the inode. * 3657153 (3622323) Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics with the fsadm_vxfs(1M) command. * 3657156 (3604750) The kernel loops during the extent re-org. * 3657157 (3617191) Checkpoint creation takes a lot of time. * 3657158 (3601943) Truncating corrupted block map of a file may lead to an infinite loop. * 3657159 (3633067) While converting from ext3 file system to VxFS using vxfsconvert, it is observed that many inodes are missing.. * 3665980 (2059611) The system panics due to a NULL pointer dereference while flushing bitmaps to the disk. * 3665984 (2439261) When the vx_fiostats_tunable value is changed from zero to non-zero, the system panics. * 3665990 (3567027) During the File System resize operation, the "fullfsck flag is set. * 3666007 (3594386) On RHEL6u5, stack overflows lead to system panic. * 3666008 (3616907) System is unresponsive causing the NMI watchdog service to stall. * 3666010 (3233276) With a large file system, primary to secondary migration takes longer duration. * 3683470 (3682138) The VxVM symbols are released before the VxFS modules unload time. * 3687679 (3685391) Execute permissions for a file not honored correctly. * 3697142 (3697141) Added support for Sles12 * 3697966 (3697964) The vxupgrade(1M) command fails to retain the fs_flags after upgrading a file system. * 3698165 (3690078) The system panics at vx_dev_strategy() routine due to stack overflow. * 3706864 (3709947) The SSD cache fails to go offline due to additional slashes "//" in the dev path of the cache device. * 3710794 (3710792) Unmount fails when the mount point contains a special character(colon). * 3715567 (3715566) VxFS fails to report an error when the maxlink and nomaxlink options are set on file systems having disk layout version (DLV) lower than 10. * 3716627 (3622326) During a checkpoint promote, a file system is marked with a fullfsck flag because the corresponding inode is marked as bad. * 3717895 (2919310) During stress testing on cluster file system, an assertion failure was hit because of a missing linkage between the directory and the associated attribute inode. * 3718542 (3269553) VxFS returns inappropriate message for read of hole via Oracle Disk Manager (ODM). * 3721458 (3721466) After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock. * 3726403 (3739618) sfcache command with "-i" option maynot show filesystem cache statistic periodically. * 3727166 (3727165) Enhance RHEV support for SF devices for identification in Guest * 3729704 (3719523) 'vxupgrade' retains the superblock replica of old layout versions. * 3733811 (3729030) The fsdedupschd daemon failed to start on RHEL7. * 3733812 (3729030) The fsdedupschd daemon failed to start on RHEL7. * 3737330 (3737329) Added support for RHEL7.1 * 3743913 (3743912) Users could create sub-directories more than 64K for disk layouts having versions lower than 10. * 3744425 (3744424) Rebundled the fix "Openssl: Common Vulnerability and Exposure (CVE) CVE- 2014-3566 (POODLE)" as 6.2.1 * 3745651 (3642314) Umount operation reports error code 255 in case of write-back cache. * 3749727 (3750187) Internal noise testing hits debug assert. * 3749776 (3637636) Cluster File System (CFS) node initialization and protocol upgrade may hang during the rolling upgrade. * 3755796 (3756750) VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline. DETAILS OF INCIDENTS FIXED BY THE PATCH --------------------------------------- This patch fixes the following incidents: Patch ID: 6.2.1.300 * 3817229 (Tracking ID: 3762174) SYMPTOM: When fsfreeze is used together with vxdump, the fsfreeze command gets timeout and vxdump command fails. DESCRIPTION: The vxdump command may try to read mount list file to get information of the corresponding mount points. This behavior results in taking a file system active level, in order to synchronize with file system reinit. But in case of fsfreeze, taking the active level will never succeed, since the file system is already freezed, so this causes a deadlock and finally results in the fsfreeze timeout. RESOLUTION: Don't use fsfreeze and vxdump command together. * 3896150 (Tracking ID: 3833816) SYMPTOM: In a CFS cluster, one node returns stale data. DESCRIPTION: In a 2-node CFS cluster, when node 1 opens the file and writes to it, the locks are used with CFS_MASTERLESS flag set. But when node 2 tries to open the file and write to it, the locks on node 1 are normalized as part of HLOCK revoke. But after the Hlock revoke on node 1, when node 2 takes the PG Lock grant to write, there is no PG lock revoke on node 1, so the dirty pages on node 1 are not flushed and invalidated. The problem results in reads returning stale data on node 1. RESOLUTION: The code is modified to cache the PG lock before normalizing it in vx_hlock_putdata, so that after the normalizing, the cache grant is still with node 1.When node 2 requests PG lock, there is a revoke on node 1 which flushes and invalidates the pages. * 3896151 (Tracking ID: 3827491) SYMPTOM: Data relocation is not executed correctly if the IOTEMP policy is set to AVERAGE. DESCRIPTION: Database table is not created correctly which results in an error on the database query. This affects the relocation policy of data and the files are not relocated properly. RESOLUTION: The code is modified fix the database table creation issue. Therelocation policy based calculations are done correctly. * 3896154 (Tracking ID: 1428611) SYMPTOM: 'vxcompress' command can cause many GLM block lock messages to be sent over the network. This can be observed with 'glmstat -m' output under the section "proxy recv", as shown in the example below - bash-3.2# glmstat -m message all rw g pg h buf oth loop master send: GRANT 194 0 0 0 2 0 192 98 REVOKE 192 0 0 0 0 0 192 96 subtotal 386 0 0 0 2 0 384 194 master recv: LOCK 193 0 0 0 2 0 191 98 RELEASE 192 0 0 0 0 0 192 96 subtotal 385 0 0 0 2 0 383 194 master total 771 0 0 0 4 0 767 388 proxy send: LOCK 98 0 0 0 2 0 96 98 RELEASE 96 0 0 0 0 0 96 96 BLOCK_LOCK 2560 0 0 0 0 2560 0 0 BLOCK_RELEASE 2560 0 0 0 0 2560 0 0 subtotal 5314 0 0 0 2 5120 192 194 DESCRIPTION: 'vxcompress' creates placeholder inodes (called IFEMR inodes) to hold the compressed data of files. After the compression is finished, IFEMR inode exchange their bmap with the original file and later given to inactive processing. Inactive processing truncates the IFEMR extents (original extents of the regular file, which is now compressed) by sending cluster-wide buffer invalidation requests. These invalidations need GLM block lock. Regular file data need not be invalidated across the cluster, thus making these GLM block lock requests unnecessary. RESOLUTION: Pertinent code has been modified to skip the invalidation for the IFEMR inodes created during compression. * 3896156 (Tracking ID: 3633683) SYMPTOM: "top" command output shows vxfs thread consuming high CPU while running an application that makes excessive sync() calls. DESCRIPTION: To process sync() system call vxfs scans through inode cache which is a costly operation. If an user application is issuing excessive sync() calls and there are vxfs file systems mounted, this can make vxfs sync processing thread to consume high CPU. RESOLUTION: Combine all the sync() requests issued in last 60 second into a single request. * 3896160 (Tracking ID: 3808033) SYMPTOM: After a service group is set offline via VOM or VCSOracle process is left in an unkillable state. DESCRIPTION: Whenever ODM issues an async request to FDD, FDD is required to do iodone processing on it, regardless of how far the request gets. The forced unmount causes FDD to take one of the early error branch which misses iodone routine for this particular async request. From ODM's perspective, the request is submitted, but iodone will never be called. This has several bad consequences, one of which is a user thread is blocked uninterruptibly forever, if it waits for request. RESOLUTION: The code is modified to add iodone routine in the error handling code. * 3896223 (Tracking ID: 3735697) SYMPTOM: vxrepquota reports error like, # vxrepquota -u /vx/fs1 UX:vxfs vxrepquota: ERROR: V-3-20002: Cannot access /dev/vx/dsk/sfsdg/fs1:ckpt1: No such file or directory UX:vxfs vxrepquota: ERROR: V-3-24996: Unable to get disk layout version DESCRIPTION: vxrepquota checks each mount point entry in mounted file system table. If any checkpoint mount point entry presents before the mount point specified in the vxrepquota command, vxrepquota will report errors, but the command can succeed. RESOLUTION: Skip checkpoint mount point in the mounted file system table. * 3896231 (Tracking ID: 3708836) SYMPTOM: When using fallocate together with delayed extending write, data corruption may happen. DESCRIPTION: When doing fallocate after EOF, vxfs grows the file by splitting the last extent of the file into two parts, then converts the part after EOF to a ZFOD extent. During this procedure, a stale file size is used to calculate the start offset of the newly zeroed extent. This may overwrite the blocks which contain the unflushed data generated by the extending write and cause data corruption. RESOLUTION: The code is modified to use up-to-date file size instead of the stale file size, to make sure the new ZFOD extent is created correctly. * 3896261 (Tracking ID: 3855726) SYMPTOM: Panic happens in vx_prot_unregister_all(). The stack looks like this: - vx_prot_unregister_all - vxportalclose - __fput - fput - filp_close - sys_close - system_call_fastpath DESCRIPTION: The panic is caused by a NULL fileset pointer, which is due to referencing the fileset before it's loaded, plus, there's a race on fileset identity array. RESOLUTION: Skip the fileset if it's not loaded yet. Add the identity array lock to prevent the possible race. * 3896267 (Tracking ID: 3861271) SYMPTOM: Due to the missing inode clear action, a page can also be in a strange state. Also, inode is not fully quiescent which leads to races in the inode code. Sometime this can cause panic from iput_final(). DESCRIPTION: We're missing an inode clear operation when a Linux inode is being de-initialized on SLES11. RESOLUTION: Add the inode clear operation on SLES11. * 3896269 (Tracking ID: 3879310) SYMPTOM: The file system may get corrupted after the file system freeze during vxupgrade. The full fsck gives following errors: UX:vxfs fsck: ERROR: V-3-20451: No valid device inodes found UX:vxfs fsck: ERROR: V-3-20694: cannot initialize aggregate DESCRIPTION: The vxupgrade requires file system to be frozen during it's functional operation. It may happen that the corruption can be detected while freeze is in progress and full fsck flag can be set on the file system. However, this doesn't stop vxupgrade to proceed. At later stage of vxupgrade, after structures related to new disk layout are updated on disk, vxfs frees up and zeroes out some of the old metadata inodes. If an error occurs after this point (because of full fsck being set), the file system completely needs to go back to previous version, at the tile of full fsck. Since the metadata corresponding to previous version is already cleared, the full fsck cannot proceed and gives error. RESOLUTION: Check for full fsck flag after freezing the file system during vxupgrade. Also, disable the file system if an error occurs after writing new metadata on disk. This will force the newly written metadata to be loaded in memory on next mount. * 3896270 (Tracking ID: 3707662) SYMPTOM: Race between reorg processing and fsadm timer thread (alarm expiry) leads to panic in vx_reorg_emap with the following stack:: vx_iunlock vx_reorg_iunlock_rct_reorg vx_reorg_emap vx_extmap_reorg vx_reorg vx_aioctl_full vx_aioctl_common vx_aioctl vx_ioctl fop_ioctl ioctl DESCRIPTION: When the timer expires (fsadm with -t option), vx_do_close() calls vx_reorg_clear() on local mount which performs cleanup on reorg rct inode. Another thread currently active in vx_reorg_emap() will panic due to null pointer dereference. RESOLUTION: When fop_close is called in alarm handler context, we defer the cleaning up untill the kernel thread performing reorg completes its operation. * 3896273 (Tracking ID: 3558087) SYMPTOM: When stat system call is executed on VxFS File System with delayed allocation feature enabled, it may take long time or it may cause high cpu consumption. DESCRIPTION: When delayed allocation (dalloc) feature is turned on, the flushing process takes much time. The process keeps the get page lock held, and needs writers to keep the inode reader writer lock held. Stat system call may keeps waiting for inode reader writer lock. RESOLUTION: Delayed allocation code is redesigned to keep the get page lock unlocked while flushing. * 3896277 (Tracking ID: 3691633) SYMPTOM: Remove RCQ Full messages DESCRIPTION: Too many unnecessary RCQ Full messages were logging in the system log. RESOLUTION: The RCQ Full messages removed from the code. * 3896281 (Tracking ID: 3830300) SYMPTOM: Heavy cpu usage while oracle archive process are running on a clustered fs. DESCRIPTION: The cause of the poor read performance in this case was due to fragmentation, fragmentation mainly happens when there are multiple archivers running on the same node. The allocation pattern of the oracle archiver processes is 1. write header with O_SYNC 2. ftruncate-up the file to its final size ( a few GBs typically) 3. do lio_listio with 1MB iocbs The problem occurs because all the allocations in this manner go through internal allocations i.e. allocations below file size instead of allocations past the file size. Internal allocations are done at max 8 Pages at once. So if there are multiple processes doing this, they all get these 8 Pages alternately and the fs becomes very fragmented. RESOLUTION: Added a tunable, which will allocate zfod extents when ftruncate tries to increase the size of the file, instead of creating a hole. This will eliminate the allocations internal to file size thus the fragmentation. Fixed the earlier implementation of the same fix, which ran into locking issues. Also fixed the performance issue while writing from secondary node. * 3896285 (Tracking ID: 3757609) SYMPTOM: High CPU usage because of contention over ODM_IO_LOCK DESCRIPTION: While performing ODM IO, to update some of the ODM counters we take ODM_IO_LOCK which leads to contention from multiple of iodones trying to update these counters at the same time. This is results in high CPU usage. RESOLUTION: Code modified to remove the lock contention. * 3896303 (Tracking ID: 3762125) SYMPTOM: Directory size sometimes keeps increasing even though the number of files inside it doesn't increase. DESCRIPTION: This only happens to CFS. A variable in the directory inode structure marks the start of directory free space. But when the directory ownership changes, the variable may become stale, which could cause this issue. RESOLUTION: The code is modified to reset this free space marking variable when there's ownershipchange. Now the space search goes from beginning of the directory inode. * 3896304 (Tracking ID: 3846521) SYMPTOM: cp -p is failing with EINVAL for files with 10 digit modification time. EINVAL error is returned if the value in tv_nsec field is greater than/outside the range of 0 to 999, 999, 999. VxFS supports the update in usec but when copying in the user space, we convert the usec to nsec. So here in this case, usec has crossed the upper boundary limit i.e 999, 999. DESCRIPTION: In a cluster, its possible that time across nodes might differ.so when updating mtime, vxfs check if it's cluster inode and if nodes mtime is newer time than current node time, then accordingly increment the tv_usec instead of changing mtime to older time value. There might be chance that it, tv_usec counter got overflowed here, which resulted in 10 digit mtime.tv_nsec. RESOLUTION: Code is modified to reset usec counter for mtime/atime/ctime when upper boundary limit i.e. 999999 is reached. * 3896306 (Tracking ID: 3790721) SYMPTOM: High CPU usage on the vxfs thread process. The backtrace of such kind of threads usually look like this: schedule schedule_timeout __down down vx_send_bcastgetemapmsg_remaus vx_send_bcastgetemapmsg vx_recv_getemapmsg vx_recvdele vx_msg_recvreq vx_msg_process_thread vx_kthread_init kernel_thread DESCRIPTION: The locking mechanism in vx_send_bcastgetemapmsg_process() is inefficient. So that every time vx_send_bcastgetemapmsg_process() is called, it will perform a series of down-up operation on a certain semaphore. This can result in a huge CPU cost when multiple threads have contention on this semaphore. RESOLUTION: Optimize the locking mechanism in vx_send_bcastgetemapmsg_process(), so that it only do down-up operation on the semaphore once. * 3896308 (Tracking ID: 3695367) SYMPTOM: Unable to remove volume from multi-volume VxFS using "fsvoladm" command. It fails with "Invalid argument" error. DESCRIPTION: Volumes are not being added in the in-core volume list structure correctly. Therefore while removing volume from multi-volume VxFS using "fsvoladm", command fails. RESOLUTION: The code is modified to add volumes in the in-core volume list structure correctly. * 3896310 (Tracking ID: 3859032) SYMPTOM: System panics in vx_tflush_map() due to NULL pointer dereference. DESCRIPTION: When converting VxFS using vxconvert, new blocks are allocated to the structural files like smap etc which can contain garbage. This is done with the expectation that fsck will rebuild the correct smap. but in fsck, we have missed to distinguish between EAU fully EXPANDED and ALLOCATED. because of which, if allocation to the file which has the last allocation from such affected EAU is done, it will create the sub transaction on EAU which are in allocated state. Map buffers of such EAUs are not initialized properly in VxFS private buffer cache, as a result, these buffers will be released back as stale during the transaction commit. Later, if any file-system wide sync tries to flush the metadata, it can refer to these buffer pointers and panic as these buffers are already released and reused. RESOLUTION: Code is modified in fsck to correctly set the state of EAU on disk. Also, modified the involved code paths as to avoid using doing transactions on unexpanded EAUs. * 3896311 (Tracking ID: 3779916) SYMPTOM: vxfsconvert fails to upgrade layout verison for a vxfs file system with large number of inodes. Error message will show some inode discrepancy. DESCRIPTION: vxfsconvert walks through the ilist and converts inode. It stores chunks of inodes in a buffer and process them as a batch. The inode number parameter for this inode buffer is of type unsigned integer. The offset of a particular inode in the ilist is calculated by multiplying the inode number with size of inode structure. For large inode numbers this product of inode_number * inode_size can overflow the unsigned integer limit, thus giving wrong offset within the ilist file. vxfsconvert therefore reads wrong inode and eventually fails. RESOLUTION: The inode number parameter is defined as unsigned long to avoid overflow. * 3896312 (Tracking ID: 3811849) SYMPTOM: On cluster file system (CFS), due to a size mismatch in the cluster-wide buffers containing hash bucket for large directory hashing (LDH), the system panics with the following stack trace: vx_populate_bpdata() vx_getblk_clust() vx_getblk() vx_exh_getblk() vx_exh_get_bucket() vx_exh_lookup() vx_dexh_lookup() vx_dirscan() vx_dirlook() vx_pd_lookup() vx_lookup_pd() vx_lookup() On some platforms, instead of panic, LDH corruption is reported. Full fsck reports some meta-data inconsistencies as displayed in the following sample messages: fileset 999 primary-ilist inode 263 has invalid alternate directory index (fileset 999 attribute-ilist inode 8193), clear index? (ynq)y DESCRIPTION: On a highly fragmented file system with a file system block size of 1K, 2K or 4K, the bucket(s) of an LDH inode, which has a fixed size of 8K, can spread across multiple small extents. Currently in-core allocation for bucket of LDH inode happens in parallel to on-disk allocation, which results in small in-core buffer allocations. Combination of these small in-core allocations will be merged for final in memory representation of LDH inodes bucket. On two Cluster File System (CFS) nodes, this may result in same LDH metadata/bucket represented as in-core buffers of different sizes. This may result in system panic as LDH inodes bucket are passed around the cluster, or this may result in on-disk corruption of LDH inode's buckets, if these buffers are flushed to disk. RESOLUTION: The code is modified to separate the on-disk allocation and in-core buffer initialization in LDH code paths, so that in-core LDH bucket will always be represented by a single 8K buffer. * 3896313 (Tracking ID: 3817734) SYMPTOM: If file system with full fsck flag set is mounted, direct command message is printed to the user to clean the file system with full fsck. DESCRIPTION: When mounting file system with full fsck flag set, mount will fail and a message will be printed to clean the file system with full fsck. This message contains direct command to run, which if run without collecting file system metasave will result in evidences being lost. Also since fsck will remove the file system inconsistencies it may lead to undesired data being lost. RESOLUTION: More generic message is given in error message instead of direct command. * 3896314 (Tracking ID: 3856363) SYMPTOM: vxfs reports mapbad errors in the syslog as below: vxfs: msgcnt 15 mesg 003: V-2-3: vx_mapbad - vx_extfind - /dev/vx/dsk/vgems01/lvems01 file system free extent bitmap in au 0 marked bad. And, full fsck reports following metadata inconsistencies: fileset 999 primary-ilist inode 6 has invalid number of blocks (18446744073709551583) fileset 999 primary-ilist inode 6 failed validation clear? (ynq)n pass2 - checking directory linkage fileset 999 directory 8192 block devid/blknum 0/393216 offset 68 references free inode ino 6 remove entry? (ynq)n fileset 999 primary-ilist inode 8192 contains invalid directory blocks clear? (ynq)n pass3 - checking reference counts fileset 999 primary-ilist inode 5 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 5 clear? (ynq)n fileset 999 primary-ilist inode 8194 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 8194 clear? (ynq)n fileset 999 primary-ilist inode 8195 unreferenced file, reconnect? (ynq)n fileset 999 primary-ilist inode 8195 clear? (ynq)n pass4 - checking resource maps DESCRIPTION: While processing the VX_IEZEROEXT extop, VxFS frees the extent without setting VX_TLOGDELFREE flag. Similarly, there are other cases where the flag VX_TLOGDELFREE is not set in the case of the delayed extent free, this could result in mapbad errors and invalid block counts. RESOLUTION: Since the flag VX_TLOGDELFREE need to be set on every extent free, modified to code to discard this flag and treat every extent free as delayed extent free implicitly. * 3901379 (Tracking ID: 3897793) SYMPTOM: Panic happens because of race where the mntlock ID is cleared while mntlock flag still set. DESCRIPTION: Panic happened because of race where mntlockid is null even after mntlock flag is set. Race is between fsadm thread and proc mount show_option thread. The fsadm thread deintialize mntlock id first and then removes mntlock flag. If other thread race with this fsadm thread, then it is possible to have mntlock flag set and mntlock id as a NULL. The fix is to remove flag first and deintialize mntlock id later. RESOLUTION: The code is modified to remove mntlock flag first. * 3903657 (Tracking ID: 3857254) SYMPTOM: Assert failure because of missed flush before taking filesnap of the file. DESCRIPTION: If the delayed extended write on the file is not completed but the snap of the file is taken, then the inode size is not updated correctly. This will trigger internal assert because of incorrect inode size. RESOLUTION: The code is modified to flush the delayed extended write before taking filesnap. * 3904841 (Tracking ID: 3901318) SYMPTOM: VxFS module failed to load on RHEL7.3. DESCRIPTION: Since RHEL7.3 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for RHEL7.3. * 3905056 (Tracking ID: 3879761) SYMPTOM: Performance issue observed due to contention on vxfs spin lock vx_worklist_lk. DESCRIPTION: ODM IOs are performed asynchronously, by queuing the ODM work items to the worker threads. It wakes up more number of worker threads than required after enqueuing the ODM work items which leads to contention of vx_worklist_lk spinlock. RESOLUTION: Modified the code such that, it will wake up one worker thread if only one workitem is enqueued. * 3906148 (Tracking ID: 3894712) SYMPTOM: ACL permissions are not inherited correctly on cluster file system. DESCRIPTION: The ACL counts stored on a directory inode gets reset every time directory inodes ownership is switched between the nodes. When ownership on directory inode comes back to the node, which previously abdicated it, ACL permissions were not getting inherited correctly for the newly created files. RESOLUTION: Modified the source such that the ACLs are inherited correctly. * 3906846 (Tracking ID: 3872202) SYMPTOM: VxFS internal test hits an assert. DESCRIPTION: In page create case VxFS was taking the ipglock twice in a thread, due to which the VxFS test hit the internal assert. RESOLUTION: Removed the ipglock from vx_wb_dio_write(). * 3906961 (Tracking ID: 3891801) SYMPTOM: Internal test hit debug assert. DESCRIPTION: Got an debug assert while creating page in shared page cache for zfod extent which is same as creating for HOLEs, which VxFS don't do. RESOLUTION: Added a check for page creation so that we don't create shared pages for zfod extent. * 3907350 (Tracking ID: 3817734) SYMPTOM: If file system with full fsck flag set is mounted, direct command message is printed to the user to clean the file system with full fsck. DESCRIPTION: When mounting file system with full fsck flag set, mount will fail and a message will be printed to clean the file system with full fsck. This message contains direct command to run, which if run without collecting file system metasave will result in evidences being lost. Also since fsck will remove the file system inconsistencies it may lead to undesired data being lost. RESOLUTION: More generic message is given in error message instead of direct command. * 3907359 (Tracking ID: 3907722) SYMPTOM: kernel BUG at fs/dcache.c:964 DESCRIPTION: There is a race between vxfs dcache pruner and umount thread in RHEL7/SLES12 kernel. This race causes kernel BUG at fs/dcache.c:964. RESOLUTION: Added code to close the race between vxfs dcache pruner and umount thread. Patch ID: 6.2.1.200 * 3864963 (Tracking ID: 3853338) SYMPTOM: Files on VxFS are corrupted while running the sequential write workload under high memory pressure. DESCRIPTION: VxFS may miss out writes sometimes under excessive write workload. Corruption occurs because of the race between the writer thread which is doing sequential asynchronous writes and the flusher thread which flushes the in-core dirty pages. Due to an overlapping write, they are serialized over a page lock. Because of an optimization, this lock is released, leading to a small window where the waiting thread could race. RESOLUTION: The code is modified to fix the race by reloading the inode write size after taking the page lock. * 3865600 (Tracking ID: 3865599) SYMPTOM: VxFS module failed to load on SLES12 SP1. DESCRIPTION: Since SLES12 SP1 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES12 SP1. Patch ID: 6.2.1.100 * 3753724 (Tracking ID: 3731844) SYMPTOM: umount -r option fails for vxfs 6.2 with error "invalid options" DESCRIPTION: Till 6.2 vxfs did not have a umount helper on linux. We added a helper in 6.2, because of this, each call to linux's umount also gets called to the umount helper binary. Due to this the -r option, which was only handled by the linux native umount, is forwarded to the umount.vxfs helper, which exits while processing the option string becase we don't support readonly remounts. RESOLUTION: To solve this, we've changed the umount.vxfs code to not exit on "-r" option, although we do not support readonly remounts, so if umount -r actually fails and the os umount attempts a readonly remount, the mount.vxfs binary will then exit with an error. This solves the problem of linux's default scripts not working for our fs. * 3754492 (Tracking ID: 3761603) SYMPTOM: Full fsck flag will be set incorrectly at the mount time. DESCRIPTION: There might be possibility that extop processing will be deferred during umount (i.e. in case of crash or disk failure) and will be kept on disk, so that mount can process them. During mount, inode can have multiple extop set. Previously if inode has trim and reorg extop set during mount, we were incorrectly setting fullfsck. This patch avoids this situation. RESOLUTION: Code is modified to avoid such unnecessary setting of fullfsck. * 3756002 (Tracking ID: 3764824) SYMPTOM: Internal cluster file system(CFS) testing hit debug assert DESCRIPTION: Internal debug assert is seen when there is a glm recovery while one of the secondary nodes is doing mount, specifically when glm recovery happens between attaching a file system and mounting file system. RESOLUTION: Code is modified to handle glm reconfiguration issue. * 3765324 (Tracking ID: 3736398) SYMPTOM: Panic in the lazy unmount path during deinit of VxFS-VxVM API. DESCRIPTION: The panic is caused when an exiting thread drops the last reference to a lazy-unmounted VxFS file-system where that fs is the last VxFS mount in the system. The exiting thread does unmount, which then calls into VxVM to de-initialize the private FS-VM API(as it is the last VxFS mounted fs). The function to be called in VxVM is looked-up via the files under /proc, this requires an opening of a file but the exit processing has removed the structs needed by the thread to open a file. RESOLUTION: The solution is to cache the de-init function (vx_fsvm_api_deinit) when the VxFS-VxVM API is initialized, so no function look-up is needed during an unmount. The cached function pointer can then be called during the last unmount bypassing the need to open the file by the exiting thread. * 3765998 (Tracking ID: 3759886) SYMPTOM: In case of nested mount, force umount of parent leaves stale child entry in /etc/mtab even after subsequent umount of child. DESCRIPTION: On rhel6 and sles11, in case of nested mount, if parent mount (say /mnt1) was removed/umounted forcefully, then child mounts (like /mnt1/dir) also get umounted but the "/etc/mtab" entry was not getting updated accordingly for child mount. Previously it was possible to remove such child entries from "/etc/mtab" by using os's umount binary. But from shikra on words, we have added helper umount binary in "/sbin/umount.vxfs". So now os's umount binary will call this helper binary which in turn call vxumount for child umount which will fail since path was not present. Hence mtab entry will not get updated and will show child as mounted. RESOLUTION: Code is modified to update mnttab when ENOENT error is returned by umount() system call. * 3769992 (Tracking ID: 3729158) SYMPTOM: fuser and other commands hang on vxfs file systems. DESCRIPTION: The hang is seen while 2 threads contest for 2 locks -ILOCK and PLOCK. The writeadvise thread owns the ILOCK but is waiting for the PLOCK. The dalloc thread owns the PLOCK and is waiting for the ILOCK. RESOLUTION: Correct order of locking is PLOCK followed by the ILOCK. * 3793241 (Tracking ID: 3793240) SYMPTOM: Vxrestore command dumps core file because of invalid japanese strings. DESCRIPTION: Vxrestore command dumps core file because of invalid characters such as %, $ etc. are present in the japanese strings. RESOLUTION: code is modified to remove the extra characters from the Japanese message strings. * 3798437 (Tracking ID: 3812914) SYMPTOM: On RHEL 6.5 and RHEL 6.4 latest kernel patch, umount(8) system call hangs if an application watches for inode events using inotify(7) APIs. DESCRIPTION: On RHEL 6.5 and RHEL 6.4 latest kernel patch, additional OS counters were added in the super block to track inotify Watches. These new counters were not implemented in VxFS for RHEL6.5/RHEL6.4 kernel. Hence, while doing umount, the operation hangs until the counter in the superblock drops to zero, which would never happen since they are not handled in VxFS. RESOLUTION: The code is modified to handle additional counters added in super block of RHEL6.5/RHEL6.4 latest kernel. * 3808285 (Tracking ID: 3808284) SYMPTOM: fsdedupadm status Japanese text includes strange character. DESCRIPTION: The translation of FAILED string in English is incorrect in Japanese and is "I/O" which stands for The failed I / O.So the translation from English to Japanese is not correct. RESOLUTION: Corrected the translation for "FAILED" string in Japanese. * 3817120 (Tracking ID: 3804400) SYMPTOM: VRTS/bin/cp does not return any error when quota hard limit is reached and partial write is encountered. DESCRIPTION: When quota hard limit is reached, VRTS/bin/cp may encounter a partial write, but it may not return any error to up layer application in such situation. RESOLUTION: Adjust VRTS/bin/cp to detect the partial write caused by quota limit, and return a proper error to up layer application. * 3821688 (Tracking ID: 3821686) SYMPTOM: VxFS module might not get loaded on SLES11 SP4. DESCRIPTION: Since SLES11 SP4 is new release therefore VxFS module failed to load on it. RESOLUTION: Added VxFS support for SLES11 SP4. Patch ID: 6.2.1.000 * 3093833 (Tracking ID: 3093821) SYMPTOM: The system panics due to referring freed super block after the vx_unmount() function errors. DESCRIPTION: In the file system unmount process, once Linux VFS calls in to VxFS specific unmount processing vx_unmount(), it doesn't expect error from this call. So, once the vx_unmount()function returns, Linux frees the file systems corresponding super_block object. But if any error is observed during vx_unmount(), it may leave file system Inodes in vxfs inode cache as it is. Otherwise, when there is no error, the file system inodes are processed and dropped on vx_unmount(). As file systems inodes left in VxFS inode cache would still point to freed super block object, so now when these inodes on Inode cache are cleaned up to free or reuse, they may refer freed super block in certain cases, which might lead to Panic due to NULL pointer de-reference. RESOLUTION: Do not return EIO or ENXIO error from vx_detach_fset() when you unmounts the filesystem. Insted of returning error process, drop inode from the inode cache. * 3657150 (Tracking ID: 3604071) SYMPTOM: With the thin reclaim feature turned on, you can observe high CPU usage on the vxfs thread process. The backtrace of such kind of threads usually look like this: - vx_dalist_getau - vx_recv_bcastgetemapmsg - vx_recvdele - vx_msg_recvreq - vx_msg_process_thread - vx_kthread_init DESCRIPTION: In the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, the locking mechanism is inefficient. Thus every time when this routine is called, it will perform a series of down-up operation on a certain semaphore. This can result in a huge CPU cost when many threads calling the routine in parallel. RESOLUTION: The code is modified to optimize the locking mechanism in the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, so that it only does down-up operation on the semaphore once. * 3657151 (Tracking ID: 3513507) SYMPTOM: Filesystem umount() may hang due to deadlock in kernel with the following stack trace and other threads(denoted as T1): vx_async_waitmsg vx_msg_broadcast vx_cwfa_step vx_cwfreeze_common vx_cwfreeze_all vx_freeze vx_detach_fset vx_unmount generic_shutdown_super sys_umount In the meanwhile, a thread(T2) is stuck at down_read: rwsem_down_failed_common rwsem_down_read_failed call_rwsem_down_read_failed down_read do_page_fault page_fault filldir64 vx_readdir_int mntput_no_expire vx_irwlock2 vx_readdir vfs_readdir sys_getdents64 A thread(T3) is waiting for down_write rwsem_down_failed_common rwsem_down_write_failed call_rwsem_down_write_failed down_write sys_munmap A thread(T4) is doing vx_read? vx_active_common_flush vx_read activate_task vfs_read sys_read system_call_fastpath DESCRIPTION: This is a dead lock which involves multiple threads. When the umount thread(T1) is initiated, it blocks all file operations from coming into the filesystem and waits for those active threads to drop the active levels they have. One of the active thread(T2) hits a pagecache which attempts a rw semaphore of the process memory map structure with read mode. However, this semaphore is held by another thread(T4) with read mode, whilst a write request(T3) is waiting in the queue that prevents other read requests from getting the semaphore. The current read user of the semaphore(T4) is requiring the active level of the file system while holding read semaphore. But the file system is frozen by the first thread(T1). RESOLUTION: The code is modified to adjust the active level to avoid the deadlock. * 3657152 (Tracking ID: 3602322) SYMPTOM: System may panic while flushing the dirty pages of the inode. The following stack traces are observed: vx_iflush_list() vx_workitem_process() vx_worklist_process() vx_worklist_thread() and vx_vn_cache_deinit() vx_inode_deinit vx_ilist_chunkclean() vx_inode_free_list() vx_ifree_scan_list() vx_workitem_process() vx_worklist_process() vx_worklist_thread() DESCRIPTION: Panic may occur due to the synchronization problem between one thread that flushes the inode, and the other thread that frees the chunks that contain the inodes on the freelist. The thread that frees the chunks of inodes on the freelist grabs an inode, and clears/dereference the inode pointer while deinitializing the inode. This may result in the pointer dereference, if the flusher thread is working on the same inode. RESOLUTION: The code is modified to resolve the race condition by taking proper locks on the inode and freelist, whenever a pointer in the inode is dereferenced. If the inode pointer is already de-initialized to NULL, then the flushing is attempted on the next inode. * 3657153 (Tracking ID: 3622323) SYMPTOM: Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics using the fsadm_vxfs(1M) command with the following stack: - vx_irwlock - vx_clust_fset_curused - vx_getcompstats - vx_aioctl_getcompstats - vx_aioctl_common - vx_aioctl - vx_unlocked_ioctl - vx_ioctl - vfs_ioctl - do_vfs_ioctl - sys_ioctl - system_call_fastpath DESCRIPTION: When file system is mounted as read-only, part of the initial setup is skipped, including loading of few internal structures. These structures are referenced while gathering statistics for sharing and/or compression. As a result, panic occurs. RESOLUTION: The code is modified to only allow "fsadm -HS all" to gather sharing and/or compression statistics on read-write file systems. On read-only file systems, this command fails. * 3657156 (Tracking ID: 3604750) SYMPTOM: The kernel loops during the extent re-org with the following stack trace: vx_bmap_enter() vx_reorg_enter_zfod() vx_reorg_emap() vx_extmap_reorg() vx_reorg() vx_aioctl_full() $cold_vx_aioctl_common() vx_aioctl() vx_ioctl() vno_ioctl() ioctl() syscall() DESCRIPTION: The extent re-org minimizes the file system fragmentation. When the re-org request is issued for an inode with a lot of ZFOD extents, it reallocates the extents of the original inode to the re-org inode. During this, the ZFOD extent are preserved and enter the re-org inode in a transaction. If the extent allocated is big, the transaction that enters the ZFOD extents becomes big and returns an error. Even when the transaction is retried the same issue occurs. As a result, the kernel loops during the extent re-org. RESOLUTION: The code is modified to enter the Bmap (block map) of the allocated extent and then perform the ZFOD processing. If you get a committable error during the ZFOD enter, then commit the transaction and continue with the ZFOD enter. * 3657157 (Tracking ID: 3617191) SYMPTOM: Checkpoint creation may take hours. DESCRIPTION: During checkpoint creation, with an inode marked for removal and being overlaid, there may be a downstream clone and VxFS starts pulling all the data. With Oracle it's evident because of temporary files deletion during checkpoint creation. RESOLUTION: The code is modified to selectively pull the data, only if a downstream push inode exists for file. * 3657158 (Tracking ID: 3601943) SYMPTOM: The block map tree of a file is corrupted across the levels, and during truncating, inode for the file may lead to an infinite loop. There are various DESCRIPTION: For files larger than 64G, truncation code first walks through the bmap tree to find the optimal offset from which to begin the truncation. If this truncation falls within corrupted range of the bmap, actual truncation code which relies on binary search to find this offset. As a result, the truncation cannot find the offset, thus it returns empty. The output makes the truncation code to submit dummy transaction, which updates the inode of file with latest ctime, without freeing the extents allocated. RESOLUTION: The truncation code is modified to detect the corruption, mark the inode bad and, mark the file system for full-fsck. The modification makes the truncation possible for full fsck. Next time when it runs, the truncation code is able to throw out the inode and free the extents. * 3657159 (Tracking ID: 3633067) SYMPTOM: While converting from ext3 file system to VxFS using vxfsconvert, it is observed that many inodes are missing. DESCRIPTION: When vxfsconvert(1M) is run on an ext3 file system, it misses an entire block group of inodes. This happens because of an incorrect calculation of block group number of a given inode in border case. The inode which is the last inode for a given block group is calculated to have the correct inode offset, but is calculated to be in the next block group. This causes the entire next block group to be skipped when the code attempts to find the next consecutive inode. RESOLUTION: The code is modified to correct the calculation of block group number. * 3665980 (Tracking ID: 2059611) SYMPTOM: The system panics due to a NULL pointer dereference while flushing the bitmaps to the disk and the following stack trace is displayed: vx_unlockmap+0x10c vx_tflush_map+0x51c vx_fsq_flush+0x504 vx_fsflush_fsq+0x190 vx_workitem_process+0x1c vx_worklist_process+0x2b0 vx_worklist_thread+0x78 DESCRIPTION: The vx_unlockmap() function unlocks a map structure of the file system. If the map is being used, the hold count is incremented. The vx_unlockmap() function attempts to check whether this is an empty mlink doubly linked list. The asynchronous vx_mapiodone routine can change the link at random even though the hold count is zero. RESOLUTION: The code is modified to change the evaluation rule inside the vx_unlockmap() function, so that further evaluation can be skipped over when map hold count is zero. * 3665984 (Tracking ID: 2439261) SYMPTOM: When the vx_fiostats_tunable is changed from zero to non-zero, the system panics with the following stack trace: vx_fiostats_do_update vx_fiostats_update vx_read1 vx_rdwr vno_rw rwuio pread DESCRIPTION: When vx_fiostats_tunable is changed from zero to non-zero, all the incore-inode fiostats attributes are set to NULL. When these attributes are accessed, the system panics due to the NULL pointer dereference. RESOLUTION: The code has been modified to check the file I/O stat attributes are present before dereferencing the pointers. * 3665990 (Tracking ID: 3567027) SYMPTOM: During the File System resize operation, the "fullfsck flag is set with the following message:vxfs: msgcnt 183168 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol file system fullfsck flag set - vx_fs_upgrade_reorg DESCRIPTION: File system resize requires some temporary inodes to swap the old inode and the converted inode. However, before a structural inode ise processed, the "fullfsck flag is set when a failure occurs during the metadata change. The flag is cleared after the swap is successfully completed. If the temporary inode allocation fails, VxFS leaves the fullfsck flag on the disk. However, all temporary inodes can be cleaned up when not being in use, thus these temporary inodes do not result in corruption. RESOLUTION: The code is modified to clear the fullfsck flag if the structural inode conversion cannot create its temporary inode. * 3666007 (Tracking ID: 3594386) SYMPTOM: On installing VxFS 5.1SP1RP4P1HF1 on RHEL6u5, system panic occurs when ftrace(1M) is enabled and multiple files are created when large amount of space is consumed. DESCRIPTION: The system panic occurs when the stack overflow happens through the bio_alloc() function that was initiated through a routine. The routine is where the bio_alloc() function exits VxFS and call through the block layer. RESOLUTION: The code is modified by adding a new handoff routine that would hand off the bio_alloc() task to a new worker thread in case the remaining stack was insufficient to proceed. * 3666008 (Tracking ID: 3616907) SYMPTOM: While performing the garbage collection operation, VxFS causes the non-maskable interrupt (NMI) service to stall. DESCRIPTION: With a highly fragmented Reference Count Table (RCT), when a garbage collection operation is performed, the CPU could be used for a longer duration. The CPU could be busy if a potential entry that could be freed is not identified. RESOLUTION: The code is modified such that the CPU is released after a when it is idle after a specified time interval. * 3666010 (Tracking ID: 3233276) SYMPTOM: On a 40 TB file system, the fsclustadm setprimary command consumes more than 2 minutes for execution. And, the unmount operation consumes more time causing a primary migration. DESCRIPTION: The old primary needs to process the delegated allocation units while migrating from primary to secondary. The inefficient implementation of the allocation unit list is consuming more time while removing the element from the list. As the file system size increases, the allocation unit list also increases, which results in additional migration time. RESOLUTION: The code is modified to process the allocation unit list efficiently. With this modification, the primary migration is completed in 1 second on the 40 TB file system. * 3683470 (Tracking ID: 3682138) SYMPTOM: The VxVM symbols are released before the VxFS modules unload time. DESCRIPTION: On RHEL7, the VxFS module receives or adds VxVM symbol as follows: 1. Get or imports all required VxVM symbols on first access or on first mount. Refcnt/hold for VxVM module is incremented through VEKI on symbol_get request so that module doesnt get unloaded. 2. The symbols (imported in step 1) are used as required. 3. VxFS puts/releases VxVM symbols during the VxFS unload time. Refcnt is decremented/hold is released. As a result, VxVM module unloads until the VxFS module is unloaded. Manual upgrade of VxVM pkg cannot happen until the VxFS modules are unloaded RESOLUTION: The code is modified to release VxVM symbols before the VxFS modules unload time. * 3687679 (Tracking ID: 3685391) SYMPTOM: Execute permissions for a file not honored correctly. DESCRIPTION: The user was able to execute the file regardless of not having the execute permissions. RESOLUTION: The code is modified such that an error is reported when the execute permissions are not applied. * 3697142 (Tracking ID: 3697141) SYMPTOM: Added support for Sles12 DESCRIPTION: Added support for Sles12 RESOLUTION: Added support for Sles12 * 3697966 (Tracking ID: 3697964) SYMPTOM: When a file system is upgraded, the upgraded layout clears the superblock flags (fs_flags). DESCRIPTION: When a file system is upgraded, the new superblock structure gets populated with the field values. Most of these values are inherited from the old superblock. As a result, the fs_flags values are overwritten and the flags such as VX_SINGLEDEV are deleted from the superblock. RESOLUTION: The code is modified to restore the old superblock flags while upgrading the disk layout of a file system. * 3698165 (Tracking ID: 3690078) SYMPTOM: The system panics at vx_dev_strategy() routine with the following stack trace: vx_snap_strategy() vx_logbuf_write() vx_logbuf_io() vx_logbuf_flush() vx_logflush() vx_mapstrategy() vx_snap_strategy() vx_clonemap() vx_unlockmap() vx_holdmap() vx_extmaptran() vx_extmapchange() vx_extprevfind() vx_extentalloc() vx_te_bmap_alloc() vx_bmap_alloc_typed() vx_bmap_alloc() vx_get_alloc() vx_cfs_pagealloc() vx_alloc_getpage() vx_do_getpage() vx_internal_alloc() vx_write_alloc() vx_write1() vx_write_common_slow() vx_write_common() vx_vop_write() vx_writev() vx_naio_write_v2() vfs_writev() DESCRIPTION: The issue was observed due to low handoff limit of vx_extprevfind. RESOLUTION: The code is modified to avoid the stack overflow. * 3706864 (Tracking ID: 3709947) SYMPTOM: The SSD cache fails to go offline due to additional slashes "//" in the dev path of the cache device. DESCRIPTION: The conv_to_bdev() function authenticates whether a path contains rdsk (character device) or dsk (disk device). If rdsk is present, the fs_convto_bspec() function calls the fs_pathcanon() function to remove the additional slashes. In case the path contains dsk (disk device) then VxFS refuses to call the functions. As a result, the additional slashes appear in the final output path. RESOLUTION: The code is modified to enable VxFS to call the correct functions. * 3710794 (Tracking ID: 3710792) SYMPTOM: Unmount fails when the mount point contains a special character (colon). DESCRIPTION: For clones, the pseudo device name must contain <devicepath>:<clone name>. Thus, when using the cloned block device, the umount codepath performs an exclusive processing to unmount the device. Sometimes, the mount point itself can be created using the special character (colon). As a result, the mount point splits at the colon. RESOLUTION: The code is modified to unsplit the mount point if it is not a block device and it is a real mount point that can be searched in the mounted file systems table (mtab). * 3715567 (Tracking ID: 3715566) SYMPTOM: VxFS fails to report an error when the maxlink and nomaxlink options are set for disk layout version (DLV) lower than 10. DESCRIPTION: The maxlink and nomaxlink options allow you to enable and disable the maxlink support feature respectively. The maxlink support feature operates only on DLV version 10. Due to an issue, the maxlink and nomaxlink options wrongly appear in DLV versions lower than 10. However, when selected the options do not take effect. RESOLUTION: The code is modified such that VxFS reports an error when you attempt to set the maxlink and nomaxlink options for DLV version lower than 10. * 3716627 (Tracking ID: 3622326) SYMPTOM: During a checkpoint promote, a file system is marked with a fullfsck flag because the corresponding inode is marked as bad. DESCRIPTION: The issue was observed because VxFS skipped moving the data to clone inode. As a result, the inode is marked bad during a checkpoint promote and as a consequence the file system is marked with a fullfsck flag. RESOLUTION: The code is modified to move the correct data to the clone inode. * 3717895 (Tracking ID: 2919310) SYMPTOM: During stress testing on cluster file system, an assertion failure was hit because of a missing linkage between the directory and the associated attribute inode. DESCRIPTION: As per the designed behavior, the node which owns the inode of the file, receives the request to remove the file from the directory. If the directory has an alternate index (hash directory) present, then in the file remove receive handler, the attribute inode is read from the disk. However, VxFS does not create a linkage between the directory and the corresponding inode, which results in an assert failure. RESOLUTION: The code is modified to set the directory inodes i_dirhash field to attribute inode. This change is exercised while bringing the inode incore during file or directory removal. * 3718542 (Tracking ID: 3269553) SYMPTOM: VxFS returns inappropriate message for read of hole via ODM. DESCRIPTION: Sometimes sparse files containing temp or backup/restore files are created outside the Oracle database. And, Oracle can read these files only using the ODM. As a result, ODM fails with an ENOTSUP error. RESOLUTION: The code is modified to return zeros instead of an error. * 3721458 (Tracking ID: 3721466) SYMPTOM: After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock. DESCRIPTION: The VX_SINGLEDEV flag was introduced in disk layout version 7. The purpose of the flag is to indicate whether a file system resides only on a single device or a volume. When the disk layout is upgraded from version 6 to 7, the flag is not inherited along with the other values since it was not supported in version 6. RESOLUTION: The code is modified to set the VX_SINGLEDEV flag when the disk layout is upgraded from version 6 to 7. * 3726403 (Tracking ID: 3739618) SYMPTOM: sfcache command with the "-i" option may not show filesystem cache statistic periodically. DESCRIPTION: sfcache command with the "-i" option may not show filesystem cache statistic periodically. RESOLUTION: The code is modified to add a loop to print sfcache statistics at the specified interval. * 3727166 (Tracking ID: 3727165) SYMPTOM: Enhance RHEV support for SF devices for identification in Guest DESCRIPTION: We need a provision of assigning serial no to a device while attach. This would be passed on as the serial no of the scsi device to the guest. As a result, scsi inquiry on the device should return the provided serial no in Vendor specific data. RESOLUTION: Corresponding changes for supplying the serial number to the vxfs hook is done * 3729704 (Tracking ID: 3719523) SYMPTOM: 'vxupgrade' does not clear the superblock replica of old layout versions. DESCRIPTION: While upgrading the file system to a new layout version, a new superblock inode is allocated and an extent is allocated for the replica superblock. After writing the new superblock (primary + replica), VxFS frees the extent of the old superblock replica. Now, if the primary superblock corrupts, the full fsck searches for replica to repair the file system. If it finds the replica of old superblock, it restores the file system to the old layout, instead of creating a new one. This behavior is wrong. In order to take the file system to a new version, we should clear the replica of old superblock as part of vxupgrade, so that full fsck won't detect it later. RESOLUTION: Clear the replica of old superblock as part of vxupgrade. * 3733811 (Tracking ID: 3729030) SYMPTOM: The fsdedupschd daemon failed to start on RHEL7. DESCRIPTION: The dedup service daemon failed to start because RHEL 7 changed the service management mechanism. The daemon uses the new systemctl to start and stop the service. For the systemctl to properly start, stop, or query the service, it needs a service definition file under the /usr/lib/systemd/system. RESOLUTION: The code is modified to create the fsdedupschd.service file while installing the VRTSfsadv package. * 3733812 (Tracking ID: 3729030) SYMPTOM: The fsdedupschd daemon failed to start on RHEL7. DESCRIPTION: The dedup service daemon failed to start because RHEL 7 changed the service management mechanism. The daemon uses the new systemctl to start and stop the service. For the systemctl to properly start, stop, or query the service, it needs a service definition file under the /usr/lib/systemd/system. RESOLUTION: The code is modified to create the fsdedupschd.service file while installing the VRTSfsadv package. * 3737330 (Tracking ID: 3737329) SYMPTOM: Added support for RHEL7.1 DESCRIPTION: Added support for RHEL7.1 RESOLUTION: Added support for RHEL7.1 * 3743913 (Tracking ID: 3743912) SYMPTOM: Users could create sub-directories more than 64K for disk layouts having versions lower than 10. DESCRIPTION: In this release, the maxlink feature enables users to create sub-directories larger than 64K.This feature is supported on disk layouts whose versions are higher than or equal to 10. The macro VX_TUNEMAXLINK denotes the maximum limitation on sub-directories. And, its value was changed from 64K to 4 billion. Due to this, users could create more than 64K sub-directories for layout versions < 10 as well, which is undesirable. This fix is applicable only on platforms other than AIX. RESOLUTION: The code is modified such that now you can set the value of sub-directory limitation to 64K for layouts whose versions are lower than 10. * 3744425 (Tracking ID: 3744424) SYMPTOM: Rebundled the fix "Openssl: Common Vulnerability and Exposure (CVE) CVE-2014-3566 (POODLE)" as 6.2.1. DESCRIPTION: VRTSfsadv package uses old versions of OpenSSL which are vulnerable to POODLE(CVE-2014-3566) and Hearbleed(CVE-2014-0160). By upgrading to OpenSSL 0.9.8zc, many security vulnerabilities have been fixed. RESOLUTION: The VRTSfsadv package is built with OpenSSL 0.9.8zc.. * 3745651 (Tracking ID: 3642314) SYMPTOM: Umount operation reports an error code 255 in case of write-back cache. DESCRIPTION: Umount helper binary uses umount or umount2 system calls to unmount VxFS file system. In case of error, these system calls return -1. The umount helper returns this value to the system(OSes) umount binary which interpreted it as 255. RESOLUTION: The code is modified to maintain consistency with the operating systems behavior. In case of umount failure, it must return MOUNT_EX_FAIL which is defined as 32 for RHEL7 and 1 for RHEL6 or SLES11. * 3749727 (Tracking ID: 3750187) SYMPTOM: Internal noise testing hits debug assert. DESCRIPTION: The issue is observed becasue Linux inodes are not initialized properly. Initialization of Linux inodes are done at some VxFS entry points like vx_lookup. In write back replay, an inode is created based on the inode number in log and similar initialization is required here as well. RESOLUTION: The code is modified to have proper initialization of inodes. * 3749776 (Tracking ID: 3637636) SYMPTOM: Cluster File System (CFS) node initialization and protocol upgrade may hang during rolling upgrade with the following stack trace: vx_svar_sleep_unlock() vx_event_wait() vx_async_waitmsg() vx_msg_broadcast() vx_msg_send_join_version() vx_msg_send_join() vx_msg_gab_register() vx_cfs_init() vx_cfs_reg_fsckd() vx_cfsaioctl() vxportalunlockedkioctl() vxportalunlockedioctl() And vx_delay() vx_recv_protocol_upgrade_intent_msg() vx_recv_protocol_upgrade() vx_ctl_process_thread() vx_kthread_init() DESCRIPTION: CFS node initialization waits for the protocol upgrade to complete. Protocol upgrade waits for the flag related to the CFS initialization to be cleared. As the result, the deadlock occurs. RESOLUTION: The code is modified so that the protocol upgrade process does not wait to clear the CFS initialization flag. * 3755796 (Tracking ID: 3756750) SYMPTOM: VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline. DESCRIPTION: When FDD module is unloaded before the cache file system is taken offline, few FDD related structures in the cache file system remains to be free. As a result, memory leak is observed. RESOLUTION: The code is modified such that FDD related structure is not initialized for cache file systems. INSTALLING THE PATCH -------------------- Run the Installer script to automatically install the patch: ----------------------------------------------------------- Please be noted that the installation of this P-Patch will cause downtime. To install the patch perform the following steps on at least one node in the cluster: 1. Copy the patch fs-sles12_x86_64-Patch-6.2.1.300.tar.gz to /tmp 2. Untar fs-sles12_x86_64-Patch-6.2.1.300.tar.gz to /tmp/hf # mkdir /tmp/hf # cd /tmp/hf # gunzip /tmp/fs-sles12_x86_64-Patch-6.2.1.300.tar.gz # tar xf /tmp/fs-sles12_x86_64-Patch-6.2.1.300.tar 3. Install the hotfix(Please be noted that the installation of this P-Patch will cause downtime.) # pwd /tmp/hf # ./installVRTSvxfs621P3 [ ...] You can also install this patch together with 6.2.1 maintenance release using Install Bundles 1. Download this patch and extract it to a directory 2. Change to the Veritas InfoScale 6.2.1 directory and invoke the installmr script with -patch_path option where -patch_path should point to the patch directory # ./installmr -patch_path [] [ ...] Install the patch manually: -------------------------- #rpm -Uvh VRTSvxfs-6.2.1.300-SLES12.x86_64.rpm REMOVING THE PATCH ------------------ #rpm -e rpm_name KNOWN ISSUES ------------ * Tracking ID: 3852045 SYMPTOM: DB2 theads (db2sysc) hang, we can see from the crash dump: Many of them get stuck in vx_dio_physio(): - schedule - rwsem_down_failed_common - call_rwsem_down_read_failed - down_read - vx_dio_physio And many of them get stuck in vx_rwlock(): - schedule - rwsem_down_failed_common - call_rwsem_down_read_failed - down_read - vx_rwlock WORKAROUND: No * Tracking ID: 3896222 SYMPTOM: slow ls -l across cluster nodes. WORKAROUND: No * Tracking ID: 3896260 SYMPTOM: Oracle database start failure, with trace log like this: ORA-63999: data file suffered media failure ORA-01114: IO error writing block to file 304 (block # 722821) ORA-01110: data file 304: ORA-17500: ODM err:ODM ERROR V-41-4-2-231-28 No space left on device WORKAROUND: No * Tracking ID: 3896276 SYMPTOM: IO service times increased with IO intensive workload on high end server. WORKAROUND: No SPECIAL INSTRUCTIONS -------------------- NONE OTHERS ------ NONE