* * * READ ME * * *
                 * * * Symantec File System 6.2.1 * * *
                         * * * Patch 100 * * *
                         Patch Date: 2015-08-10


This document provides the following information:

   * PATCH NAME
   * OPERATING SYSTEMS SUPPORTED BY THE PATCH
   * PACKAGES AFFECTED BY THE PATCH
   * BASE PRODUCT VERSIONS FOR THE PATCH
   * SUMMARY OF INCIDENTS FIXED BY THE PATCH
   * DETAILS OF INCIDENTS FIXED BY THE PATCH
   * INSTALLATION PRE-REQUISITES
   * INSTALLING THE PATCH
   * REMOVING THE PATCH
   * KNOWN ISSUES


PATCH NAME
----------
Symantec File System 6.2.1 Patch 100


OPERATING SYSTEMS SUPPORTED BY THE PATCH
----------------------------------------
AIX 6.1
AIX 7.1


PACKAGES AFFECTED BY THE PATCH
------------------------------
VRTSvxfs


BASE PRODUCT VERSIONS FOR THE PATCH
-----------------------------------
   * Symantec File System 6.2
   * Symantec Storage Foundation 6.2
   * Symantec Storage Foundation Cluster File System HA 6.2
   * Symantec Storage Foundation for Oracle RAC 6.2
   * Symantec Storage Foundation HA 6.2


SUMMARY OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
Patch ID: 6.2.1.100
* 3754492 (3761603) Internal assert failure because of invalid extop processing 
at the mount time.
* 3756002 (3764824) Internal cluster file system(CFS) testing hit debug assert
* 3769992 (3729158) Deadlock due to incorrect locking order between write advise
and dalloc flusher thread.
Patch ID: 6.2.1.000
* 3657150 (3604071) High CPU usage consumed by the vxfs thread process.
* 3657152 (3602322) Panic while flushing the dirty pages of the inode
* 3657153 (3622323) Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics with the fsadm_vxfs(1M) command.
* 3657156 (3604750) The kernel loops during the extent re-org.
* 3657157 (3617191) Checkpoint creation takes a lot of time.
* 3657158 (3601943) Truncating corrupted block map of a file may lead to an infinite loop.
* 3665980 (2059611) The system panics due to a NULL pointer dereference while
flushing bitmaps to the disk.
* 3665984 (2439261) When the vx_fiostats_tunable value is changed from zero to
non-zero, the system panics.
* 3665986 (2874054) The vxconvert(1M) command fails to convert Logical Volume Manager (LVM) disk
groups to VxVM disk groups with V-3-21784.
* 3665990 (3567027) During the File System resize operation, the "fullfsck flag is set.
* 3666003 (3451725) While performing alternate disk upgrade, the VRTSvxfs
packages report vxportal'  'vxdrv'   mkdev: 0514-517 failed errors.
* 3666010 (3233276) With a large file system, primary to secondary migration takes longer duration.
* 3697966 (3697964) The vxupgrade(1M) command fails to retain the fs_flags after upgrading a file system.
* 3701351 (3718114) Assert failure is hit in the fsck(8M) utility because of an uninitialized structure object.
* 3715567 (3715566) VxFS fails to report an error when the maxlink and nomaxlink options are set on file systems having disk layout version (DLV) lower than 10.
* 3718284 (3723701) Disable writeback cache on clusters in FSS scenario
* 3718542 (3269553) VxFS returns inappropriate message for read of hole via Oracle Disk Manager (ODM).
* 3721458 (3721466) After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock.
* 3724389 (3723693) Internal noise testing found debug assert in cluster file system
* 3725347 (3725346) Trimming of underlying SSD volume was not supported for AIX and Solar is using 
"fsadm -R -o ssd" command.
* 3726403 (3739618) sfcache command with "-i" option maynot show filesystem cache statistic periodically.
* 3729111 (3729104) Man pages changes missing for smartiomode option of
mount_vxfs (1M)
* 3729704 (3719523) 'vxupgrade' retains the superblock replica of old layout versions.
* 3743913 (3743912) Users could create sub-directories more than 64K for disk layouts having versions lower than 10.
* 3755796 (3756750) VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline.


DETAILS OF INCIDENTS FIXED BY THE PATCH
---------------------------------------
This patch fixes the following Symantec incidents:

Patch ID: 6.2.1.100

* 3754492 (Tracking ID: 3761603)

SYMPTOM:
Full fsck flag will be set incorrectly at the mount time.

DESCRIPTION:
There might be possibility that extop processing will be deferred 
during umount (i.e. in case of crash or disk failure) and will be kept on 
disk, so that mount can process them. During mount, inode can have multiple 
extop set. Previously if inode has trim and reorg extop set during mount, we 
were incorrectly setting fullfsck. This patch avoids this situation.

RESOLUTION:
Code is modified to avoid such unnecessary setting of fullfsck.

* 3756002 (Tracking ID: 3764824)

SYMPTOM:
Internal cluster file system(CFS) testing hit debug assert

DESCRIPTION:
Internal debug assert is seen when there is a glm recovery while one 
of the secondary  nodes is doing mount, specifically when glm recovery happens 
between attaching a file system and mounting file system.

RESOLUTION:
Code is modified to handle glm reconfiguration issue.

* 3769992 (Tracking ID: 3729158)

SYMPTOM:
fuser and other commands hang on vxfs file systems.

DESCRIPTION:
The hang is seen while 2 threads contest for 2 locks -ILOCK and
PLOCK. The writeadvise thread owns the ILOCK but is waiting for the PLOCK.
The dalloc thread owns the PLOCK and is waiting for the ILOCK.

RESOLUTION:
Correct order of locking is PLOCK followed by the ILOCK.

Patch ID: 6.2.1.000

* 3657150 (Tracking ID: 3604071)

SYMPTOM:
With the thin reclaim feature turned on, you can observe high CPU usage on the vxfs thread process. The backtrace of such kind of threads usually look like this:
	 
	 - vx_dalist_getau
	 - vx_recv_bcastgetemapmsg
	 - vx_recvdele
	 - vx_msg_recvreq
	 - vx_msg_process_thread
	 - vx_kthread_init

DESCRIPTION:
In the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, the locking mechanism is inefficient. Thus every time when this routine is called, it will perform a series of down-up operation on a certain semaphore. This can result in a huge CPU cost when many threads calling the routine in parallel.

RESOLUTION:
The code is modified to optimize the locking mechanism in the routine to get the broadcast information of a node which contains maps of Allocation Units (AUs) for which node holds the delegations, so that it only does down-up operation on the semaphore once.

* 3657152 (Tracking ID: 3602322)

SYMPTOM:
Panic while flushing the dirty pages of the inode with backtrace,

do_page_fault
error_exit
vx_iflush
vx_workitem_process
vx_worklist_process
vx_worklist_thread
vx_kthread_init
kernel_thread

DESCRIPTION:
The race between the vx_iflush and vx_ilist_chunkclean on the 
same 
inode. The vx_ilist_chunkclean takes the inode and clears the inode pointers 
while deiniting, which causes NULL pointer dereference in the flusher 
thread.

RESOLUTION:
Resolve the race by taking the ilock, along with icache lock, 
whenever we dereference a pointer in the inode. If the inode pointer is NULL 
already/deinitialized then goto the next inode and try to flush it.

* 3657153 (Tracking ID: 3622323)

SYMPTOM:
Cluster Filesystem mounted as read-only panics when it gets sharing and/or compression statistics using the fsadm_vxfs(1M) command with the following stack:
	 
	- vx_irwlock
	- vx_clust_fset_curused
	- vx_getcompstats
	- vx_aioctl_getcompstats
	- vx_aioctl_common
	- vx_aioctl
	- vx_unlocked_ioctl
	- vx_ioctl
	- vfs_ioctl
	- do_vfs_ioctl
	- sys_ioctl
	- system_call_fastpath

DESCRIPTION:
When file system is mounted as read-only, part of the initial setup is skipped, including loading of few internal structures. These structures are referenced while gathering statistics for sharing and/or compression. As a result, panic occurs.

RESOLUTION:
The code is modified to only allow "fsadm -HS all" to gather sharing and/or compression statistics on read-write file systems. On read-only file systems, this command fails.

* 3657156 (Tracking ID: 3604750)

SYMPTOM:
The kernel loops during the extent re-org with the following stack trace:
vx_bmap_enter()
vx_reorg_enter_zfod()
vx_reorg_emap()
vx_extmap_reorg()
vx_reorg()
vx_aioctl_full()
$cold_vx_aioctl_common()
vx_aioctl()
vx_ioctl()
vno_ioctl()
ioctl()
syscall()

DESCRIPTION:
The extent re-org minimizes the file system fragmentation. When the re-org 
request is issued for an inode with a lot of ZFOD extents, it reallocates the 
extents of the original inode to the re-org inode. During this, the ZFOD extent 
are preserved and enter the re-org inode in a transaction. If the extent 
allocated is big, the transaction that enters the ZFOD extents becomes big and 
returns an error. Even when the transaction is retried the same issue occurs. 
As a result, the kernel loops during the extent re-org.

RESOLUTION:
The code is modified to enter the Bmap (block map) of the allocated extent and 
then perform the ZFOD processing. If you get a committable error during the 
ZFOD enter, then commit the transaction and continue with the ZFOD enter.

* 3657157 (Tracking ID: 3617191)

SYMPTOM:
Checkpoint creation may take hours.

DESCRIPTION:
During checkpoint creation, with an inode marked for removal and being overlaid, there may be a downstream clone and VxFS starts pulling all the data. With Oracle it's evident because of temporary files deletion during checkpoint creation.

RESOLUTION:
The code is modified to selectively pull the data, only if a downstream push inode exists for file.

* 3657158 (Tracking ID: 3601943)

SYMPTOM:
The block map tree of a file is corrupted across the levels, and during truncating, inode for the file may lead to an infinite loop.


There are various

DESCRIPTION:
For files larger than 64G, truncation code first walks through the bmap tree to find the optimal offset from which to begin the truncation. 
If this truncation falls within corrupted range of the bmap, actual truncation 
code which relies on binary search to find this offset. As a result, the truncation cannot find the offset, thus it returns empty. The output makes the truncation code to submit dummy transaction, which updates the inode of file with latest ctime, without freeing the extents allocated.

RESOLUTION:
The truncation code is modified to detect the corruption, mark the inode bad and, mark the file system for full-fsck. The modification makes the truncation  possible for full fsck. Next time when it runs, the truncation code is able to throw out the inode and free the extents.

* 3665980 (Tracking ID: 2059611)

SYMPTOM:
The system panics due to a NULL pointer dereference while flushing the
bitmaps to the disk and the following stack trace is displayed:


vx_unlockmap+0x10c
vx_tflush_map+0x51c
vx_fsq_flush+0x504
vx_fsflush_fsq+0x190
vx_workitem_process+0x1c
vx_worklist_process+0x2b0
vx_worklist_thread+0x78

DESCRIPTION:
The vx_unlockmap() function unlocks a map structure of the file
system. If the map is being used, the hold count is incremented. The
vx_unlockmap() function attempts to check whether this is an empty mlink doubly
linked list. The asynchronous vx_mapiodone routine can change the link at random
even though the hold count is zero.

RESOLUTION:
The code is modified to change the evaluation rule inside the
vx_unlockmap() function, so that further evaluation can be skipped over when map
hold count is zero.

* 3665984 (Tracking ID: 2439261)

SYMPTOM:
When the vx_fiostats_tunable is changed from zero to non-zero, the
system panics with the following stack trace:
vx_fiostats_do_update
vx_fiostats_update
vx_read1
vx_rdwr
vno_rw
rwuio
pread

DESCRIPTION:
When vx_fiostats_tunable is changed from zero to non-zero, all the
incore-inode fiostats attributes are set to NULL. When these attributes are
accessed, the system panics due to the NULL pointer dereference.

RESOLUTION:
The code has been modified to check the file I/O stat attributes are
present before dereferencing the pointers.

* 3665986 (Tracking ID: 2874054)

SYMPTOM:
The vxconvert(1M) command fails to convert LVM disk groups to VxVM disk
groups with the following  error:
VxFS  ERROR V-3-21784: can't malloc

DESCRIPTION:
While converting an LVM volume to a VxVM volume, vxconvert(1M) allocates memory
with the malloc() function to do the conversion. The memory allocated is based
on the extent size. When the extent size is big, malloc is not able to find a
large chunk of free memory. As a result, the conversion fails with the above
mentioned error.

RESOLUTION:
The code is modified to break down the big contiguous malloc request into
smaller sizes, so that it could be fulfilled using non-contiguous memory chunks.

* 3665990 (Tracking ID: 3567027)

SYMPTOM:
During the File System resize operation, the "fullfsck flag is set with the following message:vxfs: msgcnt 183168 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/vol file system fullfsck flag set - vx_fs_upgrade_reorg

DESCRIPTION:
File system resize requires some temporary inodes to swap the old inode and the converted inode. However, before a structural inode ise processed, the "fullfsck flag is set when a failure occurs during the metadata change. The flag is cleared after the swap is successfully completed.

If the temporary inode allocation fails, VxFS leaves the fullfsck flag on the 
disk. However, all temporary inodes can be cleaned up when not being in use, 
thus these temporary inodes do not result in corruption.

RESOLUTION:
The code is modified to clear the fullfsck flag if the structural inode conversion cannot create its temporary inode.

* 3666003 (Tracking ID: 3451725)

SYMPTOM:
While performing alternate disk upgrade, the VRTSvxfs packages report
vxportal' 'vxdrv' mkdev: 0514-517 failed errors.

DESCRIPTION:
During alternate disk installation (Live Upgrade) and while
defining vxportal0 device for ODM configuration in vxextadm, it is observed that
the vxportal0 exists. Due to which, vxextadm fails and all previous
configurations are cleared. 
During the clean up, the /dev/portal device is removed resulting in the error
message.

RESOLUTION:
The code is modified to add a check that validates if "vxportal0"
and "qio0" are defined before they are added to the ODM.

* 3666010 (Tracking ID: 3233276)

SYMPTOM:
On a 40 TB file system, the fsclustadm setprimary command consumes more than 2 minutes for execution. And, the unmount operation consumes more time causing a primary migration.

DESCRIPTION:
The old primary needs to process the delegated allocation units while migrating
from primary to secondary. The inefficient implementation of the allocation unit
list is consuming more time while removing the element from the list. As the file system size increases, the allocation unit list also increases, which results in additional migration time.

RESOLUTION:
The code is modified to process the allocation unit list efficiently. With this modification, the primary migration is completed in 1 second on the 40 TB file system.

* 3697966 (Tracking ID: 3697964)

SYMPTOM:
When a file system is upgraded, the upgraded layout clears the superblock flags (fs_flags).

DESCRIPTION:
When a file system is upgraded, the new superblock structure gets populated with the field values. Most of these values are inherited from the old superblock. As a result, the fs_flags values are overwritten and the flags such as VX_SINGLEDEV are deleted from the superblock.

RESOLUTION:
The code is modified to restore the old superblock flags while upgrading the disk layout of a file system.

* 3701351 (Tracking ID: 3718114)

SYMPTOM:
Assert failure is hit in the fsck(8M) utility because of an uninitialized structure object.

DESCRIPTION:
The issue was observed because an uninitialized structure object was passed to the function.

RESOLUTION:
The code is modified to initialize the structure field using the correct value.

* 3715567 (Tracking ID: 3715566)

SYMPTOM:
VxFS fails to report an error  when the maxlink and nomaxlink options are set for disk layout version (DLV) lower than 10.

DESCRIPTION:
The maxlink and nomaxlink options allow you to enable and disable the maxlink support feature respectively. The maxlink support feature operates only on DLV version 10. Due to an issue, the maxlink and nomaxlink options wrongly appear in DLV versions lower than 10. However, when selected the options do not take effect.

RESOLUTION:
The code is modified such that VxFS reports an error when you attempt to  set the maxlink and nomaxlink options for DLV version lower than 10.

* 3718284 (Tracking ID: 3723701)

SYMPTOM:
Disable writeback cache on cluster in a Flexible Storage Sharing (FSS) scenario.

DESCRIPTION:
Consider a two-node cluster whose nodes are N1 and N2. Each of the two nodes has its own SSD and they use each other's SSD as their remote cache. In this scenario, if either of the nodes goes down, its impossible to recover the data and data loss may occur.

RESOLUTION:
The code is modified to disable the writeback cache in the case of FSS on CFS environment.

* 3718542 (Tracking ID: 3269553)

SYMPTOM:
VxFS returns inappropriate message for read of hole via ODM.

DESCRIPTION:
Sometimes sparse files containing temp or backup/restore files are created outside the Oracle database. And, Oracle can read these files only using the ODM. As a result, ODM fails with an ENOTSUP error.

RESOLUTION:
The code is modified to return zeros instead of an error.

* 3721458 (Tracking ID: 3721466)

SYMPTOM:
After a file system is upgraded from version 6 to 7, the vxupgrade(1M) command fails to set the VX_SINGLEDEV flag on a superblock.

DESCRIPTION:
The VX_SINGLEDEV flag was introduced in disk layout version 7.
The purpose of the flag is to indicate whether a file system resides only on a single device or a volume.
When the disk layout is upgraded from version 6 to 7, the flag is not inherited along with the other values since it was not supported in version 6.

RESOLUTION:
The code is modified to set the VX_SINGLEDEV flag when the disk layout is upgraded from version 6 to 7.

* 3724389 (Tracking ID: 3723693)

SYMPTOM:
Internal noise testing found the debug assert in cluster file system

DESCRIPTION:
During the extended hash removal, sometimes VxFS cannot unlock the attribute inode read and write lock.

RESOLUTION:
The code is modified to unlock attribute inode read and write lock during the extended hash removal.

* 3725347 (Tracking ID: 3725346)

SYMPTOM:
Trimming of underlying SSD volume was not supported for AIX and Solar using "fsadm -R -o 
ssd" command.

DESCRIPTION:
The fsadm command with the -o ssd option ("fsadm -R -o ssd") is used to initiate the 
TRIM command on an underlying SSD volume, which was not supported on AIX and Solaris.

RESOLUTION:
The code is modified on AIX and Solaris to support the TRIM command on an underlying 
SSD volume.

* 3726403 (Tracking ID: 3739618)

SYMPTOM:
sfcache command with the "-i" option may not show filesystem cache statistic periodically.

DESCRIPTION:
sfcache command with the "-i" option may not show filesystem cache statistic periodically.

RESOLUTION:
The code is modified to add a loop to print sfcache statistics at the specified interval.

* 3729111 (Tracking ID: 3729104)

SYMPTOM:
Man pages changes missing for smartiomode option of mount_vxfs (1M)

DESCRIPTION:
smartiomode option for mount_vxfs is missing in manpage.

RESOLUTION:
Modified the man page changes to reflect smartiomode option for
mount_vxfs.

* 3729704 (Tracking ID: 3719523)

SYMPTOM:
'vxupgrade' does not clear the superblock replica of old layout versions.

DESCRIPTION:
While upgrading the file system to a new layout version, a new superblock inode is allocated and an extent is allocated for the replica superblock. After writing the new superblock (primary + replica), VxFS frees the extent of the old superblock replica.
Now, if the primary superblock corrupts, the full fsck searches for replica to repair the file system. If it finds the replica of old superblock, it restores the file system to the old layout, instead of creating a new one. This behavior is wrong.
In order to take the file system to a new version, we should clear the replica of old superblock as part of vxupgrade, so that full fsck won't detect it later.

RESOLUTION:
Clear the replica of old superblock as part of vxupgrade.

* 3743913 (Tracking ID: 3743912)

SYMPTOM:
Users could create sub-directories more than 64K for disk layouts having versions lower than 10.

DESCRIPTION:
In this release, the maxlink feature enables users to create sub-directories larger than 64K.This feature is supported on disk layouts whose versions are higher than or equal to 10. The macro VX_TUNEMAXLINK denotes the maximum limitation on sub-directories. And, its value was changed from 64K to 4 billion. Due to this, users could create more than 64K sub-directories for layout versions < 10 as well, which is undesirable.
This fix is applicable only on platforms other than AIX.

RESOLUTION:
The code is modified such that now you can set the value of sub-directory limitation to 64K for layouts whose versions are lower than 10.

* 3755796 (Tracking ID: 3756750)

SYMPTOM:
VxFS may leak memory when File Design Driver (FDD) module is unloaded before the cache file system is taken offline.

DESCRIPTION:
When FDD module is unloaded before the cache file system is taken offline, few FDD related structures in the cache file system remains to be free. As a result, memory leak is observed.

RESOLUTION:
The code is modified such that FDD related structure is not initialized for cache file systems.



INSTALLING THE PATCH
--------------------
Run the Installer script to automatically install the patch:
-----------------------------------------------------------
To install the patch perform the following steps on at least one node in the cluster:
1. Copy the patch fs-aix-Patch-6.2.1.100.tar.gz to /tmp
2. Untar fs-aix-Patch-6.2.1.100.tar.gz to /tmp/hf
    # mkdir /tmp/hf
    # cd /tmp/hf
    # gunzip /tmp/fs-aix-Patch-6.2.1.100.tar.gz
    # tar xf /tmp/fs-aix-Patch-6.2.1.100.tar
3. Install the hotfix
    # pwd /tmp/hf
    # ./installVRTSvxfs621P1 [<host1> <host2>...]

Install the patch manually:
--------------------------
Please refer to  Install guide for install instructions


REMOVING THE PATCH
------------------
Run the Uninstaller script to automatically remove the patch:
------------------------------------------------------------
To uninstall the patch perform the following step on at least one node in the cluster:
    # /opt/VRTS/install/uninstallVRTSvxfs621P1 [<host1> <host2>...]

Remove the patch manually:
-------------------------
Please refer to  Install guide for uninstall instructions


KNOWN ISSUES
------------
* Tracking ID: 3817229

SYMPTOM: When fsfreeze is used together with vxdump, the fsfreeze command
timeout and vxdump command failed.

WORKAROUND: Don't use fsfreeze and vxdump command together.



SPECIAL INSTRUCTIONS
--------------------
NONE


OTHERS
------
NONE