Date: 2013-03-01
OS: SLES
OS Version: 10 SP3
Symantec FileStore 
5.6 RP1 P4 HF4 Patch Upgrade README

CONTENTS
I.   OVERVIEW
II.  UPGRADE PROCEDURE
III. FIXES IN THE NEW PATCH
IV.  KNOWN ISSUES 
V. 	 NEW FEATURES
VI.  APPENDIX

PATCH ID                        : N/A
PATCH NAME                      : SFSFS-patch-5.6RP1P4HF4_rc10_2013_03_01.tar.gz
BASE PACKAGE NAME               : Symantec FileStore
BASE PACKAGE VERSION            : 5.6
OBSELETE PATCHES                : N/A
SUPERCEDED PATCHES              : N/A
INCOMPATIBLE PATCHES            : N/A
SUPPORTED OS                    : SLES
SUPPORTED OS VERSION            : SLES 10 SP3
CREATION DATE                   : 2013-03-01
CATEGORY                        : enhancement, performance issue
REBOOT REQUIRED                 : Yes
SUPPORTS ROLLBACK               : NO


I. OVERVIEW:
------------
Symantec FileStore provides a scalable clustered storage solution. This document provides release information for the patch.


II. UPGRADE PROCEDURE:
----------------------
After you have installed or synchronized a new Symantec FileStore patch into your cluster, the list of available commands may change. Please login again to the CLI to access the updated features. 

IMPORTANT: There is a downtime for the services during an upgrade. The actual downtime will be a little longer than it takes to reboot the system. To avoid data loss, Symantec FileStore recommends that customers stop 
I/O processing completely during a patch upgrade.  

After you apply this patch, you cannot uninstall it. The 5.6RP1P4HF4 patch can only be installed on 5.6, 5.6P1, 5.6P2, 5.6P3, 5.6RP1, 5.6RP1P1,  5.6RP1P2, 5.6RP1P3, 5.6RP1P4, 5.6RP1P4HF1, 5.6RP1P4HF2, or 5.6RP1P4HF3.

If you are upgrading on a replication source cluster:
	1)	Pause running jobs. 
	2)	Upgrade the cluster.
	3)	Resume all paused replication jobs.
If you are upgrading on a replication target cluster:
	1)	On the target cluster, check "echo listfset | fsdb -t vxfs /dev/vx/rdsk/sfsdg/<replication target fs> | grep -w \"\" | wc -l"
		a.	If the command returns 0. 
		b.	If the job is in the running state, pause the job from the source console.
		c.	Go to step 3.
 
	2)	If the above command returns non zero:
		a.	If the job is in a running state, pause the job from the source console.
		b.	Umount the checkpoint first using the command "cfsumount /vx/<destination fs checkpoint>".  
		c.	Fsclustadm -v showprimary <destination fs>.
		d.	Except for the file system primary node, run "hagrp offline <vrts_destination fs group> -sys <non primary node(s)>".
		e.	Once you have run "hagrp -offline" on the non-primary nodes, run "vxumount -f -o mntunlock=VCS <destination fs>" on the primary node. 
		
	3)	Upgrade the cluster. After upgrading the online file systems that are offline in 2(b) and 2(d) steps, run the following commands after the upgrade:
		a) Run this command on all nodes "hagrp -online <vrts_destination fs group> -sys <nodename>".
		b) cfsmount /vx/<destination fs checkpoint>.
		c) Resume all paused replication jobs.

To install the patch:
1. Login as master:
   su - master
2. Start the patch install:
   upgrade patch install

IMPORTANT: Full upgrade instructions are included in the Symantec FileStore 5.6 Release Notes. Please note the following revisions
When you upgrade:

Symantec recommends that you remove I/O fencing before upgrading any cluster node or exporting your current configuration. Use the Storage> fencing off command first followed by the Storage> fencing destroy command to remove I/O fencing. 
This step is not required, but it is suggested for a clean upgrade. 

III. FIXES IN THE NEW PATCH:
----------------------------

Etrack Incidents: 2691918, 2826357, 2826641, 2894048, 2910223, 2913654, 2937444, 2944960, 2964049, 2967357, 2967537, 2968005, 2968102, 2968318, 2973403, 2974581, 2974923, 2977508, 3009621, 3017343, 3021347, 3036803, 3037229, 3037276, 3037304, 3038084, 3046542, 3060153, 3061557, 3065830, 3067221, 3079279, 3079365, 3086289
  
Errors/Problems Fixed:

        2691918    Symantec Filestore 5.6RP1P2 - File system utilization is not shown correctly in GUI (does not refresh).
        2826357    Tier policy run and tier schedule runs may fail sometimes.
        2826641    Full fsck flag was set.
        2894048    Needs new option to skip new snapshot creation when the file system is using replication jobs of no-named-fset removal or other check point operations. This new option causes duplicated email alerts to be sent.
        2910223    Integrate snapshot best practice into CLISH operation.
        2913654    `fsppadm enforce` running on multiple nodes is needed for tier jobs to complete safely.
        2937444    Mapuser cannot create files in the CIFS homedir share.
        2944960    Volume state should be considered before bringing the file system online.
        2964049    File system utilization is not updated correctly in GUI (does not refresh).	   
        2967357    Cli>antivirus liveupgrade start` failed. Error message is "Requested free space was 1, 048.687 (in MB)" though /tmp has enough free space.
        2967537    Unable to share file systems using CIFS from the Web GUI when in CTDB mode.
        2968005    "snapshot + dst" combination may cause "fullfsck flag set". Scheduled run per day seems to be 100% reproducible.
        2968102    Fullfsck flag set during LAN cable pull test.
        2968318    Tier policy run and schedule spends too much time on slave nodes.
        2973403    CIFS local user show user cannot handle a hyphen.
        2974581    Need log rotate for Symantec FileStore related logs.
        2974923    fsckptadm/fsppadm is frozen at vx_*() -> scheduled_timeout() on different two systems.
        2977508    fullfsck flag was set in vx_attr_iget().
        3009621    Request for current usage size check with fs shrinkby command.
        3017343    nfsd lockup.
        3032147    CIFS server status error while disabling/enabling DNS server. 
        3036803    For CIFS share, `cli>storage fs alert set fullspace` is changed to read_only but fs_mode is always changed to 1777.
        3037229/3046542    For multiple CIFS shares,`cli>storage fs alert set fullspace` works when set to read_only but it works partially when setting read_only back to read_write.
        3037276    fs_space.sh may have possibility to mis-detect/count dup_pid_count. autosnap_policy.sh had the same issue and it was already fixed.
        3037304    `cli>storage fs alert set fullspace` seems to have a design not to change rw to ro if there is 500GB free blocks or more in the file system. Doc/man page should mention about this design issue.
        3038084    It takes 30 minutes for fs_space.sh.
        3060153    SAV cannot detect the virus file from cab container file.
        3061557    msg-severity for skipping new snapshot creation was changed from ALERT(HF3) to INFO(HF4RC8). This should be ALERT because this is an important event.
        3065830    Mismatches between the Symantec FileStore GUI and the Symantec FileStore CLI.
        3067221    Some files could not be accessed with "No such device or address" on all CFS slave nodes.
        3079279    `/usr/bin/snmptrap` in event_notify.sh occurs userName error for replication events, but it works for storage/etc events.
        3079365    Data was written in root partition with replication destination mount separate.
        3086289    Request execution order change to "fsckptadm create" firstly and "fsckpatadm -s remove" secondary in autosnap_policy.sh. 

IV. KNOWN ISSUES:
-----------------

Etrack: 3061565

Symptom:  When attempting to destroy snapshots using File Systems > Snapshots > Destroy, the operation takes too long using the Symantec FileStore Management Console (GUI).

Description: When deleting snapshots using the Destroy button on the File Systems > Snapshots page, this operation may take more time depending on the user configuration. 

Resolution: Use the CLI command Storage> snapshot destroy <snapshot_name> <fs_name> instead of the Symantec FileStore Management Console (GUI). 

Etrack: 3058933

Symptom: Special permission bits (setuid and setgid attributes) of files are not 
transmitted to the destination cluster by Symantec FileStore Replication. 

Description: On SLES10 SP3, changing ownership of files clears the setuid bit 
of the file. The Posix documentation notes the following about "chown" 
behavior: 

"Unless chown is invoked by a process with appropriate privileges, the set-user-
ID and set-group-ID bits of a regular file will be cleared upon successful
completion; the set-user-ID and set-group-ID bits of other file types may be 
cleared."

Since Symantec FileStore Replication runs using root privileges, it needs to preserve 
these special permissions on the destination cluster.  At the time of applying 
the metadata attributes (for example, permissions, owner), Symantec FileStore 
Replication needs to follow an order. Currently, Symantec FileStore Replication
sets the permission of the file first, and then tries to set the owner of the 
file. Setting ownership after applying permissions will reset the setuid bit, 
and Symantec FileStore Replication should first set the ownership of the file and then 
apply permissions on the destination files.

Resolution: The Symantec FileStore 5.6 RP1P5 release will contain the required fix. 
Please upgrade to Symantec FileStore 5.6 RP1P5.

Etrack: 3060268

Symptom: The Symantec FileStore Management Console (GUI) may not update data immediately after an operation.

Description: Start an operation from the Symantec FileStore GUI.  The updated data may not be reflected in the
Symantec FileStore GUI after displaying a success message. The Symantec FileStore GUI may not reflect the data generated 
by the Symantec FileStore CLI immediately.

Resolution:  Wait for a while and then refresh the page by pressing the "F5" key. If the update does not appear, 
click the Refresh link provided in the Symantec FileStore GUI.


Etrack: 3060306

Symptom: The "Fullspace" option is missing from the Symantec FileStore Management Console (GUI) File Systems > File Systems > More > Set Alert dialog.

Description: When using the Symantec FileStore Management Console (GUI), the "Fullspace" option is missing from the File Systems > File Systems > More > Set Alert dialog.

Resolution: Use the Storage> fs alert set fullspace option in the Symantec FileStore CLI for setting this option.


Etrack: 3052313  

Symptom: NFS client, which is specified when creating an NFS share, may not be able to access NFS after a long time continuous load.

Description: While exporting an NFS share to a specific client by IP, after a long time continuous load, the NFS client may not be able to access the NFS share.

Resolution: This is very likely an NFS known issue. A Symantec FileStore representative is following up with Novell/Linux-NFS. The workaround is to restart the NFS service from the Symantec FileStore CLI.

Etrack: 3036067  

Symptom: Storage tier schedule will be removed after importing the configuration.

Description: If you save a system configuration by running 'system config export local' and the system has a tier schedule, the schedule may be lost after importing it later.

Resolution: Login using the support user and go to the directory /opt/VRTSnasgw/conf/backup that holds the saved configuration file, for example: 
Cluster7_01:/opt/VRTSnasgw/conf/backup # ls
test.cfg

Uncompress the test.cfg file, and you will see a new directory cfsfiler.config within cfsfiler.config/storage_schedules. There is a file storage_sched_conf storing the storage schedule. 
You can find the lost tier schedules here, and then add them manually to the crontab.

Etrack: 3035544  

Symptom: After `storage fs set fullspace 85` exceeded 85% full, both NFS/CIFS shares were not changed to read-only.

Description: For fullspace protection, the file system size is checked every five minutes.  During this five-minute interval, if the usage of the file system grows to more than the fullspace limit, 
the NFS/CIFS shares are changed to read-only.  Note that if the file system is a small size and I/O from CIFS/NFS fills the file system fast, then the file system usage can be bigger than the threshold (or even to 100% if I/O was 
really fast) before being changed to read-only. This is by design.

If the file system reaches more than the threshold limit, during this time period, the exported file system operates in read-write mode until the 
respective cron job is executed to set the file system to read-only mode.


Etrack: 3051906  

Symptom: When `antivirus scanaction clean delete`, scan-clean/delete failed for eicar.cab (MicroSoft CAB format).

Description: Symantec AntiVirus does not support the delete operation for some types of container files. The non-supported container file types for the delete operation are the following extensions: 
.arj, .cab, .rar, and .bzip2. Since the .cab file type is not a supported container file for the scanaction delete operation, this is expected behavior. 

Resolution: Avoid setting the delete operation as the second action when you want to scan such container files. The alternative is to use the quarantine as a secondary action policy. SAV always 
quarantines the infected files no matter what the file type.

Etrack: 3067744
Symptom: Bond devices do not work with 802.3ad mode.

Description: The SUSE Linux Enterprise Server 10 (SLES10) operating system cannot provide the correct status information for NIC bonding with mode 4(802.3ad). Veritas Cluster Server (VCS) always marks the NIC as ONLINE, 
which results in the VIP not failing over, even if the network cables are pulled.

Resolution: This is an operating system issue; upgrade to Symantec FileStore 5.7, which is using the SUSE Linux Enterprise Server 11 (SLES11) operating system. This issue is fixed in the SLES11 operating system.



V. NEW FEATURES:
------------------

Etrack: 3051890

Changes to rules for removing Symantec FileStore snapshots

The following changes for removing FileStore snapshots have been implemented:

. Skip new snapshot creation when the number of no-named-fsets exceeds the tunable value.
Note: The no-named-fset number is fixed to the number 0 because Symantec FileStore guarantees that only one snapshot operation is on-going, so it can be deleted.
. Remove one snapshot at a time.
. Enhancement for multi-threading for removing Symantec FileStore snapshots. 
Note: This is an underlying Veritas File System (VxFS) enhancement that is 
transparent to Symantec FileStore.  Internally, VxFS spawns multi-threads to remove 
Symantec FileStore snapshots.
. Enhancement to the System> option modify cfsmount_ontimeout. 

The following are the recommended operations/schedules for removing Symantec FileStore snapshots:

. Remove the oldest snapshot first.
Note: When multiple schedules are created for a single file system, Symantec 
FileStore can delete only the oldest snapshot for that schedule, which may not be 
the oldest for the file system.
Note: The time it takes to remove a snapshot depends on the inode number, blocks used in the checkpoint, I/O, speed of the storage, and so on. Removing the snapshot may take some time for certain scenarios. 
During this time period, snapshot create and destroy operations are not allowed from the Symantec FileStore CLI.
. Set a longer time for the System> option modify cfsmount_ontimeout command on heavily-loaded snapshot systems. 
. Set the proper value for the Storage> fs alert set fullspace tunable to avoid filling up the file system.
. Avoid to set multiple-snapshot-schedules onto one same filesystem because it may have possibility to break rules above.



Etrack: 3053679

. Using the Symantec FileStore CLI, you can obtain the current inode cache usage:

	Cluster.System> option show ninodes
	INODE_CACHE_SIZE
	----------------
	600000
	CURRENT_INODE_CACHE_USAGE
	-------------------------
	756

. Using the Symantec FileStore CLI system.1 manpage, you can obtain information on how to estimate the inode cache:

	Cluster.System> man option
	The maximum memory estimation according to specified vxfs_ninode will display to user as reference.
	Memory estimation can be calculated using the following principles:
	Each inode has a total of 4.75K size
	- About 1.33k of size (3 inodes in 1 page)
	- About 7 GLM locks with each GLM lock takes about 500 bytes
	Example: 500k active incore inode
	- 500k * 4.75k = 2.38G

. When modifying ninodes, the Symantec FileStore CLI displays the potential maximum memory if all the inodes are active in the core.
  You can check it against the system total memory.
	Cluster.System> option modify ninodes 2000343
	SFS option ERROR V-288-0 The configured inode cache can take maximum 9278M memory if all inodes are incore
	
	Note: The modify ninodes command requires a cluster-wide reboot.

Etrack: 3053682

. You have the option to choose whether or not to remove the oldest snapshots if the fullspace threshold value is reached.

  	clusterp2.Storage> fs alert set fullspace 74 fs1 removable=
	removable=yes removable=no
   By default, Symantec FileStore will not remove the oldest snapshot.

.  When you specify to remove the oldest snapshot when the fullspace threshold is reached, and if the oldest snapshot is not 'removable', you can choose  
   to set the 'removable' flag for the snapshot when setting the fullspace alert:
	   
	 Cluster.Storage> fs alert set fullspace 90 TEST removable=yes
	 SFS fs WARNING V-288-0 Oldest snapshot TEST-snap1 of TEST is not flagged to removable. Do you want to set removable flag now(y/n)?
	 
Note: If you choose the "removable=yes" option when setting the Storage> fs alert set fullspace alert to delete the oldest snapshot when hitting the fullspace threshold, it is expected behavior that the oldest snapshot is deleted.

Note: The oldest snapshot will not be removed whenever the respective snapshot is in the online state




VI. APPENDIX:
------------------
Etracks:
3062586 output of "df -i" differs from the actual number of files.
3064151 `df -i` shows wrong/too-many number of inodes when FS has snapshots.

Please check document "Symantec_VxFS_df_command_expectations.pdf" for above "df -i" behaviors in FileStore 5.6RP1P4HF4.