glmdump - report stuck GLM locks in a Cluster File System

glmdump [ stuck ] [ -q ] [ -p max_parallel ] [ -d delay ] [ -t temp_dir ] [ -r|-h ] [ node ...]

VRTSglm

The glmdump command displays information about GLM locks that may be useful for diagnosing Cluster File System deadlocks.

The glmdump command is intended primarily for troubleshooting by Veritas personnel. The command’s functionality and output are subject to change.

stuck	Displays information about stuck locks. Currently, the stuck keyword can be omitted. Future versions of glmdump may add other keywords.

-d delay	Sets the delay to the given number of seconds. The glmdump command checks all locks twice, with an intervening delay, and prints information about locks that are unchanged and have a waiting thread. The default delay is 10 seconds. Conceivably, a very slow system could cause some locks to appear stuck, when they are not. In doubtful cases, use a longer delay, or run the command again to see if the lock persists in the output and in the same state.
-p max_parallel
	Limit number of subprocesses that can be spawned while communicating with other nodes to max_parallel.
-r	Uses rsh for communicating with other nodes. Normally glmdump tries ssh, then rsh if ssh fails. If ssh or rsh is configured to ask for a password, the user must to supply the password when requested. If ssh or rsh hangs, glmdump also hangs. With -r, glmdump skips ssh. If rsh fails, glmdump skips that node.
-h	Uses hacli which is a VCS utility, in place of rsh or ssh. hacli goes over private LLT network for communicating with other nodes. Throws error, if LLT is not configured across the nodes.
-t temp_dir	Stores temporary files that get created while fetching lock statistics in temp_dir. Default location is ’/var/VRTS/glm/’.
-q	Prints lock information only; omits the progress messages that the command normally prints.
node ...	Checks locks on the listed nodes. If no nodes are specified, glmdump reads and displays information from all nodes listed in the /etc/llthosts file.

The following text is an example of glmdump output:

lockname 0xc701b58 0xabcdefab00010001 0x3e7 0x20002 type=range port=f
 holder node=server1 primary node=0 range=0,MAXOFF,EX
 info vol=0xc701b58 type=inode subtype=rwlock inumber=131074 fset=999 ias=0
 last_locker node=server1 thread=0xffff98167b5cd230
 proxy node=server1 address=0xffff98173ef0e600
 proxy node=server2 address=0xffff8f8b2fb23600
 proxy node=server3 address=0xffff94279f387400
 waiter node=server1 primary node=0 range=0,MAXOFF,EX thread=0xffff98167b5cd230
 waiter node=server1 primary node=0 range=0,MAXOFF,SH thread=0xffff98166fb60000
 waiter node=server1 primary node=0 range=0,MAXOFF,SH thread=0xffff98167b5ce2a0
 waiter node=server1 primary node=0 range=0,MAXOFF,SH thread=0xffff98173f499070
 waiter node=server1 primary node=0 range=0,MAXOFF,SH thread=0xffff98173f878000
 waiter node=server2 primary node=0 range=0,MAXOFF,SH thread=0xffff8f8b379bd230
 waiter node=server3 primary node=0 range=0,MAXOFF,SH thread=0xffff942755a5c1c0
 waiter node=server3 primary node=0 range=0,MAXOFF,SH thread=0xffff9427b0b320e0
 waiter node=server3 primary node=0 range=0,MAXOFF,SH thread=0xffff9427b632e2a0
 waiter node=server3 primary node=0 range=0,MAXOFF,SH thread=0xffff9427d190a0e0
 waiter node=server3 primary node=0 range=0,MAXOFF,SH thread=0xffff9427ea700000

These lines describe a single lock. If more than one lock is displayed, a blank line is used as a separator.

Holder lines describe lock holders. In this example, lock has been acquired in EX mode on server1. Lock Primary for this particular rwlock is on node id 0. If the thread ID of the holder is available, the ID is included.

Waiter lines describe threads waiting for the lock. For this lock, there are five waiters on server1, one waiter on sever2 and five waiters on server3.

Last locker lines give the thread ID of the last locker for this lock. The last locker is the last thread to get the lock on the given node. If the thread holds an exclusive lock, then the last locker is the thread ID of the holder. Otherwise, the last locker might be one of the holders, or the thread might already be unlocked.

proxy lines give the address of the lock proxy on each node. This can be useful information for someone looking at stack traces to further diagnose the problem.

The output from glmdump may be useful in further diagnosing a hang, or in deciding to reboot or collect dumps from a subset of the cluster nodes. Nodes with lock holders are typically more interesting, and more central to the hang, than nodes with waiters. On the other hand, glmdump displays a limited view of a deadlock, and so there is no guarantee that glmdump has identified the critical nodes.