glmdump - report stuck GLM locks in a Cluster File System
glmdump [ stuck ] [ -q ] [ -d delay ] [ -r ] [ node ...]
The glmdump command displays information about GLM locks that may be useful for diagnosing Cluster File System deadlocks.
The glmdump command is intended primarily for troubleshooting by Veritas personnel. The commands functionality and output are subject to change.
stuck Displays information about stuck locks. Currently, the stuck keyword can be omitted. Future versions of glmdump may add other keywords.
-d delay Sets the delay to the given number of seconds. The glmdump command checks all locks twice, with an intervening delay, and prints information about locks that are unchanged and have a waiting thread. The default delay is 10 seconds. Conceivably, a very slow system could cause some locks to appear stuck, when they are not. In doubtful cases, use a longer delay, or run the command again to see if the lock persists in the output and in the same state. -r Uses rsh for communicating with other nodes. Normally glmdump tries ssh, then rsh if ssh fails. If ssh or rsh is configured to ask for a password, the user must to supply the password when requested. If ssh or rsh hangs, glmdump also hangs. With -r, glmdump skips ssh. If rsh fails, glmdump skips that node. -q Prints lock information only; omits the progress messages that the command normally prints. node ... Checks locks on the listed nodes. If no nodes are specified, glmdump reads and displays information from all nodes listed in the /etc/llthosts file.
The following text is an example of glmdump output:
lockname 0xc70b798 0xabcdefab00020001 0x3e7 0x8006e type=regular port=f holder node=server2 level=SH count=2 info vol=0xc70b798 type=inode subtype=glock inumber=524398 fset=999 ias=0 last_locker node=server2 thread=0xffff8100564507e0 proxy node=server2 address=0xffff810049b97c00 proxy node=server3 address=0xffff8103fc2386f8 waiter node=server3 level=EX count=1
These lines describe a single lock. If more than one lock is displayed, a blank line is used as a separator.
Holder lines describe lock holders. In this example, there are two holders on node server2. If the thread ID of the holder is available, the ID is included.
Waiter lines describe threads waiting for the lock. For this lock, there is one waiter on node server3.
Last locker lines give the thread ID of the last locker for this lock. The last locker is the last thread to get the lock on the given node. If the thread holds an exclusive lock, then the last locker is the thread ID of the holder. Otherwise, the last locker might be one of the holders, or the thread might already be unlocked.
proxy lines give the address of the lock proxy on each node. This can be useful information for someone looking at stack traces to further diagnose the problem.
The output from glmdump may be useful in further diagnosing a hang, or in deciding to reboot or collect dumps from a subset of the cluster nodes. Nodes with lock holders are typically more interesting, and more central to the hang, than nodes with waiters. On the other hand, glmdump displays a limited view of a deadlock, and so there is no guarantee that glmdump has identified the critical nodes.