glmdebug - GLM debugging utility
glmdebug [-acgmpqrRxv] [-b bucket] [-f debug_filename][-l locklevel] [-L lockname] [-n nodeid][-T thread_id] port
The glmdebug command displays the current status of locks in the system. The Veritas Group Lock Manager (GLM) provides synchronization services to clients of the Veritas Group Membership and Atomic Broadcast (GAB) driver. Information can be dumped selectively by using the the available options.The default (no options specified) displays information about all existing locks in the system. On each node, there is an array of ports. A port is specified by an alphabet from a-z. This "port" field is a required parameter for this command.Thus, a lock name with its port constitute a unique entity.Each port has a master hash table and a proxy hash table associated with it. Each hash table contains 2048 buckets.Each bucket contains a list of lock structures for locks that have hashed into this bucket. The information dumped for each lock consists of the various fields in the lock structure.Many threads may make lock requests simultaneously and they may not all be granted yet. Each lock has its own request queue, grant queue and cache queue.The requests that are granted are stored on the grant queue of the lock. The requests that are still pending grant access are stored on the request queue for that lock. The state about each request on any queue is stored in the glm_req_t structure This is unique to each lock request. This information is printed out in the second display level.glmdebug has three levels of display. The first level shows information about the lock that is common across all the request/grant/cache queue structures for the specified lock. This is the default display level and no option is required to get this display. The following is the first-level display for the proxy hash table:
------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|master| Name |Qsize |Qsize|Qsize |_level |_new |_level |id | -------------------------------------------------------------
The following is the first-level display for the master hash table:
------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|reqst | Name |Qsize |Qsize|Qsize |_level |_new |_level |node | -------------------------------------------------------------
glmdebug prints the following information for each lock:
o lock name : The default is an ASCII display which shows only the printable characters. To see the full GLM_NAMESIZE bytes of the lock name, use the "-x" option."-x" would give the lock name is hex and will dump out all bytes even if they are zero. o grant queue size : The number of nodes or threads on this node that have been granted this lock. The size will be non-zero if the lock requests by these nodes or threads had not been in conflict with each other and other existing grants. o request queue size : The number of nodes or threads that have outstanding lock requests for this lock. These requests have not been granted yet because of conflicts with existing grants. o cache queue size : The number of nodes or threads which have to be granted this lock but have not completed the grant processing yet. o ll_cache_level : The grant level that has been cached by the proxy. o ll_cache_new : The cache level to be updated to. o ll_grant level : The highest level that has been granted to a thread. o master ID, or requesting node ID : On the proxy hash table, this field gives the node that is the master node for this lock.On the master it shows the first node on the request queue. The second level of display provides information about individual request structures on the grant/cache/request queues for the specified locks. The second level of display is invoked using the -q option.
------------------------------------------------------ Lockname |lr_old |lr_level|Que_type |Bucket|lr_nid| ------------------------------------------------------
The fields above are explained below.
o Lockname : Default display gives the ASCII characters for GLM_NAMESIZE bytes of the lockname.See all the bytes by using the -x option. o lr_old : The level that was held by this thread previously. o lr_level : The level now requested by this thread. o Que_type : This field was defined for this utility. There are four possible values:
GRANT: This request has been granted and is stored on the grant queue of this lock. REQUEST: This request has not been granted yet and is awaiting grant access on the request queue of this lock. CACHE: This request is being granted but has not finished the grant process yet. Such a transitioning request is stored on the cache queue of the lock. RGC_EMPTY: This lock does not have any threads requesting access. Hence, its grant,request and cache queues are empty. However, this lock has not been de-initialized yet. The third level of display is the verbose level (invoked with the -v option). The verbose level displays all the fields of the glm_ilock_t structure and glm_req_t structure for all specified lock requests. It produces a particularly gory level of detail and might be needed only to examine various flag values and pointers.
# glmdebug -v t
-a Displays lock names in ASCII.Only the printable characters (ASCII 0x21-0x70E) are displayed.To see all the bytes use the "-x" option. -b bucket Displays all locks in the specified bucket. -c Displays request elements in the cache queue for the specified lock.These requests have not completed their grant process yet. -f debug_filename Redirects output to the file debug_filename. -g Displays request elements in the grant queue for the specified lock.All these requests have been granted the shown lock levels as they were not in conflict with each other. -l locklevel Displays all requests on either the request,grant or cache queues that have the specified level as their proxy cached level (ll_cache_level).Enter the level to be matched as an integer ( e.g. : GLM_EX = 96) -L lockname Displays all matching locks that contain the substring lockname in the lock name.Will return matching lock names for both the proxy hash table and the master hash table unless otherwise specified. -m Displays information about locks in the master hash table only. -n nodeid Returns locks that this node has been granted or is waiting for. -p Displays information about locks in the proxy hash table only. -q Displays request element information for the specified locks. This invokes the second level of display.The information displayed is particular to each lock request that is either on the grant queue, cache queue or request queue for that lock. -r Displays lock requests in the request queue for the specified locks.These have not been granted because of the requests conflicting with existing grants. -R Displays locks that have the recursive flag set. -T thread_ptr Displays lock requests that have this thread as their owner. (lr_thread) field. -v Verbose listing for the specified locks. Prints all fields of the glm_ilock_t and glm_req_t structures for the specified locks. -x Displays lock names in hexadecimal. Displays all GLM_NAMESIZE bytes of the lock name.
All the options explained above can be used with each other in any combination. This enables the user to have a lot of control over what information is dumped out. Here are some examples. The -L option displays information for locks that contain a specified substring. For example, to print out locks that had the substring "lock3", enter:
#glmdebug -L lock3 t
Trying to match lock3 ------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|master| Name |Qsize |Qsize|Qsize |_level |_new |_level |id | ------------------------------------------------------------- PROXY HASH TABLE ------------------------------------------------------------ lock3b |1 |2 |0 | EX:0 | EX:0 | EX:0 |0 | ------------------------------------------------------------ Lock Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|master| Name Qsize |Qsize|Qsize |_level |_new |_level |id | ------------------------------------------------------------ MASTER HASH TABLE ------------------------------------------------------------ lock3b |1 |2 |0 | EX:0 | EX:0 | EX:0 |0 | ------------------------------------------------------------ lock3a |1 |0 |0 | SH:0 | SH:0 | SH:0 |0 | ------------------------------------------------------------
Only the printable characters from the lockname are displayed, that is, if the ascii value is between 0x21-0x70e, only then is the character printed out. If you want to see all the bytes, use the -x option. -x displays all the GLM_NAMESIZE bytes of the lockname.
#glmdebug -x -L 3b t
Trying to match 3b -------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|master| Name |Qsize |Qsize|Qsize |_level |_new |_level |id | -------------------------------------------------------------- PROXY HASH TABLE -------------------------------------------------------------- 6c6f636b33620000000000000000000000000000000000000000000000000000 |1 |2 |0 | EX:0 | EX:0 | EX:0 |0 | -------------------------------------------------------------- -------------------------------------------------------------- MASTER HASH TABLE ------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|reqst | Name |Qsize |Qsize|Qsize |_level |_new |_level |node | ------------------------------------------------------------- 6c6f636b33620000000000000000000000000000000000000000000000000000 |1 |0 |0 | NL:0 | NL:0 | EX:0 |0 | -------------------------------------------------------------
This produces lock names in hex and returns locks that contain the string "3b".
In the next example, the same information is requested. However, the levels of display requested are different.
#glmdebug -r -p t ----------------------------------------------------------------- Lock |Grant |Reqst|Cache |ll_cache|ll_cache|ll_grant|master| Name |Qsize |Qsize|Qsize |_level |_new |_level |id | ----------------------------------------------------------------- PROXY HASH TABLE ----------------------------------------------------------------- bowfingerlock |1 |1 |0 | EX:0 | EX:0 | EX:0 |0 | ----------------------------------------------------------------- lock3b |1 |2 |0 | EX:0 | EX:0 | EX:0 |0 | -----------------------------------------------------------------
#glmdebug -q -r -p t -------------------------------------------------------------- Lockname |lr_old |lr_level|Que_type |Bucket|lr_nid| -------------------------------------------------------------- PROXY HASH TABLE -------------------------------------------------------------- bowfingerlock | NL:0 | SH:0 |REQUEST |1890 |0 | -------------------------------------------------------------- lock3b | NL:0 | SH:0 |REQUEST |1900 |0 | -------------------------------------------------------------- lock3b | NL:0 | SH:0 |REQUEST |1900 |0 | --------------------------------------------------------------
The requested information is for all requests on the request queues of all locks on the proxy hash table. In the first case, the information that is particular to the lock is displayed.No distinction is made between individual requests. The information is common across all the requests for this lock.The second level is at a finer granularity. In the second case, the second level of display is invoked by using "-q". The fields that are particular to each request are shown.
One ioctl call returns only one request structure, so there are multiple ioctl calls to fetch all the requested lock req structures. The status of the queues can change between two ioctl calls, and this may result in the returned information being a merger of snapshots of the queues at different times.
gabconfig (1M), gabdisk (1M), gabsync (1M)
| VCS 7.4.1 | glmdebug(1M) |