 | ||
| About GAB client process failure | ||
|---|---|---|
| Prev | VCS performance consideration when a system panics | Next |
If a GAB client process such as HAD fails to heartbeat to GAB, the process is killed. If the process hangs in the kernel and cannot be killed, GAB halts the system. If the -k option is used in the gabconfig command, GAB tries to kill the client process until successful, which may have an effect on the entire cluster. If the -b option is used in gabconfig, GAB does not try to kill the client process. Instead, it panics the system when the client process fails to heartbeat. This option cannot be turned off once set.
HAD heartbeats with GAB at regular intervals. It registers with GAB for a heartbeat timeout of 30 seconds (default value). You can configure the VCS environment variables, VCS_GAB_TIMEOUT_SECS andVCS_GAB_PEAKLOAD_TIMEOUT_SECS, to ensure that GAB exhibits a dynamic behavior to determine the load average of a node (per CPU load). Using the variable values and the average system load, GAB decides the grace period after which it kills HAD.
If the average load on the node is minimum and HAD hangs in the kernel such that it cannot heartbeat with GAB within the VCS_GAB_TIMEOUT_SECS timeout, GAB tries to kill HAD by sending a SIGABRT signal. Upon an unsuccessful attempt, GAB retries till the number of retries reaches the gab_kill_ntries-1 value. In case GAB cannot kill HAD with a SIGABRT signal, GAB sends a SIGKILL and closes the port. When the average load is minimum, GAB does not dynamically adapt to the load and hence does not consider the VCS_GAB_PEAKLOAD_TIMEOUT_SECS timeout value to determine the grace period to keep HAD alive.
If the average load on the node is high, HAD cannot communicate with GAB because of CPU load or delays in its I/O path with file systems. Depending on the average load, the operating system sends a load average number to GAB. The load average number ranges from 5 (minimum load) through 10 (maximum load). GAB uses the load average number to compute a grace period that adapts exponentially based on the load within the user specified bounds of the VCS_GAB_TIMEOUT_SECS and VCS_GAB_PEAKLOAD_TIMEOUT_SECS variables. GAB waits for HAD to send heartbeats during the grace period after which it kills HAD by sending a SIGABRT signal. Even after a SIGABRT signal, if GAB does not succeed, it sends a SIGKILL and closes the port.
Tunables considered by GAB to calculate the timeout period for HAD:
GAB considers the VCS_GAB_TIMEOUT_SECS timeout to calculate the timeout period for HAD if both the VCS_GAB_TIMEOUT_SECS and VCS_GAB_TIMEOUT timeouts are set.
GAB considers the VCS_GAB_TIMEOUT timeout if the VCS_GAB_TIMEOUT_SECS timeout is not set.
GAB cannot exponentially adapt to determine the grace period for HAD if the VCS_GAB_PEAKLOAD_TIMEOUT_SECS timeout is not set or if its value is the same as the VCS_GAB_TIMEOUT_SECS timeout.
By default, GAB tries to kill HAD five times before closing the port. The number of times GAB tries to kill HAD is a kernel tunable parameter, gab_kill_ntries, and is configurable. The minimum value for this tunable is 3 and the maximum is 10.
Port closure is an indication to other nodes that HAD on this node has been killed. Should HAD recover from its stuck state, it first processes pending signals. Here it receive the SIGKILL first and get killed.
After GAB sends a SIGKILL signal, it waits for a specific amount of time for HAD to get killed. If HAD survives beyond this time limit, GAB panics the system. This time limit is a kernel tunable parameter, gab_isolate_time, and is configurable. The minimum value for this timer is 16 seconds and maximum is 4 minutes.