lltconfig - Low Latency Transport (LLT) Protocol configuration utility
lltconfig -c [-f file]
lltconfig [-v ] [-C clusterid] [-n systemid] [-i low_high] [-x low_high] [-o]
lltconfig -t devtag -d device -b ether [-s SAP] [-m mtu] [-l] [-Q]
lltconfig -t devtag -d device -b udp [-s port] [-m mtu] -I IPaddr -B bcast [-l] [-Q]
lltconfig -t devtag -d device -b udp6 [-s port] [-m mtu] -I IPaddr [-B mcast] [-l] [-Q]
lltconfig -t devtag -d device -b rdma -s port [-m mtu] -I IPaddr -B bcast [-l]
lltconfig [-u device_tag]
lltconfig [-T timer:value]
lltconfig [-F limit:value]
lltconfig -a list | flush | delete | set [system][device_tag][address]
lltconfig -L disable | enable
lltconfig -K 0 | 1 | 10 | 2 | 20
lltconfig -E 0 | 1 | 3
lltconfig -A 0 | 1
lltconfig -k enable | disable | status
lltconfig -j 0 | 1 | 2
The lltconfig utility initializes and maintains the configuration of the LLT protocol stack. It is responsible for plumbing the LLT protocol over the network drivers (e.g. ethernet, udp, udp6 etc) to which it is connected. lltconfig utility also manages many internal protocol parameters.
At system startup, lltconfig reads the /etc/llttab file to determine the local system ID and network devices, links the drivers, checks if there is another node with the same node id and then sets parameter information into LLT and starts the protocol running.
The lltconfig command without options reports the running status of the LLT protocol. The lltconfig -c command listens for 5 seconds on each link to see if it can detect a duplicate node in the cluster. If it detects a duplicate node it prints an error message and exits.
-a list | flush | delete | set [system][device_tag][address] Display or manipulate MAC addresses associated with specific systems on specific network links. The option list displays the address table. flush deletes all automatically learned addresses. delete removes one address as specified by the systemid and the device_tag. set adds one address for the system with the systemid on the link specified by device_tag. address should consist of hexadecimal digits separated by colons (:) or dots (.), depending on the link type. -b link_type Choose the link type required. The supported values for the link type are ether, udp and udp6. On selected versions of Linux, link type rdma is also supported. -B bcast Specify the broadcast address for link type udp and rdma. This option can be used to specify the multicast address for link type udp6. -C Set the clusterid. This option is needed only if more than one cluster is sharing network hardware being used by LLT. In this case, each cluster needs its own clusterid (or, alternately, a unique SAP), so that the clusters do not interfere with each other. Systems with different cluster ids cannot communicate with each other. The default cluster ID is 0. -c Configure the LLT protocol from the /etc/llttab file. -d device Configure a network link below LLT. The device is the network device path. For link type ether, it is followed by a colon (:) and an integer which specifies the unit or PPA used by LLT to attach. For Linux, the device can be specified in two ways. Either it can be specified as an interface name, i.e. eth0, eth1, etc., or in the more preferred way which is to specify it as eth-<mac-address>, i.e. eth-xx:xx:xx:xx:xx:xx. For link types udp and udp6, the device is the udp and udp6 device path respectively. For link type rdma, the device_name is udp. On Linux, the device can be specified as udp or udp6. -f Specify an alternate configuration file to use instead of /etc/llttab. This option is valid only with the -c option. -F query | limit:value Query or change the values of the flow control limits. Valid limit fields are-
highwater: This is the maximum number of packets LLT will transfer without flow controlling the client. When the number of packets in per-node transmit queue reaches highwater, LLT stops transmitting packets to that node by flow controlling the clients.
lowwater: When LLT has flow controlled the client, it will not start transmission again till the number of packets in transmit queue drops to lowwater.
rporthighwater: When a ports receive queue has rporthighwater number of packets in it, LLT stops accepting any more packets on that port. These packets will be dropped, and will be retransmitted again by the sender.
rportlowwater: Once a ports receive queue becomes full, LLT waits till the number of packets in it drops to rportlowwater. At this point, LLT again starts accepting packets on that port.
window: This is the maximum number of un-ACKed packets LLT will put in flight.
ackval: LLT sends acknowledgement of a packet by piggybacking an ACK packet on the next outbound data packet to the sender node. If there are no data packets on which to piggyback the ACK packet, LLT waits for ackval number of packets before sending an explict ACK. to the sender.
sws: This is used to avoid silly window syndrome.
linkburst: LLT sends packets on all the configured links in round-robin manner. linkburst is the number of back-to-back packets sent on a link before the next link is chosen. If linkburst is 0 then all the packets of a particular destination (node, port) are sent on a particular link.
Use lltconfig -F query to display the current flow control settings. The value is specified in number of packets.
-h Display a help message. -i low-high Set a range, low-high, of system ids valid for participation in the cluster. This command alters the limits of system ids that applications may use to prevent them from trying to communicate with non-existent systems. The default is to include 0-nn, where nn is the maximum supported systemid as determined by the kernel configuration. -I IPaddr Specify the IP address. This option is valid for link types udp and udp6 only. -l Used only with the -d option, this option specifies that the network link is to be used only as a last resort for sending data, although it is used to send heartbeats. -m mtu Specify the maximum transmission unit to use for packets on the network links. This number must be less than or equal to the lowest MTU of all the network links. Packets with size greater than this number are appropriately fragmented by LLT before transmission. -n systemid Set the systemid. systemid may be an integer in the range of valid systemids. It may also be a symbolic name, which is translated via /etc/llthosts to a systemid, or it may be a filename beginning with a slash (/), in which case the first word from the file is used as a symbolic name and translated via /etc/llthosts to a systemid. Systemids must be unique within a cluster. If LLT detects a configuration in which another system is using the same systemid, it disables the protocol until the system is rebooted. -o Override flag. Specify that values such as systemid need to be overwritten. It can also be used to force LLT to configure a link even if a duplicate node is detected. -Q Configure a link as "hidden". The "hidden" link is not visible to the LLT clients and is not considered in the cluster membership computation. It is used only by the LLT for exchanging some LLT-specific information with peer nodes. This type of link is needed when you enable the LLT "Faster detection of link failure" feature. -s sap Specify the SAP to bind on the network links for link type ether. This option can be used to specify port for link types udp and udp6. -T timer:value Query or change the values of the protocol timers. Valid timer fields are:
heartbeat: Send heartbeat packets repeteadly to peer nodes after every heartbeat timer interval. The default value is 50.
heartbeatlo: Send heartbeat packets repeteadly to peer nodes after every heartbeatlo timer interval. The default value is 100.
peertrouble: Mark a link of a peer node as "troubled", if do not receive any packet on that link for this timer interval. Once a link is marked as "troubled", LLT will not send any data on that link till there is at least one active link available. The default value is 200.
peertroublelo: Mark a low-pri link of a peer node as "troubled", if do not receive any packet on that link for this timer interval. Once a link is marked as "troubled", LLT will not send any data on that link till the link is available. The default value is 400.
peerinact: Mark a link of a peer node as "inactive", if do not receive any packet on that link for this timer interval. Once a link is marked as "inactive", LLT will not send any data on that link. The default value is 3200 on AIX and 1600 on other platforms. This timer value should always be greater than peertrouble timer value.
rpeerinact: Mark RDMA channel of a RDMA link as "inactive", if the node does not receive any packet on that link for this timer interval. Once RDMA channel is marked as "inactive", LLT does not send any data on the RDMA channel of that link, however, it may continue to send data over non-RDMA channel of that link until peerinact expires. You can view the status of the RDMA channel of a RDMA link using lltstat -nvv -r command. The default value of this attribute is 700. This timer value should always be greater than peertrouble timer value and less than peerinact value. This attribute is supported only on selected versions of Linux.
timetoreqhb: Explicitly request a heartbeat on this link from a peer node if no packets have come on this link from that node for this timer interval. Its value is capped to 200 less than the peerinact timer value. This timer is set to peeriact - 200 every time the peerinact timer is set. Therefore, its default value is 3200 - 200 = 3000 on AIX, and 1600 - 200 = 1400 on other platforms. If the peerinact is set to less than equal to 200, this timer is set to 0. If this timer is 0, then this request heartbeat mechanism is disabled. If this mechanism is used, then the link can be declared as "inactive" even before the peerinact time. Thus, this mechanism can be used to reduce the peer node death detection time.
reqhbtime: Time interval between two successive special heartbeat requests. See the timetoreqhb parameter for more information on special heartbeat requests. The default value is 40.
timetosendhb: Send a heartbeat to keep the node alive when llt timer doesnt run at regular intervals. This option specifies the amount of time to wait before sending a heartbeat in case of timer not running. Setting the value to 0 will disable the mechanism. The default value is 200.
sendhbcap: This option specifies a cap on which the out of context heartbeats will be stopped. The default value is 3 mins.
oos: If received packets remain out-of-sequence for this timer interval, send a NAK to the sender. The default value is 10.
retrans: Retransmit a packet if acknowledgement is not received for this timer interval. The default value is 10.
service: Call LLT service routine (which delivers messages to LLT clients) after every service timer interval. The default value is 100.
arp: Expire stored MAC addresses of peer nodes after this timer interval. The default value is 0, which means that the MAC address never expires.
arpreq: Send an arp request when this timer expires to detect other peer nodes in the cluster. The default value is 3000.
linkstable: This option specifies the amount of time to wait before processing the link-down event for any link of the local node. LLT receives link-down events from the OS when the LLT "Faster detection of link failure" feature is enabled. The default value is 200.
Use lltconfig -T query to display the current timer settings. value is specified in 1/100ths of a second, and is not used with the query option.
-t device_tag Used with the -d, -L options, this option specifies a tag used to identify a particular link in subsequent commands, and is displayed by lltstat(1M). -u device_tag Unlink the LLT protocol from the network device indicated by device_tag. Before you unlink the LLT protocol, disable the link using the -L option for the LLT peerinact time. -k enable | disable | status Display or manipulate the mode of LLT kernel tunables logging. User initiated changes to LLT kernel tunables are logged to dmesg buffer based on the logging mode of the tunable. Logging mode can be either ENABLED to enable the logging or DISABLED to disable the logging. The option enable enables the logging of user initiated LLT tunable changes. disable disables the logging of user initiated LLT tunable changes. status displays the current logging mode. -j 0 | 1 | 2 Set the LLT link failure detection level.
When set to 0 (the default), the link failure is detected and processed after the LLT "peerinact" time period.
When set to 1, enables the LLT "Faster detection of link failure" feature and with this the link failure is detected immediately and processed after the LLT "linkstable" time period. For this, you need to configure one additional link as LLT "hidden" link. You can configure your public link as LLT "hidden" link. The "hidden" link is used only for LLT internal purposes and not visible to the LLT clients.
When set to 2, provides the same functionality as the option 1 but this option must be used only when you have two-node cluster connected via cross-over cable (no switch involved in the physical connections). In this case, you need not configure the extra LLT "hidden" link as mentioned for option 1.
Use the lltstat -c command to display the current configured value.
The above functionality for the options 1 and 2 is available only on Solaris and HP-UX platforms.
-U Unlink the LLT protocol from the all network devices. -V Print the LLT current and maximum supported protocol version information. -M Print the LLT driver version information. -N List all the used cluster ids in the network.
This option must not be used when LLT is configured. This option may not list all the cluster ids if the LLT broadcast message frequency on the remote clusters is decreased or if the LLT broadcast feature is disabled.
-W Print the LLT supported, broadcast, and current protocol version information. -v Enable verbose output. -x low-high Set a range, low-high, of systemids not valid for participation in the cluster. This option alters the limits of systemids that applications may use to prevent attempts to communicate with non-existent systems. -L disable | enable Disables or enables a specfic LLT link. Used along with -t option. A Link when disabled does not send or recv any LLT traffic. A Link has to be disabled before unlinking from LLT protocol. -K 0|1|10|2|20 Set checksum mode.
When set to 1, LLT checksums each packet it sends to peer to guard against packet corruption on-the-wire. LLT will also offload checksum calculation to hardware if the underlying NIC supports it. In case checksum verification fails on the receiver LLT will drop that packet causing the sender to retransmit it.
Setting to 10 is same as setting to 1 except that LLT will strictly do checksums in software and will NOT offload checksumming to NIC even if it is capable of doing so.
When set to 2, LLT also checksums the whole data buffer submitted by the client to be verified by the peer before delivering it to peer-client. In case the checksum verification fails on the receiver, LLT will panic the machine. This is purposefully done to help in analysis of memory corruption from a crash dump.
Setting to 20 is same as setting to 2 except that LLT will strictly do checksums in software and will NOT offload checksumming to NIC even if it is capable of doing so.
Level 2 and level 20 checksums should only be used when diagnosing memory corruption under the advisement of the support center, since it does have the ability to panic the machine.
The default is 10 (checksums on) as LLT is a reliable transport and we need to guarantee packet accuracy. Level 10 checksums may be disabled if the private network is known to be reliable. There may be some peformance improvement due to the CPU cycles needed to perform the checksum.
Currently checksum offloading is only implemented on Linux and only for transmitting packets.
-E 0|1|3 Set trace level.
If set to 1 (the default), LLT will trace all events (upcalls, flow-control, link and connection state changes) in an internal circular buffer (called as trace buffer).
When set to 3, LLT will also trace packets that are received or transmitted. This has an overhead and may impact performance. Hence should be used only to debug.
Setting to 0 disables tracing.
-A 0|1 Enable strict source address checking.
If set to 1, LLT will check the source address of incoming packets and drop packets from unknown sources. When set to 0 this check is not performed.
This option is available only when UDP links are configured. For ethernet its a no-op.
LLT_LINK_TIMEOUT lltconfig listens for 5 seconds on each link to detect if another node in the cluster has the same node id. To change the default value, set this environment variable to a value in seconds.
When LLT and GAB are running under a cluster manager other than VCS, configure LLT and GAB as per the cluster managers supplementary documentation on LLT and GAB.
-f option is applicable only in VCS environment.