The SM
Role of Subnet Manager
The SM is the software entity that performs the following functions:
Discovers the topology of the subnet that it manages.
Assigns a common Subnet ID (aka Subnet Prefix) to all of the ports in its subnet.
Assigns an address to each port in its subnet (an address that is unique within that subnet). This address is referred to as a port's Local ID, or LID.
Establishes the possible paths between all endnodes in the subnet.
Sweeps (i.e., scans) its subnet on a regular basis looking for topology changes (devices added or removed, changes in the operational state of links, etc.).
Where Does the SM Reside?
The SM may reside within or behind any device in the subnet, but most typically would consist of a program running on a processor node.
How the SM Communicates With Devices
Subnet Management Interface and SMPs
Refer to Figure 9-3 on page 166. A special QP, QP0, is used by the SM:
To send Subnet Management request Packets (SMPs) to QP0 on the target device. The request packet supplies the target device with:
- -
Attribute ID.
Identifies the device attribute that some action (i.e., method) is to be performed on.
- -
Method.
Specifies the action to be taken on the indicated attribute.
- -
Attribute Modifier.
Some attribute/method combinations also require the specification of an attribute modifier. As an example, when sending a request to access a port's PortInfo attribute, the attribute modifier identifies the port on the destination device.
- -
Data.
The usage of this field is defined by the attribute/method combination. As an example, on an attribute Set operation (i.e., a write to an attribute), the data field contains the data to be written into the attribute.
- -
Attribute ID.
Identifies the device attribute that some action (i.e., method) is to be performed on.
To receive Subnet Management response Packets (SMPs). Upon receipt of an SMP request packet, the destination device performs the specified method on the specified attribute and then returns an SMP response packet to the SM. The response packet indicates the results of the requested action.
QP0 is referred to as the Subnet Management Interface (SMI). An SMP is a form of Management Datagram, or MAD.
Multiport CAs and Routers Implement Multiple SMIs
Each port on a CA or router implements an SMI (i.e., QP0) to receive SMP request packets and to transmit SMP response packets. If a CA or a router implements an SM either within or behind itself, the SM uses the SMI on one of the device's ports to send and receive SMPs.
Switch Implements One SMI on Port 0
On a switch, only switch port 0 (its management port) implements an SMI. An SMP request packet received on any other switch port with its DLID set to port 0's LID address is internally forwarded to port 0's SMI for processing. If the packet's DLID does not match port 0's LID address, the SMP packet is treated like any packet enroute to its final destination. The switch performs a lookup in its Forwarding Table using the packet's DLID to determine through which of its ports the packet will be forwarded.
It should be noted that an SM may be implemented within a switch. In this case, the SM uses the switch management port's SMI to send and receive SMPs.
Subnet Management Agent
Each CA, router, and switch implements one Subnet Management Agent (SMA). When an SMP request packet is received on any of the device's SMIs, the SMP is passed to the device's SMA for processing of the request. The SMA performs the operation defined by the SMP and then returns an SMP response packet to the SMI that had received the request SMP. The SMI transmits the response SMP back to the SM.
In the case where another SM sends an SMP to access the SMInfo attribute of the SM residing within a device, the SMI passes the SMP directly to the device's SM (rather than to its SMA) for processing.
An Example SM Attribute Access
The SM performs an attribute access in the following manner (refer to Figure 9-1 on page 164, Figure 9-2 on page 165, and to Figure 9-3 on page 166):
Build MAD in memory. The SM builds a 256-byte MAD in main memory (item C). The content of this MAD specifies the following:
- Class = Subn (indicating that this MAD is to be delivered to the target port's SMA for processing).
- Method = the action to be performed on the targeted attribute (e.g., Get or Set).
- Attribute ID = ID of attribute to be acted upon.
- Attribute Modifier (if applicable).
- Data (if it's an attribute write).
Post WR to QP0's SQ. The SM executes a Post Send Request verb call (item D) to post a WR to the SQ (item J) of the QP0 associated with the port that it will use to transmit the SMP. QP0 is a special-purpose QP tasked with sending and receiving SMPs. The WR specifies the start memory address that contains the MAD to be sent. It also specifies the LID address of the destination port to which the SMP is to be delivered and to which QP behind that port (SMPs must be delivered to the port's QP0; this is the port's Subnet Management Interface, or SMI, QP).
Packet Transmitted. The QP0 SQ Logic (item L) reads the 256-byte MAD from main memory and places it in the data payload field of an SMP. The SMP is forwarded to the HCA port associated with this QP0 for transmit.
- The BTH:Opcode field indicates that this is a Send-Only UD packet (all MADs are UD packets).
- The packet's DLID is set to the LID address of the destination port.
- The BTH:DestQP is set to QP0 (the destination port's SMI).
- The VL field is set to 15 (SMPs always travel over VL15).
Sent to SMA. The SMI sees that the MAD's Class = Subn, so it delivers the 256-byte MAD to the device's SMA (Subnet Management Agent; see Figure 9-3 on page 166) for processing.
Response packet formed. The SMA then forms an SMP response packet and sends it back to the receiving port's SMI for transmission. The SLID and DLID fields that were in the SMP request packet are reversed in the response packet's LRH:SLID and LRH:DLID fields for the return journey. The BTH:DestQP field is set to QP0 (the HCA port's SMI QP).
Response deposited in memory. The SMI's RQ Logic uses the Scatter Buffer List in the top RQ WQE to determine where the 256-byte MAD response is to be deposited in memory (item C) and a CQE is created on the CQ (item F) associated with QP0's RQ. The completion status of the operation can be found in the CQE's Status field:
- -
Status field:
16-bit field. Operation completion status. Only valid in a response MAD generated by the target in response to a request:
- Lower 8 bits = status common to all classes. These are:
Busy, MAD discarded (not an error).
Redirect required (not an error).
Class not supported.
Method not supported.
Method/Attribute combination not supported.
One or more Attribute fields contain bad values.
- Upper 8 bits = class-specific status.
- -
Status field:
16-bit field. Operation completion status. Only valid in a response MAD generated by the target in response to a request:
Figure 9-1. Example (left part)
Figure 9-2. Example (right part)
Figure 9-3. SM, SMI, SMA Relationship
There Can Be More Than One SM
General
Each subnet must have at least one SM tasked with managing that subnet. As a fallback mechanism, a subnet may optionally contain more than one SM. However, to avoid bedlam, only one SM can be active during a given period of time. That SM is referred to as the Master SM, while the others are referred to as Standby SMs.
During the discovery process, all SMs begin sweeping the fabric in order to discover the subnet topology. If a SM discovers one or more additional SMs, they negotiate with each other to determine which of them will be the Master SM. The remainder of the SMs then assume the role of Standby SMs.
Standby SMs Monitor Master's “Heartbeat”
During normal operation, the Standby SMs (if there are any) periodically perform a read of an activity indicator (the SMInfo.ActCount attribute element) within the Master SM to determine if it's still alive. If the standby SMs determine that the master has died (i.e., there has been no change in the ActCount since the last time it was read), they then negotiate with each other to determine which one will assume the master's role.