QoS within the Subnet: SL and VLs

Problem: Port's Transmitter/Receiver Are Shared Resources

Refer to Figure 25-4 on page 619. Software may set up multiple QPs (RC, UC, or UD) and/or multiple EECs (RD) that use the same CA port to send and receive message packets. This obviously creates a resource contention problem.

Figure 25-4. CA Port's Transmitter and Receiver Are Shared Resources


Assume that multiple CA QPs and/or EECs simultaneously need to send (i.e., transmit) packets through the shared port's Link Layer transmit logic. It's obvious that the port can only transmit one packet at a time, so how would it pick which QP/EEC packet to send first, which one second, and so on?

Conversely, the port Link Layer receive buffer receives a stream of inbound packets that are possibly each part of a different message targeting a different destination QP or EEC within the CA. How would it sort them out and route each packet to its final destination?

Single Buffer Pair Works, But Isn't the Best Answer

Although it is legal to implement a port's Link Layer with a single transmit buffer and a single receive buffer for sending and receiving data packets (together, they are referred to as Virtual Lane 0, or VL0), it is not the most efficient approach.

Transmit Logic Operation

The simplest port Link Layer implementation would employ a single FIFO data packet transmit buffer approach.

Some message packet streams may require faster transmission than others. Using the single transmit buffer approach, however, there is no way for the transmit buffer logic to prioritize the transmission of the packets associated with the various message packet streams it is receiving from multiple internal QP/EEC sources. The order in which it would accept the packets generated by the various sources (QPs and/or EECs) is implementation-specific. For example, it might accept one packet at a time from each of the various QPs and/or EECs in a round-robin manner. The packets associated with various messages end up intermingled in the port's Link Layer transmit buffer and are transmitted over the link in the same order that they were received.

Receive Logic Operation

Assuming that a port's Link Layer implements a single FIFO data packet receive buffer, it places all packets into this buffer in order of arrival. An inbound packet may be targeting a RC or UC QP, an EEC, an UD QP, or a raw QP. The receive buffer logic routes each packet to its final destination in the local CA in the following manner:

  • Destination is a RC or UC QP. A RC or UC QP only receives messages from a single, remote QP of the same type. The remote QP only sends one message at a time to its companion QP. The receiving port's Link Layer receive buffer logic routes each RC or UC message packet to the destination QP using the QPN specified in each packet's BTH:DestQP field.

  • Destination is a RD EEC. A local EEC in the receiving CA only receives messages from a single, remote EEC. The remote EEC only sends one message at a time to its companion EEC. The receiving port's Link Layer receive buffer logic routes each inbound RD message packet to the destination EEC using the EECN specified in each packet's RDETH (RD Extended Transport Header) field. In turn, the EEC routes each packet of the message to the target RD QP using the QPN field in the packet's BTH:DestQP.

  • Destination is a UD QP. An UD message always consists of a single packet and is routed to its destination QP by the receive buffer logic using the QPN in the packet's BTH:DestQP.

  • If the packet is a raw packet, there is no BTH and therefore no BTH:DestQP field. If the receiving port only implements one raw QP that matches the type of raw packet just received (i.e., raw IPv6 or raw EtherType), then the packet is obviously delivered to that QP. On the other hand, if the port implements multiple raw QPs of the same type as the packet, the method used by the CA logic to pick to which of the QPs the packet is delivered is implementation-specific.

Port's Link Layer May Support Multiple Buffer Pairs

General

The SL and VLs were introduced earlier in “SL and VL Fields” on page 611. At a minimum, each port's Link Layer will implement two transmit/receive buffer pairs:

  • Data buffer pair. Holds outbound and inbound data packets (packets other than link-level Flow Control Packets and SMPs; link-level Flow Control packets are not stored in any of the Link Layer's transmit/receive buffer pairs). This buffer pair is referred to as Virtual Lane 0 (VL0).

  • SMP buffer pair. Holds outbound and inbound SMPs. This buffer pair is referred to as Virtual Lane 15 (VL15).

Refer to Figure 25-5 on page 621. Optionally, the Link Layer may implement more than one data VL (i.e., data buffer pairs):

  • It may implement one data VL, referred to as VL0.

  • It may implement two data VLs, referred to as VL[1:0].

  • It may implement four data VLs, referred to as VL[3:0].

  • It may implement eight data VLs, referred to as VL[7:0].

  • It may implement 15 data VLs, referred to as VL[14:0].

Figure 25-5. Multiple Transmit/Receive Buffer Pairs


The port Link layer shown in Figure 25-5 on page 621 implements eight data VLs.

VLCap and OperationalVLs

Each port implements the following two attribute elements:

  • PortInfo.VLCap. This 4-bit, read-only attribute element specifies the number of VL buffer pairs implemented in this port's Link Layer. It is encoded as follows:

    - 1h = VL0 is implemented.

    - 2h = VL[1:0] are implemented.

    - 3h = VL[3:0] are implemented.

    - 4h = VL[7:0] are implemented.

    - 5h = VL[14:0] are implemented.

    - 6h-Fh are reserved.

  • PortInfo.OperationalVLs. This 4-bit, read/write attribute element is used by the SM, if necessary, to limit the number of buffer pairs the port is permitted to use (when the port attached to the other end of the link supports fewer VLs than this port). It is encoded as follows:

    - 1h = Only VL0 can be used. This is the default value (until the SM determines the number of VLs supported by both ends of the link and programs the port's SL-to-VL mapping table).

    - 2h = VL[1:0] may be used.

    - 3h = VL[3:0] may be used.

    - 4h = VL[7:0] may be used.

    - 5h = VL[14:0] may be used.

    - 6h-Fh are reserved.

Each Port May Implement SL-to-VL Mapping Table

CA and Router Port SL to VL Mapping

Each port on a CA or a router that implements more than one data VL must implement a separate SLtoVLMappingTable attribute for each port. If a CA or router only implements one data VL (VL0) per port, it's optional whether or not these tables are implemented.

A port's SLtoVLMappingTable attribute is accessed when an SMP is received by the port's SMI (i.e., its QP0) specifying an access to this attribute. When a packet must be output by the port, the Network Layer forwards the packet to the Link Layer's VL logic.

Table 25-2 on page 623 illustrates the format of a CA or Router port's SLtoVLMappingTable attribute. When a packet is originated in a CA, or must be passed on to an exit port within a router, the exit port's Link Layer must select the data VL transmit buffer the packet is to be placed in. This is accomplished in the following manner:

  • CA. Using the packet's LRH:SL value, the CA exit port's Link Layer performs a lookup in the port's SLtoVLMappingTable attribute to select the data VL transmit buffer. The packet is placed in that data VL transmit buffer.

  • Router. In a router, the following actions take place:

    - The router's Network Layer uses the packet's GRH:DGID to perform a lookup in its Routing Table. This yields the router port to which the packet must be passed for re-transmission.

    - The router's Network Layer uses the packet's GRH:TClass value to choose which SL to use in the subnet that the packet is being injected into.

    - The router exit port's Link Layer uses the packet's new LRH:SL value to perform a lookup in the exit port's SLtoVLMappingTable attribute. The value in the selected table entry selects the data VL transmit buffer. The packet is placed in that data VL transmit buffer.

Table 25-2. CA and Router Port's SLtoVLMappingTable Attribute Format
Entry (decimal)VL mapped ToDescription
0Specifies a data VL.During configuration, the SM specifies which data VL transmit buffer a data packet is to be sent to (based on the packet's LRH:SL value). If the SM sets an entry to 15 (the VL reserved strictly for SMPs) and a data packet's SL value selects that entry, the packet is discarded.
1Specifies a data VL.
2Specifies a data VL.
3Specifies a data VL.
4Specifies a data VL.
5Specifies a data VL.
6Specifies a data VL.
7Specifies a data VL.
8Specifies a data VL.
9Specifies a data VL.
10Specifies a data VL.
11Specifies a data VL.
12Specifies a data VL.
13Specifies a data VL.
14Specifies a data VL.
15Specifies a data VL.

Switch Port SL-to-VL Mapping

A switch that implements more than one data VL on each port must implement an array of SLtoVLMappingTable attributes (one per port). If a switch only implements one data VL (VL0) per port, it's optional whether or not these tables are implemented. When a packet arrives at an input port, the switch takes the following actions:

  • Using the packet's DLID field, the switch performs a lookup in its Forwarding Table to determine through which port the packet must be forwarded.

  • The switch then selects the SLtoVLMappingTable attribute associated with that switch port.

  • The switch uses the combination of the packet's LRH:SL field and the port number that the packet arrived on to perform a lookup in the selected mapping table. The packet is placed in the switch exit port's Link Layer data VL buffer specified by that table entry.

For a more detailed description of the switch's tables, refer to “One SLtoVLMappingTable Per Port” on page 667.

SM Sets Up SLtoVLMappingTable and VLArbitrationTable

During configuration, the SM programs the SLtoVLMappingTables in all CAs, routers, and switches. In addition, it also programs the VLArbitrationTable attribute for all CA, router, and switch ports. For a detailed description, refer to “Detailed Description of VL Arbitration” on page 628.

SM Builds Path Information Records in SA

For a detailed description of a switch's Forwarding Tables, refer to “Switch Performs Packet Forwarding” on page 662.

Switch Forwarding Tables Provide Routing within Subnet

As described earlier in “LID's Purpose: Packet Routing Within Subnet” on page 134, each switch implements a Forwarding Table attribute in its Link Layer. During configuration, the SM programs the Forwarding Table in each switch of the subnet so as to route packets from the source port to the destination port within the subnet.

Can Be Multiple Routes between Source/Destination Ports

As further described in “Why Assign a LID Range to a Port?” on page 137, the SM may assign a range of LID addresses to each CA and router port. It can then set up the Forwarding Tables within switches to permit a number of different routes (referred to as multipathing) to be taken to get to the same destination port based on which of the destination port's LID addresses is used as the DLID in the packet.

PathRecords Stored in SA Database

After configuring the subnet, the SM builds the SA database. Among other information, this database contains a series of PathRecords (see “PathRecord” on page 975), each of which describes all of the operational characteristics of a path between two ports on CAs or routers in the subnet. A PathRecord contains the information necessary to set up a connection between two QPs or two EECs. This includes the SL that should be used in packets traversing that path.

Prior to Connection Setup, Get Available Paths from SA

Prior to setting up local and remote QPs or EECs that will be used to exchange messages between two CAs, software makes a call to the SA database to obtain all of the possible paths between the local and remote port. The SA will return all of the PathRecords that apply.

Pick Your Path

Software then picks the desired path and uses the contents of that PathRecord to set up the local and remote QPs or EECs for the connection.

Specify SL When Setting Up Connection

Software uses the SL obtained from the selected PathRecord when it initially sets up the connection (in the case of RC, UC, and RD). How the SL to be inserted in each packet is specified is defined by the type of connection being established. The earlier section entitled “SL Indicates Desired QoS and Is Specified by Software” on page 611 defines how the SL is specified for RC, UC, RD, UD, and raw QPs.

Example Scenario

Refer to Figure 25-6 on page 627. This example is intended to illustrate the basic roles played by SL-to-VL mapping, VL arbitration (see “Detailed Description of VL Arbitration” on page 628), and packet transmission and reception over the selected VL. The example follows the step-by-step progress of a packet:

  • from its source port,

  • through a switch,

  • and on to the destination port.

Figure 25-6. Example Packet Transit from Source to Destination


The packet's transit from source to destination proceeds as follows:

  1. VL transmit buffer selected. The packet is originated by a QP within the source CA and a lookup is performed in the output port's SLtoVLMappingTable to select the data VL transmit buffer into which the packet will be stored for transmission. The SL value supplied by the originating QP or EEC is used to select the data VL transmit buffer (VL2 in this case). For a detailed description of the port's SL-to-VL mapping, refer to “CA and Router Port SL to VL Mapping” on page 622.

  2. Flow control credits received. Reception of link-level flow control credits (from the switch port on the other end of the link; specifically from VL2's receive buffer) enables the CA port's data VL2 transmit buffer to transmit one or more packets to the VL2 receive buffer on the other end of the link (in the switch port). For a detailed description of link-level flow control, refer to “Link-Level Flow Control” on page 637.

  3. VL arbitration performed. VL arbitration is then performed to determine when the VL2 transmit buffer has permission to transmit packets. For a detailed description of the VL arbitration, refer to “Detailed Description of VL Arbitration” on page 628.

  4. Packet is transmitted. The packet is then transmitted over VL2 to the switch port at the other end of the link. The packet contains the SL supplied earlier by software, as well as the number of the data VL receive buffer into which the packet must be placed on the receiving end of the link (VL2).

  5. Packet received by switch port. The packet is received into the appropriate VL receive buffer (as indicated by the packet's LRH:VL field) in the switch port's Link Layer (data VL2's receive buffer in this case).

  6. Forwarding table lookup performed. The switch selects an exit port by performing a Forwarding Table lookup using the packet's LRH:DLID field. For a detailed description of the switch's SL-to-VL mapping, refer to “Switch Port SL-to-VL Mapping” on page 623 and “One SLtoVLMappingTable Per Port” on page 667.

  7. VL selection. To select the exit port data VL transmit buffer into which the packet will be placed, the switch performs an SLtoVLMappingTable lookup using the combination of the packet's LRH:SL value and the port number that the packet was received on (see Figure 25-27 on page 668). The packet is placed into the exit port's selected data VL transmit buffer (VL0 in this case).

    Figure 25-27. Switch SLtoVLMappingTable Operation

  8. Flow control credits received. Reception of link-level flow control credits (from the final destination port's VL0 receive buffer) enables the VL0 transmit buffer to transmit one or more packets to the VL0 receive buffer on the other end of the link (in the destination CA).

  9. VL arbitration. The switch exit port's Link Layer logic performs VL arbitration to determine when VL0's transmit buffer has permission to transmit packets.

  10. Packet is transmitted. The packet is then transmitted over the selected data VL (VL0) to the destination port on the destination CA. As before, the packet contains the SL supplied earlier by software, as well as the number of the data VL receive buffer into which the packet must be placed on the receiving end of the link (VL0).

  11. Packet received by destination CA port. The packet is received into the appropriate VL receive buffer in the CA's input port (the VL0 receive buffer in this case).

  12. The packet is delivered to the destination QP or EEC for processing.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset