Link Layer Overview

Detailed Link Layer Description

This section provides an overview of the Link Layer. A detailed description of the Link Layer can be found in the chapter entitled “Detailed Description of the Link Layer” on page 599.

Link Layer Functions

The functions performed by the Link Layer are:

• In CAs, performs packet formatting (LRH and CRCs) and, in switches, guides the packets (using switch Forwarding Tables) along the proper path between the requester and responder within the subnet.

• Implements two or more Virtual Lane (VL) transmit/receive buffer pairs. See “Virtual Lanes” on this page.

• The SL (Service Level) field within each packet defines the desired QoS as the packet is transported through the subnet. Each port the packet passes through on its journey uses its respective SL-to-VL mapping table (set up by the SM) to choose which data VL transmit buffer the packet will be placed in, and therefore which data VL the packet will use on the respective link.

• Flow Control implementation (see “Link-Level Flow Control” on page 123).

• IBA and Raw packet multicast operations within the subnet. See “IBA and Raw Packet Multicast” on page 124.

• Subnet multipathing (refer to “Why Assign a LID Range to a Port?” on page 137 for an introduction).

Virtual Lanes

Problem Solved by VLs

Required QoS Is Application-Specific

The importance of how fast a message reaches its destination is entirely application-specific. While some applications may experience poor performance or even malfunction if their messages are not delivered quickly enough, other applications may not require that their messages be delivered quite so quickly. Stated another way, some applications demand a higher QoS than others. As described earlier in “Desired Local Quality of Service” on page 43, each packet in a message contains an SL field wherein software specifies the relative importance of this particular message stream. The manner in which software specifies the desired QoS is specified as follows:

- RC/UC. Software specifies the desired QoS for messages generated by the QP when the QP is set up.

- RD. Software specifies the desired QoS for messages generated by the EEC when the local EEC is set up within a CA.

- UD. Software specifies the QoS desired for messages sent to destination UD QPs residing behind a particular remote CA port when the Address Handle that identifies the remote port is created.

Traffic Jam at a Port: Who to Let Through First

Refer to Figure 6-10 on page 121. Occasions will arise when a number of packets from various sources are queued up for transmission through the same port. Although the example illustrates a case wherein multiple packet streams are queued up awaiting transmission through a switch port, the same situation can occur in a CA when a number of QPs and/or EECs have message packets to be streamed out through the same CA port.

Port Must Have Some Type of Priority Scheme

It should be obvious that each port must incorporate some type of priority scheme in deciding the order in which traffic streams are gated through to the link. The next section introduces this scheme.

The VL Solution

A VL Consists of a Send/Receive Buffer Pair

Refer to Figure 6-11 on page 123. In the Link Layer, each port implements two or more pairs of send/receive buffer pairs. A buffer pair is referred to as a Virtual Lane (VL). At a minimum, a port must implement at least two VLs:

- VL0 as a data buffer used to send/receive data packets.

- VL15 as an SMP buffer dedicated to the send/receive of SMP packets.

The number of data VLs implemented on a port is design-specific and is indicated by the PortInfo.VLCap attribute element. Valid values are:

- One data VL (VL0).

- Two data VLs (VL0:1).

- Four data VLs (VL0:3).

- Eight data VLs (VL0:7).

- 15 data VLs (VL0:14).

On Packet Arrival, SL-to-VL Mapping Performed

Each port that implements more than one data VL must implement the SLtoVLMappingTable attribute. The SM sets up these tables in CAs, switches, and routers during subnet configuration. As mentioned earlier (see “Required QoS Is Application-Specific” on page 119), each packet contains an SL field that defines the QoS desired by the message sender. When a packet other than an SMP is sent to a port's Link Layer for transmission, the port's Link Layer logic consults the table to determine which data VL transmit buffer to place the packet in.

Multiple Transmit Buffers; Just One Physical Transmit Link

Refer to Figure 6-11 on page 123. It should be obvious that a port must implement a method to determine in what order the packets currently posted in the data VL transmit buffers are sent to the port's Physical Layer for transmission.

Port Contains VL Arbitration Logic

At a given instant in time, a port's Link Layer may have a number of packets buffered up in the various data VL transmit buffers. The port's Link Layer implements a programmable arbitration scheme that permits the configuration software to set up separate low- and high-priority arbitration rotations. This arbitration scheme enforces a fairness algorithm so that packets posted in high-priority transmit buffers cannot dominate the transmit link to the detriment of those packets currently posted in the low-priority transmit buffers.

Detailed SL/VL Mapping and VL Arbitration

A detailed description of SL-to-VL mapping and VL arbitration can be found in “QoS within the Subnet: SL and VLs” on page 617 and “Detailed Description of VL Arbitration” on page 628.

On Transmit, Packet Contains VL Selector

When a packet is output onto the physical transmit link, the Link Layer tags it with a 4-bit VL field to indicate which transmit buffer it is being sent from. On the receiving end, the receiving port's Link Layer logic must accept the packet into the corresponding VL receive buffer.

Link-Level Flow Control

Data VL Flow Control

A particular data VL's send buffer is only permitted to transmit a packet if the corresponding VL receive buffer on the other end of the link has sufficient buffer space available to accept it. On a regular basis, each data VL receiver sends credits (in special Flow Control packets) indicating its available buffer space to its respective transmit buffer on the other end of the link.

VL15 Does Not Use Flow Control

Note that the VL15 (the SMP VL) transmit buffer is not subject to flow control. This means that an SMP packet may be sent at a moment when the VL15 receive buffer on the other end of the link does not have room. In that case, the VL15 receive logic is permitted to silently drop the packet.

It should be obvious that the sender of the SMP packet would need to know that either the request packet did not arrive at its destination, or that the response was dropped on its way back to the SM. This is handled by a timeout and retry mechanism (covered later in the book).

IBA and Raw Packet Multicast

When a switch port's Link Layer receives a packet, it determines if the LRH:DLID address is a unicast or multicast address (see “LID Address Space” on page 133). If it's a multicast address, then the Link Layer takes one of the following actions:

• Switch implements the MulticastForwardingTable attribute. In this case, the Link Layer uses the packet's DLID and performs a lookup in the MulticastForwardingTable to determine which switch port(s) to forward the packet through. In the event that the DLID does not exist in the table, the packet is forwarded through the port indicated in the SwitchInfo.DefaultMulticastPrimaryPort attribute element. Refer to “Switch Multicast Packet Forwarding” on page 675 and to “Multicasting” on page 563 for a detailed description of multicast operations.

• Switch doesn't implement the MulticastForwardingTable attribute. In this case, the Link Layer takes one of the following actions:

  - If the packet arrived at a port other than the one indicated in the SwitchInfo.DefaultMulticastPrimaryPort attribute element, then the packet is forwarded through the port indicated in this attribute element.

  - If the packet arrived at the port indicated in the SwitchInfo.DefaultMulticastPrimaryPort attribute element, then the packet is forwarded through the port indicated in the SwitchInfo.DefaultMulticastNotPrimaryPort attribute element.