Link-Level Flow Control

Problem: Transmitting a Packet to a Full Buffer

The Link Layer of each port on a CA, router, or switch implements one or more transmit/receive buffer pairs (referred to as VLs) that are used to transmit and receive data packets. A data packet is defined as any packet other than an SMP or a link-level Flow Control Packet (described in this section).

At a given moment, one or more of a port's data VL transmit buffers may have one or data packets queued up for transmission. Assume that the data VL arbiter (described earlier) has decided that it is a specific data VL's turn to transmit and the arbitration table entry has sufficient Weight left for the packet to be transmitted.

The VL transmit buffer transmits the packet to the corresponding data VL receive buffer in the port at the other end of the physical link. If that receive buffer does not have sufficient room to hold the packet, it will be discarded. The link-level Flow Control mechanism prevents this from happening.

Solution: Receivers Report Space Available

Space Availability Is Reported in FCPs

On a periodic basis, each of a port's data VL receive buffers causes a special packet to be transmitted to the corresponding data VL transmit buffer in the port at the other end of the physical link. This special packet is referred to as a Flow Control Packet (FCP) and it has the format defined in Table 25-3 on page 639. The fields are defined as follows:

  • Operand (Op). Any packet received with a reserved operand value is discarded. Only operands 0h and 1h are currently defined:

    - An operand of 0h indicates a normal FCP. When the Link Layer is in the LinkArm or LinkActive state, FCPs are sent with the normal flow control operand.

    - An operand value of 1h indicates a flow control initialization packet. When the Link Layer is in the LinkInitialize state, FCPs are sent with the flow control initialization operand.

  • FCTBS (Flow Control Total Blocks Sent). This field is not applicable when the FCP is sent from the Receiver to the Transmitter. Transmission errors in data packets, or in the exchange of flow control information in FCPs, can result in inconsistencies in the flow control state perceived by the transmitter and receiver. To ensure that the Receiver's ABR (Adjusted Blocks Received; see “Terminology” on page 640) matches the Transmitter's FCTBS, the Transmitter periodically sends a FCP to the Receiver with the correct FCTBS value. The receiver uses this data to update its ABR to match the Transmitter's FCTBS.

  • VL. Via the packet's VL field, the FCP identifies which VL receive buffer sent this credit packet (i.e., reporting available buffer space) and to which of the VL transmit buffers the credits should be delivered. If the Transmitter is the one sending the FCP, the FCTBS field is valid and the VL field indicates to which data VL receive buffer the FCTBS value should be delivered.

  • FCCL (Flow Control Credit Limit). Generated by receiver side logic. It is the sum of the Receiver's current Free Space Count and the total number of blocks that have been received since link initialization.

  • LPCRC. 16-bit Link Packet CRC that covers the first four bytes of the packet.

FCPs are periodically sent to the data VL transmit buffers by the remote port's data VL receive buffers while the Link Layer is in the LinkInitialize, LinkArm, or LinkActive states.

Buffer Space Availability Is Reported in FCB Units

A data VL receive buffer reports how much space it currently has available in units referred to as Flow Control Blocks (FCBs). One FCB = 64 bytes of buffer space. The amount of receive buffer space it will take to hold a given data packet is defined to be the size of the data packet in bytes (from the byte after the LRH up to but not including the VCRC, inclusive) ÷ 64 bytes (the FCB size), and rounded up to the next integral value.

Frequency of FCP Transmission

An FCP is transmitted by each data VL receive buffer prior to the passing of 65,536 symbol times since the last time an FCP for the given VL was transmitted. A symbol time is defined as the time required to transmit 8 bits of information onto the link. FCPs may be transmitted as often as necessary to return credits and enable efficient utilization of the link.

Table 25-3. Link-Level Flow Control Packet (FCP) Format
bits ==>31:2423:1615:87:0
bytes
0–3OpFCTBSVLFCCL
4–7LPCRCReserved

Only Applies to Data VLs, Not VL15

Only a port's data VL receive buffers generate FCPs to report buffer credits to their respective data VL transmit buffers in the port at the other end of the physical link. Credits are not reported to the VL15 transmit buffer by its companion VL15 receive buffer in the port at the other end of the physical link.

This means that the VL15 (SMP) transmit buffer transmits SMPs without knowing if there is buffer space available to hold the packets. It is therefore possible that an SMP will be dropped by the receiving port if its VL15 receive buffer is currently full. This is especially true because the specification states:

“CAs and routers shall provide a minimum of a single packet buffer per port for VL15 on each port for reception.”

FCPs Are Never Forwarded by a Switch or a Router

Link packets are a private exchange between the Link Layers of the two ports connected by a physical Link. As such, they are never forwarded through another port on a switch or a router. The only type of link packet currently defined is the FCP.

Terminology

The following terms are used throughout the description of the link-level flow control mechanism. They are provided here for ease of reference.

  • FCP (Flow Control Packet):

    - Sent periodically by the data VL receive buffer to its respective data VL transmit buffer on the other end of the physical link. Tells the transmit buffer how much space is free in the receive buffer.

    - Sent periodically by the data VL transmit buffer to its respective data VL receive buffer on the other end of the physical link. The receive buffer uses the FCTBS value to update its ABR register.

  • Transmitter. Refers to the data VL transmit buffer sourcing data packets onto a given link. Each data VL transmit buffer on one end of the link is associated with its respective data VL receive buffer on the other end. The following terms relate to the Transmitter:

    - NP (Next Packet). Indicates the number of FCBs in the next data packet to transmit.

    - FCTBS (Flow Control Total Blocks Sent). This is a 12-bit register that the Transmitter side logic uses to keep track of how many blocks have been transmitted since link initialization.

    - CR. Running count of total blocks sent (FCTBS) plus blocks in packet (NP) about to be sent.

    - CL (Credit Limit). The Transmitter sets this equal to the latest FCCL count sent by receiver in a FCP. CL indicates the total amount of data that the transmitter has been authorized to send since link initialization.

  • Receiver. Refers to the data VL receive buffer that receives data packets from its respective data VL transmit buffer on the other end of link. The following terms relate to the Receiver:

    - FCCL (Flow Control Credit Limit). Generated by receiver side logic. It is the sum of the Receiver's current Free Space Count and the total number of blocks that have been received. See CL.

    - ABR (Adjusted Blocks Received). Each time a FCP is received from the Transmitter, the Receiver's ABR is set to the value in packet's FCTBS field.

Receiver's Credit Limit

Receive Buffer Implementation Background

The number of data blocks that a given receive buffer can hold is dependent on three factors:

  • The overall size of the buffer.

  • The types and sizes of packets that are received.

  • The efficiency of the buffer implementation.

As an example, a particular receive buffer may be designed in such a manner that buffer space is allocated in chunks of a fixed size (larger than a single 64-byte block). Let's assume that the buffer space is allocated in 128-byte chunks.

Assume that the buffer is currently empty and a small packet arrives with a payload of less than one block. When it is placed in the buffer, one full 128-byte chunk is used (and a portion of it is wasted). The more small packets that are received, the faster the buffer fills up and the more wasted space it has.

Credit Limit Calculation

The Receiver's Flow Control Credit Limit (FCCL) value is calculated as follows:

  • If the current receiver buffer state would permit reception of 2048 or more blocks (128KB) from all combinations of valid IBA packets (see “Receive Buffer Implementation Background” on page 641) without discard, then the 12-bit FCCL = the total number of blocks already received (ABR) since link initialization + 2048.

  • Otherwise the FCCL = ABR + the actual number of blocks the receiver is capable of receiving from all combinations of valid IBA packets without discard. For example, if a receiver is capable of buffering more data when large packets arrive than when small packets arrive, then the receiver must use the smaller capacity in calculating the FCCL.

An Example

The example is based on Figure 25-11 on page 645 and Figure 25-12 on page 645 and the following set of assumed startup conditions.

Figure 25-11. Example Flow Control Scenario


Figure 25-12. Example Flow Control Scenario (continued)


Assumptions

The following operational conditions are assumed at the Receiver:

  • The total available buffer space at the Receiver = 3072 blocks (3072 X 64 = 196,608 bytes).

  • No data has been received yet.

  • The Receiver maintains a Free Space Counter (currently set to 3072):

    - As each packet is accepted into the data VL receive buffer, the number of FCBs in the packet is subtracted from the Free Space Counter.

    - When each packet is then forwarded from the data VL receive buffer, the number of FCBs in the packet is added to the Free Space Counter.

and these operational conditions are assumed at the Transmitter:

  • No data packets have been transmitted from this data VL transmit buffer since link initialization.

  • The Transmitter verifies that the remote data VL receive buffer has sufficient free space available in the following manner:

    - CL – CR. It subtracts its CR (the running count of total FCBs transmitted since link initialization + the number of FCBs in the packet about to be transmitted) from its CL (the latest FCCL count sent by the remote data VL Receiver).

    - If CL – CR ≥ 0, the remote data VL receive buffer has sufficient space and the packet can be sent.

  • In case the Receiver had dropped one or more blocks in its ABR (Adjusted Blocks Received) count, the Transmitter periodically sends its FCTBS (Flow Control Total Blocks Sent) value to the Receiver to ensure that the Receiver has the correct total number of blocks sent since link initialization.

Step-by-Step Explanation

  1. The Receiver currently has sufficient free buffer space to hold 3072 FCBs (3072 X 64 = 196,608 bytes).

  2. ABR (Adjusted Blocks Received) = the total number of data blocks received since link initialization. ABR is currently 0.

  3. FCCL (Flow Control Credit Limit) = Free Space (3072 blocks) + ABR (no FCBs have been transferred so far). It would appear then that FCCL = 3072 + 0 = 3072. Note, however, that if the Free Space Count is greater than 2048, the value 2048 (rather than the actual Free Space Count) is added to the ABR to yield FCCL (see “Credit Limit Calculation” on page 642). FCCL therefore = ABR (0) + 2048 = 2048.

  4. During initialization, the Receiver sends its FCCL value (currently = 2048) to its companion remote data VL Transmitter in a FCP. Upon receipt of the FCP, the respective data VL transmit buffer sets its Credit Limit (CL) = the FCCL received in the FCP. In this case, the transmit buffer's CL is set to 2048.

  5. Since no data packets have been transmitted from this data VL transmit buffer yet, its FCTBS (Flow Control Total Blocks Sent) = 0.

  6. The Transmitter wishes to send a data packet containing 10 FCBs (10 X 64 = 640 bytes) in its payload. It determines if it has sufficient credit to do so by subtracting its CR (FCTBS + NP = 0 + 10 = 10) from its current CL (2048) = 2038. Since CR ≤ CL, the packet can therefore be sent.

  7. Transmitter transmits the packet and adds 10 to its FCTBS (0 + 10 = 10 total FCBs sent).

  8. When the Receiver accepts the data packet, it takes the following actions:

    - Stores the packet in the data VL receive buffer.

    - Adds 10 to its current ABR (0) = 10.

    - Subtracts 10 from its Free Space Count (3072) = 3062.

    - Recomputes its FCCL value: 2048 + ABR (10) = 2058.

  9. When the Transmitter is ready to send its next packet, the same process is repeated:

    - The next packet has a payload of 5 FCBs (5 X 64 = 190 bytes).

    - The Transmitter determines whether it has sufficient credit to do so by subtracting its CR (FCTBS + NP = 10 + 5 = 15) from its current CL (2048) = 2038. Since CR ≤ CL, the packet can be sent.

    - FCTBS is updated to reflect the total FCBs sent since link initialization (FCTBS + NP = 15).

  10. The packet is transmitted.

  11. When the Receiver accepts the data packet, it takes the following actions:

    - Stores the packet in the data VL receive buffer.

    - Adds 5 to its current ABR (10) = 15.

    - Subtracts 5 from its Free Space Count (3062) = 3057.

    - Recomputes its FCCL value: 2048 + ABR (15) = 2063.

  12. The processes just described continue until the Receiver's Free Space Count decreases to < 2048. The example continues at the point just prior to this change.

  13. Assume that:

    - 1024 blocks have been received.

    - The Receiver VL buffer hasn't offloaded any of the received blocks from its receive buffer to a higher layer yet.

    - The Receiver has recently sent a FCP (see Table 25-3 on page 639) to the Transmitter containing:

    - FCTBS = na. This field is na when the FCP is sent from the Receiver to the Transmitter. Transmission errors in data packets, or in the exchange of flow control information in FCPs, can result in inconsistencies in the flow control state perceived by the transmitter and receiver. To ensure that the Receiver's ABR matches the Transmitter's FCTBS, the Transmitter periodically sends a FCP to the Receiver with the correct FCTBS value. The receiver uses this data to update its ABR to match the Transmitter's FCTBS.

    - FCCL = 3072.

    - The FCTBS and ABR counts agree (either because no packets were lost or discarded or because the ABR was updated with the correct FCTBS).

  14. The Transmitter and Receiver counts would have the values indicated in Figure 25-12 on page 645.

  15. The Transmitter again performs a credit check and finds ample room in the receive buffer for the next packet (with a payload of 3 FCBs). It transmits the next packet and adds 3 to the FCTBS count (1024) yielding an updated FCTBS of 1027.

  16. Upon packet reception, the Receiver updates its ABR from 1024 to 1027 and its Free Space Count from 2048 to 2045.

  17. For the first time since link initialization, the Receiver has less than 2048 blocks of free space (it has 2045). The updated FCCL value = the actual Free Space Count (2045) + ABR (1027) = 3072. Because the Free Space Count decreases and the ABR increases with each block received, the FCCL will not change until the Receiver starts offloading blocks from its receive buffer to a higher layer.

Receive Buffer Full Example

The ABR, FCCL, FCTBS, and Free Space Count registers are all 12 bits wide. When the current register contents is FFFh and one is added, the register rolls over to 000h.

Refer to Figure 25-13 on this page. The example assumes that the transmitter has sent 3072 blocks and that the receiver has not yet offloaded any data from its buffer to a higher layer. The current state of the transmitter and receiver are shown in the figure.

  1. Note that the receiver's FCCL is still at 3072 and the Free Space Count = 0.

  2. The next packet to be transmitted contains a payload one FCB in size. When the packet's FCB count (1) is added to the FCTBS, the CR value becomes 3073 (FCTBS of 3072 + NP of 1 = 3073). The credit check results in CL < CR, prohibiting the transfer of the data packet.

  3. The data packet cannot be sent until the receiver offloads at least one block of data from its receive buffer to a higher layer. When this occurs, this will increase the Free Space Count and therefore the FCCL (FCCL = Free Space Count + ABR). The new FCCL (in this case, 3073) is sent to the transmitter the next time that the receiver sends it an FCP.

Figure 25-13. Example Flow Control Receive Buffer Full Scenario


FCCL Rollover Condition

Refer to Figure 25-14 on page 648. In this example, the receiver is offloading data from its receive buffer to a higher layer faster than the remote transmitter is sending data to it. The last FCCL value sent to the transmitter in an FCP was 4090. Since then, the receive buffer has offloaded 5 additional blocks (FCCL = 4095), thereby increasing the Free Space Count by five.

  1. The Receiver offloads another block from its receive buffer, causing the Free Space Count to increase to 510. When the new FCCL value is calculated (Free Space Count of 510 + ABR of 3586 = 4096), the FCCL rolls over to 000h.

  2. The Transmitter still has a CL of 4090 (the last FCCL value sent by receiver in a FCP). When the next data packet is ready for transmission (and this example assumes that its payload is one FCB—64 bytes—in size), the Transmitter performs the credit check and detects a “go” condition (4090 – 3587 = 503). Note that the Transmitter is unaware that six additional blocks have been offloaded from the receive buffer.

  3. The data packet is sent and the Transmitter's FCTBS is increased to 3587.

  4. When the data packet arrives at the Receiver, the normal actions are taken:

    - The packet data is placed in the receive buffer.

    - The ABR is increased by the payload size of the received packet (one FCB in this case).

    - The Free Space Count is decreased by the payload size of the received packet.

    There is no change in the FCCL value (because the ABR increased by one and the Free Space Count decreased by one).

  5. Now assume that the Receiver sends an FCP to the Transmitter.

  6. The FCCL in the FCP updates the Transmitter's CL to 000h.

  7. When the Transmitter has another data packet ready to send, the credit check (CL – CR) yields a negative result. Note, however, that the binary subtraction performed by adding the ones complement of the CR to the CL value yields 509 blocks of free space (an accurate reflection of the Receiver's Free Space Count):

    CL = 000h.
CR = E03h (3587 decimal).

     000
    +1FD (1's compliment of E03h)
    1FDh (509 decimal)

Figure 25-14. Example Flow Control FCCL Rollover Scenario


..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset