Requester QP's SQ Logic PSN Generation and Verification

Start PSN Assignment

General

Refer to Figure 11-2 on page 212. When a QP is created, one of the values that must be set is the SQ Logic's Start PSN value. This is the PSN that will be inserted into the first request packet generated by the SQ Logic. It may be any value in the range from 000000h through FFFFFFh.

Select a Start PSN Value That Precludes Stale Packets

The specification stresses that a value should be selected that:

“minimizes the chance that a packet from a previous connection could fall within the valid PSN window.”

The specification is referring to either a request packet received by a QP's RQ Logic or a response packet received by a QP's SQ Logic in response to a request packet issued at an earlier time. The “valid PSN window” is the range of PSNs up to 2²³ (8M) in size consisting of the Ack'd PSN range (also referred to as the duplicate response region) plus the unAck'd region (i.e., up to the PSN of the request packet most recently transmitted by the SQ Logic).

When a QP is created, the verb layer assigns it a 24-bit QP Number (QPN). It is possible that a QP that had been in use earlier in time had the same QPN assigned as a new one just created and in the process of being set up. Alternately, the connection management software might use the Modify QP verb call to reset a local and remote QP that have been actively sending messages to each other.

The SQ Logic of earlier QPs had been assigned Start PSNs and, during their lifetimes, had transmitted a number of request packets with ascending PSNs in each packet. Subsequently, those QPs may have been deleted (i.e., destroyed) or reset and then set up again while they still had some request packets, or their respective responses, in flight. These orphan request and/or Ack packets are referred to as “stale” packets.

When a new QP and a remote QP that it communicates with are then created (or are reset) and put into play, if their respective SQ Logics were assigned Start PSNs that fell within the range of PSNs of any in-flight orphan requests and Acks, the orphan requests or Acks could be mistaken for valid requests or valid responses to requests issued by this new QP. The specification therefore urges software to assign a Start PSN to a new QP's SQ Logic so that any possible orphan requests or Acks that may be received by a QP's RQ Logic or SQ Logic will fall within the Invalid range (i.e., before the Start PSN or after the PSN in the most recently issued request packet).

No Protection From Stale Packets

If the connection management software were to tear down (i.e., destroy) the connection between two QPs and then chooses to reuse the two QPs too soon (i.e., before enough time has passed for any stale request and/or response packets to be dropped in the fabric), stale request or response packets with PSNs that fall within the valid areas might arrive at a QP's SQ Logic or RQ Logic. They will be perceived as valid request or response packets and will mortally screw things up.

If this is permitted to happen, there is no protection against it. It is the responsibility of the connection management software to avoid reusing QPs for a period long enough to ensure that there is no possibility that a stale packet could arrive at the SQ Logic or RQ Logic. The specification refers to this period of time as the “Time Wait” period.

Request Packet PSN Generation

As it transmits request packets, the SQ Logic assigns each request packet a PSN based on the following rules:

• The first request packet's PSN = the Start PSN assigned to the SQ Logic.

• For each subsequently issued request packet:

  - If the previously issued request packet was not an RDMA Read request packet, then the PSN inserted in the current request packet = the previous PSN + 1.

  - If the previously issued request packet was an RDMA Read request packet, then the PSN inserted in the current request packet = PSN of the RDMA Read request packet + the number of expected RDMA Read response packets.

Ack'd (aka Duplicate) Region

As the SQ Logic issues request packets and subsequently receives responses for each of them, the upper end of the Ack'd region grows upward.

Maximum Number of Outstanding Requests

As it issues each request packet, the RC QP's SQ Logic (or EEC Send Logic) doesn't wait for the respective response packet before issuing the next request packet. It continues to stream request packets and, eventually, it will begin to receive the stream of corresponding response packets. The maximum number of outstanding unAck'd request packets (of all types—Sends, RDMA Reads and Writes, and Atomic requests) that the SQ Logic is permitted to transmit before it must stall is 8M (2²³; see the unAck'd PSN range in Figure 11-2 on page 212). This is based on a worst-case scenario with the following characteristics:

• The requester QP's SQ Logic initiates an RDMA Write or a Send operation to write a 2GB message into the remote CA's local memory.

• Packets transmitted between the source and destination ports can not have a data payload bigger than that specified as the path's PMTU (see “Maximum Data Payload Size” on page 42.

• Assuming that the path PMTU were 256 bytes, a 2GB message transfer would consist of 8,388,608 request packets (for a Send or RDMA Write) or 8,388,608 RDMA Read response packets.

• The requester QP's SQ Logic starts transmitting the request packets.

• If the requester had transmitted all 8,388,608 request packets associated with the message and had not yet received any Acks for the message, it must stall at that point and not issue any additional request packets (for a new message transfer) until it starts receiving Acks.

Ack Verification

As the SQ Logic receives each response packet, it checks to make sure that it falls within the unAck'd region. Assuming it does, the SQ Logic advances the upper end of its Ack'd region accordingly.

Detecting Duplicate Acks

General

As will be seen later in the book (e.g., see “Packet Delivery Delays” on page 389), in certain circumstances the SQ Logic will retransmit (i.e., retry) a request packet. As a result, the remote QP's RQ Logic may receive a duplicate copy of a request after it had already issued a response to the first copy. The SQ Logic would end up receiving a second copy of the response packet with the same PSN as that received in the first copy of the response packet. Upon receipt of the first copy of the response, the SQ Logic would have advanced the upper end of its Ack'd region, so the second copy of the response packet has a PSN that falls within the Ack'd region (i.e., the duplicate response region). The SQ Logic is required to handle a duplicate response in a graceful manner by quietly dropping it.

Worst-Case Duplicate Range Size

In “Maximum Number of Outstanding Requests” on page 208, a worst-case scenario of a 2GB Send or RDMA Write consisting of 8,388,608 packets was described. Assume that the requester had transmitted the entire 2GB message and had received responses for all but the last one. At this moment in time then, the size of the Duplicate Region is 8,388,607 PSNs. Assume that the requester now experiences a timeout waiting for the last Ack. It would then rewind to that request packet and retry (i.e., retransmit) it.

The specification states that the requester may either rewind to the point of failure (back up by only one packet in this case) or that it may rewind its SQ to the beginning of the message and start over. If the requester in this example opts to do that, the first 8,388,607 request packets it resends will fall within the Duplicate Region and will be gracefully handled by the responder QP's RQ Logic. For a Send or an RDMA Write (as in this case), this would require the responder to issue Acks but not actually rewrite the message data in its local memory.

Because the above scenario may occur, the requester QP's SQ Logic is required to always remember the PSNs of up to the last 8,388,608 request packets it has transmitted and, if it should receive any Acks in the duplicate region, to treat them as duplicate Acks.

Sometimes Duplicate Ack Is Used to Deliver Credits

The remote QP's RQ Logic may deliberately issue a duplicate Ack packet with the same PSN as the last Ack packet it had returned in order to deliver End-to-End flow control credits to the SQ Logic. In this case, the SQ Logic recovers the credit information from the Ack packet and then silently drops the packet. For a detailed description of End-to-End flow control, refer to “End-to-End Flow Control” on page 417.

Detecting Ghost Acks

As noted in Figure 11-2 on page 212, a “Ghost” Ack may be received that has a PSN that falls within the Invalid range. This can occur when the SQ Logic receives a response packet that has been floating around in the fabric after being transmitted by a QP that was subsequently destroyed. Such a response packet may have a PSN that falls within one of the following ranges:

• The Ack'd (i.e., duplicate) region. The SQ Logic is required to handle this gracefully by just quietly dropping it (but—as described in “Sometimes Duplicate Ack Is Used to Deliver Credits” on page 210—if the Ack packet contains credits, the credits and the MSN must be recovered from the packet before dropping it).

• The invalid range. In this case, the packet is quietly dropped (it should be noted, however, that there is one exception—“Providing Initial Credits” on page 419).

• The unAck'd portion of the valid range. In this case, the ghost response packet might appear to be a valid response packet (if its opcode is an expected one). As stated earlier in “No Protection From Stale Packets” on page 207, this stale packet would be interpreted as a valid response packet and really screw things up.

SQ Logic PSN Rollover

Over the lifetime of a QP, its SQ Logic may generate many, many request packets. At some point, it may reach the upper boundary of the overall PSN range and issue a request packet with a PSN of FFFFFFh. This brings up the question of what PSN it will insert in the next request packet it generates.

The specification does not describe the actions taken by the QP's SQ Logic in this eventuality. It is the author's opinion that after generating a PSN of all ones, the SQ Logic's PSN will roll over to zero and continue growing upwards from there.