End-to-End Flow Control
The end-to-end credit mechanism applies only to the RC transport service.
Retries Increase Fabric Traffic
Only the Send and RDMA Write With Immediate operations require that a RQ WQE be present in the responder QP's RQ to handle them. If either type of request packet were launched when there are no WQEs posted to the target RQ, it results in the return of an RNR Nak and the need to retransmit the request packet.
Don't Launch Request unless RQ Logic Is Ready for It
The preferred strategy would therefore be to only launch a Send or an RDMA Write With Immediate request packet if it is known beforehand that there are one or more WQEs presently posted to the responder QP's RQ. That brings up the question: How would the requester QP's SQ Logic know whether or not any WQEs are currently posted to the responder QP's RQ?
Is Credit Reporting and Acceptance Required?
The following rules apply:
The RQ Logic of a RC QP in an HCA must report the number of WQEs currently posted on its RQ (referred to as the number of credits) to the SQ Logic of the QP in the remote CA.
It is optional whether or not the RQ Logic of a RC QP in a TCA reports the number of WQEs currently posted to its RQ.
If the responder QP's RQ Logic does report the number of WQEs currently posted to its RQ, the requester QP's SQ Logic is required to accept this value.
Credit Support Established during Connection Setup
At connection establishment time, the CMs associated with the two CAs exchange information about the two QPs in the REQ and REP communications messages. One of the items exchanged is a bit (the End-to-End Flow Control bit) that indicates whether or not the RQ Logic of each of the two QPs will supply credits to the other QP's SQ Logic.
It should be noted that the specification does not say how the CM would tell a QP's SQ Logic whether or not the remote QP's RQ Logic will supply credits. The author would have expected this to be a parameter supplied when making the Modify QP verb call, but it's not listed as an input parameter. It's a mystery! (Late word has it that this will result in a change to the specification.)
RQ Logic Reports Number of RQ WQEs in Ack Packet
General
Figure 17-6 on page 369 illustrates the format of an Ack packet. The upper 8-bits of the AETH is the Syndrome field and its format is defined in Table 17-3 on page 370. The 5-bit Credit Count field is only present in a positive Ack packet. As indicated earlier, the Credit Count indicates the number of WQEs currently posted to the RQ Logic's RQ. The encoding of the Credit Count field is defined in Table 17-8 on page 421. A value of 11111b indicates that the RQ Logic is not providing credit information.
Credits Provided in Normal Ack Packet
A QP's RQ Logic supplies its available credits in each Ack packet that it returns to the other QP's SQ Logic. This is referred to as “piggybacking” credits in an already scheduled Ack packet.
Credits Provided in Unsolicited Ack Packet
A QP's RQ Logic can also supply its available credits in an unsolicited Ack packet. An unsolicited Ack packet is presented to the requester QP's SQ Logic as a duplicate of the most recently sent positive Ack packet (i.e., it uses the same PSN). The BTH:Opcode of an unsolicited Ack packet, however, is always Acknowledge even if the most recently delivered positive response packet was an RDMA Read response packet or an Atomic response packet. The protocol does not permit the responder to send duplicate Naks, so an unsolicited Ack cannot be sent by resending a Nak packet. An unsolicited Ack packet can be sent at any time and appears to be a duplicate Ack to the requester QP's SQ Logic. Upon receipt of an unsolicited Ack, the requester QP's SQ Logic saves the Credit Count and the MSN (Message Sequence Number; more later) and then discards the packet.
Providing Initial Credits
Before any messages have been transferred between the two QPs, no Ack packets have been transmitted, so the SQ Logic has not yet received any credits from the RQ Logic in the remote QP. If a WR were to be posted to the SQ specifying a Send operation or an RDMA Write With Immediate operation, the SQ Logic must behave as defined in “SQ Logic's Behavior When No Credits Are Available” on page 420.
In order to deliver its initial credits to the SQ Logic, the remote RQ Logic creates an unsolicited Ack packet in the following manner. The value “1” is subtracted from the RQ Logic's assigned ePSN and an Ack packet with the resultant PSN is transmitted with the initial credit count and an MSN of zero (more on the MSN later). As an example, if the ePSN was initialized to 000000h, an unsolicited Ack packet with a PSN of FFFFFFh is transmitted by the RQ Logic. This initial unsolicited Ack packet appears to be a “ghost” Ack packet to the SQ Logic that receives it (i.e., its PSN falls within the Invalid PSN range).
SQ Logic's Behavior When No Credits Are Available
If a QP's RQ Logic doesn't provide initial credits to the remote QP's SQ Logic, or any time that the SQ Logic has no credits available, the SQ Logic operates under the following constraints:
If the SQ Logic has a Send operation to perform, the SQ Logic can only transmit the first request packet for that message. It must then stop transmission and wait for an Ack packet. To ensure that the responder generates a response, the requester must set the BTH:AckReq bit in that request packet. Upon receipt of the Ack packet, the SQ Logic recovers the Credit Count and MSN from the packet's AETH and discards the Ack packet. If the received credit count indicates that the responder QP's RQ Logic has at least one WQE posted, the requester QP's SQ Logic can transmit the remaining packets of the Send operation.
If the SQ Logic has an RDMA Write With Immediate operation to perform, it may transmit all of the request packets associated with the message before it must stop transmission and wait for an Ack packet. This is permitted because the RQ Logic doesn't need to use a WQE on the RQ until it receives the final request packet containing the immediate data. Upon receipt of the message's final request packet, the RQ Logic writes the packet's data payload to local memory, retires the top WQE on the RQ, creates a RQ CQE, and stores the 32-bit immediate data value in the CQE. To ensure that the responder generates a response, the requester sets the AckReq bit in the last request packet of the message.
If the SQ Logic has an RDMA Write Without Immediate, an RDMA Read, or an Atomic operation to perform, the request packets associated with that message transfer may be sent without regard to the availability of credits (because these operations do not need a RQ WQE on the other end).
SQ WQEs Are Internally Numbered
Refer to Figure 17-18 on page 423. As each WQE is posted to a QP's SQ, the SQ Logic internally assigns it a 24-bit ID referred to as its Send Sequence Number, or SSN. The first WQE posted has an SSN of 000000h, the second one an SSN of 000001h, and so on. In the example pictured, requests (i.e., WQEs) 1 through 32 have SSNs of 000000h through 00001Fh, respectively:
Message transfer requests 1 through 25 have been completed.
Message transfer requests 26 through 35 have yet to be transmitted.
Figure 17-18. MSN + Credit Count = LSN
MSN Indicates Last Message Completed by RQ Logic
When a RC QP is first created, its MSN and SSN are initialized as follows:
The RQ Logic sets its MSN = 000000h.
The SQ Logic sets its SSN = 000000h. The first WQE posted to the SQ has an SSN of one. As each WR is posted to the SQ, the SQ Logic assigns it the next sequential SSN.
To the SQ Logic receiving Acks, the MSN delivered in each Ack packet represents the SSN of the message transfer most recently completed by the remote QP's RQ Logic.
MSN + Credits = LSN
In Figure 17-18 on page 423, the remote QP's RQ Logic has returned an Ack packet with a Credit Count of 6 and an MSN of 000018h. When the SQ Logic receives an Ack packet and recovers the Credit Count and MSN delivered in it, it adds the Credit Count (indicating the number of WQEs currently posted to the remote QP's RQ) to the MSN. This yields the Limit Sequence Number, or LSN.
In the example, adding the MSN (000018h) and the Credit Count (6) yields an LSN of 1Eh. This gives the SQ Logic permission to transmit the request packets for messages 26 through 31 to the remote QP's RQ Logic. In the worst-case scenario, message transfer requests 26 through 31 could all be Send or RDMA Write With Immediate operations and could all be transmitted because the RQ Logic has indicated that it has six WQEs currently posted to its RQ to handle inbound Send or RDMA Write With Immediate operations. Message transfer requests 26 through 31 are therefore referred to as unlimited WQEs.
Figure 17-19 on page 425 summarizes the SQ Logic's Credit Count initialization, credit recovery from Ack packets, and credit checking.
Figure 17-19. Requester QP's SQ Logic Credit Recovery and Credit Check
Limited WQEs
If the SQ Logic completes transmitting all of the request packets associated with messages 26 through 31 and message transfer request 32 requires the presence of a RQ WQE on the remote QP's RQ, then the SQ Logic must behave as defined in “SQ Logic's Behavior When No Credits Are Available” on page 420. For this reason, if any of message transfer requests 32 through 35 are Send or RDMA Write With Immediate operations, they are referred to as limited WQEs. Unless the LSN is ≥ to their respective SSNs, they must behave as defined in “SQ Logic's Behavior When No Credits Are Available” on page 420.
LSN + 1 for Each SQ WQE That Doesn't Need RQ WQE
The SQ Logic also adds one to the LSN for each message transfer request posted to the SQ that doesn't require a RQ WQE at the remote end to handle the transfer. This would include RDMA Read, RDMA Write Without Immediate, and Atomic operations.
As an example, assume that message transfer requests 27 and 29 are not Send or RDMA Write With Immediate operations, while 26, 28, 30, and 31 are. The LSN would then be 000020h, rather than 1Eh (MSN of 000018h + 6 +1 +1).
Obey the Ordering Rules
The ordering rules dictate that the SQ Logic processes its WQEs in the order that they were posted. It is never permitted to process them out of order. For this reason, the specification states the following:
“The requester may not search the send queue looking for requests which don't consume a receive WQE and transmit those requests out of order, nor may the requester violate the rules governing fenced WQEs.”
When a WR is posted to a QP's SQ, if the Fence Indicator bit is set in the WR, then all RDMA Read and Atomic WRs posted to the SQ before this WR must complete before the SQ Logic can begin execution of this WR.
No Credit Check on Retry Due to Lost Packet
Consider the following scenario:
The requester QP's SQ Logic has a Send or an RDMA Write With Immediate operation to perform.
The responder QP's RQ Logic has indicated it has sufficient credits to handle it.
The SQ Logic starts transmitting the request packets associated with this message.
At some point, one or more of the request or corresponding response packets are ost while en route.
The SQ Logic has a Transport Timer timeout and therefore has to retry the request packet.
The SQ Logic does not need to check for an available credit before the retransmission. Insufficient End-to-End Flow Control credits only limit the transmission of new request messages.
RQ WQE's Scatter Buffer List May Not Be Large Enough
The requester QP's SQ Logic can launch a Send operation to the responder QP's RQ Logic if the RQ Logic has indicated that it has a credit (i.e., a RQ WQE) to handle the incoming message. However, just because a RQ WQE is present does not necessarily mean that its Scatter Buffer List defines an area of memory large enough to hold the entire message.
If the message is larger than the buffers intended to hold it, the RQ Logic will generate an error to its local software upon receiving the Send request packet whose data payload overruns the final Scatter buffer.
The RQ Logic will respond to that request packet with an Invalid Request Nak (see “Reason for Invalid Request Nak” on page 404).
The RQ Logic takes the actions defined in “RQ Logic Behavior after Returning Invalid Request Nak” on page 405.
Upon receipt of the Nak, the requester QP's SQ Logic takes the actions defined in “Requester's Reaction on Receipt of Invalid Request Nak” on page 407.
RQ Logic Credit Calculation
Refer to Figure 17-20 on page 428. The RQ Logic initializes its Credit Count to zero when the QP is first created or whenever the QP is commanded (via a Modify QP verb call) to return to the Reset state. While the QP is in the Initialize, RTR, or RTS states, the Credit Count is incremented each time a WR is posted to the QP's RQ. While the QP is in the RTR or RTS state, each time the first request packet of a new Send or RDMA Write With Immediate operation arrives, the RQ Logic checks for an available credit (i.e., WQE on the RQ) and takes the following actions:
If no credit is available, the RQ Logic returns an RNR Nak to the requester QP's SQ Logic.
If a credit is available, the RQ Logic takes the following actions:
- Decrements its Credit Count by one.
- Returns a response packet with its adjusted Credit Count.
Figure 17-20. Responder QP's RQ Logic Credit Calculation
