QP State Before Any Messages Transferred. It is always helpful to establish the state of things just before normal operation begins. This section initiates the discussion by establishing the state of the local and remote QPs after they have been created and set up, but before any messages have been transferred between the two.

Standard Operation in Fast, Error-Free Environment. This section describes how messages are transferred in a “perfect” environment. It assumes the following set of conditions:

- No delay in request packet delivery. The request packets issued by the requester QP's SQ Logic make their way to the destination QP's RQ Logic in an expeditious manner.

- Quick generation of responses. The responder QP's RQ Logic processes each request packet rather quickly and issues the corresponding response packet back to the requester QP.

- No delay in response packet delivery. The response packet makes its way back to the requester QP's SQ Logic quickly.

- No packets lost. No request or response packets are “lost” in the fabric due to errors of any sort.

- No error conditions are encountered, neither within the two CAs nor in the packet flight path.

- No traffic reduction. The responder QP's RQ Logic does not implement Acknowledge coalescing (more on this later). In other words, the responder QP's RQ Logic is designed to issue a response (or, possibly, multiple responses in the case of an RDMA Read request) for each request packet received.

Traffic Reduction. This section describes the Ack coalescing mechanism that can be used to decrease the amount of response traffic injected into the fabric. Support for Ack coalescing is optional for the RQ Logic and mandatory for the SQ Logic.

Packet Delivery Delays. The next phase of the discussion assumes the following:

- No request or response packets are lost in transit. This section assumes that all request packets generated by the requester QP's SQ Logic eventually arrive at the responder QP's RQ Logic. Likewise, it assumes that all responses generated by the responder QP's RQ Logic eventually arrive at the requester QP's SQ Logic.

- Delivery of request or response packets may be delayed. The message request and/or response packets may be delayed along the flight path between the requester and responder QPs. This can result in some interesting situations that the requester and responder QPs must handle in a graceful manner. This section describes the possible causes of packet delivery delays, the resulting situations, and how the QPs must handle each situation.

- No error conditions are encountered, neither within the two CAs nor in the packet flight path.

- Ack coalescing. The responder QP's RQ Logic may or may not implement Acknowledge coalescing.

Packet Loss. Due to various conditions that might arise as a packet traverses the fabric between the requester and responder QPs, a request or response packet may be dropped along the way. This section describes the possible causes of packet loss, how the requester or responder detects packet loss, and how the loss is dealt with.

Nak Errors. Under certain error conditions, the responder QP's RQ Logic returns a Nak (Negative Ack) rather than a positive Ack, an RDMA Read response, or an Atomic response. This section provides a detailed description of each Nak, its possible causes, and the actions taken by both the requester QP's SQ Logic upon receipt of the Nak and the responder QP's RQ Logic after sending it.

End-to-End Flow Control. Upon arrival at the responder QP's RQ Logic, some messages require the existence of a previously posted RQ WQE to handle the incoming message. If no WQE is posted, the message transfer is rejected and the requester QP's SQ Logic will have to retry it. The End-to-End Flow Control feature permits the responder QP's RQ Logic to tell the requester QP's SQ Logic how many WQEs are currently posted to its RQ. The SQ Logic then only launches a message that requires a RQ WQE if a RQ WQE exists to handle it.

SQ Logic Can Use MSN to Complete WQEs. This section provides a detailed description of the Message Sequence Number (MSN) returned by the responder QP's RQ Logic and how it is used by the requester QP's SQ Logic.

Additional Reference Information. The author has chosen to place some necessary (but dry) information in this section.