Completion Queues (CQs) and CQEs
Purpose of CQ
After a message transfer completes, either with or without error, the WQE is retired from the respective queue (SQ or RQ) and a completion notice is recorded in the CQ associated with the respective queue (SQ or RQ). The entry created in the CQ is referred to as a Completion Queue Entry, or CQE (pronounced “cookie”). Each CQE contains the 64-bit WR ID that was assigned to the original message transfer WR posted to the SQ or the RQ by software. This permits software to associate a CQE with a previously issued message transfer WR.
CQ-Related Verbs
General
The verbs that are associated with CQs are:
Create CQ.
Query CQ.
Resize CQ.
Destroy CQ.
Set Completion Event Handler.
Poll For Completion.
Request Completion Notification.
The sections that follow provide a description of each of these verbs.
Create CQ
Input Parameters
The caller specifies the following input parameters:
- HCA Handle.
- The minimum size of the CQ.
Action Taken
The HCA attempts to create a CQ of at least the requested size. If the requested size is greater than the size supported by the HCA, an error is returned. Otherwise, the HCA creates a CQ of at least the requested size. The HCA assigns a unique handle to the newly created CQ.
Output Parameters
On successful completion, the verb returns the following output parameters:
- The handle of the newly created CQ.
- The actual size of the CQ.
Query CQ
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- CQ handle.
Action Taken
The verb returns the size of the specified CQ.
Output Parameters
On successful completion, the verb returns the following output parameter: The actual size of the CQ.
Resize CQ
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- CQ handle.
- The new, desired CQ size.
Action Taken
The verb adjusts the size of the CQ to, at a minimum, the newly requested size.
Output Parameters
On successful completion, the verb returns the following output parameters:
- The adjusted size of the CQ.
HCA May Not Support Dynamic CQ Resizing
It should be noted that the HCA may not support the ability to dynamically resize a CQ. Software can determine the HCA's capabilities by executing the Query HCA verb.
Destroy CQ
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- CQ handle.
Action Taken
The verb destroys the indicated CQ. It should be noted that an error is returned if the CQ is still associated with one or more WQs.
Output Parameters
On completion, the verb returns success or failure.
Set Completion Event Handler
General
For a complete description, see “Notification of CQE Posting” on page 286.
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- Entry point of the handler to be called when a CQE is posted to any CQ for which completion notification has been requested.
Action Taken
The verb registers the indicated handler with the HCA's device driver. The driver will then call the handler each time the HCA indicates (typically via an interrupt) that a CQE has been posted to a CQ for which CQE posting notification has been requested (see “Request Completion Notification” on page 286).
Output Parameters
On completion, the verb returns success or failure.
Poll for Completion
General
For a complete description, see “Notification of CQE Posting” on page 286.
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- CQ handle.
Action Taken
The verb checks the indicated CQ for newly posted CQEs. If any are present, the oldest unprocessed CQE is returned to the caller. If no new CQEs have been posted to the specified CQ, an error is returned.
Output Parameters
On successful completion, the verb returns the CQE that was most recently posted to the specified CQ.
Request Completion Notification
General
For a complete description, see “Notification of CQE Posting” on this page and “Solicited and Unsolicited Events” on page 288.
Input Parameters
The caller specifies the following input parameters:
- HCA handle.
- CQ handle.
- Type of completion notification requested (next completion or a solicited completion).
Action Taken
The verb adds the indicated CQ to the list of CQs for which completion notification has been requested. The completion event handler (see “Set Completion Event Handler” on page 285) will be called when a CQE of the specified type (next completion or a solicited completion) has been posted to the specified CQ.
Output Parameters
On completion, the verb returns success or failure.
CQ/WQ Association
Prior to creating a QP, the Create CQ verb is called (once or twice) to create CQ(s) of a specified size. This verb creates a CQ and returns a CQ Handle to the caller. When the Create QP verb is then called, the handle (or handles) of the CQ(s) associated with the QP's SQ and RQ are specified. A CQ can be simultaneously associated with one or more WQs (SQ and/or RQ) on the same or different QPs.
The special SMI and GSI QPs (QP0 and QP1) can share a CQ, but they may not share a CQ with any other type of QP.
Notification of CQE Posting
Software may register to be automatically informed whenever a message transfer completes. This is accomplished in the following manner:
Execute the Set Completion Event Handler verb to register a handler to be called when a CQE is posted to a CQ. See “Set Completion Event Handler” on page 285.
Execute the Request Completion Notification verb, specifying a CQ and the type of completions to be notified of (see “Request Completion Notification” on page 286).
When the next CQE of the specified type is posted to a CQ for which notification has been requested, the handler is called. The HCA and CQ handles that identify the CQ to which a CQE of the desired type has been posted are passed to handler.
In the handler, the Poll for Completion verb is executed specifying the CQ.
The verb returns the oldest unprocessed CQE from the specified CQ. The contents of a CQE are detailed in the next section.
CQE Contents
If the message transfer was not successful, some of the CQE contents are undefined (as noted below). The CQE contents are as follows:
The 64-bit WR ID from WR. This is always valid regardless of the message transfer's completion status.
The operation type specified in the WR. The valid types are:
- Send (for WRs posted to the SQ).
- RDMA Write (for WRs posted to the SQ).
- RDMA Read (for WRs posted to the SQ).
- Compare and Swap If Equal (for WRs posted to the SQ).
- Atomic Fetch and Add (for WRs posted to the SQ).
- Memory Window Bind (for WRs posted to the SQ).
- Send Data Received (for WRs posted to the RQ).
- RDMA Write with Immediate Data received (for WRs posted to the RQ).
The number of bytes transferred.
If immediate data is present in the CQE, the Immediate Data indicator bit is set to indicate this.
The remote node address and QP. Note that this only true for a RD or UD QP. The information returned is:
- LID address of the port that sent the packet.
- SL in the packet.
- Source QPN from the packet's DETH:SrcQP field.
- Local EECN (only applies to RD, not UD).
- LID address of the port that the packet arrived on (only applicable to UD, not RD).
The GRH Present indicator bit (only present if this is an UD QP):
- If the bit is set to one, then the first 40 bytes of Scatter buffer(s) contain the incoming message's GRH.
- If the bit is cleared to zero, then the first 40 bytes of the Scatter buffer(s) are undefined. The actual message data begins after the first 40 bytes.
The P_Key index is present in the CQE if this is a GSI QP.
The message transfer's completion status is reflected in the CQE's Status field.
The Freed Resource Count. This only applies to RD RQs. It is always valid, regardless of the message transfer's completion status. If the count = 0, it indicates that no RQ WQEs were retired when this CQE was generated (the specification says this, but if no RQ WQEs were retired then how did this CQE get generated?). If the count is greater than zero, software can assume the counter indicates the number of WQEs retired from the RD RQ. This is useful for software to keep track of the number of available WRs that can be outstanding. For most implementations, this count is expected to be one in each CQE.
Solicited and Unsolicited Events
General
When the Post Send Request verb is executed to post a Send or an RDMA Write With Immediate WR to the SQ, the Solicited Event Indicator is specified in the WR:
If = 0, then the BTH:SE bit = 0 in the “Last” or “Only” request packet of the Send, Send With Immediate, or RDMA Write With Immediate message transfer.
If = 1, then the BTH:SE bit = 1 in the “Last” or “Only” request packet of the Send, Send With Immediate, or RDMA Write With message transfer.
As described earlier (in “Request Completion Notification” on page 286), the Request Completion Notification verb is executed to request that the CQ event handler be called when the next CQE of the specified type (Solicited or Unsolicited Event) is posted to the specified CQ. The next two subsections define a Solicited versus an Unsolicited Event.
Solicited Event
A Solicited Event occurs when either:
an incoming Send or RDMA Write With Immediate message with the BTH:SE bit set to one causes a successful message transfer CQE to be posted to the CQ associated with a RQ.
an unsuccessful message transfer CQE is posted to a CQ associated with a RQ.
Unsolicited Event
An Unsolicited Event occurs when either:
any other successful message transfer RQ CQE is posted to the CQ associated with a RQ.
any successful message transfer SQ CQE is posted to a CQ associated with a SQ.