This section provides a detailed description of the MADs sent between the CMs associated with two CA's to setup, modify, or tear down a communications channel between either:
A QP in one CA and a QP in the other CA (if RC or UC).
An EEC in one CA and an EEC in the other CA (if RD).
In the description of each field in a message, the words “local” and “remote” are frequently used (e.g., the local CM and the remote CM). The use of these two terms can be confusing. They are defined as follows:
Local. Refers to one of the following:
- The CM sending the message.
- Logic (e.g., a QP) within the CA associated with the CM that is sending the message.
Remote. Refers to one of the following:
- The CM receiving the message.
- Logic (e.g., a QP) within the CA associated with the CM receiving the message.
When a software application local to a CA wishes to establish a communications channel with a service provided by a remote CA, it issues a request to the local CA's CM. In turn, the CM initiates a communications establishment message exchange with the remote CA's CM, starting with the sending of the REQ message (see Table 36-4 on page 1075) and Figure 36-1 on page 1075.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from REQ sender's point of view. REQ sender:
|
Reserved | 32-bit field. |
ServiceID | RC/UC/RD. 64-bit value. Sent by the REQ sender to indicate the type of service it wishes to establish a connection with.
CM receiving REQ uses the supplied ServiceID to determine if the desired service exists within its associated subsystem. If the service exists, the receiving CM either:
|
Local CA GUID | RC/UC/RD. 64-bit value. EUI-64 GUID of the REQ sender's CA. This is provided in case both CM's simultaneously send REQ messages to each other. In that case, the two CM's must compare the GUIDs of their associated CA's to determine which will take the active role and which the passive role for the remainder of the communications establishment message exchange. For more information, refer to “Active Client to Active Client” on page 1112. |
Local CM Q_Key | RC/UC/RD. 32-bit value. This is the Q_Key of the GSI (QP1) the CM issuing the REQ message is using to send and receive CM messages. The CM receiving the REQ message must use this Q_Key in the DETH:Q_Key field of all message MADs it sends to the CM that issued the REQ message. |
Local Q_Key | RD. 32-bit value. The CM sending the REQ MAD must supply the other CA's CM with the QPN (supplied in the REQ message's Local QPN field) and Q_Key (supplied in the REQ message's Local Q_Key field) of the local RD QP that will serve as its end of the communications channel being established. The receiving CM supplies this QPN and Q_Key to the software associated with the requested service on its end. That service-related software then uses that Q_Key and QPN in any WRs that it posts to the SQ of its QP. |
Local QPN | RC/UC/RD. 24-bit value. QPN of QP at REQ sender's end of connection.
|
Offered Responder Resources | RC/UC/RD. 8-bit value. Depth of the special queue that the REQ sender's newly created CA
QP uses to hold and process incoming RDMA Read/Atomic requests. The CM sending the REQ message obtains this value using Query HCA verb (if its associated CA is an HCA).
If = 0, this indicates that the REQ sender's CA QP doesn't support inbound RDMA Read or Atomic requests. If the queue depth indicated by this value proves insufficient for the CM receiving the REQ message, it responds to the REQ with a REJ. Conversely, if the REQ recipient's CA queue depth (indicated in the REP message) is insufficient for the REQ sender, the REQ sender responds to the REP with a REJ. Receipt of a REJ message discontinues the connection establishment process. |
Local EECN | RD. 24-bit value. This is the EECN that identifies the EEC on the REQ sender's end of the RDC. When a new RDC is being created (i.e., RDC Exists bit in this REQ message = 0), the receiving CM creates a new EEC in its CA and programs this EECN into the context of the newly created EEC. |
Offered Initiator Depth | RC/UC/RD. 8-bit value. Depth of the special queue that the REQ sender's CA QP uses to issue outgoing RDMA Read and Atomic requests to the receiving CA's QP. The CM sending the REQ message obtains this value using the Query HCA verb (if its associated CA is an HCA). The depth chosen must not exceed the depth of the corresponding queue on the receiving CA's QP or EEC. Note that the depth of the receiving CA's QP or EEC special receive queue is returned in the REP message. |
Remote EECN | RD. 24-bit value. EECN of EEC on REQ recipient's end of RDC:
|
Remote CM Response Timeout | RC/UC/RD. 5-bit unsigned value. This value tells the CM receiving the REQ message how quickly it must generate a response whenever it receives a CM message from the REQ sender. The response time is defined as
If the REQ sender cannot respond to a subsequent message received from the other CM within this time period, it must send an MRA (Message Receipt Acknowledgement); see “MRA (Message Receipt Acknowledgment) MAD” on page 1096) to prevent a timeout on the message sender's part. |
Transport Service Type | RC/UC/RD. 2-bit value. Indicates type of QP (and possibly EEC) to be created on CA receiving REQ:
|
End-to-End Flow Control | RD. 1-bit value. Indicates whether REQ sender's newly created QP
RQ Logic:
It should be noted that Table 85 in Volume 1 of the specification indicates this applies to both RC and RD, but it only applies to RC (RD does not support End-to-End Flow Control). For more information on End-to-End Flow Control, refer to “End-to-End Flow Control” on page 417. |
Starting PSN | RC/UC/RD. 24-bit value. This is the start PSN the REQ sender has assigned to the Send Logic of its newly created QP or EEC. The receiving CM creates a QP or EEC in its CA and assigns this value as the ePSN of the newly created QP's or EEC's Receive Logic. The value should be chosen to minimize the chance that a packet from a previous connection could fall within the valid PSN window. For more information, refer to “Select a Start PSN Value That Precludes Stale Packets” on page 207. |
Local CM Response Timeout | RC/UC/RD. 5-bit unsigned value. This value tells the CM receiving the REQ message how long it should wait for a response whenever it sends a subsequent communications message (e.g., a REP message) to the CM that sent the REQ message. The response time is defined as
and includes the round-trip packet flight time plus the message processing time on the other end. If the REQ sender cannot respond to a subsequent message received from the other CM within this time period, it must send an MRA to prevent a timeout on the message sender's part. |
Retry Count | RC/UC/RD. 3-bit value. Indicates the maximum number of times the QP or EEC to be created in the receiving CM's CA should retry a request packet for which:
|
P_Key | RC/UC/RD. 16-bit value. P_Key to be used for channel being established. When the receiving CM creates the new QP or EEC, it must assign a Primary P_Key Index value that selects this P_Key in the P_KeyTable attribute of the port that the QP or EEC uses to send and receive messages. |
Path Packet Payload MTU | RC/UC/RD. 4-bit value. Specifies the maximum packet payload size (i.e., PMTU), in bytes, for the channel being established. Applies to both primary and alternate paths.
|
RDC Exists | RD. 1-bit value.
|
RNR Retry Count | RC/UC/RD. 3-bit value. Indicates the total number of times the newly created QP or EEC in the receiving CM's CA is to retry request packets for which it receives RNR Naks. After this count is exhausted, the QP's or EEC's Send Logic will post an error CQE on the Send Logic's CQ. |
Max CM Retries | RC/UC/RD. 4-bit value. Maximum times either CA's CM can resend a REQ, REP, or DREQ message. After resending the maximum times without a response, the sending party terminates the connection establishment sequence by sending a REJ message indicating it timed out. |
Reserved | 4-bit field. |
Primary Local Port LID | RC/UC/RD. 16-bit value. The LID of a port on the CA sending the REQ that the new QP or EEC on the sender's end will use to send and receive messages. The receiving CM programs this LID address into the context of the QP or EEC about-to-be-created. |
Primary Remote Port LID | RC/UC/RD. 16-bit value. The CM sending the REQ message indicates a LID address of the port on the receiving CA. The receiving CM uses this value to identify which port number on its CA is to be programmed into the context of the about-to-be-created QP or EEC. The QP or EEC will use that port to send and receive messages.
In addition, the receiving CM programs the QP or EEC Context with the Source Path Bits value (refer to “Source Port's LID Address” on page 45) that will select the specified port LID address to be inserted in the LRH:SLID field of all outbound packets generated by the QP or EEC. The receiving CM may send a REJ message to reject this port, and may optionally suggest an acceptable port [see “REJ (Reject) MAD” on page 1098]. The CM sending the REQ message is responsible for ensuring that the Primary Remote Port LID and the Primary Remote Port GID field of REQ refer to same port. |
Primary Local Port GID | RC/UC/RD. 128-bit value. GID address of the port that will be used by the newly created QP or EEC in the REQ sender's CA to send and receive messages. The receiving CM programs this address into the context of the QP or EEC about-to-be-created as the DGID address. If both CAs reside in the same subnet (as indicated by the REQ:Primary Subnet Local field), this field is 0. |
Primary Remote Port GID | RC/UC/RD. 128-bit value. If the two CAs reside in the same subnet (as indicated by the REQ:Primary Subnet Local field), this value is 0. If they are in different subnets, this is the GID address of the port on the receiving CM's CA that the about-to-be-created QP or EEC will use to send and receive messages. The receiving CM uses this value to identify the Source GID Index value that will be programmed into the context of the about-to-be-created QP or EEC. The index value selects the specified GID address to be inserted in the GRH:SGID field of all outbound packets generated by the QP or EEC. Also see the description of the Primary Remote Port LID field in this table. |
Primary Flow Label | RC/UC/RD. 20-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:FlowLabel field of all outbound packets generated by the QP or EEC. |
Reserved | 4-bit field. |
Primary Packet Rate (IPD) | RC/UC/RD. 8-bit value. Tells the receiving CM what IPD value to program into the context of the about-to-be-created QP or EEC. It defines the maximum rate at which the QP or EEC may transmit packets. See “Static Rate Control” on page 589 for more information. |
Primary TClass | RC/UC/RD. 8-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:TClass field of all outbound packets generated by the QP or EEC. |
Primary Hop Limit | RC/UC/RD. 8-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:HopLmt field of all outbound packets generated by the QP or EEC. |
Primary SL | RC/UC/RD. 4-bit value. The receiving CM programs this value into the context of the about-to-be-created QP or EEC. It defines the value placed in the LRH:SL field of all outbound packets generated by the QP or EEC. |
Primary Subnet Local | RC/UC/RD. 1-bit value. Tells the receiving CM whether the two CAs reside in the same or different subnets:
|
Reserved | 3-bit field. |
Primary Local Ack Timeout | RC/UC/RD. 5-bit unsigned value. Transport (Ack) timeout for use by the receiving CA's QP or EEC
Calculated by the REQ sender based on (2 X Packet LifeTime + Local CA's response generation delay). The receiving CM may program this value into the context of the about-to-be-created QP or EEC as the Transport Timer timeout. The receiving CM is not required to use this value, but is strongly encouraged to do so. PacketLifeTime represents the maximum expected flight time for a packet to traverse the path between the two CAs. PacketLifeTime is contained in the SA's PathRecord.
|
Reserved | 3-bit field. |
Alternate Path Info | Optional (because APM is optional). State of supplied Alternate Local Port LID field indicates whether or not an alternate path is supplied.
If valid, alternate path information consists of:
If the failover information is accepted, each CM causes their respective, newly created QP or EEC to be placed in the ReArm Migration state so that they are ready to perform a path migration. Receiving CM may accept connection request, but reject supplied alternate path information. It does this by sending REP message with one of the following values in the 2-bit Failover Accepted bit field:
|
Reserved | 3-bit field. |
PrivateData | RC/UC/RD. 736-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a request based on the private data. The format and meaning of the PrivateData field is specific to the ServiceID and the message type, and is not specified within the CM specification. |
Refer to Figure 36-1 on page 1075. Upon receipt of a REQ MAD [refer to “REQ (Request) MAD” on page 1074] from a remote CA's CM, the local CA's CM determines whether the requested service is available on the this CA. If it isn't, the CM sends a REJ MAD [see “REJ (Reject) MAD” on page 1098] back to the other CM stating the reason for the rejection.
If the requested service is supported by the local CA, the local CM creates an EEC and/or a QP to handle its end of the communications channel, programs that QP's or EEC's context with information supplied in the REQ MAD, transitions the newly created QP or EEC to the RTR state, and then sends a REP MAD (see Table 36-5 on page 1089) back to the requester with information about the newly created QP of EEC and QP.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from REP sender's point of view. REP sender:
|
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the REP message. This would be the same Communication ID received in the REQ message. Values in Local and Remote Communication ID fields in CM MADs are exchanged in requests and replies. The pair of IDs is used to reference a specific connection during establishment, failover management, and release. CM messages with invalid IDs are not processed and are rejected. |
Local Q_Key | RD. 32-bit value. In response to the receipt of the REQ message, the CM creates a RD
QP and assigns it a Q_Key. This field contains that Q_Key. Upon receipt of the REP message, the other CM passes this Q_KEY and the QPN returned in the REP:Local QPN field to the applications software that requested the establishment of a communications channel with a service in the other CA. Whenever the applications software then wishes to pass a message to the remote RD
QP associated with the service, it supplies the following information in a WR posted to the local RD QP's SQ:
|
Local QPN | RC/UC/RD. 24-bit value. This is the QPN of the QP created by the CM upon receipt of the REQ message.
|
Reserved | 8-bit field. |
Local EECN | RD. 24-bit value. This field only has meaning if the CM sending the REP message had created a new EEC in its CA at the behest of the REQ message sender (i.e., the REQ:RDC Exists field = 0). It then creates the new EEC and returns its EECN in this field of the REP message. The other CM then programs this EECN into the context of the EEC in its CA that will act as the other end of the newly created RDC. |
Reserved | 8-bit field. |
Starting PSN | RC/UC/RD. 24-bit value. This is the start PSN that the CM sending the REP message has assigned to its newly created QP's or EEC's Send Logic. Upon receipt of the REP message, the other CM programs this value into the ePSN in the context of its newly created QP or EEC. |
Reserved | 8-bit field. |
Responder Resources | RC/UC/RD. 8-bit value. Depth of the special queue that the REP sender's newly created CA
QP or EEC uses to hold and process incoming RDMA Read/Atomic requests. The CM sending the REP message obtains this value using Query HCA verb (if its associated CA is an HCA).
If = 0, this indicates that the REP sender's newly created CA QP doesn't support inbound RDMA Read or Atomic requests. If the queue depth indicated by this value proves insufficient for the CM receiving the REP message, it responds to the REP with a REJ. Receipt of a REJ message discontinues the connection establishment process. |
Initiator Depth | RC/UC/RD. 8-bit value. Depth of the special queue that the REP sender's newly created CA QP or EEC uses to issue outgoing RDMA Read and Atomic requests to the receiving CA's QP. The CM sending the REP message obtains this value using the Query HCA verb (if its associated CA is an HCA). The depth chosen must not exceed the depth of the corresponding queue on the receiving CA's QP or EEC. Note that the depth of the receiving CA's QP or EEC special receive queue was supplied in the REQ message. |
Target ACK Delay | RC/UC/RD. 5-bit unsigned value.
It represents the maximum expected time interval between REP sender's reception of a message and its transmission of the associated Ack or Nak. REQ sender uses this value along with its best estimate of network propagation delay (i.e., packet flight time) to determine how long to wait before timing out a packet transmission to REP sender. |
Failover Accepted | RC/UC/RD. 2-bit value. Indicates whether REQ recipient accepted or rejected the alternate path information contained in the REQ message. By sending the REP message, the REQ recipient accepts the connection request, but it may still reject the proposed failover port.
|
End-To-End Flow Control | RC. 1-bit value. Indicates whether REP sender's newly created QP
RQ Logic:
It should be noted that Table 89 in Volume 1 of the specification indicates this applies to both RC and RD, but it only applies to RC (RD does not support End-to-End Flow Control). For more information on End-to-End Flow Control, refer to “End-to-End Flow Control” on page 417. |
RNR Retry Count | RC/UC/RD. 3-bit value. Indicates the total number of times the newly created QP or EEC in the receiving CM's CA is to retry request packets for which it receives RNR Naks. After this count is exhausted, the QP's or EEC's Send Logic will post an error CQE on the Send Logic's CQ. |
Reserved | 8-bit field. |
PrivateData | RC/UC/RD. 1632-bit field. Data opaque to the CM protocol, passed from the REP sender to the REP recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Refer to Figure 36-1 on page 1075. When the CM requesting the establishment of a communications channel with a remote CA's CM receives a REP MAD from the remote CM, it uses the information in the REP MAD to finish programming the context of the newly created local QP or EEC and transitions the local QP or EEC to the RTS state. At this point, the newly created QP or EEC in the remote CA is in the RTR state.
The remote QP or EEC can be transitioned to the RTS state in one of two ways:
The local CM can send an RTU MAD to the remote CM. Upon receipt of the RTU MAD, the remote CA's CM transitions the newly created QP or EEC to the RTS, fully operational state.
Alternatively, the CM can forego sending the RTU MAD. Instead, the local applications software that requested the establishment of the communications channel can post a WR to the SQ of the newly created local QP, causing the QP's SQ Logic to transmit a message to the remote QP's RQ Logic. Upon receipt of the first request packet, the remote QP, or QP and EEC automatically transition from the RTR to the RTS state.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the RTU message. This CM must use the same ID in all CM MADs associated with the establishment and release of this communications channel. This CM must not reuse this ID (for another connection) for the life of this connection or while any messages related to this connection could still be in the fabric. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the RTU. This is the same as the Local Communication ID received in the earlier REP message. Values in the Local and Remote Communication ID fields in CM MADs are exchanged in requests and replies. The pair of IDs is used to reference a specific connection during establishment, failover management, and release. CM messages with invalid IDs are not processed and are rejected. |
PrivateData | RC/UC/RD. 1792-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
The DREQ MAD (see Table 36-7 on page 1095) can be sent by the CM at either end of the connection to tear down (i.e., destroy) the connection. Upon receipt of the DREQ MAD, the other CM validates the Communications IDs associated with the connection, destroys the local QP or EEC, and responds by sending a DREP MAD—see “DREP (Disconnect Reply) MAD” on page 1095.
Field | Description |
---|---|
Local Communications ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREQ message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the DREQ message. |
Remote QPN or Remote EECN | This is the QPN or EECN of the remote CA's QP or EEC. Provides additional validation that the Local and Remote Communication ID pair reference is correct. |
Reserved | 8-bit field. |
PrivateData | Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Refer to Table 36-8 on this page and to “DREQ (Disconnect Request) MAD” on page 1094.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREP message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the DREP message. |
PrivateData | Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
When a CM receives a MAD message for which the appropriate response is a REP, RTU, APR, or REJ MAD, it may be some time before it is ready to send that response. The CM that had sent the MAD had triggered its CM Response Timeout while it awaits a response MAD. If the CM that had received the MAD takes too much time before sending a response MAD, the other CM's CM Response Timeout will expire, causing it to retry the transmit of its original MAD. To prevent this, the CM should send an MRA MAD as a placeholder for its eventual response. When the MRA is received by the CM awaiting the response, the MRA's ServiceTimeout field (see Table 36-9 on this page) tells the CM the maximum time required before the MRA sender will be ready to send its real response.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREP message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the MRA message. |
Message MRAed | RC/UC/RD. 2-bit value. Identifies the message type that the MRA issuer needs more time to process before it send its real response:
|
Reserved | 6-bit field. |
ServiceTimeout | RC/UC/RD. 5-bit unsigned value. The maximum time required before the MRA sender will be ready to send a REP, RTU, APR, or REJ message (as appropriate). This value is expressed as
from the time the MRA is posted to the CM's GSI SQ. The recipient of the MRA must wait the specified amount of time plus the PacketLifetime (i.e., the packet flight time) after receiving this message before timing out. |
Reserved | 3-bit field. |
PrivateData | RC/UC/RD. 1776-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Upon receipt of a communications establishment MAD, a CM may choose to issue a REJ response MAD (see Table 36-10 on this page) indicating that it intends to break off the communications establishment message exchange. The reason for the rejection is indicated in the REJ MAD's Reason field. The possible reasons for rejection are listed in Table 36-11 on page 1099.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the REJ message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the REJ message. |
Message rejected | RC/UC/RD. 2-bit value indicating the type of message type being rejected:
|
Reserved | 6-bit field. |
Reject Info Length | RC/UC/RD. 7-bit field. If ≠ 0,this is the length in bytes of the Additional Reject Information (ARI) field (defined in this table). |
Reserved | 1-bit field. |
Reason | RC/UC/RD. 16-bit field. Error code indicating the reason for the REJ sender's termination of the communication establishment process. See Table 36-11 on page 1099. |
Additional Reject Information (ARI) | RC/UC/RD. 576-bit field. Contents defined by the error code delivered in the Reason field. See Table 36-11 on this page. |
PrivateData | RC/UC/RD. 1184-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Code | Reason | Description | Meaning of ARI Field |
---|---|---|---|
1 | No QP available | The REQ message required the recipient to allocate a QP and none were available. | NA |
2 | No EEC available | The REQ message required the recipient to allocate an EEC and none were available. | NA |
3 | No resources available | The REQ message required the recipient to allocate resources other than a QP or an EEC and none were available. | NA |
4 | Timeout | The CM protocol timed out waiting for a message. | NA |
5 | Unsupported request | Receiving CM does not support this request. | NA |
6 | Invalid Communication ID | The recipient received a CM message in which the Local Communication ID, Remote Communication ID, or both were invalid. | NA |
7 | Invalid Communication Instance | The communications tuple (i.e., the combination of the Local Communication ID, Remote Communication ID, and QPN or EECN) does not refer to any valid connection. | NA |
8 | Invalid ServiceID | The recipient of the REQ message does not recognize or does not support the service associated with the specified ServiceID. | NA |
9 | Invalid Transport Service Type | The recipient of the REQ message did not recognize the requested Transport Service Type. | NA |
10 | Stale connection | The recipient of the REQ determined that it already had a connection with the “Local QPN” or “Local EECN” specified in the REQ. Upon receiving a REJ for this reason, the REJ recipient places the QP or EEC to be placed into the TimeWait state. The TimeWait timer is set to twice the PacketLifeTime value (i.e., the source-to-destinaation round-trip flight time) plus the remote's Ack Delay. The CM is responsible for placing a QP or EEC in the TimeWait state, for maintaining it in that state for a period not less than the TimeWait period, and for removing it afterward. Receipt of a DREQ while in the TimeWait state does not affect the TimeWait timer. | NA |
11 | RDC does not exist | Upon receipt of a REQ message requesting the creation of a new RD QP using a preexistent EEC, the CM that received the REQ determined that the specified EEC does not exit. | NA |
12 | Primary Remote Port GID rejected | The recipient of the REQ message could not (or would not) accept the Primary Remote Port GID for its end of the connection. See the description of Primary Remote Port GID in Table 36-4 on page 1075. | GID of acceptable port. |
13 | Primary Remote Port LID rejected | The recipient of the REQ message could not (or would not) accept the Primary Remote Port LID for its end of the connection. See the description of Primary Remote Port LID in Table 36-4 on page 1075. | LID of acceptable port |
14 | Invalid Primary SL | The recipient of the REQ message does not support the requested Primary SL. | Acceptable SL |
15 | Invalid Primary Traffic Class | The recipient of the REQ message does not support the requested Primary Traffic Class. | Acceptable Traffic Class |
16 | Invalid Primary Hop Limit | The recipient of the REQ message could not (or would not) accept the Primary Hop Limit. | Acceptable Hop Limit |
17 | Invalid Primary Packet Rate | The recipient of the REQ message could not adjust its transmitter to send as slowly as would be required to comply with the requested Primary Inter-Packet Delay (IPD). | Minimum acceptable IPD |
18 | Alternate Remote Port GID rejected | The recipient of the REQ message could not (or would not) accept the Alternate Remote Port GID for its end of the connection. | GID of acceptable port |
19 | Alternate Remote Port LID rejected | The recipient of the REQ message could not (or would not) accept the Alternate Remote Port LID for its end of the connection. | LID of acceptable port |
20 | Invalid Alternate SL | The recipient of the REQ message does not support the requested Alternate SL. | Acceptable SL |
21 | Invalid Alternate Traffic Class | The recipient of the REQ message does not support the requested Alternate Traffic Class. | Acceptable Traffic Class |
22 | Invalid Alternate Hop Limit | The recipient of the REQ message could not (or would not) accept the Alternate Hop Limit. | Acceptable Hop Limit |
23 | Invalid Alternate Packet Rate | The recipient of the REQ message could not adjust its transmitter to send as slowly as would be required to comply with the requested Alternate Inter-Packet Delay (IPD). | Minimum acceptable IPD |
24 | Port and CM Redirection | The recipient of the REQ message supports the requested ServiceID, but at the endpoint specified by the ARI. Any further CM messages should be sent to that CA port as well. | ARI contains ClassPortInfo data structure (see Table 28-7 on page 794) describing where to send subsequent CM messages, and also describing the GID of the port to propose in the new REQ |
25 | Port Redirection | The recipient of the REQ message supports the requested ServiceID, but at the port specified by the ARI. Further CM messages must be sent to the port to which the original REQ was sent. | GID of port to propose in new REQ |
26 | Invalid Path MTU | The recipient of the REQ message cannot support the PMTU specified. | Maximum acceptable PMTU |
27 | Insufficient Responder Resources | The size of the special queue (Responder Resources) in the REP message for RDMA Read and Atomic requests was insufficient. | NA |
28 | Consumer Reject | The consumer (i.e., software) decided to reject the communication establishment attempt for reasons other than those listed in the other entries of this table. Typically, this happens based on rejection of information in the PrivateData field of a message. | Defined by the consumer software |
29 | RNR Retry Count Reject | The recipient of the message rejects the RNR NAK Retry Count value. | NA |
A CM would need to send new alternate path information to the remote CM under two circumstances:
APM was not initially enabled when the communications channel was created, but software now wishes to program the remote QP or EEC with alternate path information and transition it to the ReArm state. Software can program the local QP or EEC with alternate path information directly using the Modify QP or Modify EEC verb (if the local CA is an HCA).
APM was previously enabled on both ends of the connection, and the path was subsequently migrated to the alternate path (either due a hardware problem or because software triggered the migration). When software is informed that the connection between the two QPs or EECs has migrated to the alternate path (via a call to the Asynchonous Event Hander), it may wish to program new alternate path information into both the local and remote QPs or EECs.
The CM sends a LAP MAD (see Table 36-12 on this page) to the remote CM to load new alternate path information into the context of the remote QP or EEC. Upon receipt of the LAP MAD, the remote CM verifies that new alternate path information is different than the alternate path information currently contained in the remote QP's or EEC's context.
Assuming that it is different, the CM verifies that all fields in the LAP are good values. It then loads the new alternate path information into the context of the remote QP or EEC and changes the migration state of the QP or EEC from the Migrated state to the ReArm state. The remote CM then sends an APR MAD [see “APR (Alternate Path Response) MAD” on page 1107] to the CM that sent the LAP MAD. The APR MAD tells the LAP sender the status of the operation.
For a detailed description of APM, refer to “Automatic Path Migration” on page 575.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the LAP message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the LAP message. |
Local CM Q_KEY | RC/UC/RD. 32-bit value. Q_Key of the GSI (QP1) used by the CM sending the LAP message. |
Remote QPN or Remote EECN | This is the QPN or EECN of the QP or EEC in the CA receiving the LAP message. Provides additional validation that the Local and Remote Communication ID pair reference is correct. |
Remote CM Response Timeout | RC/UC/RD. 5-bit unsigned value. Time, expressed as:
within which remote CM (the LAP recipient) must transmit a response to the LAP message. If the remote CM cannot respond to the LAP (with an APR message) within this time period, it must send an MRA to prevent a timeout and a retry on the LAP sender's part. |
Reserved | 3-bit field. |
Reserved | 32-bit field. |
Alternate Path Info | New alternate path information to be programmed into the context of the remote QP or EEC. It consists of:
|
Reserved | 3-bit field. |
PrivateData | RC/UC/RD. 1,344-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Refer to Table 36-13 on this page and to “Loading a New Path or a Tertiary Path” on page 586.
Field | Description |
---|---|
Local Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the APR message. |
Remote Communication ID | RC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the APR message. |
Additional Information | 576-bit field. |
AP Status | Alternate Path Status. 4-bit value. Valid values are:
|
Reserved | 4-bit field. |
PrivateData | RC/UC/RD. 1208-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Refer to Table 36-14 on this page and “UD Connection Issues” on page 202.
Field | Description |
---|---|
RequestID | UD. 32-bit value identifying request. Recipient of request message returns this value unchanged in SIDR_REP. If the SIDR_REQ message is resent (due to a timeout awaiting a response), the sender must use same RequestID. |
Reserved | 32-bit field. |
ServiceID | UD. 64-bit value. ID specifying service being requested. These include, but are not limited to, service numbers defined for typical TCP services. |
PrivateData | UD. 1728-bit field. Data opaque to the CM protocol, passed from the SIDR_REQ sender to the SIDR_REQ recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |
Refer to Table 36-15 on this page and “UD Connection Issues” on page 202.
Field | Description |
---|---|
RequestID | UD. 32-bit value identifying the SIDR_REQ message that this a reply to. |
QPN | UD. 24-bit value. The CM sending the message places the QPN and Q_Key of the QP on which the requested service (identified by the ServiceID) is supported. Valid only if so indicated by this MAD's Status field. Also see the description of the Q_Key field. |
Status | UD. 8-bit value. Indicates whether the QPN field is valid. If the QPN is invalid, the Status field indicates the reason that the QPN has not been provided.
|
ServiceID | UD. 64-bit ID value that was contained in request. |
Q_Key | See the description of the QPN field. |
ClassPortInfo | UD. 576-bit field. ClassPortinfo data structure (see Table 28-7 on page 794) describing where to redirect SIDR_REQ. Only valid if the Status field indicates that the request must be redirected. |
PrivateData | UD. 1120-bit field. Data opaque to the CM protocol, passed from the SIDR_REP sender to the SIDR_REP recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification. |