Memory Regions

Definition of a Memory Region

A memory region is a collection of memory pages within the local HCA's memory. When the region is created by software (via a verb call), the caller defines a combination of local and, possibly, remote access permissions within the memory region being created:

• Local access permission grants logic within the local CA (e.g., QPs) authority to perform the defined access types within the region.

• Remote access permission grants QPs within a remote CA authority to perform the defined access types within the region. When a remote QP sends a request to access the local memory associated with the destination QP, the destination QP must have been granted the appropriate access rights to access local memory on behalf of the remote QP. In other words, in order for remote QPs to access this CA's local memory, the local CA must also have been granted local access rights.

Before QPs within a HCA can access its memory, one or more memory regions must be set up that permit local and, possibly, remote access.

Virtual Memory Regions

Definition of a Virtual Memory Region

When software tells the verbs to create a virtual memory region, it is left up to the OS memory management routine to select which physical memory pages are assigned to the region (for the alternative, see “Physical Memory Regions” on page 307).

Creating a Virtual Region

A virtual region is created by performing a Register Memory Region verb call. Some of the input parameters provided by software include:

• The desired start virtual memory address (64 bits):

  - It can be any byte-aligned 64-bit address.

  - It does not have to be a page-aligned start address.

  - If the start address isn't page-aligned, the region starts at an offset within the first physical page assigned to the region by the OS memory management routine.

  - Note that if the requested virtual address range is already in use (i.e., the OS memory management routine has already mapped it to physical pages in memory for another requester), the OS will assign a different virtual address range to this region.

• Region length (in bytes):

  - This defines the end address of the virtual address range assigned to the region.

  - The region may end in the middle of the last page assigned to region.

• Access control attributes:

  - Local read access is always automatically granted.

  - Enable/disable local write access.

  - Enable/disable remote write access (requires enabling of local write access as well).

  - Enable/disable remote read access.

  - Enable/disable remote atomic operation access (requires enabling of local write access as well).

  - Enable/disable memory window binding (window binding is described in “Memory Windows” on page 308).

Some of the output parameters returned from the verb call include:

• L_Key. Local access key (aka Region ID). This key must be specified when local logic within the HCA wishes to access the region. When specifying the Gather Buffer List or Scatter Buffer List in a WR, each entry in the supplied list is specified by start virtual memory address, buffer length, and the L_Key assigned to that memory area when that memory region was created.

• R_Key. Remote access key (aka Region ID). This key is only returned if remote access rights were requested. The R_Key is supplied in an RDMA or Atomic request packet sent by a remote QP when it desires access to the region.

• Region handle. The handle must be specified when making any subsequent verb calls to manipulate this region's characteristics.

Actions Taken by Verb to Create a Region

Refer to Figure 15-1 on page 304. The following steps are performed in order to create a virtual memory region:

1.	Software executes the Register Memory Region verb to create a region.
2.	The Register Memory Region verb in turn calls the OS memory allocation routine to request a buffer of the requested size.
3.	The OS memory allocation routine locates pages in physical main memory whose sum total size ≥ the requested size of the region.
4.	The pages allocated by the OS memory management routine almost certainly do not occupy a contiguous physical memory address range.
5.	When calling the OS memory management routine, the verb must specify that the pages allocated to the region must be pinned in memory by the OS. This prevents the OS from swapping these pages out to mass storage.
6.	The OS memory management routine returns the start physical address of each page to the verb.
7.	The verb creates (within the HCA or the HCA's local memory) a table mapping the VA (Virtual Address) range defined (by the start VA and the region length) to the series of physical pages assigned to the newly created region.

Region Windowing May Be Enabled When Region Is Created

In order to enable the binding of windows (see “Memory Windows” on page 308) to a region, this must be specified when the region is created (using the Register Memory Region or Register Physical Memory Region verbs).

Characteristics of a Region

• Multiple regions may partially or fully overlap the same physical memory locations.

• A region is only related to the HCA to which it is registered (not to any other HCA).

• The region registration verbs must issue a pin request to the OS memory management routine each a time region is registered (even if some of the pages were previously allocated and pinned by previous region registrations).

• When all of the regions utilizing a page have been deregistered, the OS must unpin the page. This means that the OS memory management routine must track how many pin requests were received for a page.

• Only privileged code (i.e., kernel code) is permitted to make the region-oriented verb calls.

• When software commands the HCA (through the verbs) to create a QP or CQ, the verbs may build data/control structures associated with the new QP or CQ in memory managed by the OS. In this case:

  - The verbs must instruct the OS memory management routine to pin the pages containing these structures.

  - Local HCA read/write access to these private data/control structures is implicitly granted.

  - Software external to the HCA (the specification uses the phrase “above the verb layer”) cannot access these structures.

  - Although the structures may be in system memory, they may not fall within an HCA region.

Region-Oriented Verbs

The region-oriented verbs are:

• Register Memory Region (described in “Creating a Virtual Region” on page 300 and “Actions Taken by Verb to Create a Region” on page 301).

• Reregister Memory Region. Allows modification of a virtual region's VA, length, PD (covered later in this chapter), and access attributes.

• Register Physical Memory Region (described in “Physical Memory Regions” on page 307). Unlike the Register Memory Region verb, this verb allows specification of the page list, the number of pages, the IOVA, the region length, the offset of the IOVA in the first page, the PD, and the access control attributes.

• Reregister Physical Memory Region (described in “Physical Memory Regions” on page 307). Allows modification of the page list, the number of pages, the IOVA, the region length, the offset of the IOVA in the first page, the PD, and the access control attributes.

• Register Shared Memory Region (described in “Shared Memory Regions” on page 303). Creates a new region associated with the same physical memory locations as a preexistent region.

• Query Memory Region. Returns the L_Key, R_Key, access attributes, and PD associated with the region.

• Deregister Memory Region. Destroys the region. The verb calls the OS memory management routine and instructs it to destroy the virtual-to-physical mappings associated with the region's assigned pages.

Shared Memory Regions

Multiple regions can be set up that share the same physical pages in memory. This is accomplished by calling the Register Shared Memory Region verb. This verb call has the following characteristics:

• It creates a new region associated with the same physical memory locations as that of a preexistent region.

• The VA, PD, and access rights need not be same as those of the existing region. However, the lengths are the same by definition.

• The new region shares the processor mapping resources (e.g., Page Table entries) to the extent possible. A number of regions can share the same mapping resources.

• The regions created by this verb behave identically to memory regions created by the other memory registration verbs.

Register/Reregister/Deregister Are High Overhead

All three of these operations are high overhead:

• The initial creation of a region involves a switch from User to Kernel mode (and back again) with all of the attendant hardware/software overhead.

• Reregistering a region also involves a mode switch.

• Destruction (i.e., deregistration) of a region also involves a mode switch.

Example Virtual Region

Refer to Figure 15-2 on page 306. This example assumes the following:

• Software has executed a Register Memory Region verb call.

• The requested 64-bit start virtual address is 141200h.

• The length of the region is 10000d bytes.

• For this example, the access control attributes can be any desired.

• The processor page size is 4KB.

As a result, the verb makes a call to the OS memory management routine specifying the following:

• The requested 64-bit start virtual address is 141200h.

• The length of the region is 10000d bytes.

• The pages allocated by the OS must be pinned in memory.

• The requested access control attributes can be any desired.

Note that the requested virtual start address (141200h) is not aligned on a physical page start address (i.e., an address divisible by the page size of 4KB).

The OS memory management routine takes the following actions:

The verb then creates a virtual-to-physical address mapping table either within the HCA or within its local memory.

Physical Memory Regions

Software Specifies Physical Pages Assigned to Region

When a virtual memory region is created (by executing a Register Memory Region verb call), the caller leaves it up to the OS memory management routine to determine which pages in physical memory will be mapped into the region's virtual address space.

In some circumstances, the caller requires that specific physical memory address ranges are included in the region's virtual address space. In this case, a physical, rather than a virtual, region is created.

Use Register Physical Memory Region Verb Call

The input parameters supplied when calling the Register Physical Memory Region verb are (this is a partial list):

• A list of the physical pages (the specification refers to them as “buffers”) to be included in the region. Note that each page must begin and end on an HCA-supported page boundary. The page size is dictated by the processor type and, if the processor supports more than one page size, the selected page size.

• The total number of pages in the list.

• The IOVA (IO Virtual Address) requested to be assigned to the first byte of the region. Note that if the requested virtual address range is already in use (i.e., the OS memory management routine has already mapped it to physical pages in memory for another requester), then the OS will assign a different virtual address range to this region.

• The region length (in bytes). The region end address is defined by the length and can be any location within the last physical page in the list.

• The offset of the IOVA within the first page of the region.

• The Access Control attributes:

  - Local read access is always automatically granted.

  - Enable/disable Local Write Access.

  - Enable/disable Remote Write Access (requires enabling of Local Write Access as well).

  - Enable/disable Remote Read Access.

  - Enable/disable Remote Atomic operation access (requires enabling of Local Write Access as well).

  - Enable/disable Memory Window Binding (see “Memory Windows” on page 308).

The output parameters supplied by the verb are:

• The region handle. The handle must be specified when making any subsequent verb calls to manipulate this region.

• The actual IOVA (may or may not be the same as the requested IOVA).

• L_Key and, possibly, R_Key. Access keys (aka Region IDs). Note that the R_Key is only returned if remote access rights were requested. The R_Key is supplied in an RDMA or atomic request packet when a remote QP desires access to the region.

HCA Creates Mapping Table

As result of registering (i.e., creating) the physical region, the verb creates the virtual-to-physical page mapping table to map addresses within the assigned IOVA range to the corresponding locations within physical memory.

Physical and Virtual Region Pages May Be Shared

One or more pages in a physical region may also be part of a virtual region. In this case, those pages must remain pinned (by the OS) until the page(s) are no longer in use by any registered region.