This appendix introduces the concept of managing native Peripheral Component Interconnect Express (PCIe) features (10GbE Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) Express and zEnterprise Data Compression (zEDC) Express). This concept requires the use of an integrated firmware processor (IFP) and resource groups that are associated to the physical location of the feature cards.

This appendix describes how these features are implemented into the PCIe I/O structure of the z13. The following components and functions manage the native PCIe features:

This appendix includes the following sections:

There are feature card types introduced on zEC12, which require a new design to manage these adapters. The native PCIe features are 10GbE RoCE Express and zEDC Express.

These adapters are plugged in a PCIe I/O drawer together with existing I/O features, but they are managed in a different way than the existing I/O adapters and features. The native PCle feature cards are exclusively plugged into the PCIe I/O drawer and have a physical channel ID (PCHID) assigned according to the physical location in the PCIe I/O drawer.

On existing features that were plugged into an I/O drawer or I/O cage, all adapter layer functions are integrated into the adapter hardware. For the new features introduced by zEC12 central processor complex (CPC), the adapter layer function is now handled by an integrated firmware processor (IFP).

This section covers the following topics:

For traditional PCIe I/O adapters, such as the Open Systems Adapter (OSA) and Fibre Channel connection (FICON) cards, the diagnostic program and device drivers are downloaded from the Support Element (SE) to the application-specific integrated circuit (ASIC) chips that are on those cards. With the introduction of the IFP and the native PCIe adapters, which do not have an ASIC chip, the device drivers for these native PCIe adapters were moved to the operating systems. The adapter layer function runs on the IFP and uses two resource groups.

All virtualization, recovery, diagnostics, failover, concurrent firmware (FW) updates, and so on function on traditional I/O features are performed on the adapter level. For the native PCIe features, these functions are done by the IFP.

The IFP is used to manage native PCIe adapters that are installed in a PCIe I/O drawer. On previous systems, this processor was not used but known as a reserved processor. The IFP is allocated from a pool of processor units (PUs) available for the whole system. Because IFP is exclusively used to manage native PCIe adapters, it is not taken from the pool of PUs that can be characterized for the client’s use.

If a native PCIe feature is present in the system, the IFP is initialized and allocated during the systems’ power-on reset (POR) phase. Although the IFP is allocated to one of the physical PUs, it is not visible to the client. In an error or failover scenario, the IFP acts like any other PU (that is, sparing).

To manage the PCIe features, the IFP has two resource groups allocated. The two resource groups handle the adapter layer function of the native PCIe feature cards. Each I/O domain in a PCIe I/O drawer is assigned to one of the two resource groups. There are four I/O domains in the PCIe I/O drawer, where I/O domain 0 and 2 are handled by resource group 1, and I/O domain 1 and 3 are handled by resource group 2. Figure G-1 shows the relationships among the I/O domains, I/O slots, and the resource groups that are managed by the IFP.

Up to five PCIe I/O drawers are supported on the z13. Native PCIe features of the same type are configured to different resource groups, PCIe I/O drawers, and I/O domains to prevent a single point of failure. Other than the existing PCIe feature cards (FICON, OSA, and Crypto), each I/O domain supports a total of two native PCIe feature cards.

The following native PCIe features are supported in the PCIe I/O drawer:

Only zEDC Express and 10GbE RoCE Express features are managed by the IFP, but the Flash Express feature is counted when configuring native PCIe features into the PCIe I/O drawer.

The following management functions are provided for the native PCIe features by the IFP:

For earlier I/O features, parts of the management function were included on the adapter. The IFP performs all management tasks on the native PCIe features:

Microcode Change Level (MCL) upgrades on native PCIe adapters or on the code of the resource groups require the specific adapter or all native PCIe adapters managed by the specific resource group (depending on the type of u-code that it applies) to be offline during activation of the MCL. However, to maintain availability, MCLs can only be applied to one resource group at a time. While one resource group is offline, the second resource group and all adapters in it remain active. An MCL application for a native PCIe adapter or resource group is not possible if an error condition exists within the other resource group.

If an error in one of the resource groups or features assigned to one of the resource groups occurs, the IFP manages error recovery and collects error data. The error data is sent by the IFP to the SE, which then provides a message on the SE and the Hardware Management Console (HMC). If an error requires maintenance, a call home to the IBM Support system is initiated by the HMC.

Any maintenance action on a native PCIe feature is managed by the IFP, including testing or replacing a feature card. Before configuring a feature offline, the IFP ensures that the same type of feature is available in the same or the other resource group (if applicable).

There is a limit on the number of features that you can order for a specific native PCIe feature. There is also a maximum number of PCIe features. This maximum is based on the maximum number of physical native PCIe cards per PCIe I/O drawer, which is eight, but also on the presence of the Flash Express feature. The Flash Express feature is different from the two other native PCIe features because it does not use the IFP or resource groups. However, remember that one Flash Express feature takes up two slots in the PCIe I/O drawer, therefore limiting the remaining slots for the other native PCIe features.

The z13 system can have up to five PCIe I/O drawers for a maximum of 40 slots for native PCIe features (10GbE RoCE Express and zEDC Express) and the Flash Express feature. Up to 32 slots can be used for the maximum of 16 slots for 10GbE RoCE Express, eight slots for the zEDC Express feature, and up to eight slots for Flash Express.

Table G-1 shows the dependencies and the maximum number of native PCIe features installable in the PCIe I/O drawer.

Each Flash Express feature (FC 0402) occupies two slots in the PCIe I/O drawer. Each 10GbE RoCE Express (FC 0411) or zEDC Express (FC 0420) feature occupies one slot. If one Flash Express feature is installed in a z13 with one PCIe drawer, it allows the installation of up to six 10GbE RoCE or zEDC features in this PCIe I/O drawer. Any additional 10GbE RoCE or zEDC feature (total of more than six) requires a second PCIe I/O drawer.

During the ordering process of the native PCIe adapters, such as the zEDC Express and 10GbE RoCE Express features, features of the same type are evenly spread across two resource groups (resource group 1 and resource group 2) for availability and serviceability. Figure G-2 shows a sample of the PCHID report for a configuration with four each of the previously mentioned features and how they are spread across resource group 1 (RG1) and resource group 2 (RG2). Although Flash Express features are counted as native PCIe cards for the total number of Native PCIe features, they are not part of any resource group.

The native PCIe features are not part of the traditional channel subsystem (CSS). They do not have a channel-path identifier (CHPID) assigned, but they have a PCHID assigned according to the physical location in the PCIe I/O drawer.

To define the native PCIe adapters in the HCD or HMC, a new I/O configuration program (IOCP) FUNCTION statement is introduced that includes several feature-specific parameters. Figure G-3 shows examples of the specific statements for the 10GbE RoCE Express and zEDC Express features. This example defines two zEDC features (PCHID 100 and 12C) and two 10GbE RoCE Express features (PCHID 11C and 144).

The FUNCTION identifier (FID) is a hexadecimal number between 000 and FFF that you use to assign a PCHID to the FUNCTION to identify the specific hardware feature in the PCIe I/O drawer. Because the FUNCTION is not related to a channel subsystem, all LPARs on a zEnterprise CPC can be defined to it. However, a FUNCTION cannot be shared between LPARs. It is only dedicated or reconfigurable by using the PART parameter. The TYPE parameter is new for z13 and is required.

If you want several LPARs to be able to use a zEDC Express feature (the 10GbE RoCE Express feature cannot be shared between LPARs), you need to use a Virtual Function (VF) number. A VF number is a number between 1 and n, where n is the maximum number of LPARs that the feature supports, which is 15 for the zEDC Express feature and 31 for the RoCE Express feature.

The physical network identifier (PNETID) is required to set up the Shared Memory Communications over Remote Direct Memory Access (SMC-R) communication between two 10GbE RoCE Express features. Each FUNCTION definition supports up to four PNETIDs.