In this chapter we provide information about the security characteristics of the IBM z/OS Shared Memory Communications over RDMA (SMC-R) function of z/OS 2.1. It is based on the assumption that readers have a basic understanding of TCP/IP protocols and the related z/OS implementation of those protocols.

SMC-R is an open protocol that is defined in the informational RFC titled Shared Memory Communications over RDMA, which is available on the following web page:

In this chapter, we focus exclusively on the IBM z/OS implementation of the SMC-R protocol. Our primary purpose is to describe how SMC-R communications are secured and the way that existing z/OS Communications Server network security features apply to SMC-R.

The chapter includes the following sections:

This section provides an overview of the Shared Memory Communications over RDMA support that was introduced in z/OS Communications Server 2.1.

Remote Direct Memory Access (RDMA) is a communications technology that enables a host to make a subset of its memory directly available to a remote host. By doing so, data can be transferred between hosts efficiently and without any help from the CPU on the source or target host. Historically, RDMA has been confined to high-performance computing environments where the cost of maintaining RDMA-capable network fabrics such as InfiniBand was justified given the emphasis of performance over cost. However, RDMA is now available on standard Ethernet-based networks by using the industry (InfiniBand Trade Association) standard referred to as RDMA over Converged Ethernet (RoCE). With RoCE, the cost of adopting RDMA is lower because it can flow over the Ethernet fabrics that are already in place to carry IP network communications. Both standard TCP/IP and RDMA traffic can flow over the same physical LAN fabric at the same time, but RDMA network interface cards (RNICs, also referred to as RoCE host channel adapters (HCAs), are required to do so. On System z, the 10Gb RoCE Express adapter serves as the RNIC.

z/OS Communications Server 2.1 introduces a new capability that combines the performance benefits of RDMA with the widely-used TCP/IP sockets programming interface. This function, called Shared Memory Communication - RDMA (SMC-R) allows your TCP sockets applications to benefit from direct, high-speed, low-latency, memory-to-memory (peer-to-peer) communications over RDMA transparently - no changes are required in your application programs.

SMC-R provides an enterprise class of services for RDMA that are designed for enterprise class data center networks. Communicating peers (the z/OS TCP/IP stacks) dynamically learn about the shared memory capability by using traditional TCP/IP connection establishment flows. With this awareness, the TCP/IP stacks can switch from TCP network flows to more efficient direct memory access flows that use RDMA. The application programs are unaware of the switch to shared memory communications.

The remainder of this section will describe relevant characteristics of SMC-R communications in only enough detail to provide a basis for the later security discussion. For a more complete description of the z/OS SMC-R implementation, see Chapter 10 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

Shared Memory Communications over RDMA is a hybrid protocol that uses RDMA technology within an existing IP network topology. SMC-R connections are established and operate transparently to applications within the context of their TCP socket connections. IP communications occur over OSA adapters (as they have in the past) and the associated SMC-R connections are established over the RNICs. As such, the RNICs must be attached to the same network infrastructure as the OSAs.

SMC-R’s reliance on existing IP network topology and TCP connection setup preserves critical TCP/IP operational and network management features, including compatibility with transport layer load balancers (for example, Sysplex Distributor), preserving the IP security model (IP filters, VLANs, TLS/SSL, and so forth), and minimal (or zero) topology changes to accommodate the use of RDMA. This reliance on IP topology is a foundational element of the SMC-R security landscape.

For two nodes to be eligible to communicate with SMC-R, several criteria must be met:

The direct-access requirements are based on the fact that the underlying RDMA connections are non-routable. This means that SMC-R connections are not routable as well. The direct-access requirements ensure that a direct communication path exists at layer 2 between the SMC-R capable nodes, with no intervening IP router. The additional VLAN requirement further confines the traffic within the physical LAN fabric in cases where VLANs are in use.

The topology requirements are illustrated in Figure 5-1. As you can see, the SMC-R enabled TCP connections between HOST A and HOST B are allowed because both hosts have OSA and RoCE adapters connected to the same physical LAN fabric, VLAN and IP subnet. On the other hand, even though HOST C is attached to the same physical LAN fabric, it cannot establish any IP connections to HOST A or HOSTB without an intervening IP router because it is connected to a different VLAN and IP subnet.

Because SMC-R connection processing uses existing IP topology (TCP/IP connection setup) SMC-R connections transparently inherit the same VLAN and IP subnet connection eligibility attributes of the associated TCP connection. When VLANs are in use, SMC-R connections then become VLAN qualified.

SMC-R is enabled by specifying the SMCR parameter of the GLOBALCONFIG statement in the TCP/IP profile data set and including one or more Peripheral Component Interconnect Express (PCIe) function ID (PFID) values. Each PFID value represents an RNIC adapter that is configured by using the traditional hardware configuration definition (HCD) tools. TCP/IP activates the RNICs when the first SMC-R capable IP interface is started. IPAQENET or IPAQENET6 interfaces with the OSD channel path ID type can be configured for SMC-R capability.

Any TCP connections that traverse SMC-R capable IP interfaces are eligible for SMC-R communications. The decision about whether an eligible connection uses SMC-R communications is reached during traditional TCP connection establishment. The sequence of flows that determine whether or not to use SMC-R on a given TCP connection is called Rendezvous processing, which is illustrated in Figure 5-2.

The Rendezvous exchange of information occurs in three stages:

TCP connections are still established using the standard three-way handshake mechanism. When SMC-R communications are enabled, the client adds TCP options settings in the SYN request to indicate that it supports SMC-R protocols. The server, when SMC-R communications are enabled, likewise responds with TCP options settings for SMC-R in the SYN-ACK response. No additional exchange of information is required in this stage of the rendezvous processing.

After the TCP three-way handshake succeeds, the client and server negotiate the use of SMC-R for this TCP connection by using SMC-R CLC messages that flow as in-band data over the TCP connection. Conceptually, these flows are similar to the TLS/SSL handshake processing that occurs after the TCP connection is established, but they occur before any data is allowed to flow over the TCP connection (including the TLS/SSL handshake).

The CLC messages exchange the following information:

Using the RoCE credentials exchanged in phase 2, an RDMA connection called an SMC-R Link is established between the two peers across Reliable Connected Queue Pairs (RC QPs). SMC-R LLC messages are then exchanged across the SMC-R Link to confirm that the RoCE information is correct and that the RC QPs that comprise the SMC-R link have connectivity. This stage is skipped if an existing RoCE connection is used for this TCP connection.

Because the z/OS TCP/IP stack does not allow the client and server applications to exchange application data before or during rendezvous processing, the TCP connection can revert to IP protocols if there is a failure during the setup of the SMC-R communications. However, after the RoCE connection is confirmed by using the LLC messages, the TCP connection is committed to using SMC-R protocols and cannot fall back to using IP protocols if SMC-R communications encounter an error.

Even though application data is sent out of band from the TCP connection with SMC-R communications, the TCP connection remains active to preserve the connection state for monitoring and management functions, load balancers, and so on, and to support various stack functions, including connection termination processing.

This section examines SMC-R connections from a security perspective and compares them to IP packets and TCP connections within a single LAN segment, because that is the scope of an SMC-R connection.

First, let us consider the protection of application-level data as it traverses the LAN. Many organizations need to protect application-level data (transactions, database query results, transferred files, and such) as it crosses any network between two nodes. When those applications use TCP-based protocols (FTP, HTTP, CICS transactions, and so on), Transport Layer Security (TLS) protocols can be used to provide endpoint authentication, data authentication, data integrity and data privacy (encryption) protections for the application data. Because this protection is done above the transport layer, it can be used for SMC-R enabled connections just as effectively as it can for TCP connections over a regular IP network.

We have explained that application data can be protected over SMC-R enabled links by using the TLS, just as it can be over regular TCP/IP. However, if you’re wondering about protecting the TCP and IP headers that contain the application data, this section gives you a closer look at the types of attacks that can be directed at the lower-level networking headers and then discusses protection scenarios as they apply to regular IP networks and those carrying RDMA traffic.

TCP/IP, RDMA, and the underlying Ethernet-based link layer protocols must all carry enough information in each transmission unit to ensure the data is properly handled as it traverses the network and after it arrives at its destination. Specifically, such a protocol provides the information that you need for the following purposes:

Ethernet provides the source and destination identity/location information in the form of MAC addresses. Along with the MAC addresses, Ethernet uses an associated upper-layer addressing value (an IP address in the case of IP or a Global ID (GID) in the case of RDMA) to assist in the routing of a given frame through the LAN and to the proper device driver after it arrives at its destination. Other control information is included in each Ethernet frame header.

IP provides the source and destination identity/location information through IP addresses and the control information through various flags or headers. In the IP cases relevant to our discussion, TCP provides the upper-layer tagging in the form of TCP port information and also provides its own control information. Other control information is included in each IP packet header.

When an IP packet traverses an Ethernet-based LAN, it is susceptible to different kinds of attack techniques, including the following types:

There are a few of ways to mitigate the risks involved in these types of IP-based attacks:

SMC-R connections are expected to be established on the second type of LAN, where the physical access is controlled tightly enough to obviate the need for cryptographic protection of network and transport layer headers.

Because SMC-R is based on RoCE, which in turn is based on RDMA, it relies on the RoCE and RDMA protocols that use the Ethernet MAC and the RDMA GID for their source and destination identity/location. The RC QPs that comprise the SMC Link provide the session level information (analogous to a TCP connection). The RC QPs are identified by source and destination queue pairs. An SMC Link is further bound to a specific area of memory in each partner called a remote memory buffer which is identified by a special token and key, called RToken and RKey, respectively. All of this information is carried in the various session establishment flows over IP and RoCE as described and also over the RNICs during the underlying RDMA connection establishment flows.

Although the SMC-R sessions can be attacked by using techniques similar to those described for IP, these connections are no more or less vulnerable to such attacks than regular TCP/IP connections. However, because there is no RDMA analog to IPSec, SMC-R connections should be enabled only on LAN segments that are physically secure.

If an organization does not trust in the physical security of a LAN segment enough to flow data without IPSec protection, then that segment should not be enabled for SMC-R.

Although the preceding point should be used as a general guideline regarding the appropriateness of SMC-R traffic over any given LAN segment, bear in mind that the use of IPSec to protect a TCP session actually disables the use of SMC-R on z/OS, as described in subsequent sections.

One other important consideration in the use of SMC-R is its effect on firewalls and deep packet inspection (DPI) devices, such as bump-in-the-wire intrusion prevention devices. These devices process Ethernet frames, IP packets, and other well-known protocols built upon IP, and they are usually deployed at network zone boundaries along with a Layer 3 (IP) router.

Because SMC-R traffic is not routable (due to its RDMA roots), it is not expected to reach traditional Layer 3 firewalls. Likewise, it would be unusual for SMC-R traffic to reach a non-traditional firewall or a DPI device.

However, such devices might be incompatible with RDMA traffic, so it is important to read product documentation for the specific device to learn how the device behaves if it encounters SMC-R (RDMA) traffic.

This section provides an overview of the key security features of z/OS Communications Server and how each relates to SMC-R. Note that this section discusses only the subset of security features that are specifically relevant to the SMC-R functions in the TCP/IP stack. It is not a comprehensive survey of all the Communications Server security functions.

The TCPIP profile data set’s INTERFACE statement allows you to specify whether or not a given IPAQENET or IPAQENET6 interface (with the OSD channel path ID type) is enabled for SMC-R by specifying the SMCR or NOSMCR parameter. If not specified, the default is to enable SMC-R for these types of interfaces.

You can use the TCPIP profile data set’s PORT and PORTRANGE statements to exclude a port or set of ports from SMC-R eligibility by specifying the NOSMCR parameter. This can be a useful feature for disabling SMC-R for a specific z/OS TCP server application.

z/OS Communications Server provides two types of SAF resources that allow system administrators to control access of z/OS user IDs to network resources. One controls access to different network zones (hosts, subnets, and networks), and the other controls access to local TCP ports.

EZB.NETACCESS.sysname.tcpname.zonename resources control which z/OS user IDs are allowed to access different network zones. User ID with permission to a NETACCESS resource are allowed to send and receive data to and from the network zone represented by the zone name. Network access control provides an additional layer of security on top of any authentication and authorization mechanisms that are used in the network or at the peer system by preventing unauthorized users from communicating with the peer network resource.

EZB.PORTACCESS.sysname.tcpname.portname resources determine which z/OS user IDs are allowed to bind to specific TCP ports. User IDs with permission to a PORTACCESS resource are allowed to bind to the TCP port represented by the resource.

Because both NETACCESS and PORTACCES controls are enforced at the socket layer, they apply fully to SMC-R enabled connections.

For more information about NETACCESS and PORTACCESS resources, see Chapter 3 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

z/OS Communications Server’s IP security function includes IP packet filtering that controls the flow of IP packets into and out of the TCP/IP stack. An administrator can define IP security policy to permit or deny any type of IP traffic from entering/exiting the TCP/IP stack. Both local and routed traffic are supported and filter rules can be defined using a wide variety of criteria.

Basic permit/deny filters for local traffic are honored for TCP connections that use SMC-R. A permit filter allows TCP connections to be established while a deny filter prevents them. This is true regardless of the use of SMC-R. After a TCP connection is successfully established, any change in IP filtering configuration that installs a deny filter for the connection will immediately cause the connection to be dropped for both IP and SMC-R traffic.

Given that RoCE (and hence, SMC-R) traffic is not routable, IP filters that apply to routed traffic do not apply to SMC-R traffic.

For more information about IP filtering, see Chapter 19 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

In addition to IP filtering, Communications Server's IP security function provides a complete IPSec implementation. IPSec is a cryptography-based set of technologies that allow participating peers to establish a wide variety of VPN tunnels. As described earlier, IPSec protects entire IP packets (not just the application data within those packets) and is completely transparent to application programs. z/OS applies IPSec protection under the direction of IP filter rules that specify an action of protect with IPSec.

Because IPSec protocols are bound tightly to IP packet formats, they are simply not compatible with SMC-R. As such, when any TCP connection is protected by an IPSec filter rule, z/OS makes that connection ineligible for SMC-R. Furthermore, if an SMC-R enabled TCP connection is already established when such an IPSec filter rule is installed, then that connection will be terminated.

For more information about IPSec, see Chapter 19 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL) are cryptography-based technologies that are applied just above the transport layer, typically under direction of the application program that is generating the traffic to be protected. z/OS offers two RFC-compliant TLS/SSL implementations:

Because TLS (and SSL) protection is applied to application data just above the transport layer (before any TCP segmentation, which is true even for AT-TLS), it is completely compatible with SMC-R. The TCP/IP stack, including the SMC-R function sees it all simply as application data.

For more information about System SSL, see z/OS Cryptographic Services System Secure Sockets Layer Programming Version 2 Release 1, SC41-7495. For more information about AT-TLS, see Chapter 22 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

To download a copy of the documentation for the IBMJSSE2 provider, see the Security documentation on IBM developerWorks®. In addition to this cross platform information, z/OS specific information for JSSE2 can be found in the z/OS JSSE Reference Guide.

The Secure Shell (SSH) protocol is widely-used in to cryptographically protect traffic between UNIX environments (including z/OS UNIX). SSH operates at above the transport layer, so it applies to SMC-R enabled TCP connections in the same manner as TLS/SSL.

The Ported Tools for z/OS offering includes a version of OpenSSH for z/OS. For more information about this OpenSSH implementation, see the IBM Ported Tools for z/OS: OpenSSH User's Guide Version 1 Release 2, SA23-2246.

Many security protocols exist at the application layer and some applications have their own security features. For example, Web Services Security (WS-Security), Domain Name System Security (DNS-SEC), security features within SNMP and so forth. As with TLS/SSL and SSH, the fact that these are applied above the transport layer means that it all just looks like application data to the TCP/IP stack and to SMC-R. Because of this, application layer security functions will continue to work over SMC-R enabled connection.

z/OS Communications Server Intrusion Detection Services (IDS) provide reporting and, in some cases, defensive protections, for a wide variety of intrusion-related events that fall into the following categories:

Scan detection and traffic regulation are both enforced during TCP connection setup and therefore complete before SMC-R enablement is ever considered for a given connection.

Attack detection includes a wide range of checks. Because most of the TCP-related checks apply to inbound TCP packets, they are not relevant to the direct memory transfers employed by SMC-R communications. However, two attack types do apply to SMC-R enabled connections. Let's examine each of these in more detail.

You can configure IDS policy to detect when a TCP connection's send or receive queues become constrained due to the amount or age of the data on the queue. When a queue becomes constrained, the IDS rule can either reset the TCP connection or continue to monitor the condition until the queue is no longer constrained. Send or receive queues for SMC-R enabled TCP connections are considered to be constrained in the following circumstances:

You can configure IDS policy to detect attacks that are designed to consume system resources by creating many TCP connections and causing them to stall, making them unable to send data. A global stall condition is in effect when at least 50% of the active TCP connections are stalled and at least 1000 TCP connections are active. A global TCP stall IDS rule can either reset stalled connections or continue to monitor the condition. TCP connections that traverse SMC-R links are considered during global TCP stall detection. Such a connection is considered to be stalled when the TCB is write-blocked.

For more information about IDS functions, see Chapter 18 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.

Multilevel security is an enhanced security environment in which the z/OS security server and trusted resource managers enforce mandatory access control policies in addition to the usual discretionary access control policies. To participate in an MLS environment, the user IDs associated with tasks trying to access z/OS resources and those resource profiles in the SERVAUTH class need to have security labels defined.

MLS is supported for SMC-R enabled TCP connections with one exception. If a TCP connection requires MLS packet tagging (that is, if the source and destination zones are both SYSMULTI), then SMC-R will be disabled for that connection. If an SMC-R enabled TCP connection is already established and a dynamic configuration update introduces a requirement for MLS packet tagging on that connection, it will be dropped.

For more information about MLS, see Chapter 4 in the z/OS Communications Server IP Configuration Guide Version 2 Release 1, SC27-3650.