System upgrades
This chapter provides an overview of IBM z13 upgrade capabilities and procedures, with an emphasis on capacity on demand (CoD) offerings. The upgrade offerings to the z13 systems were developed from previous IBM z Systems. In response to client demands and changes in market requirements, many features were added. The provisioning environment gives you unprecedented flexibility and more control over cost and value.
For detailed tutorials about all aspects of system upgrades, go to the IBM Resource Link1 website. Click Resource Link → Client Initiated Upgrade Information, and then select Education. Select your particular product from the list of available systems:
The growth capabilities that are provided by the z13 include the following benefits:
•Enabling exploitation of new business opportunities
•Supporting the growth of dynamic, smart, and cloud environments
•Managing the risk of volatile, high-growth, and high-volume applications
•Supporting 24 x 7 application availability
•Enabling capacity growth during lockdown periods
•Enabling planned-downtime changes without availability impacts
This chapter includes the following sections:
8.1 Upgrade types
The types of upgrades for a z13 are summarized in this section.
8.1.1 Overview of upgrade types
Upgrades can be categorized as described in the following section.
Permanent and temporary upgrades
Permanent and temporary upgrades are different types of upgrades in different situations. For example, a growing workload might require more memory, more I/O cards, or more processor capacity. However, only a short-term upgrade might be necessary to handle a peak workload, or to temporarily replace a system that is down during a disaster or data center maintenance. The z13 offers the following solutions for such situations:
•Permanent:
 – Miscellaneous equipment specification (MES):
The MES upgrade order always is performed by IBM personnel. The result can be either real hardware or installation of Licensed Internal Code Configuration Control (LICCC) to the system. In both cases, installation is performed by IBM personnel.
 – Customer Initiated Upgrade (CIU):
Using the CIU facility for a system requires that the online CoD buying feature (FC 9900) is installed on the system. The CIU facility supports only LICCC upgrades.
•Temporary:
All temporary upgrades are LICCC-based. The one billable capacity offering is On/Off Capacity on Demand (On/Off CoD). The two replacement capacity offerings available are Capacity Backup (CBU) and Capacity for Planned Event (CPE).
 
Tip: An MES provides system upgrades that can result in more enabled processors, a different central processor (CP) capacity level, and in additional processor drawers, memory, PCIe I/O drawers, and I/O features (physical upgrade). Additional planning tasks are required for nondisruptive logical upgrades. An MES is ordered through your IBM representative and installed by IBM service support representatives (IBM SSRs).
Concurrent and nondisruptive upgrades
Depending on the impact on the system and application availability, upgrades can be classified in the following manner:
•Concurrent
In general, concurrency addresses the continuity of operations of the hardware part of an upgrade. For example, whether a system (hardware) is required to be turned off during the upgrade. For more information, see 8.2, “Concurrent upgrades” on page 313.
•Non-concurrent
This type of upgrade requires turning off the hardware that is being upgraded. Examples include model upgrades from any z13 model to the z13 NE1 model, and certain physical memory capacity upgrades.
•Disruptive
An upgrade is considered disruptive when resources modified or added to an operating system image require that the operating system be restarted to configure the newly added resources.
•Nondisruptive
Nondisruptive upgrades do not require the software or operating system to be restarted for the upgrade to take effect. Therefore, even concurrent upgrades can be disruptive to operating systems or programs that do not support the upgrades while being nondisruptive to others. For more information, see 8.8, “Nondisruptive upgrades” on page 346.
8.1.2 Terminology that is related to CoD for z13 systems
Table 8-1 lists the most frequently used terms that are related to Capacity on Demand for z13 systems.
Table 8-1 CoD terminology
Term
 
Description
Activated capacity
 
Capacity that is purchased and activated. Purchased capacity can be greater than the activated capacity.
Billable capacity
 
Capacity that helps handle workload peaks, either expected or unexpected. The one billable offering that is available is On/Off Capacity on Demand (OOCoD).
Capacity
 
Hardware resources (processor and memory) that can process the workload can be added to the system through various capacity offerings.
Capacity Backup
(CBU)
 
Capacity Backup allows you to place model capacity or specialty engines in a backup system. CBU is used in an unforeseen loss of system capacity because of an emergency.
Capacity for planned event (CPE)
 
Used when temporary replacement capacity is needed for a short-term event. CPE activates processor capacity temporarily to facilitate moving systems between data centers, upgrades, and other routine management tasks. CPE is an offering of Capacity on Demand.
Capacity levels
 
Can be full capacity or subcapacity. For the z13 system, capacity levels for the CP engine are 7, 6, 5, and 4:
•1 - 99 in decimal and A0 - E1, where A0 represents 100 and E1 represents 141, for capacity level 7nn.
•1 - 30 for capacity levels 6yy and 5yy.
•0 - 30 for capacity levels 4xx. An all Integrated Facility for Linux (IFL) or an all integrated catalog facility (ICF) system has a capacity level of 400.
Capacity setting
 
Derived from the capacity level and the number of processors.
For the z13 system, the capacity levels are 7nn, 6yy, 5yy, and 4xx, where xx, yy, or nn indicates the number of active CPs.
The number of processors can have the following ranges:
•1 - 99 in decimal and A0 - E1, where A0 represents 100 and E1 represents 141, for capacity level 7nn.
•1 - 30 for capacity levels 6yy and 5yy.
•0 - 30 for capacity levels 4xx. An all IFL or an all ICF system has a capacity level of 400.
Customer Initiated Upgrade (CIU)
 
A web-based facility where you can request processor and memory upgrades by using the IBM Resource Link and the system’s Remote Support Facility (RSF) connection.
Capacity on Demand (CoD)
 
The ability of a computing system to increase or decrease its performance capacity as needed to meet fluctuations in demand.
Capacity Provisioning Manager (CPM)
 
As a component of z/OS Capacity Provisioning, CPM monitors business-critical workloads that are running on z/OS on z13 systems.
Customer profile
 
This information is on Resource Link, and contains client and system information. A customer profile can contain information about more than one system.
Full capacity CP feature
 
For z13, feature (CP7) provides full capacity. Capacity settings 7nn are full capacity settings.
High-water mark
 
Capacity that is purchased and owned by the client.
Installed record
 
The LICCC record is downloaded, staged to the Support Element (SE), and is installed on the central processor complex (CPC). A maximum of eight different records can be concurrently installed and active.
Model capacity identifier (MCI)
 
Shows the current active capacity on the system, including all replacement and billable capacity.
For the z13, the model capacity identifier is in the form of 7nn, 6yy, 5yy, or 4xx, where xx, yy, or nn indicates the number of active CPs:
•1 - 99 in decimal and A0 - E1, where A0 represents 100 and E1 represents 141, for capacity level 7nn.
•yy can have a range of 01 - 30.
•xx can have a range of 00 - 30. An all IFL or an all ICF system has a capacity level of 400.
Model Permanent Capacity Identifier (MPCI)
 
Keeps information about the capacity settings that are active before any temporary capacity is activated.
Model Temporary Capacity Identifier (MTCI)
 
Reflects the permanent capacity with billable capacity only, without replacement capacity. If no billable temporary capacity is active, Model Temporary Capacity Identifier equals the MPCI.
On/Off Capacity on Demand (CoD)
 
Represents a function that allows a spare capacity in a CPC to be made available to increase the total capacity of a CPC. For example, On/Off CoD can be used to acquire more capacity for handling a workload peak.
Features on Demand (FoD)
 
FoD is a new centralized way to entitle flexibly features and functions on the system. FoD contains, for example, the IBM z BladeCenter Extension (zBX) High Water Marks (HWM). HWMs refer to highest quantity of blade entitlements by blade type that the client has purchased. On z196 and z114, the HWMs are stored in the processor and memory LICCC record. On z13 and zEC12, the HWMs are stored in the FoD record.
Permanent capacity
 
The capacity that a client purchases and activates. This amount might be less capacity than the total capacity purchased.
Permanent upgrade
 
LIC that is licensed by IBM to enable the activation of applicable computing resources, such as processors or memory, for a specific CIU-eligible system on a permanent basis.
Purchased capacity
 
Capacity that is delivered to and owned by the client. It can be higher than the permanent capacity.
Permanent/
Temporary entitlement record
 
The internal representation of a temporary (TER) or permanent (PER) capacity upgrade that is processed by the CIU facility. An entitlement record contains the encrypted representation of the upgrade configuration with the associated time limit conditions.
Replacement capacity
 
A temporary capacity that is used for situations in which processing capacity in other parts of the enterprise is lost. This loss can be a planned event or an unexpected disaster. The two replacement offerings available are Capacity for Planned Events and Capacity Backup.
Resource Link
 
The IBM Resource Link is a technical support website that provides a comprehensive set of tools and resources. It is available at the IBM Systems technical support website:
Secondary approval
 
An option, which is selected by the client, that requires second approver control for each CoD order. When a secondary approval is required, the request is sent for approval or cancellation to the Resource Link secondary user ID.
Staged record
 
The point when a record that represents a capacity upgrade, either temporary or permanent, is retrieved and loaded on the SE disk.
Subcapacity
 
For the z13, CP features (CP4, CP5, and CP6) provide reduced capacity relative to the full capacity CP feature (CP7).
Temporary capacity
 
An optional capacity that is added to the current system capacity for a limited amount of time. It can be capacity that is owned or not owned by the client.
Vital product data (VPD)
 
Information that uniquely defines system, hardware, software, and microcode elements of a processing system.
8.1.3 Permanent upgrades
Permanent upgrades can be obtained by using these processes:
•Ordered through an IBM marketing representative
•Initiated by the client with the CIU on the IBM Resource Link
 
Tip: The use of the CIU facility for a system requires that the online CoD buying feature (FC 9900) is installed on the system. The CIU facility itself is enabled through the permanent upgrade authorization feature code (FC 9898).
Permanent upgrades that are ordered through an IBM representative
Through a permanent upgrade, you can accomplish these tasks:
•Add processor drawers
•Add Peripheral Component Interconnect Express (PCIe) drawers and features
•Add model capacity
•Add specialty engines
•Add memory
•Activate unassigned model capacity or Integrated Facilities for Linux (IFLs)
•Deactivate activated model capacity or IFLs
•Activate channels
•Activate cryptographic engines
•Change specialty engine (recharacterization)
•Add zBX model 004 entitlement features
 
Considerations: Most of the MESs can be concurrently applied without disrupting the existing workload. For more information, see 8.2, “Concurrent upgrades” on page 313. However, certain MES changes are disruptive, such as model upgrades from any z13 model to the z13 NE1 model.
The only permanent upgrade available for the zBX model 004, supported by z13, is the addition of zBX Blades entitlement features.
Memory upgrades that require dual in-line memory module (DIMM) changes can be made nondisruptively if there are multiple CPC drawers and the flexible memory option is used.
Permanent upgrades that are initiated through CIU on the IBM Resource Link
Ordering a permanent upgrade by using the CIU application through Resource Link allows you to add capacity to fit within your existing hardware:
•Add model capacity
•Add specialty engines
•Add memory
•Activate unassigned model capacity or IFLs
•Deactivate activated model capacity or IFLs
8.1.4 Temporary upgrades
System z13 offers three types of temporary upgrades:
•On/Off Capacity on Demand (On/Off CoD):
This offering allows you to add temporarily more capacity or specialty engines to cover seasonal activities, period-end requirements, peaks in workload, or application testing. This temporary upgrade can be ordered only by using the CIU application through Resource Link.
•Capacity Backup (CBU):
This offering allows you to replace model capacity or specialty engines in a backup system that is used in an unforeseen loss of system capacity because of a disaster.
•Capacity for Planned Event (CPE):
This offering allows you to replace model capacity or specialty engines because of a relocation of workload during system migrations or a data center move.
CBU or CPE temporary upgrades can be ordered by using the CIU application through Resource Link or by calling your IBM marketing representative.
Temporary upgrade capacity changes can be billable or a replacement.
Billable capacity
To handle a peak workload, you can activate up to double the purchased capacity of any processor unit (PU) type temporarily. You are charged daily.
The one billable capacity offering is On/Off Capacity on Demand (On/Off CoD).
Replacement capacity
When a processing capacity is lost in another part of an enterprise, replacement capacity can be activated. It allows you to activate any PU type up to your authorized limit.
The following offerings are the two replacement capacity offerings:
•Capacity Backup
•Capacity for Planned Event
8.2 Concurrent upgrades
Concurrent upgrades on the z13 can provide more capacity with no system outage. In most cases, with prior planning and operating system support, a concurrent upgrade also can be nondisruptive to the operating system.
The benefits of the concurrent capacity growth capabilities that are provided by the z13 include, but are not limited to, these benefits:
•Enabling the exploitation of new business opportunities
•Supporting the growth of smart and cloud environments
•Managing the risk of volatile, high-growth, and high-volume applications
•Supporting 24 x 7 application availability
•Enabling capacity growth during lockdown or frozen periods
•Enabling planned-downtime changes without affecting availability
This capability is based on the flexibility of the design and structure, which allows concurrent hardware installation and Licensed Internal Code (LIC) control over the configuration.
The subcapacity models allow more configuration granularity within the family. The added granularity is available for models that are configured with up to 30 CPs, and provides 90 extra capacity settings. Subcapacity models provide for CP capacity increase in two dimensions that can be used together to deliver configuration granularity. The first dimension is adding CPs to the configuration. The second is changing the capacity setting of the CPs currently installed to a higher model capacity identifier.
The z13 allows the concurrent and nondisruptive addition of processors to a running logical partition (LPAR). As a result, you can have a flexible infrastructure in which you can add capacity without pre-planning. This function is supported by z/OS, z/VM, and z/VSE. There are two ways to accomplish this addition:
•With planning ahead for the future need of extra processors. In the LPAR’s profile, reserved processors can be specified. When the extra processors are installed, the number of active processors for that LPAR can be increased without the need for a partition reactivation and IPL.
•Another (easier) way is to enable the dynamic addition of processors through the z/OS LOADxx member. Set the DYNCPADD parameter in member LOADxx to ENABLE. The z13 supports dynamic processor addition in the same way that the zEC12, z196, and z10 support it. The operating system must be z/OS V1R10 or higher.
Another function concerns the system assist processor (SAP). When more SAPs are concurrently added to the configuration, the SAP-to-channel affinity is dynamically remapped on all SAPs on the system to rebalance the I/O configuration.
8.2.1 Model upgrades
The z13 has a machine type and model, and model capacity identifiers:
•The machine type and model are 2964-Nvv.
The vv can be 30, 63, 96, C9, or E1. The model number indicates how many PUs (vv) are available for client characterization (C9 stands for 129, where C equals 12 in decimal and E1 stands for 141, where E equals 14 in decimal). Model N30 has one processor drawer that is installed, model N63 contains two processor drawers, model N96 contains three processor drawers, and models NC9 and NE1 contain four processor drawers.
•The model capacity identifiers are 4xx, 5yy, 6yy, or 7nn.
The xx is a range of 00 - 302, yy is a range of 01 - 30, and nn is a range of 01 - 99, A0 - E1, where A0 represents the decimal number 100, combining the hexadecimal A with decimal 0 and E1 represents the decimal number 141, obtained combining the hexadecimal E equals 14, and the decimal digit 1. A z13 with 141 client usable processors is a z13 7E1. The model capacity identifier describes how many CPs are characterized (xx, yy, or nn) and the capacity setting (4, 5, 6, or 7) of the CPs.
A hardware configuration upgrade always requires more physical hardware (processor drawers, PCIe I/O drawers, or both of them3). A system upgrade can change either, or both, of the system model and the MCI.
Consider the following model upgrade information:
•LICCC upgrade:
 – Does not change the system model 2964-Nvv because more processor drawers are not added
 – Can change the model capacity identifier, the capacity setting, or both
•Hardware installation upgrade:
 – Can change the system model 2964-Hvv, if one or more processor drawers are added
 – Can change the model capacity identifier, the capacity setting, or both
The system model and the model capacity identifier can be concurrently changed. Concurrent upgrades can be performed for both permanent and temporary upgrades.
 
Tip: A model upgrade can be concurrent by using concurrent drawer add (CDA), except for upgrades to Model NE1.
Licensed Internal Code upgrades (MES ordered)
The LICCC provides for system upgrades without hardware changes by activation of additional (previously installed) unused capacity. Concurrent upgrades through LICCC can be performed for these resources:
•Processors (CPs, ICFs, IBM z Integrated Information Processors (zIIPs), IFLs, and SAPs) if unused PUs are available on the installed processor drawers, or if the model capacity identifier for the CPs can be increased.
•Memory, when unused capacity is available on the installed memory cards. Plan-ahead memory and the flexible memory option are available to give you better control over future memory upgrades. For more information, see 2.4.6, “Flexible Memory Option” on page 55, and 2.4.7, “Pre-planned memory” on page 56.
Concurrent hardware installation upgrades (MES ordered)
Configuration upgrades can be concurrent when installing the following resources:
•Processor drawers (which contain processors, memory, and fanouts). Up to three processor drawers can be added concurrently on the model z13 N30.
•PCIe fanouts.
•I/O cards, when slots are still available on the installed PCIe I/O drawers.
•PCIe I/O drawers.3
•zBX features. However, the upgrade from a zBX Model 002 or zBX Model 003 to a zBX model 004 is disruptive to zBX operations.
The concurrent I/O upgrade capability can be better used if a future target configuration is considered during the initial configuration.
Concurrent PU conversions (MES ordered)
The z13 supports concurrent conversion between all PU types, which includes SAPs, to provide flexibility to meet changing business requirements.
 
Important: The LICCC-based PU conversions require that at least one PU, either CP, ICF, or IFL, remains unchanged. Otherwise, the conversion is disruptive. The PU conversion generates an LICCC that can be installed concurrently in two steps:
1. Remove the assigned PU from the configuration.
2. Activate the newly available PU as the new PU type.
LPARs also might have to free the PUs to be converted. The operating systems must have support to configure processors offline or online so that the PU conversion can be done nondisruptively.
 
Considerations: Client planning and operator action are required to use concurrent PU conversion. Consider the following information about PU conversion:
•It is disruptive if all current PUs are converted to different types.
•It might require individual LPAR outages if dedicated PUs are converted.
Unassigned CP capacity is recorded by a model capacity identifier. CP feature conversions change (increase or decrease) the model capacity identifier.
8.2.2 Customer Initiated Upgrade facility
The CIU facility is an IBM online system through which you can order, download, and install permanent and temporary upgrades for z Systems. Access to and use of the CIU facility requires a contract between the client and IBM, through which the terms and conditions for use of the CIU facility are accepted. The use of the CIU facility for a system requires that the online CoD buying feature code (FC 9900) is installed on the system. Although it can be installed on your z13 at any time, generally it is added when ordering a z13. The CIU facility itself is controlled through the permanent upgrade authorization feature code, FC 9898.
After you place an order through the CIU facility, you receive a notice that the order is ready for download. You can then download and apply the upgrade by using functions that are available through the Hardware Management Console (HMC), along with the Remote Support Facility (RSF). After all the prerequisites are met, the entire process, from ordering to activation of the upgrade, is performed by the client.
After download, the actual upgrade process is fully automated and does not require any onsite presence of IBM SSRs.
CIU prerequisites
The CIU facility supports LICCC upgrades only. It does not support I/O upgrades. All additional capacity that is required for an upgrade must be previously installed. Additional processor drawers or I/O cards cannot be installed as part of an order that is placed through the CIU facility. The sum of CPs, unassigned CPs, ICFs, zIIPs, IFLs, and unassigned IFLs cannot exceed the client (characterized) PU count of the installed processor drawers. The total number of zIIPs can be twice the number of purchased CPs.
CIU registration and contract for CIU
To use the CIU facility, a client must be registered and the system must be set up. After you complete the CIU registration, access to the CIU application is available through the IBM Resource Link website:
As part of the setup, provide one resource link ID for configuring and placing CIU orders and, if required, a second ID as an approver. The IDs are then set up for access to the CIU support. The CIU facility allows upgrades to be ordered and delivered much faster than through the regular MES process.
To order and activate the upgrade, log on to the IBM Resource Link website and start the CIU application to upgrade a system for processors or memory. Requesting a client order approval to conform to your operational policies is possible. You can allow the definition of more IDs to be authorized to access the CIU. Additional IDs can be authorized to enter or approve CIU orders, or only view existing orders.
Permanent upgrades
Permanent upgrades can be ordered by using the CIU facility. Through the CIU facility, you can generate online permanent upgrade orders to concurrently add processors (CPs, ICFs, zIIPs, IFLs, and SAPs) and memory, or change the model capacity identifier. You can do so up to the limits of the installed processor drawers on an existing system.
Temporary upgrades
The base model z13 describes permanent and dormant capacity (Figure 8-1) using the capacity marker and the number of PU features installed on the system. Up to eight temporary offerings can be present. Each offering has its own policies and controls, and each can be activated or deactivated independently in any sequence and combination. Although multiple offerings can be active at any time, if enough resources are available to fulfill the offering specifications, only one On/Off CoD offering can be active at any time.
Figure 8-1 The provisioning architecture
Temporary upgrades are represented in the system by a record. All temporary upgrade records are resident on the SE hard disk drive (HDD). The records can be downloaded from the RSF or installed from portable media. At the time of activation, you can control everything locally. Figure 8-1 shows a representation of the provisioning architecture.
The authorization layer enables administrative control over the temporary offerings. The activation and deactivation can be driven either manually or under control of an application through a documented application programming interface (API).
By using the API approach, you can customize, at activation time, the resources that are necessary to respond to the current situation, up to the maximum that is specified in the order record. If the situation changes, you can add or remove resources without having to go back to the base configuration. This process eliminates the need for temporary upgrade specification for all possible scenarios. However, for CPE, the ordered configuration is the only possible activation.
In addition, this approach enables you to update and replenish temporary upgrades, even in situations where the upgrades are already active. Likewise, depending on the configuration, permanent upgrades can be performed while temporary upgrades are active. Figure 8-2 shows examples of the activation sequence of multiple temporary upgrades.
Figure 8-2 Example of temporary upgrade activation sequence
If R2, R3, and R1 are active at the same time, only parts of R1 can be activated because not enough resources are available to fulfill all of R1. When R2 is deactivated, the remaining parts of R1 can be activated as shown.
Temporary capacity can be billable as On/Off CoD, or replacement capacity as CBU or CPE:
•On/Off CoD is a function that enables concurrent and temporary capacity growth of the system.
On/Off CoD can be used for client peak workload requirements, for any length of time, and has a daily hardware and maintenance charge. The software charges can vary according to the license agreement for the individual products. See your IBM Software Group representative for exact details.
On/Off CoD can concurrently add processors (CPs, ICFs, zIIPs, IFLs, and SAPs), increase the model capacity identifier, or both. It can do so up to the limit of the installed processor drawers of an existing system, and is restricted to twice the currently installed capacity. On/Off CoD requires a contractual agreement between you and IBM.
You decide whether to either pre-pay or post-pay On/Off CoD. Capacity tokens inside the records are used to control activation time and resources.
•CBU is a concurrent and temporary activation of more CPs, ICFs, zIIPs, IFLs, and SAPs, an increase of the model capacity identifier, or both.
CBU cannot be used for peak workload management in any form. As stated, On/Off CoD is the correct way to do that. A CBU activation can last up to 90 days when a disaster or recovery situation occurs. CBU features are optional, and require unused capacity to be available on installed processor drawers of the backup system. They can be available as unused PUs, as a possibility to increase the model capacity identifier, or as both.
A CBU contract must be in place before the special code that enables this capability can be loaded on the system. The standard CBU contract provides for five 10-day tests4 (the CBU test activation) and one 90-day activation over a five-year period. Contact your IBM representative for details.
You can run production workload on a CBU upgrade during a CBU test. At least an equivalent amount of production capacity must be shut down during the CBU test. If you already have existing CBU contracts, you also must sign an Amendment (US form #Z125-8145) with IBM to allow you to run production workload on a CBU upgrade during your CBU tests.
•CPE is a concurrent and temporary activation of extra CPs, ICFs, zIIPs, IFLs, and SAPs, an increase of the model capacity identifier, or both.
The CPE offering is used to replace temporary lost capacity within a client’s enterprise for planned downtime events, for example, with data center changes. CPE cannot be used for peak load management of client workload or for a disaster situation.
The CPE feature requires unused capacity to be available on installed processor drawers of the backup system. The capacity must be available either as unused PUs, as a possibility to increase the model capacity identifier on a subcapacity system, or as both. A CPE contract must be in place before the special code that enables this capability can be loaded on the system. The standard CPE contract provides for one 3-day planned activation at a specific date. Contact your IBM representative for details.
8.2.3 Summary of concurrent upgrade functions
Table 8-2 summarizes the possible concurrent upgrades combinations.
Table 8-2 Concurrent upgrade summary
Type
Name
Upgrade
Process
Permanent
MES
CPs, ICFs, zIIPs, IFLs, SAPs, processor drawer, memory, and I/Os
Installed by IBM SSRs
Online permanent upgrade
CPs, ICFs, zIIPs, IFLs, SAPs, and memory
Performed through the CIU facility
Temporary
On/Off CoD
CPs, ICFs, zIIPs, IFLs, and SAPs
Performed through the OOCoD facility
CBU
CPs, ICFs, zIIPs, IFLs, and SAPs
Performed through the CBU facility
CPE
CPs, ICFs, zIIPs, IFLs, and SAPs
Performed through the CPE facility
8.3 Miscellaneous equipment specification (MES) upgrades
MES upgrades enable concurrent and permanent capacity growth. MES upgrades allow the concurrent adding of processors (CPs, ICFs, zIIPs, IFLs, and SAPs), memory capacity, and I/O ports, and entitlements to the IBM z BladeCenter Extension. For subcapacity models, MES upgrades allow the concurrent adjustment of both the number of processors and the capacity level. The MES upgrade can be performed by using LICCC only, by installing more processor drawers, by adding PCIe I/O drawers, by adding I/O5 features, or by a combination:
•MES upgrades for processors are done by any of the following methods:
 – LICCC assigning and activating unassigned PUs up to the limit of the installed processor drawers.
 – LICCC to adjust the number and types of PUs, to change the capacity setting, or both.
 – Installing more processor drawers, and LICCC assigning and activating unassigned PUs on the installed processor drawers.
•MES upgrades for memory are done by one of the following methods:
 – Using LICCC to activate more memory capacity up to the limit of the memory cards on the currently installed processor drawers. Plan-ahead and flexible memory features enable you to have better control over future memory upgrades. For more information about the memory features, see these descriptions:
 – Installing more processor drawers and using LICCC to activate more memory capacity on installed processor drawers.
 – Using the Enhanced CPC Drawer Availability (EDA), where possible, on multi-drawer systems to add or change the memory cards.
•MES upgrades for I/O5 are done by installing more I/O5 features and supporting infrastructure, if required, on PCIe drawers that are already installed, or installing more PCIe drawers to hold the new cards.
•Entitlement MES upgrades for the IBM z BladeCenter Extension can be performed only through your IBM representative.
An MES upgrade requires IBM SSRs for the installation. In most cases, the time that is required for installing the LICCC and completing the upgrade is short.
To better use the MES upgrade function, carefully plan the initial configuration to allow a concurrent upgrade to a target configuration. The availability of PCIe I/O drawers improves the flexibility to perform unplanned I/O configuration changes concurrently.
The Store System Information (STSI) instruction gives more useful and detailed information about the base configuration and temporary upgrades. You can more easily resolve billing situations where independent software vendor (ISV) products are in use.
The model and model capacity identifiers that are returned by the STSI instruction are updated to coincide with the upgrade. For more information, see “Store System Information instruction” on page 348.
 
Upgrades: The MES provides the physical upgrade, resulting in more enabled processors, different capacity settings for the CPs, and more memory, I/O ports, I/O adapters, and I/O drawers. Additional planning tasks are required for nondisruptive logical upgrades. For more information, see “Guidelines to avoid disruptive upgrades” on page 350.
8.3.1 MES upgrade for processors
An MES upgrade for processors can concurrently add CPs, ICFs, zIIPs, IFLs, and SAPs to a z13 by assigning available PUs on the processor drawers, through LICCC. Depending on the quantity of the additional processors in the upgrade, more processor drawers might be required, and can be concurrently installed before the LICCC is enabled. With the subcapacity models, more capacity can be provided by adding CPs, by changing the capacity identifier on the current CPs, or by doing both.
 
Limits: The sum of CPs, inactive CPs, ICFs, zIIPs, IFLs, unassigned IFLs, and SAPs cannot exceed the maximum limit of PUs available for client use. The number of zIIPs cannot exceed twice the number of purchased CPs.
Figure 8-3 is an example of an MES upgrade for processors, showing two upgrade steps.
Figure 8-3 MES for processor example
A model N30 (one processor drawer), model capacity identifier 708 (eight CPs), is concurrently upgraded to a model N63 (two processor drawers), with model capacity identifier (MCI) 738 (38 CPs). The model upgrade requires adding a processor drawer and assigning and activating 38 PUs as CPs. Then, model N63, MCI 738, is concurrently upgraded to a capacity identifier 739 (39 CPs) with two IFLs. This process is done by assigning and activating three more unassigned PUs (one as CP and two as IFLs). If needed, more LPARs can be created concurrently to use the newly added processors.
The example in Figure 8-3 on page 321 was used to show how the addition of PUs as CPs and IFLs, and the addition of a processor drawer, works. In reality, the addition of a processor drawer to a z13 Model N30 upgrades the machine model to N63. In addition, one of the two spare PUs on CPC drawer 0 is moved over to CPC drawer 1 to have one spare PU on each CPC drawer. After the second CPC drawer addition, CPC drawer 0 has 31 configurable PUs and CPC drawer 1 has 32 configurable PUs, which allow 63 PUs to be characterized on the new N63 model.
 
Consideration: Up to 141 logical processors, including reserved processors, can be defined to an LPAR. However, do not define more processors to an LPAR than the target operating system supports.
Table 8-3 describes the number of processors that are supported by various z/OS and z/VM releases.
Table 8-3 Number of processors that are supported by the operating system
Operating system
Number of processors that are supported
z/OS V1R10 with PTFs
64
z/OS V1R11 with PTFs
100
z/OS V1R12 with PTFs
100
z/OS V1R13 with PTFs
100
z/OS V2R1
141 PUs per z/OS LPAR in non-SMT mode and 128 PUs per z/OS LPAR in SMT mode. For both, the PU total is the sum of CPs and zIIPs
z/VM V5R4 - z/VM V6R3
32
z/VSE
z/VSE Turbo Dispatcher can use up to 4 CPs, and tolerates up to 10-way LPARs
z/TPF
86 CPs
Linux on z Systems -141 CPs
SUSE SLES 10: 64 CPs or IFLs
SUSE SLES 11: 64 CPs or IFLs
Red Hat RHEL 5: 80 CPs or IFLs
Red Hat RHEL 6: 80 CPs or IFLs
Software charges, which are based on the total capacity of the system on which the software is installed, are adjusted to the new capacity after the MES upgrade.
Software products that use Workload License Charges (WLC) might not be affected by the system upgrade. Their charges are based on partition usage, not on the system total capacity. For more information about WLC, see 7.16, “Software licensing” on page 299.
8.3.2 MES upgrades for memory
MES upgrades for memory can concurrently add more memory in the following ways:
•Enabling, through LICCC, more capacity up to the limit of the currently installed DIMM memory cards
•Concurrently installing more CPC drawers and LICCC-enabling memory capacity on the new CPC drawers.
The Preplanned Memory Feature is available to allow better control over future memory upgrades. For more information about plan-ahead memory features, see 2.4.6, “Flexible Memory Option” on page 55 and 2.4.7, “Pre-planned memory” on page 56.
If the z13 is a multiple processor drawer configuration, you can use the EDA feature to remove a processor drawer and add DIMM memory cards. It also can be used to upgrade the already installed memory cards to a larger size. You can then use LICCC to enable the additional memory. With proper planning, more memory can be added nondisruptively to z/OS partitions and z/VM partitions. If necessary, new LPARs can be created nondisruptively to use the newly added memory.
 
 
 
 
 
Concurrency: Upgrades requiring DIMM changes can be concurrent by using the EDA feature. Planning is required to see whether this is a viable option for your configuration. Using the flexible memory option and the Preplanned Memory Feature (FC 1996 for the 16-GB increment, or FC 1990 for the 32-GB increment) ensures that EBA can work with the least disruption.
The one-processor drawer model N30 has a minimum of 320 GB physical installed memory. The client addressable storage in this case is 256 GB. If you require more, an additional memory upgrade can install up to 2.5 TB of memory. It does so by changing the existing DIMM sizes and adding more DIMMs in all available slots in the processor drawer. You can also add memory by concurrently adding a second processor drawer with sufficient memory into the configuration and then using LICCC to enable that memory.
An LPAR can dynamically take advantage of a memory upgrade if reserved storage is defined to that LPAR. The reserved storage is defined to the LPAR as part of the image profile. Reserved memory can be configured online to the LPAR by using the LPAR dynamic storage reconfiguration (DSR) function. DSR allows a z/OS operating system image and z/VM partitions to add reserved storage to their configuration if any unused storage exists.
The nondisruptive addition of storage to a z/OS and z/VM partition requires that pertinent operating system parameters have been prepared. If reserved storage is not defined to the LPAR, the LPAR must be deactivated, the image profile changed, and the LPAR reactivated. This process allows the additional storage resources to be available to the operating system image.
8.3.3 MES upgrades for I/O
MES upgrades for I/O can concurrently add more I/O features by using one of the following methods:
•Installing more I/O features on an already installed PCIe I/O drawer.
•The installed PCIe I/O drawer must provide the number of I/O slots that are required by the target configuration.
•Adding a PCIe I/O drawer to hold the new I/O features.
 
Tip: Up to two I/O drawers are supported if carried forward on an upgrade from a zEC12 or from a z196. They only support the FICON Express8 feature in the z13.
For more information about I/O drawers and PCIe I/O drawers, see 4.2, “I/O system overview” on page 138.
Table 8-4 gives an overview of the number of I/O drawers and PCIe I/O drawers that can be present in a z13.
Table 8-4 I/O drawers and PCIe drawer summary
Description
New build
Carry-forward
MES add
I/O drawer
0
0 - 2
0
PCIe I/O drawer
0 - 5
0 - 5
0 - 5
Table 8-5 lists the number of FICON Express8 features that can be carried forward.
Table 8-5 Number of FICON Express8 features and drawers on a carry-forward
Number of FICON Express8 features on a carry-forward
Number of I/O drawers
1 - 8
1
9 - 16
2
 
Consideration: The maximum number of original I/O features on a carry-forward is 16. Also, the only supported carry-forward feature is the 4-port FICON Express 8 card.
Depending on the number of I/O features that are carried forward on an upgrade, the configurator determines the number I/O drawers and PCIe I/O drawers.
To better use the MES for I/O capability, carefully plan the initial configuration to allow concurrent upgrades up to the target configuration. If original I/O features are removed from the I/O drawer, the configurator does not physically remove the drawer unless the I/O frame slots are required to install a new PCIe I/O drawer.
If a PCIe I/O drawer is added to an existing z13 and original features must be physically moved to another PCIe I/O drawer, original card moves are disruptive.
z/VSE, z/TPF, Linux on z Systems, and CFCC do not provide dynamic I/O configuration support. The installation of the new hardware is performed concurrently, but defining the new hardware to these operating systems requires an IPL.
 
Tip: The z13 has a hardware system area (HSA) of 96 GB. The zEC12 has a 32-GB HSA. HSA is not part of the client-purchased memory.
8.3.4 MES upgrades for the zBX
The MES upgrades for zBX model 004 can concurrently add blades if there are any slots available in the existing blade chassis and the entitlement records allow this action. Add Feature on Demand (FoD) entitlements through LICCC are available for zBX model 004 only if there are empty slots in the existing BladeCenter chassis.
Feature on Demand
FoD contains the zBX High Water Marks (HWMs). HWMs refer to the highest quantities of blade entitlements by blade type that the client has purchased. On the z196/z114, the HWMs are stored in the processor and memory LICCC record. On the z13 and zEC12, the HWMs are in the FoD LICCC record.
The current zBX installed and staged feature values can be obtained by using the Perform Model Conversion function on the SE, or from the HMC by using a Single Object Operation (SOO) to the servers’ SE. Figure 8-4 shows the window for FoD Blades feature values that are shown under the Perform Model Conversion → Features on Demand Manage function.
Figure 8-4 Features on Demand window for zBX blade feature HWMs
Only one FoD LICCC record is installed or staged at any time in the system and its contents can be viewed under the Manage window, as shown in Figure 8-4. A staged record can be removed without installing it. An FoD record can only be installed completely. There is no selective feature or partial record installation. The features that are installed are merged with the CPC LICCC after activation.
An FoD record can be installed only one time. If it is removed, a new FoD record is needed to install it again. A remove action cannot be undone.
If upgrading from an existing z196 with zBX-002, or upgrading from an existing zEC12 with zBX -003, the zBX Model 002 or the zBX model 003 must to be upgraded to a zBX Model 004. The existing zBX must be detached from z196 or the zEC12 and converted to a zBX model 004, becoming an independent node that is not associated, or not owned, by the z13. Because the system upgrade is always disruptive, the zBX upgrade is a disruptive task as well. Because the zBX model 004 is not owned by z13, the zBX upgrade does not have to be done at the same time that the CPC is upgraded. However, the zBX upgrades are disruptive to the zBX environment.
If installing a new build z13 and you plan to take over an existing zBX attached to a zEC12, zBC12, z196, or z114, the conversion of the zBX-002 or zBX-003 to the zBX-004 also can be done during the installation phase of the z13. FC 0030 must be ordered to detach the zBX from an existing z196 or z114. FC 0031 is required to reattach the zBX to the zEC12 or zBC12.
If the model zBX Model 002 still has IBM Smart Analytics Optimizer blades installed, they must be removed from the Model 002 before you order the upgrade to a Model 004.
The zBX Model 004 maintains the zBX-003 features or incorporates them if upgrading from a zBX model 002. Here are the features and functions that are available with zBX-004:
•Broadband RSF support. The HMC application LIC for current z Systems servers does not support dial modem use.
•Increased quantity of System x blade enablement to 56.
•Enables potential of 20-Gb Ethernet bandwidth by using link aggregation.
•Doubled 10 GbE cables between a BladeCenter 10GbE switch and a 10GbE Top of Rack (ToR) switch.
•Doubled 10 GbE cables between the BladeCenter 10GbE switches.
•New version of the advanced management module (AMM) in the BladeCenter chassis.
•Upgraded hypervisors and all existing zBX firmware.
8.3.5 Summary of plan-ahead features
A number of plan-ahead features exist for z13. The following list provides an overview of those features:
•Flexible memory
Flexible memory has no feature code (FC) associated with it. The purpose of flexible memory is to enable enhanced processor drawer availability. If a processor drawer must be serviced, the flexible memory is activated to accommodate the storage of the CPC drawer that is taken offline. After the repair action, the memory is taken offline again and is made unavailable for usage.
•Pre-planned memory
Pre-planned memory allows you to plan for nondisruptive memory upgrades. Any hardware that is required is pre-plugged, based on a target capacity that is specified in advance. Pre-plugged hardware is enabled by using an LICCC order when additional memory capacity is needed. FC 1990 provides 32 GB of pre-planned memory, and FC 1996 provides 16 GB of pre-planned memory. FC 1901 is used to activate previously installed pre-planned memory, and can activate all the preinstalled memory or subsets of it.
•Balanced Power Plan Ahead
Balanced Power Plan Ahead is designed to anticipate future upgrade power needs on the z13. When more processor drawers are added to the system, the power consumption also rises. If necessary, one or more bulk power regulators (BPRs) must be added. This process increases the time that is needed for the upgrade. When ordering this feature, regardless of the configuration, all six BPR pairs are installed and activated. Balanced Power Plan Ahead has FC 3003.
•Line Cord plan ahead
This option allows you to plan ahead for the second set of power cords. It is normally not configured until the addition of extra BPRs requires them. A plan-ahead option allows you to plan for a lengthy outage that is caused by installing circuit breakers or power feeds, or the routing of cables under the floor. The Line Cord plan-ahead option is FC 2000.
 
Tip: Accurate planning and the definition of the target configuration allows you to maximize the value of these plan-ahead features.
8.4 Permanent upgrade through the CIU facility
By using the CIU facility (through the IBM Resource Link on the web), you can initiate a permanent upgrade for CPs, ICFs, zIIPs, IFLs, SAPs, or memory. When performed through the CIU facility, you add the resources without having IBM personnel present at your location. You can also unassign previously purchased CPs and IFL processors through the CIU facility.
Adding permanent upgrades to a system through the CIU facility requires that the permanent upgrade enablement feature (FC 9898) is installed on the system. A permanent upgrade might change the system model capacity identifier (4xx, 5yy, 6yy, or 7nn) if more CPs are requested, or if the capacity identifier is changed as part of the permanent upgrade. However, it cannot change the system model. If necessary, more LPARs can be created concurrently to use the newly added processors.
 
Consideration: A permanent upgrade of processors can provide a physical concurrent upgrade, resulting in more enabled processors that are available to a system configuration. Therefore, more planning and tasks are required for nondisruptive logical upgrades. For more information, see “Guidelines to avoid disruptive upgrades” on page 350.
Maintenance charges are automatically adjusted as a result of a permanent upgrade.
Software charges that are based on the total capacity of the system on which the software is installed are adjusted to the new capacity after the permanent upgrade is installed. Software products that use WLC might not be affected by the system upgrade because their charges are based on an LPAR usage rather than system total capacity. For more information about WLC, see 7.16.3, “Advanced Workload License Charges (AWLC)” on page 302.
Figure 8-5 illustrates the CIU facility process on IBM Resource Link.
Figure 8-5 Permanent upgrade order example
The following sample sequence shows how to initiate an order on the IBM Resource Link:
1. Sign on to Resource Link.
2. Select Customer Initiated Upgrade from the main Resource Link page. Client and system details that are associated with the user ID are displayed.
3. Select the system to receive the upgrade. The current configuration (PU allocation and memory) is shown for the selected system.
4. Select Order Permanent Upgrade. The Resource Link limits the options to those that are valid or possible for the selected configuration (system).
5. After the target configuration is verified by the system, accept or cancel the order. An order is created and verified against the pre-established agreement.
6. Accept or reject the price that is quoted. A secondary order approval is optional. Upon confirmation, the order is processed. The LICCC for the upgrade will be available within hours.
Figure 8-6 illustrates the process for a permanent upgrade. When the LICCC is passed to the Remote Support Facility, you are notified through an email that the upgrade is ready to be downloaded.
Figure 8-6 CIU-eligible order activation example
The two major components in the process are ordering and retrieval (along with activation).
8.4.1 Ordering
Resource Link provides the interface that enables you to order a concurrent upgrade for a system. You can create, cancel, or view the order, and view the history of orders that were placed through this interface. Configuration rules enforce that only valid configurations are generated within the limits of the individual system. Warning messages are issued if you select invalid upgrade options. The process allows only one permanent CIU-eligible order for each system to be placed at a time.
For a tutorial, see this website:
Figure 8-7 shows the initial view of the Machine profile on Resource Link.
Figure 8-7 Machine profile window
The number of CPs, ICFs, zIIPs, IFLs, SAPs, memory size, and unassigned IFLs on the current configuration are displayed on the left side of the web page.
Resource Link retrieves and stores relevant data that is associated with the processor configuration, such as the number of CPs and installed memory cards. It allows you to select only those upgrade options that are deemed valid by the order process. It allows upgrades only within the bounds of the currently installed hardware.
8.4.2 Retrieval and activation
After an order is placed and processed, the appropriate upgrade record is passed to the IBM support system for download.
When the order is available for download, you receive an email that contains an activation number. You can then retrieve the order by using the Perform Model Conversion task from the SE, or through the Single Object Operation to the SE from an HMC.
In the Perform Model Conversion window, select Permanent upgrades to start the process, as shown in Figure 8-8.
Figure 8-8 z13 Perform Model Conversion window
The window provides several possible options. If you select the Retrieve and apply data option, you are prompted to enter the order activation number to initiate the permanent upgrade, as shown in Figure 8-9.
Figure 8-9 Customer Initiated Upgrade Order Activation Number window
8.5 On/Off Capacity on Demand
On/Off Capacity on Demand (On/Off CoD) allows you to enable temporarily PUs and unassigned IFLs that are available within the current hardware model. You also can use it to change capacity settings for CPs to help meet your peak workload requirements.
8.5.1 Overview
The capacity for CPs is expressed in millions of service units (MSUs). Capacity for speciality engines is expressed in number of speciality engines. Capacity tokens are used to limit the resource consumption for all types of processor capacity.
Capacity tokens are introduced to provide better control over resource consumption when On/Off CoD offerings are activated. Tokens represent the following resource consumptions:
•For CP capacity, each token represents the amount of CP capacity that results in one MSU of software cost for one day (an MSU-day token).
•For speciality engines, each token is equivalent to one speciality engine capacity for one day (an engine-day token).
Each speciality engine type has its own tokens, and each On/Off CoD record has separate token pools for each capacity type. During the ordering sessions on Resource Link, select how many tokens of each type to create for an offering record. Each engine type must have tokens for that engine type to be activated. Capacity that has no tokens cannot be activated.
When resources from an On/Off CoD offering record that contains capacity tokens are activated, a billing window is started. A billing window is always 24 hours. Billing takes place at the end of each billing window. The resources that are billed are the highest resource usage inside each billing window for each capacity type. An activation period is one or more complete billing windows. The activation period is the time from the first activation of resources in a record until the end of the billing window in which the last resource in a record is deactivated. At the end of each billing window, the tokens are decremented by the highest usage of each resource during the billing window. If any resource in a record does not have enough tokens to cover usage for the next billing window, the entire record is deactivated.
On/Off CoD requires that the Online CoD Buying feature (FC 9900) is installed on the system that you want to upgrade.
The On/Off CoD to Permanent Upgrade Option is a new offering. It is an offshoot of On/Off CoD and takes advantage of aspects of the architecture. You are given a window of opportunity to assess capacity additions to your permanent configurations by using On/Off CoD. If a purchase is made, the hardware On/Off CoD charges during this window (three days or less) are waived. If no purchase is made, you are charged for the temporary use.
The resources eligible for temporary use are CPs, ICFs, zIIPs, IFLs, and SAPs. The temporary addition of memory and I/O ports or adapters is not supported. Unassigned PUs that are on the installed processor drawers can be temporarily and concurrently activated as CPs, ICFs, zIIPs, IFLs, and SAPs through LICCC. You can assign PUs up to twice the currently installed CP capacity, and up to twice the number of ICFs, zIIPs, or IFLs. Therefore, an On/Off CoD upgrade cannot change the system model. The addition of new processor drawers is not supported. However, the activation of an On/Off CoD upgrade can increase the model capacity identifier (4xx, 5yy, 6yy, or 7nn).
8.5.2 Ordering
Concurrently installing temporary capacity by ordering On/Off CoD is possible in the following manner:
•CP features equal to the MSU capacity of installed CPs
•IFL features up to the number of installed IFLs
•ICF features up to the number of installed ICFs
•zIIP features up to the number of installed zIIPs
•SAPs up to six for model N30, 12 for an N63, 18 for an N96, and 24 for an NC9 and NE1
On/Off CoD can provide CP temporary capacity in two ways:
•By increasing the number of CPs.
•For subcapacity models, capacity can be added by increasing the number of CPs, changing the capacity setting of the CPs, or both. The capacity setting for all CPs must be the same. If the On/Off CoD is adding CP resources that have a capacity setting different from the installed CPs, the base capacity settings are changed to match.
On/Off CoD has the following limits that are associated with its use:
 – The number of CPs cannot be reduced.
 – The target configuration capacity is limited to these amounts:
 • Twice the currently installed capacity, expressed in MSUs for CPs.
 • Twice the number of installed IFLs, ICFs, and zIIPs. The number of SAPs that can be activated depends on the model. For more information, see 8.2.1, “Model upgrades” on page 313.
On/Off CoD can be ordered as prepaid or postpaid:
•A prepaid On/Off CoD offering record contains resource descriptions, MSUs, a number of speciality engines, and tokens that describe the total capacity that can be used. For CP capacity, the token contains MSU-days. For speciality engines, the token contains speciality engine-days.
•When resources on a prepaid offering are activated, they must have enough capacity tokens to allow the activation for an entire billing window, which is 24 hours. The resources remain active until you deactivate them or until one resource consumes all of its capacity tokens. When that happens, all activated resources from the record are deactivated.
•A postpaid On/Off CoD offering record contains resource descriptions, MSUs, speciality engines, and can contain capacity tokens that denote MSU-days and speciality engine-days.
•When resources in a postpaid offering record without capacity tokens are activated, those resources remain active until they are deactivated, or until the offering record expires. The record usually expires 180 days after its installation.
•When resources in a postpaid offering record with capacity tokens are activated, those resources must have enough capacity tokens to allow the activation for an entire billing window (24 hours). The resources remain active until they are deactivated, until one of the resource tokens are consumed, or until the record expires. The record usually expires 180 days after its installation. If one capacity token type is consumed, resources from the entire record are deactivated.
As an example, for a z13 with capacity identifier 502 (two CPs), there are two ways to deliver a capacity upgrade through On/Off CoD:
•The first option is to add CPs of the same capacity setting. With this option, the model capacity identifier can be changed to a 503, adding one more CP to make it a 3-way. It can also be changed to a 504, which adds two CPs, making it a 4-way.
•The second option is to change to a different capacity level of the current CPs and change the model capacity identifier to a 602 or to a 702. The capacity level of the CPs is increased, but no additional CPs are added. The 502 also can be temporarily upgraded to a 603, increasing the capacity level and adding another processor. The capacity setting 430 does not have an upgrade path through On/Off CoD.
Use the Large System Performance Reference (LSPR) information to evaluate the capacity requirements according to your workload type. LSPR data for current IBM processors is available at this website:
The On/Off CoD hardware capacity is charged on a 24-hour basis. There is a grace period at the end of the On/Off CoD day. This grace period allows up to an hour after the 24-hour billing period to either change the On/Off CoD configuration for the next 24-hour billing period or deactivate the current On/Off CoD configuration. The times when the capacity is activated and deactivated are maintained in the z13 and sent back to the support systems.
If On/Off capacity is already active, more On/Off capacity can be added without having to return the system to its original capacity. If the capacity is increased multiple times within a 24-hour period, the charges apply to the highest amount of capacity active in that period. If more capacity is added from an already active record that contains capacity tokens, the systems checks whether the resource has enough capacity to be active for an entire billing window (24 hours). If that criteria is not met, no additional resources are activated from the record.
If necessary, more LPARs can be activated concurrently to use the newly added processor resources.
 
Consideration: On/Off CoD provides a concurrent hardware upgrade, resulting in more enabled processors that are available to a system configuration. Additional planning tasks are required for nondisruptive upgrades. For more information, see “Guidelines to avoid disruptive upgrades” on page 350.
To participate in this offering, you must have accepted contractual terms for purchasing capacity through the Resource Link, established a profile, and installed an On/Off CoD enablement feature on the system. Later, you can concurrently install temporary capacity up to the limits in On/Off CoD and use it for up to 180 days. Monitoring occurs through the system call-home facility, and an invoice is generated if the capacity is enabled during the calendar month. You are billed for the use of temporary capacity until the system is returned to the original configuration. If the On/Off CoD support is no longer needed, the enablement code must be removed.
On/Off CoD orders can be pre-staged in Resource Link to allow multiple optional configurations. The pricing of the orders is done at the time you order them, and the pricing can vary from quarter to quarter. Staged orders can have different pricing. When the order is downloaded and activated, the daily costs are based on the pricing at the time of the order. The staged orders do not have to be installed in order sequence. If a staged order is installed out of sequence, and later an order that was staged that had a higher price is downloaded, the daily cost is based on the lower price.
Another possibility is to store unlimited On/Off CoD LICCC records on the SE with the same or different capacities, giving you greater flexibility to enable quickly needed temporary capacity. Each record is easily identified with descriptive names, and you can select from a list of records that can be activated.
Resource Link provides the interface to order a dynamic upgrade for a specific system. You can create, cancel, and view the order. Configuration rules are enforced, and only valid configurations are generated based on the configuration of the individual system. After you complete the prerequisites, orders for the On/Off CoD can be placed. The order process uses the CIU facility on Resource Link.
You can order temporary capacity for CPs, ICFs, zIIPs, IFLs, or SAPs. Memory and channels are not supported on On/Off CoD. The amount of capacity is based on the amount of owned capacity for the different types of resources. An LICCC record is established and staged to Resource Link for this order. After the record is activated, it has no expiration date.
However, an individual record can be activated only once. Subsequent sessions require a new order to be generated, producing a new LICCC record for that specific order. Alternatively, you can use an auto-renewal feature to eliminate the need for a manual replenishment of the On/Off CoD order. This feature is implemented in Resource Link, and you also must select this feature in the machine profile. For more information, see Figure 8-10.
Figure 8-10 Order On/Off CoD record window
8.5.3 On/Off CoD testing
Each On/Off CoD-enabled system is entitled to one no-charge 24-hour test. No IBM charges are assessed for the test, including charges that are associated with temporary hardware capacity, IBM software, and IBM maintenance. The test can be used to validate the processes to download, stage, install, activate, and deactivate On/Off CoD capacity.
This test can have a maximum duration of 24 hours, commencing upon the activation of any capacity resource that is contained in the On/Off CoD record. Activation levels of capacity can change during the 24-hour test period. The On/Off CoD test automatically terminates at the end of the 24-hour period.
In addition, you can perform administrative testing. No additional capacity is added to the system, but you can test all the procedures and automation for the management of the On/Off CoD facility.
Figure 8-11 is an example of an On/Off CoD order on the Resource Link web page.
Figure 8-11 On/Off CoD order example
The example order in Figure 8-11 is an On/Off CoD order for 0% more CP capacity (system is already at capacity level 7), and for two more ICFs, and two more zIIPs. The maximum number of CPs, ICFs, zIIPs, and IFLs is limited by the current number of available unused PUs of the installed processor drawers. The maximum number of SAPs is determined by the model number and the number of available PUs on the already installed processor drawers.
To finalize the order you have to accept Terms and Conditions for the order, as shown in Figure 8-12.
Figure 8-12 CIU order Terms and Conditions
8.5.4 Activation and deactivation
When a previously ordered On/Off CoD is retrieved from Resource Link, it is downloaded and stored on the SE HDD. You can activate the order when the capacity is needed, either manually or through automation.
If the On/Off CoD offering record does not contain resource tokens, you must deactivate the temporary capacity manually. Deactivation is accomplished from the SE, and is nondisruptive. Depending on how the additional capacity was added to the LPARs, you might be required to perform tasks at the LPAR level to remove it. For example, you might have to configure offline any CPs that were added to the partition, deactivate LPARs that were created to use the temporary capacity, or both.
On/Off CoD orders can be staged in Resource Link so that multiple orders are available. An order can be downloaded and activated only one time. If a different On/Off CoD order is required or a permanent upgrade is needed, it can be downloaded and activated without having to restore the system to its original purchased capacity.
In support of automation, an API is provided that allows the activation of the On/Off CoD records. The activation is performed from the HMC, and requires specifying the order number. With this API, automation code can be used to send an activation command along with the order number to the HMC to enable the order.
8.5.5 Termination
A client is contractually obligated to terminate the On/Off CoD right-to-use feature when a transfer in asset ownership occurs. A client also can choose to terminate the On/Off CoD right-to-use feature without transferring ownership. Application of FC 9898 terminates the right to use the On/Off CoD. This feature cannot be ordered if a temporary session is already active. Similarly, the CIU enablement feature cannot be removed if a temporary session is active. Any time the CIU enablement feature is removed, the On/Off CoD right-to-use is simultaneously removed. Reactivating the right-to-use feature subjects the client to the terms and fees that apply then.
Upgrade capability during On/Off CoD
Upgrades involving physical hardware are supported while an On/Off CoD upgrade is active on a particular z13. LICCC-only upgrades can be ordered and retrieved from Resource Link, and can be applied while an On/Off CoD upgrade is active. LICCC-only memory upgrades can be retrieved and applied while an On/Off CoD upgrade is active.
Repair capability during On/Off CoD
If the z13 requires service while an On/Off CoD upgrade is active, the repair can take place without affecting the temporary capacity.
Monitoring
When you activate an On/Off CoD upgrade, an indicator is set in vital product data. This indicator is part of the call-home data transmission, which is sent on a scheduled basis. A time stamp is placed into the call-home data when the facility is deactivated. At the end of each calendar month, the data is used to generate an invoice for the On/Off CoD that was used during that month.
Maintenance
The maintenance price is adjusted as a result of an On/Off CoD activation.
Software
Software Parallel Sysplex license charge (PSLC) clients are billed at the MSU level that is represented by the combined permanent and temporary capacity. All PSLC products are billed at the peak MSUs that are enabled during the month, regardless of usage. Clients with WLC licenses are billed by product at the highest four-hour rolling average for the month. In this instance, temporary capacity does not necessarily increase the software bill until that capacity is allocated to LPARs and used.
Results from the STSI instruction reflect the current permanent and temporary CPs. For more information, see “Store System Information instruction” on page 348.
8.5.6 z/OS capacity provisioning
The z13 provisioning capability that is combined with Capacity Provisioning Manager (CPM) functions in z/OS provides a flexible, automated process to control the activation of On/Off Capacity on Demand. The z/OS provisioning environment is shown in Figure 8-13.
Figure 8-13 The capacity provisioning infrastructure
The z/OS WLM manages the workload by goals and business importance on each z/OS system. WLM metrics are available through existing interfaces, and are reported through IBM Resource Measurement Facility™ (RMF) Monitor III, with one RMF gatherer for each z/OS system.
Sysplex-wide data aggregation and propagation occur in the RMF Distributed Data Server (DDS). The RMF Common Information Model (CIM) providers and associated CIM models publish the RMF Monitor III data.
The CPM, a function inside z/OS, retrieves critical metrics from one or more z/OS systems’ CIM structures and protocol. CPM communicates to local or remote SEs and HMCs through Simple Network Management Protocol (SNMP).
CPM has visibility of the resources in the individual offering records and the capacity tokens. When CPM activates resources, a check is run to determine whether enough capacity tokens remain for the specified resource to be activated for at least 24 hours. If insufficient tokens remain, no resource from the On/Off CoD record is activated.
If a capacity token is consumed during an activation that is driven by the CPM, the corresponding On/Off CoD record is deactivated prematurely by the system. This process occurs even if the CPM has activated this record, or parts of it. However, you do receive warning messages if capacity tokens are getting close to being fully consumed. You receive the messages five days before a capacity token is fully consumed. The five days are based on the assumption that the consumption is constant for the five days. You must put operational procedures in place to handle these situations. You can either deactivate the record manually, allow it happen automatically, or replenish the specified capacity token by using the Resource Link application.
The Capacity Provisioning Control Center (CPCC), which is on a workstation, provides an interface to administer capacity provisioning policies. The CPCC is not required for regular CPM operation. The CPCC will over time be moved into the z/OS Management Facility (z/OSMF). Parts of the CPCC are included in z/OSMF V1R13.
Capacity Provisioning Domain
The control over the provisioning infrastructure is managed by the CPM through the Capacity Provisioning Domain (CPD), which is controlled by the Capacity Provisioning Policy (CPP). The CPD is shown in Figure 8-14.
Figure 8-14 The Capacity Provisioning Domain
The CPD configuration defines the CPCs and z/OS systems that are controlled by an instance of the CPM. One or more CPCs, sysplexes, and z/OS systems can be defined into a domain. Sysplexes and CPCs do not have to be contained in a domain, but must not belong to more than one domain. Each domain has one active capacity provisioning policy. The CPCC is the CPM user interface component. Administrators work through this interface to define the domain configuration and provisioning policies. The CPCC is installed on a Microsoft Windows workstation.
CPM operates in four modes, allowing four different levels of automation:
•Manual mode:
Use this command-driven mode when no CPM policy is active.
•Analysis mode:
 – In analysis mode, CPM processes capacity-provisioning policies and informs the operator when a provisioning or deprovisioning action is required according to policy criteria.
 – In analysis mode, the operator determines whether to ignore the information or to manually upgrade or downgrade the system by using the HMC, the SE, or available CPM commands.
•Confirmation mode:
In this mode, CPM processes capacity provisioning policies and interrogates the installed temporary offering records. Every action that is proposed by the CPM must be confirmed by the operator.
•Autonomic mode:
This mode is similar to the confirmation mode, but no operator confirmation is required.
A number of reports are available in all modes that contain information about workload and provisioning status, and the rationale for provisioning guidelines. User interfaces are provided through the z/OS console and the CPCC application.
The provisioning policy defines the circumstances under which more capacity can be provisioned (when, which, and how). There are three elements in the criteria:
•A time condition is when provisioning is allowed:
 – Start time indicates when provisioning can begin
 – Deadline indicates that provisioning of more capacity is no longer allowed
 – End time indicates that deactivation of more capacity must begin
•A workload condition is which work qualifies for provisioning. It can have these parameters:
 – The z/OS systems that can run eligible work.
 – The importance filter indicates eligible service class periods, which are identified by WLM importance.
 – Performance Index (PI) criteria:
 • Activation threshold: PI of service class periods must exceed the activation threshold for a specified duration before the work is considered to be suffering.
 • Deactivation threshold: PI of service class periods must fall below the deactivation threshold for a specified duration before the work is considered to no longer be suffering.
 – Included service classes are eligible service class periods.
 – Excluded service classes are service class periods that must not be considered.
 
Tip: If no workload condition is specified, the full capacity that is described in the policy is activated and deactivated at the start and end times that are specified in the policy.
•Provisioning scope is how much more capacity can be activated, expressed in MSUs.
The number of zIIPs must be one specification per CPC that is part of the CPD. They are specified in MSUs.
The maximum provisioning scope is the maximum additional capacity that can be activated for all the rules in the CPD.
The provisioning rule is, in the specified time interval, that if the specified workload is behind its objective, up to the defined additional capacity can be activated.
The rules and conditions are named and stored in the Capacity Provisioning Policy.
For more information about z/OS Capacity Provisioning functions, see z/OS MVS Capacity Provisioning User’s Guide, SA33-8299.
Planning considerations for using automatic provisioning
Although only one On/Off CoD offering can be active at any one time, several On/Off CoD offerings can be present on the system. Changing from one to another requires stopping the active one before the inactive one can be activated. This operation decreases the current capacity during the change.
The provisioning management routines can interrogate the installed offerings, their content, and the status of the content of the offering. To avoid the decrease in capacity, create only one On/Off CoD offering on the system by specifying the maximum allowable capacity. The CPM can then, when an activation is needed, activate a subset of the contents of the offering sufficient to satisfy the demand. If more capacity is needed later, the Provisioning Manager can activate more capacity up to the maximum allowed increase.
Having an unlimited number of offering records pre-staged on the SE hard disk is possible. Changing the content of the offerings, if necessary, is also possible.
 
Remember: The CPM has control over capacity tokens for the On/Off CoD records. In a situation where a capacity token is consumed, the system deactivates the corresponding offering record. Therefore, you must prepare routines for catching the warning messages about capacity tokens being consumed, and have administrative routines in place for such a situation. The messages from the system begin five days before a capacity token is fully consumed. To avoid capacity records being deactivated in this situation, replenish the necessary capacity tokens before they are consumed.
The Capacity Provisioning Manager operates based on Workload Manager (WLM) indications, and the construct that is used is the PI of a service class period. It is important to select service class periods that are appropriate for the business application that needs more capacity. For example, the application in question might be running through several service class periods, where the first period is the important one. The application might be defined as importance level 2 or 3, but might depend on other work that is running with importance level 1. Therefore, it is important to consider which workloads to control and which service class periods to specify.
8.6 Capacity for Planned Event (CPE)
CPE is offered with the z13 to provide replacement backup capacity for planned downtime events. For example, if a server room requires an extension or repair work, replacement capacity can be installed temporarily on another z13 in the client’s environment.
 
Important: CPE is for planned replacement capacity only, and cannot be used for peak workload management.
CPE includes these feature codes:
•FC 6833: Capacity for Planned Event enablement
•FC 0116: 1 CPE Capacity Unit
•FC 0117: 100 CPE Capacity Unit
•FC 0118: 10000 CPE Capacity Unit
•FC 0119: 1 CPE Capacity Unit - IFL
•FC 0120: 100 CPE Capacity Unit - IFL
•FC 0121: 1 CPE Capacity Unit - ICF
•FC 0122: 100 CPE Capacity Unit - ICF
•FC 0125: 1 CPE Capacity Unit - zIIP
•FC 0126: 100 CPE Capacity Unit - zIIP
•FC 0127: 1 CPE Capacity Unit - SAP
•FC 0128: 100 CPE Capacity Unit - SAP
The feature codes are calculated automatically when the CPE offering is configured. Whether using the eConfig tool or the Resource Link, a target configuration must be ordered. The configuration consists of a model identifier, a number of speciality engines, or both. Based on the target configuration, a number of feature codes from the list are calculated automatically, and a CPE offering record is constructed.
CPE is intended to replace capacity that is lost within the enterprise because of a planned event, such as a facility upgrade or system relocation. CPE is intended for short duration events that last a maximum of three days. Each CPE record, after it is activated, gives you access to dormant PUs on the system for which you have a contract, as described by the feature codes. Processor units can be configured in any combination of CP or specialty engine types (zIIP, SAP, IFL, and ICF). At the time of CPE activation, the contracted configuration is activated. The general rule of two zIIPs for each configured CP is enforced for the contracted configuration.
The processors that can be activated by CPE come from the available unassigned PUs on any installed processor drawer. CPE features can be added to an existing z13 nondisruptively. A one-time fee is applied for each CPE event. This fee depends on the contracted configuration and its resulting feature codes. Only one CPE contract can be ordered at a time.
The base system configuration must have sufficient memory and channels to accommodate the potential requirements of the large CPE-configured system. Ensure that all required functions and resources are available on the system where CPE is activated. These functions and resources include CF LEVELs for coupling facility partitions, memory, and cryptographic functions, and include connectivity capabilities.
The CPE configuration is activated temporarily and provides more PUs in addition to the system’s original, permanent configuration. The number of additional PUs is predetermined by the number and type of feature codes that are configured, as described by the feature codes. The number of PUs that can be activated is limited by the unused capacity that is available on the system:
•A model N63 with 26 CPs, and no IFLs or ICFs has 37 unassigned PUs available.
•A model H96 with 38 CPs, one IFL, and one ICF has 56 unassigned PUs available.
When the planned event is over, the system must be returned to its original configuration. You can deactivate the CPE features at any time before the expiration date.
A CPE contract must be in place before the special code that enables this capability can be installed on the system. CPE features can be added to an existing z13 nondisruptively.
8.7 Capacity Backup (CBU)
CBU provides reserved emergency backup processor capacity for unplanned situations in which capacity is lost in another part of your enterprise. It allows you to recover by adding the reserved capacity on a designated z13.
CBU is the quick, temporary activation of PUs and is available in these options:
•For up to 90 contiguous days, if there is a loss of processing capacity as a result of an emergency or disaster recovery situation.
•For 10 days for testing your disaster recovery procedures or running the production workload. This option requires that an amount of z Systems workload capacity that is equivalent to the CBU upgrade capacity is shut down or otherwise made unusable during the CBU test.6
 
Important: CBU is for disaster and recovery purposes only, and cannot be used for peak workload management or for a planned event.
8.7.1 Ordering
The CBU process allows for CBU to activate CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs. To be able to use the CBU process, a CBU enablement feature (FC 9910) must be ordered and installed. You must order the quantity and type of PU that you require. Use the following feature codes:
•FC 6805: Additional test activations
•FC 6817: Total CBU years ordered
•FC 6818: CBU records ordered
•FC 6820: Single CBU CP-year
•FC 6821: 25 CBU CP-year
•FC 6822: Single CBU IFL-year
•FC 6823: 25 CBU IFL-year
•FC 6824: Single CBU ICF-year
•FC 6825: 25 CBU ICF-year
•FC 6826: Single CBU zAAP-year
•FC 6827: 25 CBU zAAP-year
•FC 6828: Single CBU zIIP-year
•FC 6829: 25 CBU zIIP-year
•FC 6830: Single CBU SAP-year
•FC 6831: 25 CBU SAP-year
•FC 6832: CBU replenishment
The CBU entitlement record (FC 6818) contains an expiration date that is established at the time of the order. This date depends on the quantity of CBU years (FC 6817). You can extend your CBU entitlements through the purchase of additional CBU years. The number of
FC 6817 per instance of FC 6818 remains limited to five. Fractional years are rounded up to the nearest whole integer when calculating this limit. If there are two years and eight months before the expiration date at the time of the order, the expiration date can be extended by no more than two years. One test activation is provided for each additional CBU year added to the CBU entitlement record.
FC 6805 allows for ordering more tests in increments of one. The total number of tests that is allowed is 15 for each FC 6818.
The processors that can be activated by CBU come from the available unassigned PUs on any installed processor drawer. The maximum number of CBU features that can be ordered is 101. The number of features that can be activated is limited by the number of unused PUs on the system:
•A model N30 with Capacity Model Identifier 410 can activate up to 40 CBU features: 20 to change the capacity setting of the existing CPs, and 20 to activate unused PUs.
•A model N63 with 15 CPs, four IFLs, and one ICF has 43 unused PUs available. It can activate up to 43 CBU features.
However, the ordering system allows for over-configuration in the order itself. You can order up to 141 CBU features regardless of the current configuration. However, at activation, only the capacity that is already installed can be activated. At activation, you can decide to activate only a subset of the CBU features that are ordered for the system.
Subcapacity makes a difference in the way that the CBU features are done. On the full-capacity models, the CBU features indicate the amount of additional capacity needed. If the amount of necessary CBU capacity is equal to four CPs, the CBU configuration is four CBU CPs.
The subcapacity models have multiple capacity settings of 4xx, 5yy, or 6yy. The standard models have the capacity setting 7nn. The number of CBU CPs must be equal to or greater than the number of CPs in the base configuration. All the CPs in the CBU configuration must have the same capacity setting. For example, if the base configuration is a 2-way 402, providing a CBU configuration of a 4-way of the same capacity setting requires two CBU feature codes. If the required CBU capacity changes the capacity setting of the CPs, going from model capacity identifier 402 to a CBU configuration of a 4-way 504 requires four CBU feature codes with a capacity setting of 5yy.
If the capacity setting of the CPs is changed, more CBU features are required, not more physical PUs. Therefore, your CBU contract requires more CBU features if the capacity setting of the CPs is changed.
CBU can add CPs through LICCC only, and the z13 must have the correct number of processor drawers that are installed to allow the required upgrade. CBU can change the model capacity identifier to a higher value than the base setting (4xx, 5yy, or 6yy), but does not change the system model. The CBU feature cannot decrease the capacity setting.
A CBU contract must be in place before the special code that enables this capability can be installed on the system. CBU features can be added to an existing z13 nondisruptively. For each system enabled for CBU, the authorization to use CBU is available for a 1 - 5 year period.
The alternative configuration is activated temporarily, and provides additional capacity greater than the system’s original, permanent configuration. At activation time, determine the capacity that you require for that situation. You can decide to activate only a subset of the capacity that is specified in the CBU contract.
The base system configuration must have sufficient memory and channels to accommodate the potential requirements of the large CBU target system. Ensure that all required functions and resources are available on the backup systems. These include CF LEVELs for coupling facility partitions, memory, and cryptographic functions, and connectivity capabilities.
When the emergency is over (or the CBU test is complete), the system must be returned to its original configuration. The CBU features can be deactivated at any time before the expiration date. Failure to deactivate the CBU feature before the expiration date can cause the system to downgrade resources gracefully to the original configuration. The system does not deactivate dedicated engines, or the last of in-use shared engines.
 
Planning: CBU for processors provides a concurrent upgrade. This upgrade can result in more enabled processors, changed capacity settings that are available to a system configuration, or both. You can activate a subset of the CBU features that are ordered for the system. Therefore, more planning and tasks are required for nondisruptive logical upgrades. For more information, see “Guidelines to avoid disruptive upgrades” on page 350.
For more information, see the z Systems Capacity on Demand User’s Guide, SC28-6846.
8.7.2 CBU activation and deactivation
The activation and deactivation of the CBU function is your responsibility and does not require the onsite presence of IBM SSRs. The CBU function is activated/deactivated concurrently from the HMC by using the API. On the SE, CBU is activated either by using the Perform Model Conversion task or through the API. The API enables task automation.
CBU activation
CBU is activated from the SE, by using the HMC and SSO (single object operation) to the SE, using the Perform Model Conversion task, or through automation by using the API on the SE or the HMC. During a real disaster, use the Activate CBU option to activate the 90-day period.
Image upgrades
After CBU activation, the z13 can have more capacity, more active PUs, or both. The additional resources go into the resource pools and are available to the LPARs. If the LPARs must increase their share of the resources, the LPAR weight can be changed or the number of logical processors can be concurrently increased by configuring reserved processors online. The operating system must be able to concurrently configure more processors online. If necessary, more LPARs can be created to use the newly added capacity.
CBU deactivation
To deactivate the CBU, the additional resources must be released from the LPARs by the operating systems. In some cases, this process is a matter of varying the resources offline. In other cases, it can mean shutting down operating systems or deactivating LPARs. After the resources are released, the same facility on the HMC/SE is used to turn off CBU. To deactivate CBU, select the Undo temporary upgrade option from the Perform Model Conversion task on the SE.
CBU testing
Test CBUs are provided as part of the CBU contract. CBU is activated from the SE by using the Perform Model Conversion task. Select the test option to initiate a 10-day test period. A standard contract allows one test per CBU year. However, you can order more tests in increments of one up to a maximum of 15 for each CBU order.
 
Tip: The CBU test activation is done the same way as the real activation, using the same SE Perform a Model Conversion panel and then selecting the Temporary upgrades option. The HMC panels are changed to avoid real CBU activations by setting the test activation as the default option.
The test CBU must be deactivated in the same way as the regular CBU. Failure to deactivate the CBU feature before the expiration date can cause the system to degrade gracefully back to its original configuration. The system does not deactivate dedicated engines, or the last of in-use shared engines.
CBU example
An example of a capacity backup operation is 12 CBU features that are installed on a backup model N63 with model capacity identifier 708. When a production model N30 with model capacity identifier 708 has an unplanned outage, the backup system can be temporarily upgraded from model capacity identifier 708 - 720. This process allows the capacity to take over the workload from the failed production system.
Furthermore, you can configure systems to back up each other. For example, if you use two models of H20 model capacity identifier 705 for the production environment, each can have five or more features installed. If one system suffers an outage, the other one uses a temporary upgrade to recover the approximate original total capacity.
8.7.3 Automatic CBU enablement for GDPS
The IBM Geographically Dispersed Parallel Sysplex (GDPS) CBU enables automatic management of the PUs that are provided by the CBU feature if there is a system or site failure. Upon detection of a site failure or planned disaster test, GDPS concurrently add CPs to the systems in the take-over site to restore processing power for mission-critical production workloads. GDPS automation runs the following tasks:
•Runs the analysis that is required to determine the scope of the failure. This process minimizes operator intervention and the potential for errors.
•Automates authentication and activation of the reserved CPs.
•Automatically restarts the critical applications after reserved CP activation.
•Reduces the outage time to restart critical workloads from several hours to minutes.
The GDPS service is for z/OS only, or for z/OS in combination with Linux on z Systems.
8.8 Nondisruptive upgrades
Continuous availability is an increasingly important requirement for most clients, and even planned outages are no longer acceptable. Although Parallel Sysplex clustering technology is the best continuous availability solution for z/OS environments, nondisruptive upgrades within a single system can avoid system outages and are suitable to more operating system environments.
The z13 allows concurrent upgrades, which mean that dynamically adding more capacity to the system is possible. If the operating system images running on the upgraded system do not require disruptive tasks to use the new capacity, the upgrade is also nondisruptive. This process type means that power-on reset (POR), LPAR deactivation, and IPL do not have to occur.
If the concurrent upgrade is intended to satisfy an image upgrade to an LPAR, the operating system that is running in this partition must be able to concurrently configure more capacity online. z/OS operating systems have this capability. z/VM can concurrently configure new processors and I/O devices online, and memory can be dynamically added to z/VM partitions.
If the concurrent upgrade is intended to satisfy the need for more operating system images, more LPARs can be created concurrently on the z13 system. These include all resources that are needed by such LPARs. These additional LPARs can be activated concurrently.
These enhanced configuration options are available through the separate HSA, which was introduced on the zEnterprise 196.
Linux operating systems, in general, cannot add more resources concurrently. However, Linux, and other types of virtual machines that run under z/VM, can benefit from the z/VM capability to nondisruptively configure more resources online (processors and I/O).
With z/VM, Linux guests can manipulate their logical processors by using the Linux CPU hotplug daemon. The daemon can start and stop logical processors that are based on the Linux average load value. The daemon is available in Linux SLES 10 SP2 and up, and in Red Hat Enterprise Linux (RHEL) V5R4 and up.
8.8.1 Components
The following components can be added, depending on the considerations that are described here.
Processors
CPs, ICFs, zIIPs, IFLs, and SAPs can be added concurrently to a z13 if unassigned PUs are available on any installed processor drawer. The number of zIIPs cannot exceed twice the number of CPs plus unassigned CPs. Additional processor drawers also can be installed concurrently, allowing further processor upgrades.
If necessary, more LPARs can be created concurrently to use the newly added processors.
The Coupling Facility Control Code (CFCC) also can configure more processors online to coupling facility LPARs by using the CFCC image operations window.
Memory
Memory can be added concurrently up to the physical installed memory limit. Additional processor drawers also can be installed concurrently, allowing further memory upgrades by LICCC, enabling memory capacity on the new processor drawers.
Using the previously defined reserved memory, z/OS operating system images, and z/VM partitions, you can configure dynamically more memory online. This process allows nondisruptive memory upgrades. Linux on z Systems supports Dynamic Storage Reconfiguration.
I/O
I/O features can be added concurrently if all the required infrastructure (I/O slots and PCIe Fanouts) is present on the configuration. PCIe I/O drawers can be added concurrently without planning if free space is available in one of the frames and the configuration permits.
Dynamic I/O configurations are supported by certain operating systems (z/OS and z/VM), allowing nondisruptive I/O upgrades. However, having dynamic I/O reconfiguration on a stand-alone coupling facility system is not possible because there is no operating system with this capability running on this system.
Cryptographic adapters
Crypto Express5S features can be added concurrently if all the required infrastructure is in the configuration.
Special features
Special features, such as Flash Express, zEnterprise Data Compression (zEDC) Express, and RoCE, also can be added concurrently if all infrastructure is available in the configuration.
8.8.2 Concurrent upgrade considerations
By using an MES upgrade, On/Off CoD, CBU, or CPE, a z13 can be upgraded concurrently from one model to another, either temporarily or permanently.
Enabling and using the additional processor capacity is transparent to most applications. However, certain programs depend on processor model-related information, such as independent software vendor (ISV) products. Consider the effect on the software that is running on a z13 when you perform any of these configuration upgrades.
Processor identification
Two instructions are used to obtain processor information:
•Store System Information (STSI) instruction
STSI reports the processor model and model capacity identifier for the base configuration, and for any additional configuration changes through temporary upgrade actions. It fully supports the concurrent upgrade functions, and is the preferred way to request processor information.
•Store CPU ID (STIDP) instruction
STIDP is provided for compatibility with an earlier version.
Store System Information instruction
Figure 8-15 shows the relevant output from the STSI instruction. The STSI instruction returns the model capacity identifier for the permanent configuration and the model capacity identifier for any temporary capacity. This data is key to the functioning of CoD offerings.
Figure 8-15 STSI output on z13
The model capacity identifier contains the base capacity, On/Off CoD, and CBU. The Model Permanent Capacity Identifier and the Model Permanent Capacity Rating contain the base capacity of the system. The Model Temporary Capacity Identifier and Model Temporary Capacity Rating contain the base capacity and On/Off CoD.
Store CPU ID (STIDP) instruction
The STIDP instruction provides information about the processor type, serial number, and LPAR identifier, as shown in Table 8-6. The LPAR identifier field is a full byte to support more than 15 LPARs.
Table 8-6 STIDP output for z13
Description
Version
code
CPU identification number
Machine type
number
Logical partition
2-digit indicator
Bit position
0 - 7
8 - 15
16 - 31
32 - 48
48 - 63
Value
x’00’1
LPAR ID2
4-digit number that is derived from the CPC serial number
x’2964’
x’8000’3
1 The version code for z13 is x00.
2 The LPAR identifier is a two-digit number in the range of 00 - 3F. It is assigned by the user on the image profile through the SE or HMC.
3 A high-order bit that is on indicates that the LPAR ID value returned in bits 8 - 15 is a two-digit value.
When issued from an operating system that is running as a guest under z/VM, the result depends on whether the SET CPUID command was used:
•Without the use of the SET CPUID command, bits 0 - 7 are set to FF by z/VM. However, the remaining bits are unchanged, which means that they are exactly as they were without running as a z/VM guest.
•If the SET CPUID command is issued, bits 0 - 7 are set to FF by z/VM and bits 8 - 31 are set to the value that is entered in the SET CPUID command. Bits 32 - 63 are the same as they were without running as a z/VM guest.
Table 8-7 lists the possible output that is returned to the issuing program for an operating system that runs as a guest under z/VM.
Table 8-7 z/VM guest STIDP output for z13
Description
Version
code
CPU identification number
Machine type
number
Logical partition
2-digit indicator
Bit position
0 - 7
8 - 15
16 - 31
32 - 48
48 - 63
Without
SET CPUID
command
x’FF’
LPAR ID
4-digit number that is derived from the CPC serial number
x’2964’
x’8000’
With
SET CPUID
command
x’FF’
6-digit number as entered by the command SET CPUID = nnnnnn
x’2964’
x’8000’
Planning for nondisruptive upgrades
Online permanent upgrades, On/Off CoD, CBU, and CPE can be used to upgrade concurrently a z13. However, certain situations require a disruptive task to enable capacity that was recently added to the system. Some of these situations can be avoided if planning is done in advance. Planning ahead is a key factor for nondisruptive upgrades.
The following list describes the main reasons for disruptive upgrades. However, by carefully planning and reviewing “Guidelines to avoid disruptive upgrades” on page 350, you can minimize the need for these outages.
•LPAR memory upgrades when reserved storage was not previously defined are disruptive to image upgrades. z/OS and z/VM support this function.
•Installation of an I/O cage is disruptive.
•An I/O upgrade when the operating system cannot use the dynamic I/O configuration function is disruptive to that partition. Linux, z/VSE, z/TPF, and CFCC do not support dynamic I/O configuration.
Guidelines to avoid disruptive upgrades
Based on reasons for disruptive upgrades (“Planning for nondisruptive upgrades” on page 350), here are guidelines for avoiding or at least minimizing these situations, increasing the chances for nondisruptive upgrades:
•Using an SE function that is called Logical Processor add, which is under Operational Customization tasks, CPs and zIIPs can be added concurrently to a running partition. The CP and zIIP, and initial or reserved number of processors can be changed dynamically.
•The operating system that runs in the targeted LPAR must support the dynamic addition of resources and be able to configure processors online. The total number of defined and reserved CPs cannot exceed the number of CPs that are supported by the operating system. z/OS V1R11, z/OS V1R12, and z/OS V1R13 with PTFs support up to 100 processors. z/OS V2R1 supports 141 PUs per z/OS LPAR in non-SMT mode and 128 PUs per z/OS LPAR in SMT mode. For both, the PU total is the sum of CPs and zIIPs. z/VM supports up to 32 processors.
•Configure reserved storage to LPARs.
Configuring reserved storage for all LPARs before their activation enables them to be nondisruptively upgraded. The operating system that is running in the LPAR must be able to configure memory online. The amount of reserved storage can be above the CPC drawer threshold limit, even if no other CPC drawer is already installed. With z13, the current partition storage limit is 4 TB for z/OS. z/VM still supports 1 TB memory partitions.
•Consider the flexible and plan-ahead memory options.
Use a convenient entry point for memory capacity, and select memory options that allow future upgrades within the memory cards that are installed on the CPC drawers. For more information about the offerings, see this information:
Considerations when installing additional CPC drawers
During an upgrade, more processor drawers can be installed concurrently. Depending on the number of additional processor drawers in the upgrade and your I/O configuration, a fanout rebalancing might be needed for availability reasons.
8.9 Summary of Capacity on Demand offerings
The CoD infrastructure and its offerings are major features that were introduced with the z13 system. These features are based on numerous client requirements for more flexibility, granularity, and better business control over the z Systems infrastructure, operationally and financially.
One major client requirement is to eliminate the need for a client authorization connection to the IBM Resource Link system when activating an offering. This requirement is being met by the z196, zEC12, and z13. After the offerings are installed on the z13, they can be activated at any time at the client’s discretion. No intervention by IBM or IBM personnel is necessary. In addition, the activation of the Capacity Backup does not require a password.
The z13 can have up to eight offerings that are installed at the same time, with the limitation that only one of them can be an On/Off CoD offering. The others can be any combination. The installed offerings can be activated fully or partially, and in any sequence and any combination. The offerings can be controlled manually through command interfaces on the HMC, or programmatically through a number of APIs. IBM applications, ISV programs, and client-written applications can control the usage of the offerings.
Resource consumption (and therefore financial exposure) can be controlled by using capacity tokens in the On/Off CoD offering records.
The CPM is an example of an application that uses the CoD APIs to provision On/Off CoD capacity based on the requirements of the workload. The CPM cannot control other offerings.
For more information about any of the topics that are described in this chapter, see the z Systems System Capacity on Demand User’s Guide, SC28-6943.
 
1 http://www.ibm.com/servers/resourcelink/. Registration is required to access Resource Link.
2 The z13 zero CP MCI is 400. This setting applies to an all-IFL system or an all-ICF system.
3 The 8-slot I/O drawer cannot be ordered as an MES on z13. They are available as carry-forward only.
4 z13 provides more improvements in the CBU activation panels. These panels have been improved to prevent inadvertent CBU activation.
5 Other adapter types, such as zFlash, zEOC, and Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE), also can be added to the PCIe I/O drawers through an MES.
6 All new CBU contract documents contain new CBU test terms to allow execution of production workload during CBU test. Existing CBU clients must run the IBM client Agreement Amendment for IBM z Systems Capacity Backup Upgrade Tests (US form #Z125-8145).
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset