Resource Optimized High Availability
This chapter describes one feature: Resource Optimized High Availability (ROHA). This feature is a new feature of PowerHA SystemMirror Standard and Enterprise Edition V7.2.
This chapter covers the following topics:
6.1 Resource Optimized High Availability concept and terminology
With this feature, PowerHA SystemMirror can manage dynamic LPAR (DLPAR) and Capacity of Demand (CoD) resources. CoD resources are composed of Enterprise Pool CoD (EPCoD) resources and On/Off CoD resources.
Enterprise Pool CoD resources
EPCoD resources are resources that can be freely moved among servers in the same pool where the resources are best used. Physical resources (such as CPU or memory) are not moved between servers; what is moved is the privilege to use them. You can grant this privilege to any server of the pool so that you can flexibly manage the pool of resources and acquire the resources where they are most needed.
On/Off CoD resource
On/Off CoD resources are preinstalled and inactive (and unpaid for) physical resources for a given server: Processors or memory capacity. On/Off CoD is a type of CoD license enabling temporary activation of processors and memory. PowerHA SystemMirror can dynamically activate these resources and can make them available to the system so that they are allocated when needed to the LPAR through a DLPAR operation.
Dynamic logical partitioning
DLPAR represents the facilities in some IBM Power Systems that provide the ability to logically attach and detach a managed system’s resources to and from an LPAR’s operating system without restarting.
By integrating with DLPAR and CoD resources, PowerHA SystemMirror ensures that each node can support the application with reasonable performance at a minimum cost. This way, you can tune the capacity of the logical partition flexibly when your application requires more resources, without having to pay for idle capacity until you need it (for On/Off CoD), or without keeping acquired resources if you do not use them (for Enterprise Pool CoD).
You can configure cluster resources so that the logical partition with minimally allocated resources serves as a standby node, and the application is on another LPAR node that has more resources than the standby node. This way, you do not use any additional resources that the frames have until the resources are required by the application.
PowerHA SystemMirror uses the system-connected HMC to perform DLPAR operation and manage CoD resources.
Table 6-1 displays all available types of the CoD offering. Only two of them are dynamically managed and controlled by PowerHA SystemMirror: EPCoD and On/Off CoD.
Table 6-1 CoD offering and PowerHA
CoD offering
PowerHA SystemMirror V6.1 Standard and Enterprise Edition
PowerHA SystemMirror V7.1 or V7.2 Standard and Enterprise Edition
Enterprise Pool
Memory and Processor
No
Yes, from Version 7.2
On/Off CoD (temporary)
Memory
No
Yes, from Version 7.1.3 SP2
On/Off CoD (temporary)
Processor
Yes
Yes
Utility CoD (temporary)
Memory and Processor
Utility CoD automatically is performed at the PHYP/System level. PowerHA cannot play a role in the same system.
Trial CoD
Memory and Processor
Trial CoD is used if available through DLPAR operation.
CUoD (permanent)
Memory & Processor
CUoD are used if available through DLPAR operation.
PowerHA does not handle this kind of resource directly.
Trial CoD
Trial CoD are temporary resources, but they are not set to On or Off to follow dynamic needs. When Trial CoD standard or exception code is entered into the HMC, these resources are On immediately, and elapsed time starts immediately. The amount of resources that is granted by Trial CoD directly enters the available DLPAR resources. It is as though these were configured as DLPAR resources.
Therefore, PowerHA SystemMirror can dynamically control the Trial CoD resource after customer manually enters a code to activate the resource through HMC.
6.1.1 Environment requirement for Resource Optimized High Availability
Here are the requirements to implement ROHA:
•PowerHA SystemMirror V7.2 Standard Edition or Enterprise Edition
•AIX 6.1 TL09 SP5, or AIX 7.1 TL03 SP5, or AIX 7.1 TL4 or AIX 7.2 or later
•HMC requirements:
 – To use the Enterprise Pool CoD license, your system must be using HMC 7.8 firmware or later.
 – Configure the backup HMC for EPCoD with high availability.
 – For the EPCoD User Interface (UI) in HMC, the HMC must have a minimum of 2 GB of memory.
•Hardware requirements for using Enterprise Pool CoD license:
 – POWER7+: 9117-MMD (770 D model), 9179-MHD (780 D model), that uses FW780.10 or later.
 – POWER8: 9119-MME (E870), 9119-MHE (E880), that uses FW820 or later.
6.2 New PowerHA SystemMirror SMIT configuration panels for Resource Optimized High Availability
To support the ROHA feature, PowerHA SystemMirror provides some new SMIT menu and clmgr command options. These options include the following functions:
•HMC configuration
•Hardware Resource Provisioning for Application Controller
•Cluster tunables configuration
Figure 6-1 shows a summary of the SMIT menu navigation for all new ROHA panels. For the new options of clmgr command, see 6.14.1, “The clmgr interface to manage Resource Optimized High Availability” on page 233.
Figure 6-1 New Resource Optimized High Availability panels
6.2.1 Entry point to Resource Optimized High Availability
Start smit sysmirror and select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors). This panel is a menu window with a title menu option and four item menu options. Only the third item is the entry point to ROHA configuration (Figure 6-2).
           Configure User Applications (Scripts and Monitors)
 
Move cursor to desired item and press Enter.
 
Application Controller Scripts
Application Monitors
Resource Optimized High Availability
Show Cluster Applications
Figure 6-2 Entry point to the Resource Optimized High Availability menu
Table 6-2 shows the context-sensitive help for the ROHA entry point.
Table 6-2 Context-sensitive help for the Resource Optimized High Availability entry point
Name and fast path
Context-sensitive help (F1)
Resource Optimized High Availability
# smitty cm_cfg_roha
Choose this option to configure ROHA. ROHA performs dynamic management of hardware resources (memory and CPU) for the account of PowerHA SystemMirror. This dynamic management of resources uses three types of mechanism: DLPAR, On/Off CoD, and Enterprise Pool CoD. If the resources that are available on the central electronic complex are not sufficient, and cannot be obtained through a DLPAR operation, it is possible to fetch external pools of resources that are provided by CoD: Either On/Off or Enterprise Pool. On/Off CoD can result in extra costs, and a formal agreement from the user is required. The user must configure Hardware Management Consoles (HMC) to for acquisition/release of resources.
6.2.2 Resource Optimized High Availability panel
Start smit sysmirror and select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability. The next panel is a menu window with a title menu option and three item menu options. Its fast path is cm_cfg_roha (Figure 6-3).
            Resource Optimized High Availability
 
Move cursor to desired item and press Enter.
 
HMC Configuration
Hardware Resource Provisioning for Application Controller
Change/Show Default Cluster Tunables
Figure 6-3 Resource Optimized High Availability panel
Table 6-3 shows the help information for the ROHA panel.
Table 6-3 Context-sensitive help for the Resource Optimized High Availability panel
Name and fast path
Context-sensitive help (F1)
HMC Configuration
# smitty cm_cfg_hmc
This option configures the HMC that is used by your cluster configuration, and to optionally associate the HMC to your cluster’s nodes. If no HMC is associated with a node, PowerHA SystemMirror uses the default cluster configuration.
Change/Show Hardware Resource Provisioning for Application Controller
# smitty cm_cfg_hr_prov
This option changes or shows CPU and memory resource requirements for any Application Controller that runs in a cluster that uses DLPAR, CoD, or Enterprise Pool CoD capable nodes, or a combination.
Change/Show Default Cluster Tunables
# smitty cm_cfg_def_cl_tun
This option modifies or views the DLPAR, CoD, and Enterprise Pool CoD configuration parameters.
6.2.3 HMC configuration
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → HMC Configuration. The next panel is a menu window with a title menu option and seven item menu options. Its fast path is cm_cfg_hmc (Figure 6-4).
            HMC Configuration
 
Move cursor to desired item and press Enter.
 
Add HMC Definition
Change/Show HMC Definition
Remove HMC Definition
Change/Show HMC List for a Node
Change/Show HMC List for a Site
Change/Show Default HMC Tunables
Change/Show Default HMC List
Figure 6-4 HMC configuration menu
Table 6-4 shows the help information for the HMC configuration.
Table 6-4 Context-sensitive help for the HMC configuration
Name and fast path
Context-sensitive help (F1)
Add HMC Definition
# smitty cm_cfg_add_hmc
Choose this option to add an HMC and its communication parameters, and add this new HMC to the default list. All the nodes of the cluster use by default these HMC definitions to perform DLPAR operations, unless you associate a particular HMC to a node.
Change/Show HMC Definition
# smitty cm_cfg_ch_hmc
Choose this option to modify or view an HMC host name and communication parameters.
Remove HMC Definition
# smitty cm_cfg_rm_hmc
Choose this option to remove an HMC, and then remove it from the default list.
Change/Show HMC List for a Node
# smitty cm_cfg_hmcs_node
Choose this option to modify or view the list of an HMC of a node.
Change/Show HMC List for a Site
# smitty cm_cfg_hmcs_site
Choose this option to modify or view the list of an HMC of a site.
Change/Show Default HMC Tunables
# smitty cm_cfg_def_hmc_tun
Choose this option to modify or view the HMC default communication tunables.
Change/Show Default HMC List
# smitty cm_cfg_def_hmcs
Choose this option to modify or view the default HMC list that is used by default by all nodes of the cluster. Nodes that define their own HMC list do not use this default HMC list.
HMC Add/Change/Remove Definition
 
Note: Before you add HMC, you must build password-less communication from AIX nodes to the HMC. For more information, see 6.4.1, “Consideration before Resource Optimized High Availability configuration” on page 159.
To add HMC, select Add HMC Definition. The next panel is a dialog window with a title dialog header and several dialog command options. Its fast path is cm_cfg_add_hmc. Each item has a context-sensitive help window that you access by pressing F1, and can have an associated list (press F4).
Figure 6-5 shows the menu to add the HMC definition and its entry fields.
            Add HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name [] +
DLPAR operations timeout (in minutes) [] #
Number of retries [] #
Delay between retries (in seconds) [] #
Nodes [] +
Sites [] +
Check connectivity between HMC and nodes Yes
Figure 6-5 Add HMC Definition menu
Table 6-5 shows the help and information list for adding the HMC definition.
Table 6-5 Context-sensitive help and associated list for Add HMC Definition menu
Name
Context-sensitive help (F1)
Associated list (F4)
HMC name
Enter the host name for the HMC. An IP address is also accepted here. Both IPv4 and IPv6 addresses are supported.
Yes (single-selection).
The list is obtained by running the following command: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc -a IP
DLPAR operations timeout (in minutes)
Enter a timeout in minutes by using DLPAR commands run on an HMC (-w parameter). This -w parameter exists only on the chhwres command when allocating or releasing resources. It is adjusted according to the type of resources (for memory, 1 minute per gigabyte is added to this timeout). Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Number of retries
Enter a number of times one HMC command is retried before the HMC is considered as non-responding. The next HMC in the list is used after this number of retries fails. Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Delay between retries (in seconds)
Enter a delay in seconds between two successive retries. Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Nodes
Enter the list of nodes that use this HMC.
Yes (multiple-selection).
A list of nodes to be proposed can be obtained by running the following command:
odmget HACMPnode
Sites
Enter the sites that use this HMC. All nodes of the sites then use this HMC by default, unless the node defines an HMC as its own level.
Yes (multiple-selection).
A list of sites to be proposed can be obtained by running the following command:
odmget HACMPsite
Check connectivity between the HMC and nodes
Select Yes to check communication links between nodes and HMC.
<Yes>|<No>. The default is Yes.
If Domain Name Service (DNS) is configured in your environment and DNS can do resolution for HMC IP and host name, then you can use F4 to select one HMC to perform the add operation.
Figure 6-6 shows an example of selecting one HMC from the list to perform the add operation.
           HMC name
Move cursor to desired item and press Enter.
e16hmc1 is 9.3.207.130
e16hmc3 is 9.3.207.133
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-6 Select one HMC from the HMC list to perform an add HMC operation
PowerHA SystemMirror also supports entering the HMC IP address to add the HMC. Figure 6-7 shows an example of entering one HMC IP address to add the HMC.
           Add HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name [9.3.207.130] +
DLPAR operations timeout (in minutes) []
Number of retries []
Delay between retries (in seconds) []
Nodes []+
Sites []+
Check connectivity between HMC and nodes Yes
Figure 6-7 Enter one HMC IP address to add an HMC
Change/Show HMC Definition
To show or modify an HMC, select Change/Show HMC Definition. The next panel is a selector window with a selector header that lists all existing HMC names. Its fast path is cm_cfg_ch_hmc (Figure 6-8).
| HMC name
|
| Move cursor to desired item and press Enter.
|
|    e16hmc1
|    e16hmc3
|
| F1=Help F2=Refresh F3=Cancel
| Esc+8=Image Esc+0=Exit Enter=Do
| /=Find n=Find Next
Figure 6-8 Select an HMC from a list during a change or show HMC configuration
Press Enter on an existing HMC to modify it. The next panel is the one that is presented in Figure 6-9. You cannot change the name of the HMC.
           Change/Show HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name                                           e16hmc1
DLPAR operations timeout (in minutes) [5] #
Number of retries [3] #
Delay between retries (in seconds) [10] #
Nodes                             [ITSO_rar1m3_Node1 ITSO_r1r9m1_Node1] +
Sites [] +
Check connectivity between HMC and nodes Yes
Figure 6-9 Change/Show HMC Definition of SMIT menu
Remove HMC Definition
To delete an HMC, select Remove HMC Definition. The panel that is shown in Figure 6-10 is the same selector window. Press Enter on an existing HMC name to remove it. Its fast path is cm_cfg_rm_hmc.
            HMC name
 
Move cursor to desired item and press Enter.
 
e16hmc1
e16hmc3
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-10 Select one HMC to remove
Figure 6-11 shows the removed HMC definition.
           Remove HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name                                                 e16hmc1
 
Figure 6-11 Remove one HMC
Change/Show HMC List for a Node
To show or modify the HMC list for a node, select Change/Show HMC List for a Node. The next panel (Figure 6-12) is a selector window with a selector header that lists all existing nodes. Its fast path is cm_cfg_hmcs_node.
                Select a Node
 
Move cursor to desired item and press Enter.
 
ITSO_rar1m3_Node1
ITSO_r1r9m1_Node1
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next Esc+8=Image
Figure 6-12 Select a node to change
Press Enter on an existing node to modify it. The next panel (Figure 6-13) is a dialog window with a title dialog header and two dialog command options.
You cannot add or remove an HMC from this list. You can only reorder (set in the correct precedence order) the HMCs that are used by the node.
            Change/Show HMC List for a Node
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Node name ITSO_rar1m3_Node1
HMC list [e16hmc1 e16hmc3]
Figure 6-13 Change/Show HMC list for a Node
Table 6-6 shows the help information to change or show the HMC list for a node.
Table 6-6 Context-sensitive help for Change or Show HMC list for a Node
Name and fast path
Context-sensitive help (F1)
Node name
This is the node name to associate with one or more HMCs.
HMC list
The precedence order of the HMCs that are used by this node. The first in the list is tried first, then the second, and so on. You cannot add or remove any HMC. You can modify only the order of the already set HMCs.
Change/Show HMC List for a Site
To show or modify the HMC list for a node, select Change/Show HMC List for a Site. The next panel (Figure 6-14) is a selector window with a selector header that lists all existing sites. Its fast path is cm_cfg_hmcs_site.
                      Select a Site
 
Move cursor to desired item and press Enter.
 
site1
site2
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-14 Select a Site menu for Change/Show HMC List for as Site
Press Enter on an existing site to modify it. The next panel (Figure 6-15) is a dialog window with a title dialog header and two dialog command options.
          Change/Show HMC List for a Site
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Site Name site1
HMC list [e16hmc1 e16hmc3]
Figure 6-15 Change/Show HMC List for a Site menu
You cannot add or remove an HMC from the list. You can only reorder (set in the correct precedence order) the HMCs used by the site. See Table 6-7.
Table 6-7 Site and HMC usage list
Name and fast path
Context-sensitive help (F1)
Site name
This is the site name to associate with one or more HMCs.
HMC list
The precedence order of the HMCs that are used by this site. The first in the list is tried first, then the second, and so on. You cannot add or remove any HMC. You can modify only the order of the already set HMCs.
Change/Show Default HMC Tunables
To show or modify default HMC communication tunables, select Change/Show Default HMC Tunables. The next panel (Figure 6-16) is a dialog window with a title dialog header and three dialog command options. Its fast path is cm_cfg_def_hmc_tun. Each item has a context-sensitive help window (press F1) and can have an associated list (press F4).
            Change/Show Default HMC Tunables
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                                 [Entry Fields]
DLPAR operations timeout (in minutes) [10] #
Number of retries [5] #
Delay between retries (in seconds) [10]
Figure 6-16 Change/Show Default HMC Tunables menu
Change/Show Default HMC List
To show or modify the default HMC list, select Change/Show Default HMC List. The next panel (Figure 6-17) is a dialog window with a title dialog header and one dialog command option. Its fast path is cm_cfg_def_hmcs. Each item has a context-sensitive help window (press F1) and can have an associated list (press F4).
            Change/Show Default HMC List
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
HMC list [e16hmc1 e16hmc3]
Figure 6-17 Change/Show Default HMC list menu
6.2.4 Hardware resource provisioning for application controller
To provision hardware, complete the following steps:
1. Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → Hardware Resource Provisioning for Application Controller. The next panel (Figure 6-18) is a menu window with a title menu option and three item menu options.
            Hardware Resource Provisioning for Application Controller
 
Move cursor to desired item and press Enter.
 
Add Hardware Resource Provisioning to an Application Controller
Change/Show Hardware Resource Provisioning of an Application Controller
Remove Hardware Resource Provisioning from an Application Controller
Figure 6-18 Hardware Resource Provisioning for Application Controller menu
2. Choose one of the following actions:
 – To add an application controller configuration, select Add.
 – To change or show an application controller configuration, select Change/Show.
 – To remove an application controller configuration, select Remove.
In case you choose Add or Change/Show, the On/Off CoD Agreement is displayed, as shown in Figure 6-19. However, this is displayed only if the user has not yet agreed to it. If the user has already agreed to it, it is not displayed.
On/Off CoD Agreement
Figure 6-19 is a dialog panel with a dialog header and one dialog command option.
            On/Off CoD Agreement
Type or select a value for the entry field.
Press Enter AFTER making all desired changes.
[Entry Fields]
Resources Optimized High Availability management No +
can take advantage of On/Off CoD resources.
On/Off CoD use would incur additional costs.
Do you agree to use On/Off CoD and be billed
for extra costs?
Figure 6-19 On/Off CoD Agreement menu
To accept the On/Off CoD Agreement, complete the following steps:
1. Enter Yes to have PowerHA SystemMirror use On/Off Capacity On Demand (On/Off CoD) resources to perform DLPAR operations on your nodes.
2. If you agree to use On/Off CoD, you must ensure that you have entered the On/Off CoD activation code. The On/Off CoD license key must be entered into HMC before PowerHA SystemMirror can activate this type of resources.
3. In the following cases, keep the default value:
 – If there is only half Enterprise Pool CoD, keep the default value of No.
 – If there is not Enterprise Pool CoD or On/Off CoD, PowerHA manages only the server’s permanent resources through DLPAR, so also keep the default value.
This option can be modified later in the Change/Show Default Cluster Tunables panel, as shown in Figure 6-22 on page 156.
Add Hardware Resource Provisioning to an Application Controller
The panel that is shown in Figure 6-20 is a selector window with a selector header that lists all existing application controllers.
           Select Application Controller
 
Move cursor to desired item and press Enter.
 
App1
   App2
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-20 Select Application Controller menu
To create a Hardware Resource Provisioning for an Application Controller, the list displays only application controllers that do not already have hardware resource provisioning, as shown in Figure 6-21.
To modify or remove a Hardware Resource Provisioning for an Application Controller, the list displays application controllers that already have hardware resource provisioning.
Press Enter on an existing application controller to modify it. The next panel is a dialog window with a title dialog header and three dialog command options. Each item has a context-sensitive help window (press F1) and can have an associated list (press F4).
           Add Hardware Resource Provisioning to an Application Controller
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Application Controller Name App1
Use desired level from the LPAR profile No +
Optimal number of gigabytes of memory []
Optimal number of dedicated processors [] #
 
Optimal number of processing units []
Optimal number of virtual processors []
Figure 6-21 Add Hardware Resource Provisioning to an Application Controller menu
Table 6-8 shows the help for adding hardware resources.
Table 6-8 Context-sensitive help for add hardware resource provisioning
Name and fast path
Context-sensitive help (F1)
Application Controller Name
This is the application controller for which you configure DLPAR and CoD resource provisioning.
Use desired level from the LPAR profile
There is no default value. You must make one of the following choices:
•Enter Yes if you want the LPAR hosting your node to reach only the level of resources that is indicated by the desired level of the LPAR’s profile. By choosing Yes, you trust the desired level of LPAR profile to fit the needs of your application controller.
•Enter No if you prefer to enter exact optimal values for memory, processor (CPU), or both. These optimal values match the needs of your application controller, and enable you to better control the level of resources that are allocated to your application controller.
•Enter nothing if you do not need to provision any resource for your application controller.
For all application controllers having this tunable set to Yes, the allocation that is performed lets the LPAR reach the LPAR desired value of the profile.
Suppose that you have a mixed configuration, in which some application controllers have this tunable set to Yes, and other application controllers have this tunable set to No with some optimal level of resources specified. In this case, the allocation that is performed lets the LPAR reach the desired value of the profile that is added to the optimal values.
Optimal number of gigabytes of memory
Enter the amount of memory that PowerHA SystemMirror attempts to acquire for the node before starting this application controller.
This Optimal number of gigabytes of memory value can be set only if the Used desired level from the LPAR profile value is set to No.
Enter the value in multiples of ¼, ½, ¾, or 1 GB. For example, 1 represents 1 GB or 1024 MB, 1.25 represents 1.25 GB or 1280 MB, 1.50 represents 1.50 GB or 1536 MB, and 1.75 represents 1.75 GB or 1792 MB.
If this amount of memory is not satisfied, PowerHA SystemMirror takes resource group (RG) recovery actions to move the RG with this application to another node. Alternatively, PowerHA SystemMirror can allocate less memory depending on the Start RG even if resources are insufficient cluster tunable.
Optimal number of dedicated processors
Enter the number of processors that PowerHA SystemMirror attempts to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Dedicated Processing Mode.
This Optimal number of dedicated processors value can be set only if the Used desired level from the LPAR profile value is set to No.
If this number of CPUs is not satisfied, PowerHA SystemMirror takes RG recovery actions to move the RG with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
For more information about how to acquire mobile resources at the RG onlining stage, see 6.6, “Introduction to resource acquisition” on page 171.
For more information about how to release mobile resources at the RG offlining stage, see 6.7, “Introduction to release of resources” on page 181.
Optimal number of processing units
Enter the number of processing units that PowerHA SystemMirror attempts to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Shared Processing Mode.
This Optimal number of processing units value can be set only if the Used desired level from the LPAR profile value is set to No.
Processing units are specified as a decimal number with two decimal places, 0.01 - 255.99.
This value is used only on nodes that support allocation of processing units.
If this number of CPUs is not satisfied, PowerHA SystemMirror takes RG recovery actions to move the RG with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
For more information about how to acquire mobile resources at the RG onlining stage, see 6.6, “Introduction to resource acquisition” on page 171.
For more information about how to release mobile resources at the RG offlining stage, see 6.7, “Introduction to release of resources” on page 181.
Optimal number of virtual processors
Enter the number of virtual processors that PowerHA SystemMirror attempts to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Shared Processing Mode.
This Optimal number of dedicated or virtual processors value can be set only if the Used desired level from the LPAR profile value is set to No.
If this number of virtual processors is not satisfied, PowerHA SystemMirror takes RG recovery actions to move the RG with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
To modify an application controller configuration, select Change/Show. The next panel is the same selector window, as shown in Figure 6-21 on page 153. Press Enter on an existing application controller to modify it. The next panel is the same dialog window shown in Figure 6-21 on page 153 (except the title, which is different).
To delete an application controller configuration, select Remove. The next panel is the same selector window that was shown previously. Press Enter on an existing application controller to remove it.
If Use desired level from the LPAR profile is set to No, then at least the memory (Optimal number of gigabytes of memory) or CPU (Optimal number of dedicated or virtual processors) setting is mandatory.
6.2.5 Change/Show Default Cluster Tunable
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → Change/Show Default Cluster Tunables. The next panel (Figure 6-22) is a dialog window with a title dialog header and seven dialog command options. Each item has a context-sensitive help window (press F1) and can have an associated list (press F4). Its fast path is cm_cfg_def_cl_tun.
            Change/Show Default Cluster Tunables
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Dynamic LPAR
Always start Resource Groups                       Yes +
   Adjust Shared Processor Pool size if required No +
Force synchronous release of DLPAR resources No +
On/Off CoD
I agree to use On/Off CoD and be billed for Yes +
extra costs
Number of activating days for On/Off CoD requests [30]
Figure 6-22 Change/Show Default Cluster Tunables menu
Table 6-9 shows the help for the cluster tunables.
Table 6-9 Context-sensitive help for Change/Show Default Cluster Tunables
Name and fast path
Context-sensitive help (F1)
Always start Resource Groups
Enter Yes to have PowerHA SystemMirror start RGs even if there is any error in ROHA resources activation. This can occur when the total requested resources exceed the LPAR profile’s maximum or the combined available resources, or if there is a total loss of HMC connectivity. Thus, the best-can-do allocation is performed.
Enter No to prevent starting Resources Groups if any error during ROHA resources acquisition.
The default is Yes.
Adjust Shared Processor Pool size if required
Enter Yes to authorize PowerHA SystemMirror to dynamically change the user-defined Shared-Processors Pool boundaries, if necessary. This change can occur only at takeover, and only if CoD resources are activated for the central electronic complex so that changing the maximum size of a particular Shared-Processors Pool is not done to the detriment of other Shared-Processors Pools.
The default is No.
Force synchronous release of DLPAR resources
Enter Yes to have PowerHA SystemMirror release CPU and memory resources synchronously. For example, if the client must free resources on one side before they can be used on the other side. By default, PowerHA SystemMirror automatically detects the resource release mode by looking at whether Active and Backup nodes are on the same or different CECs. A leading practice is to have asynchronous release in order to not delay the takeover.
The default is No.
I agree to use On/Off CoD and be billed for extra costs
Enter Yes to have PowerHA SystemMirror use On/Off Capacity On Demand (On/Off CoD) to obtain enough resources to fulfill the optimal amount that is requested. Using On/Off CoD requires an activation code to be entered on the HMC and can result in extra costs due to the usage of the On/Off CoD license.
The default is No.
Number of activating days for On/Off CoD requests
Enter a number of activating days for On/Off CoD requests. If the requested available resources are insufficient for this duration, then the longest-can-do allocation is performed. Try to allocate the amount of resources that is requested for the longest duration. To do that, consider the overall resources that are available. This number is the sum of the On/Off CoD resources that are already activated but not yet used, and the On/Off CoD resources not yet activated.
The default is 30.
6.3 New PowerHA SystemMirror verification enhancement for Resource Optimized High Availability
The ROHA function allows PowerHA SystemMirror to automatically or manually check for environment discrepancies. The clverify tool is improved to check ROHA-related configuration integrity.
Customers can use the verification tool to ensure that their environment is correct regarding their ROHA setup. Discrepancies are called out by PowerHA SystemMirror, and the tool assists customers to correct the configuration if possible.
The results appear in the following files:
•The /var/hacmp/log/clverify.log file
•The /var/hacmp/log/autoverify.log file
The user is actively notified of critical errors. A distinction can be made between errors that are raised during configuration and errors that are raised during cluster synchronization.
As a general principal, any problems that are detected at configuration time are presented as warnings instead of errors.
Another general principle is that PowerHA SystemMirror checks only what is being configured at configuration time and not the whole configuration. PowerHA SystemMirror checks the whole configuration at verification time.
For example, when adding an HMC, you check only the new HMC (verify that it is pingable, at an appropriate software level, and so on) and not all of the HMCs. Checking the whole configuration can take some time and is done at verify and sync time rather than each individual configuration step.
General verification
Table 6-10 shows the general verification list.
Table 6-10 General verification list
Item
Configuration time
Synchronization
time
Check that all RG active and standby nodes are on different CECs. This enables the asynchronous mode of releasing resources.
Info
Warning
This code cannot run on an IBM POWER4.
Error
Error
HMC communication verification
Table 6-11 shows the HMC communication verification list.
Table 6-11 HMC communication verification list
Item
Configuration time
Synchronization
time
Only one HMC is configured per node.
None
Warning
Two HMCs are configured per node.
None
OK
One node is without HMC (if ROHA only).
None
Error
Only one HMC per node can be pinged.
Warning
Warning
Two HMCs per node can be pinged.
OK
OK
One node has a non-pingable HMC.
Warning
Error
Only one HMC with password-less SSH communication exists per node.
Warning
Warning
Two HMCs with password-less SSH communication exist per node.
OK
OK
One node exists with non-SSH accessible HMC.
Warning
Error
Check that all HMCs share the same level (the same version of HMC).
Warning
Warning
Check that all HMCs administer the central electronic complex hosting the current node. Configure two HMCs administering the central electronic complex hosting the current node. If not, PowerHA gives a warning message.
Warning
Warning
Check whether the HMC level supports FSP Lock Queuing.
Info
Info
Capacity on Demand verification
Table 6-12 shows the CoD verification.
Table 6-12 Capacity on Demand verification
Item
Configuration Time
Synchronization
Time
Check that all CECs are CoD capable.
Info
Warning
Check whether CoD is enabled.
Info
Warning
Power enterprise pool verification
Table 6-13 shows the enterprise pool verification list.
Table 6-13 Power enterprise pool verification
Item
@info
@Sync
Check that all CECs are Enterprise Pool capable.
Info
Info
Determine which HMC is the master, and which HMC is the non-master.
Info
Info
Check that the nodes of the cluster are on different pools. This enables the asynchronous mode of releasing resources.
Info
Info
Check that all HMCs are at level 7.8 or later.
Info
Warning
Check that the central electronic complex has unlicensed resources.
Info
Warning
Resource provisioning verification
Table 6-14 shows the resource provisioning verification information.
Table 6-14 Resource provisioning verification
Item
@info
@Sync
Check that for one given node the total of optimal memory (of RG on this node) that is added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that for one given node the total of optimal CPU (of RG on this node) that is added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that for one given node the total of optimal PU (of RG on this node) that is added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that the total processing units do not break the minimum processing units per virtual processor ratio.
Error
Error
6.4 Planning for one Resource Optimized High Availability cluster environment
Before completing the ROHA configuration, read the following considerations.
6.4.1 Consideration before Resource Optimized High Availability configuration
This section describes a few considerations to know before a ROHA configuration.
Tips for an Enterprise Pool license
If you have ordered IBM Power Systems Enterprise Pool license for your servers, and you want to use the resource with your PowerHA SystemMirror cluster, then you must create the Enterprise Pool manually.
Before you create the Enterprise Pool, get the configuration extensible markup language (XML) file from the IBM CoD, and the deactivation code from the IBM CoD project office at this IBM website.
The configuration XML file is used to enable and generate mobile resources.
The deactivation code is used to deactivate some of the permanent resources to inactive mode. The number is the same independent of how many mobile resources are on the server’s order.
For example, in one order, there are two Power Systems servers. Each one has 16 static CPUs, eight mobile CPUs, and eight inactive CPUs, for a total of 32 CPUs. When you power them on the first time, you see that each server has 24 permanent CPUs, 16 static CPUs, plus 8 mobile CPUs.
After you create the Enterprise Pool with the XML configuration file, you see that there are 16 mobile CPUs that are generated in the Enterprise Pool, but the previous eight mobile CPUs are still in permanent status in each server. This results in the server’s status being different from its original order. This brings some issues in future post-sales activities.
There are two steps to complete the Enterprise Pool implementation:
1. Create the Enterprise Pool with the XML configuration file.
2. Deactivate some permanent resources (the number is the same as mobile resources) to inactive with the deactivation code.
After you finish these two steps, each server has 16 static CPUs and 16 inactive CPUs, and the Enterprise Pool has 16 mobile CPUs. Then, the mobile CPUs can be assigned to each of the two servers through the HMC GUI or the command-line interface.
 
Note: These two steps will be combined into one step in the future. At the time of writing, you must perform each step separately.
How to get the deactivation code and use it
The following steps explain how to get the deactivation code and how to use it:
1. Send an email to the IBM CoD project office ([email protected]). You need to provide the following information or attach the servers order:
 – Customer name
 – Each server’s system type and serial number
 – Configuration XML file
2. In reply to this note, you receive from the CoD project office a de-activation code for the servers. The de-activation code lowers the number of activated resources to align it with your server order.
 
Note: This de-activation code updates the IBM CoD website after you receive the note. This de-activation code has RPROC and RMEM. RPROC is for reducing processor resources, and RMEM is for reducing memory resources.
3. Enter this de-activation code in the corresponding servers through the HMC, as shown in Figure 6-23 (shows the menu for Enter CoD Code).
Figure 6-23 Menu for Enter CoD Code
4. After entering the de-activation code, you must send a listing of the updated Vital Product Data (VPD) output to the CoD Project office at [email protected].
Collect the VPD by using the HMC command line, as shown in Example 6-1.
Example 6-1 Collecting the VPD information case
Collect the VPD using the HMC command line instruction for every server:
Processor:lscod -m your_system_name -t code -r proc -c mobile
Memory:lscod -m your_system_name -t code -r mem -c mobile
5. With the receipt of the lscod profile, the Project Office updates the CoD database records and closes out your request.
For more information about how to use the configuration XML file to create Power Enterprise Pool and some management concept, see Power Enterprise Pools on IBM Power Systems, REDP-5101.
Configuring redundant HMCs or add EP’s master and backup HMC
Section 6.12, “Hardware Management Console high availability introduction” on page 222 introduces HMC high availability design in PowerHA SystemMirror. For the ROHA solution, the HMC is critical, so configuring redundant HMCs is advised.
If there is a Power Enterprise Pool that is configured, configure a backup HMC for Enterprise Pool and add both of them into PowerHA SystemMirror by running the clmgr add hmc <hmc>’ command or through the SMIT menu. Thus, PowerHA SystemMirror can provide the fallover function if the master HMC fails. Section 6.12.1, “Switching to the backup HMC for the Power Enterprise Pool” on page 224 introduces some prerequisites when you set up the Power Enterprise Pool.
 
Note: At the time of writing, Power Systems Firmware supports a pair of HMCs to manage one Power Enterprise Pool: One is in master mode, and the other one is in backup mode.
Note: At the time of writing, for one Power Systems server, IBM only supports at most two HMCs to manage it.
Verifying the communication between the Enterprise Pool HMC IP and AIX LPARs
If you want PowerHA SystemMirror to control the Power Systems Enterprise Pool mobile resource for RG automatically, you must be able to ping the HMC’s host name from an AIX environment. For example, in our testing environment, the master HMC and backup HMC of Power Enterprise Pool is e16hmc1 and e16hmc3. You can get the information by using the clmgr view report roha command in AIX or by using the lscodpool command in the HMC command line, as shown in Example 6-2 and Example 6-3.
Example 6-2 Show HMC information with the clmgr view report ROHA through AIX
...
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1' --> Master HMC name of EPCoD
Backup HMC: 'e16hmc3' --> Backup HMC name of EPCoD
Enterprise pool memory
Activated memory: '100' GB
Available memory: '100' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Example 6-3 Show EPCoD HMC information with lscodpool through the HMC
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=In compliance,sequence_num=41,master_mc_name=e16hmc1,master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=1,unreturned_mobile_procs=0,mobile_mem=102400,avail_mobile_mem=60416,unreturned_mobile_mem=0
Before PowerHA SystemMirror acquires the resource from EPCoD or releases the resource back to EPCoD, PowerHA tries to check whether the HMC is accessible by using the ping command. So, AIX must be able to perform the resolution between the IP address and the host name. You can use /etc/hosts, the DNS, or other technology to achieve resolution. For example, on AIX, run ping e16hmc1 and ping e16hmc3 to check whether the resolution works.
If the HMCs are in the DNS configuration, configure these HMCs into PowerHA SystemMirror by using their names, and not their IPs.
Entering the On/Off CoD code before using the resource
If you purchased the On/Off CoD code and want to use it with PowerHA SystemMirror, you must enter the code to activate it before you use it. The menu is shown in Figure 6-23 on page 161.
No restrictions for the deployment combination with Enterprise Pool
In one PowerHA SystemMirror cluster, there is no restriction for its nodes deployment with EPCoD:
•It supports all the nodes in one server and shares mobile resources from one EPCoD.
•It supports the nodes in different servers and shares one EPCoD.
•It supports the nodes in different servers and in different EPCoDs.
•It supports the nodes in different servers and some of them in EPCoD, and others has no EPCoD.
No restrictions for the LPAR CPU type combination in one cluster
One PowerHA SystemMirror cluster supports the following combinations:
•All nodes are in dedicated processor mode.
•All nodes are in shared processor mode.
•Some of the processors are dedicated processor mode and others are shared.
In Figure 6-24, before the application starts, PowerHA SystemMirror checks the current LPAR processor mode. If it is dedicated, then two available CPUs are its target. If it is shared mode, then 1.5 available CPUs and three available VPs are its target.
Add Hardware Resource Provisioning to an Application Controller
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
[Entry Fields]
* Application Controller Name AppController1
 
Use desired level from the LPAR profile No +
Optimal number of gigabytes of memory [30]
 
Optimal number of dedicated processors [2] #
 
Optimal number of processing units [1.5]
Optimal number of virtual processors [3]
Figure 6-24 Mixed CPU type in one PowerHA SystemMirror cluster
Preferred practice after changing a partition’s LPAR name
If you change one partition’s LPAR name, the profile is changed, but AIX does not recognize this change automatically. You must shut down the partition and activate it with its profile (AIX IPL process), then after restart, the LPAR name information can be changed.
PowerHA SystemMirror gets the LPAR name from the uname -L command’s output and uses this name to do DLPAR operations through the HMC. LPAR names of the LPAR hosting cluster node are collected and persisted into HACMPdynresop so that this information is always available.
 
Note: There is one enhancement to support a DLPAR name update for AIX commands such as uname -L or lparstat -i. The requirements are as follows:
•Hardware firmware level SC840 or later (for E870 and E880)
•AIX 7.1 TL4 or 7.2 or later
•HMC V8 R8.4.0 (PTF MH01559) with mandatory interim fix (PTF MH01560)
Building password-less communication from the AIX nodes to the HMCs
In order for LPARs to communicate with the HMC, you must use SSH. All the LPAR nodes must have SSH set up.
Setting up SSH for password-less communication with the HMC requires that the user run ssh-keygen on each LPAR node to generate a public and private key pair. The public key must then be copied to the HMC’s public authorized keys file. Then, the ssh from the LPAR can contact the HMC without you needing to type in a password. Example 6-4 shows an example to set up HMC password-less communication.
Example 6-4 Setting up the HMC password-less communication
# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa):
Created directory '//.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
70:0d:22:c0:28:e9:71:64:81:0f:79:52:53:5a:52:06 root@epvioc3
The key's randomart image is:
…
 
# cd /.ssh/
# ls
id_rsa id_rsa.pub
# export MYKEY='cat /.ssh/id_rsa.pub'
# ssh [email protected] mkauthkeys -a "$MYKEY"
The authenticity of host '172.16.15.42 (172.16.15.42)' can't be established.
RSA key fingerprint is b1:47:c8:ef:f1:82:84:cd:33:c2:57:a1:a0:b2:14:f0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.15.42' (RSA) to the list of known hosts.
Keeping synchronization turned OFF for the Sync current configuration Capability setting
With later versions of the HMC, the administrator can enable the synchronization of the current configuration to the profile. If you use ROHA, the CPU and memory setting in the LPAR’s profile is modified with each operation. For consistency, turn off this synchronization when ROHA is enabled (Figure 6-25).
Figure 6-25 Sync current configuration capability
Setting the LPAR’s minimum and maximum parameters
When you configure an LPAR on the HMC (outside of PowerHA SystemMirror), you provide LPAR minimum and LPAR maximum values for the number of CPUs and amount of memory.
The stated minimum values of the resources must be available when an LPAR node starts. If more resources are available in the free pool on the frame, an LPAR can allocate up to the stated wanted values. During dynamic allocation operations, the system does not allow the values for CPU and memory to go below the minimum or above the maximum amounts that are specified for the LPAR.
PowerHA SystemMirror obtains the LPAR minimum and LPAR maximum amounts and uses them to allocate and release CPU and memory when application controllers are started and stopped on the LPAR node.
In the planning stage, you must consider how many resources are needed to satisfy all the RGs online carefully and set LPAR’s minimal and maximum parameters correctly.
Using pre-event and post-event scripts
Existing pre-event and post-event scripts that you might be using in a cluster with LPARs (before using the CoD integration with PowerHA SystemMirror) might need to be modified or rewritten if you plan to configure CoD and DLPAR requirements in PowerHA SystemMirror.
Keep in mind the following considerations:
•PowerHA SystemMirror performs all the DLPAR operations before the application controllers are started and after they are stopped. You might need to rewrite the scripts to account for this situation.
•Because PowerHA SystemMirror takes care of the resource calculations, and requests additional resources from the DLPAR operations and, if allowed, from CUoD, you can dispose the portions of your scripts that perform those functions.
•PowerHA SystemMirror considers the On/Off CoD possibility, for example, even though the cluster is configured in a single frame. If your cluster is configured within one frame, then modifying the scripts as stated before is sufficient.
•However, if a cluster is configured with LPAR nodes that are on two frames, you might still require the portions of the existing pre-event and post-event scripts that deal with dynamically allocating resources from the free pool on one frame to the node on another frame, if the application requires these resources.
 
Note: When you deal with EPCoD or On/Off CoD resources, it does not matter if there is one or two frames. For case scenarios with EPCoD or On/Off CoD, you activate (for On/Off) and acquire (for EPCoD), and modify the portion of code that deals with On/Off activation and EPCoD acquisition.
About the elapsed time of the DLPAR operation
When you plan a PowerHA SystemMirror cluster with the ROHA feature, you must consider the DLPAR release time.
While initially bringing the RG online, PowerHA SystemMirror must wait for all the resources acquisition to complete before it can start the user’s application.
While performing a takeover (fallover to the next priority node, for example), PowerHA SystemMirror tries to perform some operations (DLPAR or adjust CoD and EPCoD resource) in parallel to the release of resources on the source node and the acquisition of resources on target node if the user allows it in the tunables (the value of Force synchronous release of DLPAR resources is No).
Table 6-15 on page 167 shows the testing results of the DLPAR operation. The result might be different in other environments, particularly if the resource is being used.
There is one LPAR, its current running CPU resource size is 2C, and the running memory resource size is 8 GB. The DLPAR operation includes add and remove.
Table 6-15 Elapsed time of the DLPAR operation
Incremental value
By DLPAR
Add CPU
(in seconds)
Add memory
(in seconds)
Remove CPU
(in seconds)
Remove memory
(in minutes and seconds)
2C and 8 GB
5.5 s
8 s
6 s
88 s (1 m 28 s)
4C and 16 GB
7 s
12 s
9.8 s
149 s (2 m 29 s)
8C and 32 GB
13 s
27 s
23 s
275 s (4 m 35 s)
16C and 64 GB
18 s
34 s
33 s
526 s (8 m 46 s)
32C and 128 GB
24 s
75 s
52 s
1010 s (16 m 50 s)
48C and 192 GB
41 s
179 s
87 s
1480 s (24 m 40 s)
AIX ProbeVue maximum pinned memory setting
ProbeVue is a dynamic tracing facility of AIX. You can use it for both performance analysis and problem debugging. ProbeVue uses the Vue programming language to dynamically specify trace points and provide the actions to run at the specified trace points. This feature is enabled by default. There is one restriction regarding ProbeVue’s maximum pinned memory and DLPAR remove memory operation: The Max Pinned Memory For ProbeVue tunable cannot cross the 40% limit of system running memory.
For example, you configure one profile for an LPAR with 8 GB (minimum) and 40 GB (wanted). When you activate this LPAR, the maximum pinned memory of ProbVue is set to 4 GB (10% of system running memory), as shown in Example 6-5.
From AIX 7.1 TL4 onward, the tunables are derived based on the available system memory. MAX pinned memory is set to 10% of the system memory. It cannot be adjusted when you restart the operating system or adjust the memory size with the DLPAR operation.
Example 6-5 Current maximum pinned memory for ProbeVue
# probevctrl -l
Probevue Features: on --> ProbeVue is enabled at this time
MAX pinned memory for Probevue framework(in MB): 4096 --> this is the value we are discussing
...
Now, if you want to reduce the memory 40 - 8 GB, run the following command:
chhwres -r mem -m r1r9m1-9117-MMD-1038B9P -o r -p ITSO_S2Node1 -q 32768
The command fails with the error that is shown in Example 6-6.
Example 6-6 Error information when you reduce the memory through the DLPAR
hscroot@e16hmc3:~> chhwres -r mem -m r1r9m1-9117-MMD-1038B9P -o r -p ITSO_S2Node1 -q 32768
HSCL2932 The dynamic removal of memory resources failed: The operating system prevented all of the requested memory from being removed. Amount of memory removed: 0 MB of 32768 MB. The detailed output of the OS operation follows:
 
0930-050 The following kernel errors occurred during the
DLPAR operation.
 
0930-023 The DR operation could not be supported by one or more kernel extensions.
 
Consult the system error log for more information
....
Please issue the lshwres command to list the memory resources of the partition and to determine whether it is pending and runtime memory values match. If they do not match, problems with future memory-related operations on the managed system might occur, and it is recommended that the rsthwres command to restore memory resources be issued on the partition to synchronize its pending memory value with its runtime memory value.
From AIX, the error report also generates some error information, as shown in Example 6-7 and Example 6-8.
Example 6-7 AIX error information when you reduce the memory through the DLPAR
47DCD753 1109140415 T S PROBEVUE DR: memory remove failed by ProbeVue rec
252D3145 1109140415 T S mem DR failed by reconfig handler
Example 6-8 Detailed information about the DR_PVUE_MEM_REM_ERR error
LABEL: DR_PVUE_MEM_REM_ERR
IDENTIFIER: 47DCD753
 
Date/Time: Mon Nov 9 14:04:56 CST 2015
Sequence Number: 676
Machine Id: 00F638B94C00
Node Id: ITSO_S2Node1
Class: S
Type: TEMP
WPAR: Global
Resource Name: PROBEVUE
 
Description
DR: memory remove failed by ProbeVue reconfig handler
 
Probable Causes
Exceeded one or more ProbeVue Configuration Limits or other
 
Failure Causes
Max Pinned Memory For Probevue tunable would cross 40% limit
 
Recommended Actions
Reduce the Max Pinned Memory For Probevue tunable
 
Detail Data
DR Phase Name
PRE
Current System Physical Memory
            42949672960 -->> This is 40 GB, which is the current running memory size.
Memory that is requested to remove
            34359738368 -->> This is 32 GB, which you want to remove.
ProbeVue Max Pinned Memory tunable value
4294967296 -->> This is 4 GB, which is current maximum pinned memory for ProbeVue.
In the ROHA solution, it is possible that PowerHA SystemMirror removes memory to a low value, such as in the procedure of Automatic resource release after OS failure. To avoid this situation, consider the following items:
•If you want to enable the ProbeVue component, set the maximum pinned memory less or equal to (40% *minimum memory value of one LPAR’s profile). For example, in this case, the minimum memory size is 8 GB, so 40% is 3276.8 MB.
Therefore, you can set the maximum pinned memory size with the command that is shown in Table 6-9.
Example 6-9 Change max_total_mem_size
# probevctrl -c max_total_mem_size=3276
Attention: The command "/usr/sbin/bosboot -a" must be run for the change to take effect in the next boot.
Set it to 3276 MB, which is less than 3276.8 (8 GB*40%). This change takes effect immediately. But if you want this change to take effect after the next start, you need to run /usr/sbin/bosboot -a before the restart.
•If you do not want the ProbeVue component online, you can turn it off with the command that is shown in Example 6-10.
Example 6-10 Turn off ProbeVue
# probevctrl -c trace=off
Attention: The command "/usr/sbin/bosboot -a" must be run for the change to take effect in the next boot.
This change takes effect immediately. But if you want this change to take effect after the next start, you need to run /usr/sbin/bosboot -a before the restart.
6.4.2 Configuration steps for Resource Optimized High Availability
After finishing all of the preparations and considerations outside of PowerHA SystemMirror, you must configure PowerHA SystemMirror.
First, you must configure some generic elements for the PowerHA SystemMirror cluster:
•Cluster name
•Nodes in the cluster
•CAA repository disk
•Shared VG
•Application controller
•Service IP
•RG
•Other user-defined contents, such as pre-event or post-event
Then, you can configure the ROHA-related elements:
 – At least one HMC. Two HMCs are better.
 – Optionally, change the cluster HMC tunables.
 – Optionally, change the HMC at the node or site level.
•Optimal resources for each application controller (see 6.2.4, “Hardware resource provisioning for application controller” on page 151).
•Optionally, change the cluster ROHA tunables (see 6.2.5, “Change/Show Default Cluster Tunable” on page 156).
•Run Verify and Synchronize, review the warning or error messages, and fix them.
•Show the ROHA report by running the clmgr view report roha command and review the output.
6.5 Resource acquisition and release process introduction
This section introduces the steps of the resource acquisition and release in a ROHA solution.
6.5.1 Steps for allocation and for release
Figure 6-26 shows the steps of allocation and release. During fallover, resources are released on the active node (same as stopping RGs, which are the red numbers in the diagram) and resources are acquired on the backup node (same as starting RGs, which are the green numbers in the diagram). Figure 6-26 shows the process when CoD Pool and Enterprise Pool are used. On some CECs, none or only one or both of those reservoirs can be used.
Figure 6-26 Allocations and releases steps
For the resource releasing process, in some cases, PowerHA SystemMirror tries to return EPCoD resources before doing the DLPAR remove operation from the LPAR, and this generates unreturned resource on this server. This is an asynchronous process and is helpful to speed up RG takeover. The unreturned resource is reclaimed after the DLPAR remove operation is completed.
6.6 Introduction to resource acquisition
Figure 6-27 shows the process of acquisition for memory and processor. Resources are acquired together for a list of applications. It is a four-step process:
1. Query (blue boxes): The required resources are computed based on the LPAR configuration, and the information that is provided by the PowerHA SystemMirror state (if applications are currently running) and applications. Then, the script contacts the HMC to get information about available ROHA resources.
2. Compute (purple box): Based on this information, PowerHA SystemMirror determines the total amount of required resources that are needed on the node for the list of RGs that are to be started on the node.
3. Identify (green box): PowerHA SystemMirror determines how to perform this allocation for the node by looking at each kind of allocation to be made, that is, which part must come from Enterprise Pool CoD resources, and which part must come from On/Off CoD resources to provide some supplementary resources to the central electronic complex, and which amount of resources must be allocated from the central electronic complex to the LPAR through a DLPAR operation.
4. Apply (orange boxes): After these decisions are made, the script contacts the HMC to acquire resources. First, it acquires Enterprise Pool CoD resources and activates On/Off CoD resources, and then allocates all DLPAR resources. The amount of DLPAR resources that are allocated is persisted in the HACMPdynresop ODM object for release purposes.
Figure 6-27 Four steps to acquire resources
There are many reasons for success. The script immediately returns whether the applications are not configured with optimal resources. The script also exits if there are already enough resources allocated. Finally, the script exits when the entire process of acquisition succeeds.
However, the script can fail and return an error if one of the following situations occurs:
•The maximum LPAR size as indicated in the LPAR profile is exceeded and the Always Start RGs tunable is set to No.
•The shared processor pool size is exceeded and the Adjust SPP size if the required tunable is set to No.
•There are not enough free resources on the central electronic complex, or on the Enterprise Pool CoD, or on the On/Off CoD, and the Always Start RGs tunable is set to No.
•Any one step of the acquisition fails (see the previous four steps). Thus, successful actions that were previously performed are rolled back, and the node is reset to its initial allocation state.
In a shared processor partition, more operations must be done. For example, account for both virtual CPUs and processing units instead of only a number of processors. To activate On/Off CoD resources or acquire Enterprise Pool CoD resources, decimal processing units are converted to integers and decimal gigabytes of memory should be converted to integers.
On shared processor pool partitions, the maximum pool size can be automatically adjusted, if necessary and if authorized by the user.
6.6.1 Query
In the query step, PowerHA SystemMirror gets the information that is listed in the following sections.
Getting information from onlining apps
Onlining applications see the applications being brought online. The process is achieved by summing values that are returned by an ODM request to the HACMPserver object containing the applications resources provisioning.
Getting information from running apps
Running applications see the applications currently running on the node. The process is achieved by calling the clRGinfo command to obtain all the running applications and summing values that are returned by an ODM request on all those applications.
Getting LPAR information from HMC
The minimum, maximum, and currently allocated resources for the partition are listed through the HMC command lshwres.
Getting the DLPAR resource information from HMC
Some people think only the available resources (the query method is shown in Table 6-16) can be used for the DLPAR operation.
Table 6-16 Get server’s available resource from HMC
Memory
lshwres -m <cec> --level sys -r mem -F curr_avail_sys_mem
CPU
lshwres -m <cec> --level sys -r proc -F curr_avail_sys_proc_units
 
Tip: The lshwres commands are given in Table 6-16 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Strictly speaking, this situation is not correct. Two kinds of cases must be considered:
•There are stopped partitions on the central electronic complex.
A stopped partition still keep its resources because the resources do not appear as Available in the central electronic complex. As a matter of fact, the resources are available for other LPARs. Therefore, if you stopped a partition on the central electronic complex, the resource that stopped the partition must be available for the DLPAR operation.
Figure 6-28 shows that there is no available CPU resource when you run the lshwres command. But in fact, there is 0.5 CPU, which LPAR rar1m34 is holding, and this LPAR is in the Not Activated state. The free CPU resources are 0.5 CPU (0+0.5).
Figure 6-28 Describing the difference between available resources and free resources
•There are uncapped mode partitions in the central electronic complex.
In an uncapped shared processor partition, considering only the maximum processor unit is not correct.
Consider the following case, where one LPAR’s profile includes the following configuration:
 – Minimum processor unit: 0.5
 – Assigned processor unit: 1.5
 – Maximum processor unit: 3
 – Minimum virtual processor: 1
 – Assigned virtual processor: 6
 – Maximum virtual processor: 8
This LPAR can acquire six processor units if the workload increases, and if these resources are available in the central electronic complex. Also, this value is above the limit that is set by the Maximum processor unit, which has a value of 3.
But in any case, allocation beyond the limit of the maximum processor unit is something that is performed at the central electronic complex level, and cannot be controlled at the PowerHA SystemMirror level.
But it is true that the calculation of available resources could consider what is really being used in the CEC, and should not consider the Maximum processor unit as an intangible maximum. The real maximum comes from the number of Assigned Virtual Processor.
PowerHA SystemMirror supports the uncapped mode, but does not play a direct role in this support because this mode is used at the central electronic complex level. There is no difference in uncapped mode compared with the capped mode for PowerHA SystemMirror.
Based on the previous considerations, the formula to calculate free resources (memory and processor) for the DLAR operation is shown in Figure 6-29.
Figure 6-29 Formula to calculate free resources of one central electronic complex
 
Note: You read the level of configured resources (configurable_sys_mem in the formula), and you remove from that the level of reserved resources (sys_firmware_mem in the formula), then you end up with the level of resources that is needed to run one started partition.
Moreover, when computing the free processing units of a CEC, you consider the reserved processing units of any used Shared Processor Pool (the reserved in the formula).
Getting On/Off CoD resource information from the HMC
The available On/Off CoD resources for the CEC is listed through the HMC lscod command. The state is Available or Running (a request is ongoing). Table 6-17 shows the commands that PowerHA SystemMirror uses to get On/Off resource information. You do not need to run these commands.
Table 6-17 Get On/Off CoD resources’ status from HMC
Memory
lscod -m <cec> -t cap -c onoff -r mem -F mem_onoff_state:avail_mem_for_onoff
CPU
lscod -m <cec> -t cap -c onoff -r proc -F proc_onoff_state:avail_proc_for_onoff
 
Tip: The lscod commands are given in Table 6-17 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Acquiring Power Enterprise Pool resource information from the HMC
The available Enterprise Pool CoD resources for the pool can be acquired by running the HMC lscodpool command. Table 6-18 shows the commands that PowerHA SystemMirror uses to get the EPCoD information. You do not need to run these commands.
Table 6-18 Get the EPCoD available resources from the HMC
Memory
lscodpool -p <pool> --level pool -F avail_mobile_mem
CPU
lscodpool -p <pool> --level pool -F avail_mobile_procs
 
Tip: The lscodpool commands are given in Table 6-18 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Note: If the execution of this command fails (either because the link is down or other errors), after the last retry but before trying another HMC, PowerHA SystemMirror changes the master HMC for its Enterprise Pool.
6.6.2 Resource computation
After the query step, PowerHA SystemMirror starts performing computations to satisfy the PowerHA SystemMirror application controller’s needs. It is likely that some resources must be allocated from the CEC to the LPAR. Figure 6-30 shows the computation of the amount of resources to be allocated to the partition. This computation is performed for all types of resources, and it accounts for the following items:
•The configuration of the partition (minimum, current, and maximum amount of resources)
•The optimal resources that are configured for the applications currently running on the partition
•The optimal resources that are configured for the applications that are being brought online
In Figure 6-30, case 2b is the normal case. The currently allocated resources level matches the blue level, which is the level of resources for the application controllers currently running. PowerHA SystemMirror adds the yellow amount to the blue amount.
But in some cases, where these two levels do not match, consider having a “start fresh” policy. This policy performs a readjustment of the allocation to the exact needs of the currently running application controllers that are added to the application controllers that are being brought online (always provides an optimal amount of resources to application controllers). Those alternative cases can occur when the user has manually released (case 2a) or acquired (case 2c) resources.
Figure 6-30 Computation policy in the resource acquisition process
Here is information about the cases:
•In case 1, PowerHA SystemMirror keeps the allocated level current to satisfy your needs. This case can occur when a partition is at its profile’s wanted level, which is greater than its profile’s minimum.
•In case 2a, the readjustment consists of allocating only the missing part of the application controllers that are being brought online.
•In case 2c, the readjustment consists of allocating the missing part of the application controllers currently running that is added to the application controllers that are being brought online.
•In case 3, the needed resources cannot be satisfied by this partition. It exceeds the partition profile’s maximum. In this particular case, two behaviors can happen here depending on the Always Start RGs tunable. If enabled, PowerHA SystemMirror tries to allocate all that can be allocated, raises a warning, and continues. If disabled, PowerHA SystemMirror stops and returns an error.
In shared processor partitions, both virtual CPUs and processing units are computed. In shared processor partitions that are part of a Shared Processor Pool, the need for computation is checked against the PU/VP ratio and adjusted as needed. If it is less than need, everything is fine and the process continues. If it is greater than needed, set the Adjust SPP size if required tunable to No. The process stops and returns an error. Otherwise, it raises a warning, changes the pools size to the new size, and goes on.
6.6.3 Identifying the method of resource allocation
In the resource compute step, the amount of resources that are needed by the LPAR is computed, so now you must identify how to achieve the wanted amount. PowerHA SystemMirror considers multiple strategies in the following order:
1. Consider the CEC current free pool for DLPAR operations. This section explains how these available resources are computed.
2. If resources are still insufficient, consider the Enterprise Pool of resources, first by considering the available amount of Enterprise Pool in the local frame, then by considering the amount of Enterprise Pool that is acquired by the other frame (and see whether it can be returned back to be available on the local frame), then by considering the amount of Enterprise Pool of all frames sharing this Enterprise Pool (and see whether they can be returned back compliantly or uncompliantly to be available on the local frame).
3. If resources are still insufficient, consider the CoD pool of resources if a license was activated, and if any On/Off CoD resources are available.
When the correct strategy is chosen, there are three types of resource allocations to be done:
1. Release on other CECs: You might need to release EPCoD resources on other CEC so that these resources are made available on the local CEC.
2. Acquisition/Activation to the CEC: Resources can come from the Enterprise Pool CoD or the On/Off CoD pools.
3. Allocation to the partition: Resources come from the CEC to the LPAR.
Figure 6-31 shows the computation for the DLPAR, CoD, and Enterprise Pool CoD for the amount of resources to acquire. The computation is performed for all types of resources. In shared processor partitions, only processing units are computed this way, and accounts for the following items:
•The total amount of resources to acquire for the node (computed previously).
•The available amount of DLPAR resources on the CEC.
•The available amount of On/Off CoD resources on the CEC.
•The available amount of Enterprise Pool CoD resources in the pool to which the CEC belongs.
•The amount of Enterprise Pool CoD resources that are acquired on other frames, and that it is possible to return back.
Figure 6-31 shows the identified policy in the resource acquisition process.
Figure 6-31 Identify the policy in the resource acquisition process
There are four possible cases:
1. There are sufficient DLPAR resources to fulfill the optimal configuration. No EPCoD resources or On/Off CoD resources will be allocated to the CEC. A portion of the available DLPAR resources will be allocated to the node.
2. A portion of available Enterprise Pool CoD resources will be allocated to the CEC, and then all DLPAR resources will be allocated. No On/Off CoD resources will be activated.
Alternative case: If there are no available EPCoD resources, a portion of available On/Off CoD resources will be activated instead, and then all DLPAR resources will be allocated.
3. All available Enterprise Pool CoD resources will be allocated to the CEC, then a portion of On/Off CoD resources will be activated, and then all DLPAR resources will be allocated.
Alternative case: If there are no available EPCoD resources, a portion of available On/Off CoD resources will be activated instead, and then all DLPAR resources will be allocated (as in case 2).
4. All available Enterprise Pool CoD resources will be allocated to the CEC, then all On/Off CoD resources will be activated, and then all DLPAR resources will be allocated.
Alternative case: If the cluster has not been configured to automatically start the RGs even if resources are insufficient, do not allocate or acquire any resources because this exceeds the available resources for this CEC and exits on error instead.
In shared processor partitions, PowerHA SystemMirror accounts for the minimum ratio of assigned processing units to assigned virtual processors for the partition that is supported by the CEC. In an IBM POWER6® server, the ratio is 0.1 and in an IBM POWER7® server, the ratio is 0.05.
For example, if the current assigned processing unit in the partition is 0.6 and the current assigned virtual processor is 6, and PowerHA SystemMirror acquires virtual processors, it raises an error because it breaks the minimum ratio rule. The same occurs when PowerHA SystemMirror releases the processing units. PowerHA SystemMirror must compare the expected ratio to the configured ratio.
6.6.4 Acquiring the resource
After finishing the steps in 6.6.3, “Identifying the method of resource allocation” on page 177, PowerHA SystemMirror performs the acquire operation.
Acquiring the Power Enterprise Pool resource
The Enterprise Pool CoD resources are allocated by the HMC chcodpool command. Table 6-19 shows the commands that PowerHA SystemMirror uses to assign EPCoD resources to one server. You do not need to run these commands.
Table 6-19 Acquiring the EPCoD mobile resources
Memory
chcodpool -p <pool> -m <system> -o add -r mem -q <mb_of_memory>
CPU
chcodpool -p <pool> -m <system> -o add -r proc -q <cpu>
 
Tip: The chcodpool commands are given in Table 6-19 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Activating the On/Off CoD resources
On/Off CoD resources are activated by the HMC chcod command. Table 6-20 shows the commands that PowerHA SystemMirror uses to assign the On/Off CoD resource to one server. You do not need to run these commands.
Table 6-20 Acquire On/Off available resources
Memory
chcod -m <cec> -o a -c onoff -r mem -q <mb_of_memory> -d <days>
CPU
chcod -m <cec> -o a -c onoff -r proc -q <cpu> -d <days>
 
Tip: The chcod commands are given in Table 6-20 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Note: For acquiring the Power Enterprise Pool and the On/Off CoD resources, every amount of memory resources is expressed in MB but aligned in GB of memory (for example, 1024 or 4096), and every number of processing units is aligned on the whole upper integer.
All Power Enterprise Pool and On/Off CoD resources that are acquired are in the CEC’s free pool, and these are automatically added to the target LPAR by using DLPAR.
Allocating the DLPAR resources
DLPAR resources are allocated by using the HMC chhwres command. Table 6-21 shows the commands that PowerHA SystemMirror uses to assign resources from the server’s free pool to one LPAR. You do not need to run these commands.
Table 6-21 Assign resources from the server’s free pool to target LPAR
Dedicate Memory
chhwres -m <cec> -p <lpar> -o a -r mem -q <mb_of_memory>
Dedicate CPU
chhwres -m <cec> -p <lpar> -o a -r proc --procs <cpu>
Shared CPU
chhwres -m <cec> -p <lpar> -o a -r proc --procs <vp> --proc_units <pu>
 
Tip: The chhwres commands are given in Table 6-21 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
For shared processor partitions in a Shared-Processors Pool that is not the default pool, it might be necessary to adjust the maximum processing units of the Shared Processor Pool. To do so, use the operation that is shown in Example 6-11, which uses the HMC chhwres command. The enablement of this adjustment is authorized or not by a tunable.
Example 6-11 shows the command that PowerHA SystemMirror uses to change the Shared-Processor Pool’s maximum processing units. You do not need to run this command.
Example 6-11 DLPAR command line from HMC
chhwres -m <cec> -o s -r procpool --poolname <pool> -a max_pool_proc_units=<pu>
 
Tip: The chhwres commands are given in Example 6-11 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
6.7 Introduction to release of resources
When the RGs are stopped, PowerHA SystemMirror computes the amount of resources to be released and is responsible for performing the release of ROHA resources. There are four steps when releasing resources:
1. The query step, which appears in purple. In this step, PowerHA SystemMirror queries all the information that is needed for the compute, identify, and release steps.
2. The compute step, which appears in blue. In this step, PowerHA SystemMirror computes how many resources must be released through DLPAR. In this step, PowerHA SystemMirror uses a “fit to remaining RGs” policy, which consists in computing amounts of resources to be released by accounting for currently allocated resources and total optimal resources that are needed by RGs remaining on the node. In any case, and as it was done before, PowerHA SystemMirror does not release more than optimal resources for the RGs being released.
3. The identify step, which appears in green. In this step, PowerHA SystemMirror identifies how many resources must be removed from the LPAR, and identify how many resources must be released to the On/Off CoD and to the Power Enterprise Pool.
4. In the apply step, you remove resources from the LPAR and release resources from the CEC to On/Off CoD and Power Enterprise Pool, which appears in light green. In this step, PowerHA SystemMirror performs the DLPAR remove operation and then releases On/Off CoD resources and EPCoD resources. You can release up to the amount, but no more, of the DLPAR resources being released.
These steps are shown in Figure 6-32.
Figure 6-32 Four steps to release resources
6.7.1 Query
In the query step, PowerHA SystemMirror gets the information that is described in the following sections for the compute step.
Getting information from offlining apps
Offlining applications see the one being brought offline. At this step, check that the release of resources is needed. It means that at least one application is configured with optimal resources.
Getting information from running apps
Running applications see the ones currently running on the node. It is achieved by calling the clRGinfo binary to obtain all the running applications and summing values that are returned by an ODM request on all those applications.
Getting LPAR information from the HMC
The minimum, maximum, and currently allocated resources for the partition are listed by the HMC lshwres command.
Get On/Off CoD resource information from the HMC
The active On/Off CoD resources for the CEC are listed by the HMC lscod command. Table 6-22 shows the commands that PowerHA SystemMirror uses to get On/Off CoD information. You do not need to run these commands.
Table 6-22 Get On/Off active resources in this server from the HMC
Memory
lscod -m <cec> -t cap -c onoff -r mem -F activated_onoff_mem
CPU
lscod -m <cec> -t cap -c onoff -r proc -F activated_onoff_proc
 
Tip: The lscod commands are given in Table 6-22 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Getting Power Enterprise Pool resource information from the HMC
The allocated Enterprise Pool CoD resources for the pool are listed by the HMC lscodpool command. Table 6-23 shows the commands that PowerHA SystemMirror uses to get EPCoD information. You do not need to run these commands.
Table 6-23 Get EPCoD resource information
Memory
lscodpool -p <pool> --level pool -F mobile_mem
lscodpool -p <cec> --level sys --filter "names=server name" -F mobile_mem
CPU
lscodpool -p <pool> --level pool -F mobile_procs
lscodpool -p <cec> --level sys --filter "names=server name" -F mobile_procs
 
Tip: The lscodpool commands are given in Table 6-23 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
Resource computation
The level of resources to be left on the LPAR is computed by using the fit to remaining RGs policy. What is above this level is released, and it accounts for the following information:
1. The configuration of the LPAR (minimum, current, and maximum amount of resources).
2. The optimal resources that are configured for the applications currently running on the LPAR. PowerHA SystemMirror tries to fit to the level of remaining RGs running on the node.
3. The optimal amount of resources of the stopping RGs because you do not de-allocate more than this.
Two cases can happen, as shown in Figure 6-33:
1. Release resources to a level that enables the remaining applications to run at optimal level. PowerHA SystemMirror applies the remaining RGs policy here to compute and provide the optimal amount of resources to the remaining applications.
2. Do not release any resources because the level of currently allocated resources is already under the level that is computed by the fit to remaining RGs policy.
Figure 6-33 Resource computation in the releasing process
Releasing resources from the LPAR to the CEC
DLPAR resources are released by the HMC chhwres command. Table 6-24 shows the commands that PowerHA SystemMirror uses to release resources from the LPAR. You do not need to run these commands.
Table 6-24 Release resources from the LPAR to the CEC through the HMC
Dedicate memory
chhwres -m <cec> -p <lpar> -o r -r mem -q <mb_of_memory>
Dedicate CPU
chhwres -m <cec> -p <lpar> -o r -r proc --procs <cpu>
Shared CPU
chhwres -m <cec> -p <lpar> -o r -r proc --procs <vp> --proc_units <pu>
 
Tip: The chhwres commands are given in Table 6-24 as examples, but it is not necessary for the user to run these commands. These commands are embedded in to the ROHA run time, and run as part of the ROHA acquisition and release steps.
A timeout is given with the -w option and this timeout is set to the configured value at the cluster level (DLPAR operations timeout) added with 1 minute per GB. So, for example, to release 100 GB, if the default timeout value is set to 10 minutes, the timeout is set to 110 minutes (10 + 100).
For large memory releases, for example, instead of making one 100 GB release request, make 10 requests of a 10 GB release. You can see the logs in the hacmp.out log file.
Identifying the resource to release
The diagram that is shown in Figure 6-34 shows three cases of DLPAR, CoD, and Enterprise Pool CoD release for memory and processors.
At release, the de-allocation order is reversed, On/Off CoD resources are preferably released, preventing the user from paying for extra costs. Figure 6-34 shows the process.
Figure 6-34 Identifying the source to release
There are three cases in the identify step:
1. There are no On/Off CoD or Enterprise Pool resources that are used by the CEC. Therefore, no resources need to be released to On/Off CoD or the Enterprise Pool.
2. There are On/Off resources that are allocated to the CEC. Some of the On/Off CoD resources are deactivated up to the amount, but no more than, of the DLPAR resources being released.
3. There are both On/Off CoD resources and Enterprise Pool resources that are allocated to the CEC. Then, On/Off CoD resources are deactivated up to the amount, but no more than, of the DLPAR resources to be released.
Alternative case: If there are no On/Off CoD resources that are activated on the CEC, only return back Enterprise Pool resources to the pool.
Generating “unreturned” resources
In this step, if some EPCoD resource is identified, it is possible for PowerHA SystemMirror to release them to EPCoD immediately, and before the DLPAR remove operation even starts.
PowerHA SystemMirror raises an asynchronous process to do the DLPAR remove operation. PowerHA SystemMirror does not need to wait for the DLPAR operation to complete. So, PowerHA SystemMirror on standby mode can bring the online RGs quickly.
This asynchronous process happens only under the following two conditions:
1. If there are only two nodes in the cluster and those two nodes are on different managed systems, or if there are more than two nodes in the cluster and the operation is a move to target node and the source node is on another managed system.
2. If you set the Force synchronous release of DLPAR resources as the default, which is No, see 6.2.5, “Change/Show Default Cluster Tunable” on page 156.
About the “unreturned” resources
The unreturned resource is one function of EPCoD. This function enables you to remove Mobile CoD resources from a server that the server cannot reclaim because they are still in use; these resources become unreturned resources. From the EPCoD pool point of view, the resource is back and can be assigned to other nodes. This function can allow the standby node to acquire the resource and application to use them while the resource is being released by the primary node.
When an unreturned resource is generated, a grace period timer starts for the unreturned Mobile CoD resources on that server, and EPCoD is in Approaching out of compliance (within server grace period) status. After the releasing operation completes physically on the primary node, the unreturned resource is reclaimed automatically, and the EPCoD’s status is changed back to In compliance.
 
Note: For more information about the Enterprise Pool’s status, see the IBM Knowledge Center.
Releasing resources
This section describes the release resource concept.
Deactivating the On/Off CoD resource
CoD resources are deactivated through the HMC command-line interface chcod. PowerHA SystemMirror runs the command automatically.
Releasing (or returning back) the Enterprise Pool CoD resource
Enterprise Pool CoD resources are returned back to the pool by using the HMC chcodpool command. PowerHA SystemMirror runs the command automatically.
6.7.2 Synchronous and asynchronous mode
Because release requests take time, PowerHA SystemMirror tries to release DLPAR resources asynchronously. In asynchronous mode, the process of release is run in the background and gives priority back to other tasks.
By default, the release is asynchronous. This default behavior can be changed with a cluster tunable.
But synchronous mode is automatically computed as follows:
•All nodes of a cluster are on the same CEC.
•Otherwise, the backup LPARs of the given list of RGs are on the same CEC.
For example, if one PowerHA SystemMirror cluster includes two nodes, the two nodes are deployed on different servers and the two servers share one Power Enterprise Pool. In this case, if you are keeping asynchronous mode, you can benefit from the RG move scenarios because EPCoD’s unreturned resource feature and asynchronous release mode can reduce takeover time.
During RG offline, operations to release resources to EPCoD pool can be done even if physical resources are not free on the server at that time. The freed resources are added back to the EPCoD pool as available resources immediately so that the backup partition can use these resources to bring the RG online at once.
6.7.3 Automatic resource release process after an operating system crash
Sometimes, the ROHA resources have not been released by one node before the node failed or crashed. In this kind of cases, an automatic mechanism is implemented to release these resources when the node restarts.
A history of what was allocated for the partition is kept in the AIX ODM object database, and PowerHA SystemMirror uses it to release the same amount of resources at boot time.
 
Note: You do not need to start PowerHA SystemMirror service to activate this process after an operating system restart because this operation is triggered by the /usr/es/sbin/cluster/etc/rc.init script, which is in the /etc/inittab file.
6.8 Example 1: Setting up one Resource Optimized High Availability cluster (without On/Off CoD)
This section describes how to set up a ROHA cluster without On/Off CoD.
6.8.1 Requirement
We have two IBM Power 770 D model servers, and they are in one Power Enterprise Pool. We want to deploy one PowerHA SystemMirror cluster with two nodes that are in different servers. We want the PowerHA SystemMirror cluster to manage the server’s free resources and EPCoD mobile resource to automatically satisfy the application’s hardware requirements before we start it.
6.8.2 Hardware topology
Figure 6-35 shows the hardware topology.
Figure 6-35 Hardware topology for example 1
The topology includes the following components for configuration:
•Two Power 770 D model servers, named P770D-01 and P770D-02.
•One Power Enterprise Pool with four mobile processors and 100 GB mobile memory resources.
•The PowerHA SystemMirror cluster includes two nodes, ITSO_S1Node1 and ITSO_S2Node1.
•P770D-01 has four inactive CPUs, 140 GB of inactive memory, four available CPUs, and 5 GB of free memory.
•P770D-02 has 16 inactive CPUs, 225 GB of inactive memory, four available CPUs, and
10 GB of free memory.
•This topology also includes the profile configuration for each LPAR.
There are two HMCs to manage the EPCoD named e16hmc1 and e16hmc3. Here, e16hmc1 is the master and e16hmc3 is the backup. There are two applications in this cluster and the related resource requirement.
6.8.3 Cluster configuration
This section describes the cluster configuration.
Topology and resource group configuration
Table 6-25 shows the cluster’s attributes.
Table 6-25 Cluster’s attributes
Attribute
ITSO_S1Node1
ITSO_S2Node2
Cluster name
ITSO_ROHA_cluster
Cluster type: No Site Cluster (NSC)
Network interface
en0: 10.40.1.218
Netmask: 255.255.254.0
Gateway: 10.40.1.1
en0: 10.40.0.11
Netmask: 255.255.254.0
Gateway: 10.40.1.1
Network
net_ether_01 (10.40.0.0/23)
CAA
Unicast
Primary disk: repdisk1
Backup disk: repdisk2
Shared VG
shareVG1: hdisk18
shareVG2: hdisk19
shareVG1: hdisk8
shareVG2: hdisk9
Application controller
App1Controller:
/home/bing/app1start.sh
/home/bing/app1stop.sh
App2Controller:
/home/bing/app2start.sh
/home/bing/app2stop.sh
Service IP
10.40.1.61 ITSO_ROHA_service1
10.40.1.62 ITSO_ROHA_service2
Resource Group
RG1 includes shareVG1, ITSO_ROHA_service1, and App1Controller.
RG2 includes shareVG2, ITSO_ROHA_service2, and App2Controller.
The node order is ITSO_S1Node1 ITSO_S2Node1.
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
Resource Optimized High Availability configuration
The ROHA configuration includes the HMC, hardware resource provisioning, and the cluster-wide tunable configuration.
HMC configuration
There are two HMCs to add, as shown in Table 6-26 and Table 6-27.
Table 6-26 Configuration of HMC1
Items
Value
HMC name
9.3.207.1301
DLPAR operations timeout (in minutes)
3
Number of retries
2
Delay between retries (in seconds)
5
Nodes
ITSO_S1Node1 ITSO_S2Node1
Sites
N/A
Check connectivity between HMC and nodes
Yes (default)
1 Enter one HMC name, not an IP address, or select one HMC and then press F4 to show the HMC list. PowerHA SystemMirror also supports an HMC IP address.
Table 6-27 Configuration of HMC2
Items
Value
HMC name
9.3.207.1331
DLPAR operations timeout (in minutes)
3
Number of retries
2
Delay between retries (in seconds)
5
Nodes
ITSO_S1Node1 ITSO_S2Node1
Sites
N/A
Check connectivity between HMC and nodes
Yes (default)
1 Enter one HMC name, not an IP address, or select one HMC and then press F4 to show the HMC list. PowerHA SystemMirror also supports an HMC IP address.
Additionally, in /etc/hosts, there are resolution details between the HMC IP and the HMC host name, as shown in Example 6-12.
Example 6-12 The /etc/hosts file for example 1 and example 2
10.40.1.218 ITSO_S1Node1
10.40.0.11 ITSO_S2Node1
10.40.1.61 ITSO_ROHA_service1
10.40.1.62 ITSO_ROHA_service2
9.3.207.130 e16hmc1
9.3.207.133 e16hmc3
Hardware resource provisioning for application controller
There are two application controllers to add, as shown in Table 6-28 and Table 6-29.
Table 6-28 Configuration for HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
No (default)
Application Controller Name
AppController1
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
30
Optimal number of dedicated processors
2
Table 6-29 Configuration for HMC2
Items
Value
I agree to use On/Off CoD and be billed for extra costs
No (default)
Application Controller Name
AppController2
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
40
Optimal number of dedicated processors
6
Cluster-wide tunables
All the tunables use the default values, as shown in Table 6-30.
Table 6-30 Configuration for HMC1
Items
Value
DLPAR Start Resource Groups even if resources are insufficient
No (default)
Adjust Shared Processor Pool size if required
No (default)
Force synchronous release of DLPAR resources
No (default)
I agree to use On/Off CoD and be billed for extra costs
No (default)
Perform the PowerHA SystemMirror Verify and Synchronize Cluster Configuration process after finishing the previous configuration.
6.8.4 Showing the Resource Optimized High Availability configuration
Example 6-13 shows the output of the clmgr view report roha command.
Example 6-13 Output of the clmgr view report roha command
Cluster: ITSO_ROHA_cluster of NSC type <---NSC means No Site Cluster
Cluster tunables
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '0'
--> don’t use On/Off CoD resource in this case
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP <--this server is P770D-01
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '96'
Processing mode: Dedicated
Processors: minimum '1' desired '2' current '2' maximum '12'
ROHA provisioning for resource groups
No ROHA provisioning.
Node: ITSO_S2Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P <--this server is P770D-02
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '96'
Processing mode: Dedicated
Processors: minimum '1' desired '2' current '2' maximum '12'
ROHA provisioning for resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130' <--this HMC is master
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133' <--this HMC is backup
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP'
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '52' GB processing units '8.00'
Inactive: memory '140' GB processing units '4.00'
Available: memory '5' GB processing units '4.00'
On/Off CoD
--> this server has enabled On/Off CoD, but we don’t use them during resource group bring online or offline scenarios because we only want to simulate ONLY Enterprise Pool scenarios. Please ignore the On/Off CoD information.
On/Off CoD memory
State: 'Available'
Available: '9927' GB.days
On/Off CoD processor
State: 'Running'
Available: '9944' CPU.days
Activated: '4' CPU(s) <-- this 4CPU is assigned to P770D-01 manually to simulate 4 free processor resource
Left: '20' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC' <-- this is enterprise pool name
Hardware Management Console
9.3.207.130
9.3.207.133
Logical partition 'ITSO_S1Node1'
 
Managed System 'r1r9m1-9117-MMD-1038B9P'
Hardware resources of managed system
Installed: memory '320' GB processing units '32.00'
Configurable: memory '95' GB processing units '16.00'
Inactive: memory '225' GB processing units '16.00'
Available: memory '10' GB processing units '4.00'
On/Off CoD
--> this server has enabled On/Off CoD, but we don’t use them during resource group bring online or offline because we want to simulate ONLY Enterprise Pool exist scenarios.
On/Off CoD memory
State: 'Available'
Available: '9889' GB.days
On/Off CoD processor
State: 'Available'
Available: '9976' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Logical partition 'ITSO_S2Node1'
This 'ITSO_S2Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Enterprise pool 'DEC_2CEC'
--> shows that there is no EPCoD mobile resource is assigned to any of server
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '100' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
6.9 Test scenarios of Example 1 (without On/Off CoD)
Based on the cluster configuration in 6.5, “Resource acquisition and release process introduction” on page 170, this section introduces several testing scenarios:
6.9.1 Bringing two resource groups online
When PowerHA SystemMirror starts the cluster service on the primary node (ITSO_S1Node1), the two RGs are online. The procedure that is related to ROHA is described in Figure 6-36 on page 195.
Figure 6-36 Resource acquisition procedure to bring two resource groups online
Section 6.6, “Introduction to resource acquisition” on page 171 introduces four steps for PowerHA SystemMirror to acquire resources. In this case, the following section provides the detailed description for the four steps.
Query step
PowerHA SystemMirror queries the server, the EPCoD, the LPARs, and the current RG information. The data is shown in yellow in Figure 6-36.
Compute step
In this step, PowerHA SystemMirror computes how many resources to be added through DLPAR. It needs 7C and 46 GB. The purple table shows the process in Figure 6-36. For example:
•The expected total CPU number is as follows: 1 (Min) + 2 (RG1 requires) + 6 (RG2 requires) + 0 (running RG requires, there is no running RG) = 9C.
•Take this value to compare with LPAR’s profile needs less than or equal to the Maximum and more than or equal to the Minimum value.
•If the requirement is satisfied, and takes this value minus the current running CPU, 9 - 2 = 7, we get the CPU number to add through the DLPAR.
Identify and acquire step
After the compute step, PowerHA SystemMirror identifies how to satisfy the requirement. For CPU, it gets the remaining 4C of this server and 3C from EPCoD. For memory, it gets the remaining 5 GB of this server and 41 GB from EPCoD. The process is shown in the green table in Figure 6-36 on page 195. For example:
•There are four CPUs that are available in the server’s free pool, so PowerHA SystemMirror reserves them, and then needs another three CPUs (7 - 4).
•There are four mobile CPUs in the EPCoD pool, so PowerHA SystemMirror assigns the three CPUs from EPCoD to this server through the HMC (by running the chcodpool command). At this time, there are seven CPUs in the free pool, then PowerHA SystemMirror assigns all of them to LPAR (ITSO_S1Node1) through the DLPAR operation (by using the chhwres command).
 
Note: During this process, PowerHA SystemMirror adds mobile resources from EPCoD to the server’s free pool first, then adds all the free pool’s resources to the LPAR through DLPAR. To describe the process clearly, the free pool means only the available resources of one server before adding the EPCoD’s resources to it.
The orange tables (Figure 6-36 on page 195) show the result after the resource acquisition, and include the LPAR’s running resource, EPCoD, and the server’s resource status.
Tracking the hacmp.out log to know what is happening
By reviewing the hacmp.out log, we know that the resources (seven CPUs and 41 memory) cost 53 seconds, as shown in Example 6-14.
Example 6-14 The hacmp.out log shows the resource acquisition process for example 1
# egrep "ROHALOG|Close session|Open session" /var/hacmp/log/hacmp.out
+RG1 RG2:clmanageroha[roha_session_open:162] roha_session_log 'Open session
Open session 22937664 at Sun Nov 8 09:11:39 CST 2015
INFO: acquisition is always synchronous.
=== HACMProhaparam ODM ====
--> Cluster-wide tunables display
ALWAYS_START_RG = 0
ADJUST_SPP_SIZE = 0
FORCE_SYNC_RELEASE = 0
AGREE_TO_COD_COSTS = 0
ONOFF_DAYS = 30
===========================
------------------+----------------+
HMC | Version |
------------------+----------------+
9.3.207.130 | V8R8.3.0.1 |
9.3.207.133 | V8R8.3.0.1 |
------------------+----------------+
------------------+----------------+----------------+
MANAGED SYSTEM | Memory (GB) | Proc Unit(s) |
------------------+----------------+----------------+
Name | rar1m3-9117-MMD-1016AAP | --> Server name
State | Operating |
Region Size | 0.25 | / |
VP/PU Ratio | / | 0.05 |
Installed | 192.00 | 12.00 |
Configurable | 52.00 | 8.00 |
Reserved | 5.00 | / |
Available | 5.00 | 4.00 |
Free (computed) | 5.00 | 4.00 | --> Free pool resource
------------------+----------------+----------------+
------------------+----------------+----------------+
LPAR (dedicated) | Memory (GB) | CPU(s) |
------------------+----------------+----------------+
Name | ITSO_S1Node1 |
State | Running |
Minimum | 8.00 | 1 |
Desired | 32.00 | 2 |
Assigned | 32.00 | 2 |
Maximum | 96.00 | 12 |
------------------+----------------+----------------+
+------------------+----------------+----------------+
| ENTERPRISE POOL | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| Name | DEC_2CEC | --> Enterprise Pool Name
| State | In compliance |
| Master HMC | e16hmc1 |
| Backup HMC | e16hmc3 |
| Available | 100.00 | 4 | --> Available resource
| Unreturned (MS) | 0.00 | 0 |
| Mobile (MS) | 0.00 | 0 |
| Inactive (MS) | 140.00 | 4 | --> Maximum number to add
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| TRIAL CoD | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| State | Not Running | Not Running |
| Activated | 0.00 | 0 |
| Days left | 0 | 0 |
| Hours left | 0 | 0 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| ONOFF CoD | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| State | Available | Running |
| Activated | 0.00 | 4 | --> just ignore it
| Unreturned | 0.00 | 0 |
| Available | 140.00 | 4 |
| Days available | 9927 | 9944 |
| Days left | 0 | 20 |
| Hours left | 0 | 2 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| OTHER | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| LPAR (dedicated) | ITSO_S2Node1 |
| State | Running |
| Id | 13 |
| Uuid | 78E8427B-B157-494A-8711-7B8 |
| Minimum | 8.00 | 1 |
| Assigned | 32.00 | 2 |
+------------------+----------------+----------------+
| MANAGED SYSTEM | r1r9m1-9117-MMD-1038B9P |
| State | Operating |
+------------------+----------------+----------------+
| ENTERPRISE POOL | DEC_2CEC |
| Mobile (MS) | 0.00 | 0 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+----------------+---------+
| OPTIMAL APPS | Use Desired | Memory (GB) | CPU(s) |PU(s)/VP(s)
+------------------+----------------+----------------+----------------+---------+
| App1Controller | 0 | 30.00 | 2 | 0.00/0
| App2Controller | 0 | 40.00 | 6 | 0.00/0
+------------------+----------------+----------------+----------------+---------+
| Total | 0 | 70.00 | 8 | 0.00/0
+------------------+----------------+----------------+----------------+---------+
 
===== HACMPdynresop ODM ====
TIMESTAMP = Sun Nov 8 09:11:43 CST 2015
STATE = start_acquire
MODE = sync
APPLICATIONS = App1Controller App2Controller
RUNNING_APPS = 0
PARTITION = ITSO_S1Node1
MANAGED_SYSTEM = rar1m3-9117-MMD-1016AAP
ENTERPRISE_POOL = DEC_2CEC
PREFERRED_HMC_LIST = 9.3.207.130 9.3.207.133
OTHER_LPAR = ITSO_S2Node1
INIT_SPP_SIZE_MAX = 0
INIT_DLPAR_MEM = 32.00
INIT_DLPAR_PROCS = 2
INIT_DLPAR_PROC_UNITS = 0
INIT_CODPOOL_MEM = 0.00
INIT_CODPOOL_CPU = 0
INIT_ONOFF_MEM = 0.00
INIT_ONOFF_MEM_DAYS = 0
INIT_ONOFF_CPU = 4
INIT_ONOFF_CPU_DAYS = 20
SPP_SIZE_MAX = 0
DLPAR_MEM = 0
DLPAR_PROCS = 0
DLPAR_PROC_UNITS = 0
CODPOOL_MEM = 0
CODPOOL_CPU = 0
ONOFF_MEM = 0
ONOFF_MEM_DAYS = 0
ONOFF_CPU = 0
ONOFF_CPU_DAYS = 0
 
===== Compute ROHA Memory =====
--> compute memory process
minimal + optimal + running = total <=> current <=> maximum
8.00 + 70.00 + 0.00 = 78.00 <=> 32.00 <=> 96.00 : => 46.00 GB
============ End ==============
===== Compute ROHA CPU(s) =====
--> compute CPU process
minimal + optimal + running = total <=> current <=> maximum
1 + 8 + 0 = 9 <=> 2 <=> 12 : => 7 CPU(s)
============ End ==============
===== Identify ROHA Memory ====
--> identify memory process
Remaining available memory for partition: 5.00 GB
Total Enterprise Pool memory to allocate: 41.00 GB
Total Enterprise Pool memory to yank: 0.00 GB
Total On/Off CoD memory to activate: 0.00 GB for 0 days
Total DLPAR memory to acquire: 46.00 GB
============ End ==============
=== Identify ROHA Processor ===
--> identify CPU process
Remaining available PU(s) for partition: 4.00 Processing Unit(s)
Total Enterprise Pool CPU(s) to allocate: 3.00 CPU(s)
Total Enterprise Pool CPU(s) to yank: 0.00 CPU(s)
Total On/Off CoD CPU(s) to activate: 0.00 CPU(s) for 0 days
Total DLPAR CPU(s) to acquire: 7.00 CPU(s)
============ End ==============
--> assign EPCoD resource to server
clhmccmd: 41.00 GB of Enterprise Pool CoD have been allocated.
clhmccmd: 3 CPU(s) of Enterprise Pool CoD have been allocated.
--> assign all resource to LPAR
clhmccmd: 46.00 GB of DLPAR resources have been acquired.
clhmccmd: 7 VP(s) or CPU(s) and 0.00 PU(s) of DLPAR resources have been acquired.
The following resources were acquired for application controllers App1Controller App2Controller.
DLPAR memory: 46.00 GB On/Off CoD memory: 0.00 GB Enterprise Pool memory: 41.00 GB.
DLPAR processor: 7.00 CPU(s) On/Off CoD processor: 0.00 CPU(s) Enterprise Pool processor: 3.00 CPU(s)
INFO: received rc=0.
Success on 1 attempt(s).
 
===== HACMPdynresop ODM ====
TIMESTAMP = Sun Nov 8 09:12:31 CST 2015
STATE = end_acquire
MODE = 0
APPLICATIONS = 0
RUNNING_APPS = 0
PARTITION = 0
MANAGED_SYSTEM = 0
ENTERPRISE_POOL = 0
PREFERRED_HMC_LIST = 0
OTHER_LPAR = 0
INIT_SPP_SIZE_MAX = 0
INIT_DLPAR_MEM = 0
INIT_DLPAR_PROCS = 0
INIT_DLPAR_PROC_UNITS = 0
INIT_CODPOOL_MEM = 0
INIT_CODPOOL_CPU = 0
INIT_ONOFF_MEM = 0
INIT_ONOFF_MEM_DAYS = 0
INIT_ONOFF_CPU = 0
INIT_ONOFF_CPU_DAYS = 0
SPP_SIZE_MAX = 0
DLPAR_MEM = 46
DLPAR_PROCS = 7
DLPAR_PROC_UNITS = 0
CODPOOL_MEM = 41
CODPOOL_CPU = 3
ONOFF_MEM = 0
ONOFF_MEM_DAYS = 0
ONOFF_CPU = 0
ONOFF_CPU_DAYS = 0
============================
Session_close:313] roha_session_log 'Close session 22937664 at Sun Nov 8 09:12:32 CST 2015'
 
Important: The contents of the HACMPsynresop ODM changed in PowerHA SystemMirror V7.2.1. Although the exact form changed, the idea of persisting values into HACMPdynresop was kept, so the contents of information that is persisted into HACMPdynresop is subject to change depending on the PowerHA SystemMirror version.
Resource Optimized High Availability report update
The clmgr view report roha command output (Example 6-15) shows updates on the resources of P770D-01 and the Enterprise Pool.
Example 6-15 The update in the Resource Optimized High Availability report shows the resource acquisition process for example 1
# clmgr view report roha
...
Managed System 'rar1m3-9117-MMD-1016AAP' --> this is P770D-01 server
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '93' GB processing units '11.00'
Inactive: memory '99' GB processing units '1.00'
Available: memory '0' GB processing units '0.00'
...
 
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '59' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '1'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '41' GB
Unreturned memory: '0' GB
Activated CPU(s): '3' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Testing summary
The total time to bring the two RGs online is 68 s (from 09:11:27 to 9.12:35), and it includes the resource acquisition time, as shown in Example 6-16.
Example 6-16 The hacmp.out log shows the total time
Nov 8 09:11:27 EVENT START: node_up ITSO_S1Node1
Nov 8 09:11:31 EVENT COMPLETED: node_up ITSO_S1Node1 0
Nov 8 09:11:33 EVENT START: rg_move_fence ITSO_S1Node1 2
Nov 8 09:11:33 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 2 0
Nov 8 09:11:33 EVENT START: rg_move_acquire ITSO_S1Node1 2
Nov 8 09:11:33 EVENT START: rg_move ITSO_S1Node1 2 ACQUIRE
Nov 8 09:11:34 EVENT START: acquire_service_addr
Nov 8 09:11:34 EVENT START: acquire_aconn_service en0 net_ether_01
Nov 8 09:11:34 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Nov 8 09:11:35 EVENT START: acquire_aconn_service en0 net_ether_01
Nov 8 09:11:35 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Nov 8 09:11:35 EVENT COMPLETED: acquire_service_addr 0
Nov 8 09:11:39 EVENT COMPLETED: rg_move ITSO_S1Node1 2 ACQUIRE 0
Nov 8 09:11:39 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 2 0
Nov 8 09:11:39 EVENT START: rg_move_complete ITSO_S1Node1 2
Nov 8 09:12:32 EVENT START: start_server App1Controller
Nov 8 09:12:32 EVENT START: start_server App2Controller
Nov 8 09:12:32 EVENT COMPLETED: start_server App1Controller 0
Nov 8 09:12:32 EVENT COMPLETED: start_server App2Controller 0
Nov 8 09:12:33 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 2 0
Nov 8 09:12:35 EVENT START: node_up_complete ITSO_S1Node1
Nov 8 09:12:35 EVENT COMPLETED: node_up_complete ITSO_S1Node1 0
6.9.2 Moving one resource group to another node
There are two RGs that are running on the primary node (ITSO_S1Node1). Now, we want to move one RG from this node to the standby node (ITSO_S2Node1).
In this case, we split this move into two parts: One is the RG offline at the primary node, and the other is the RG online at the standby node.
Resource group offline at the primary node (ITSO_S1Node1)
Figure 6-37 describes the offline procedure at the primary node.
Figure 6-37 Resource group offline procedure at the primary node during the resource group move
The following sections describe the offline procedure.
Query step
PowerHA SystemMirror queries the server, EPCoD, the LPARs, and the current RG information. The data is shown in the yellow tables in Figure 6-37.
Compute step
In this step, PowerHA SystemMirror computes how many resources must be removed by using the DLPAR. PowerHA SystemMirror needs 2C and 30 GB. The purple tables show the process, as shown in Figure 6-37:
•In this case, RG1 is released and RG2 is still running. PowerHA calculates how many resources it can release based on whether RG2 has enough resources to run. So, the formula is: 9 (current running) - 1 (Min) - 6 (RG2 still running) = 2C. Two CPUs can be released.
•PowerHA accounts for that sometimes you adjust your current running resources by using a manual DLPAR operation. For example, you add some resources to satisfy another application that was not started with PowerHA. To avoid removing this kind of resource, PowerHA must check how many resources it allocated before.
The total number is those resources that PowerHA freezes so that the number is not greater than what was allocated before.
So in this case, PowerHA takes the value in the compute step to compare with the real resources this LPAR allocated before. This value is stored in one ODM object database (HACMPdryresop), and the value is 7. PowerHA SystemMirror selects the small one.
 
Identify and release step
PowerHA SystemMirror identifies how many resources must be released to EPCoD and then releases them to EPCoD asynchronously, although the resources are still in use. This process generates an unreturned resource temporarily. Figure 6-38 displays the dialog boxes that are shown on the HMC.
Figure 6-38 HMC message shows that there are unreturned resources that are generated
We can display the unreturned resources by using the clmgr view report roha command from the AIX command line, as shown in Example 6-17.
Example 6-17 Displaying unreturned resources from the AIX command line
# clmgr view report roha
...
Enterprise pool 'DEC_2CEC'
State: 'Approaching out of compliance (within server grace period)'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '89' GB -->the 30 GB has been changed to EPCoD available status
Unreturned memory: '30' GB -->the 30 GB is marked 'unreturned'
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '3' --> the 2CPU has been changed to EPCoD available status
Unreturned CPU(s): '2' --> the 2CPU is marked 'unreturned'
Used by: 'rar1m3-9117-MMD-1016AAP' -->show unreturned resource from server’s view
Activated memory: '11' GB
Unreturned memory: '30' GB
Activated CPU(s): '1' CPU(s)
Unreturned CPU(s): '2' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
From the HMC command line, you can see the unreturned resources that are generated, as shown in Example 6-18.
Example 6-18 Showing the unreturned resources and the status from the HMC command line
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level sys
name=rar1m3-9117-MMD-1016AAP,mtms=9117-MMD*1016AAP,mobile_procs=1,non_mobile_procs=8,unreturned_mobile_procs=2,inactive_procs=1,installed_procs=12,mobile_mem=11264,non_mobile_mem=53248,unreturned_mobile_mem=30720,inactive_mem=101376,installed_mem=196608
name=r1r9m1-9117-MMD-1038B9P,mtms=9117-MMD*1038B9P,mobile_procs=0,non_mobile_procs=16,unreturned_mobile_procs=0,inactive_procs=16,installed_procs=32,mobile_mem=0,non_mobile_mem=97280,unreturned_mobile_mem=0,inactive_mem=230400,installed_mem=327680
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=Approaching out of compliance (within server grace period),sequence_num=41,master_mc_name=e16hmc1,master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=3,unreturned_mobile_procs=2,mobile_mem=102400,avail_mobile_mem=91136,unreturned_mobile_mem=30720
Meanwhile, PowerHA SystemMirror triggers one asynchronous process to do the DLPAR remove operation, and it removes 2C and 30 GB resources from the LPAR into the server’s free pool. The log is written in the /var/hacmp/log/async_release.log file.
When the DLPAR operation completes, the unreturned resource is reclaimed immediately, and some messages are shown on the HMC (Figure 6-39). The Enterprise Pool’s status is changed back to In compliance.
Figure 6-39 The unreturned resource is reclaimed after the DLPAR operation
You can see the changes from HMC command line, as shown in Example 6-19.
Example 6-19 Showing the unreturned resource that is reclaimed from the HMC command line
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level sys
name=rar1m3-9117-MMD-1016AAP,mtms=9117-MMD*1016AAP,mobile_procs=1,non_mobile_procs=8,unreturned_mobile_procs=0,inactive_procs=3,installed_procs=12,mobile_mem=11264,non_mobile_mem=53248,unreturned_mobile_mem=0,inactive_mem=132096,installed_mem=196608
name=r1r9m1-9117-MMD-1038B9P,mtms=9117-MMD*1038B9P,mobile_procs=0,non_mobile_procs=16,unreturned_mobile_procs=0,inactive_procs=16,installed_procs=32,mobile_mem=0,non_mobile_mem=97280,unreturned_mobile_mem=0,inactive_mem=230400,installed_mem=327680
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=In compliance,sequence_num=41,master_mc_name=e16hmc1,
master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=3,unreturned_mobile_procs=0,mobile_mem=102400,avail_mobile_mem=91136,unreturned_mobile_mem=0
 
Note: The Approaching out of compliance status is a normal status in the Enterprise Pool, and it is useful when you need extra resources temporarily. The PowerHA SystemMirror RG takeover scenario is one of the cases.
Log information in the hacmp.out file
The hacmp.out log file records the process of the RG offlining, as shown in Example 6-20.
Example 6-20 The hacmp.out log file information about the resource group offline process
#egrep "ROHALOG|Close session|Open session" /var/hacmp/log/hacmp.out
...
===== Compute ROHA Memory =====
minimum + running = total <=> current <=> optimal <=> saved
8.00 + 40.00 = 48.00 <=> 78.00 <=> 30.00 <=> 46.00 : => 30.00 GB
============ End ==============
===== Compute ROHA CPU(s) =====
minimal + running = total <=> current <=> optimal <=> saved
1 + 6 = 7 <=> 9 <=> 2 <=> 7 : => 2 CPU(s)
============ End ==============
===== Identify ROHA Memory ====
Total Enterprise Pool memory to return back: 30.00 GB
Total On/Off CoD memory to de-activate: 0.00 GB
Total DLPAR memory to release: 30.00 GB
============ End ==============
=== Identify ROHA Processor ===
Total Enterprise Pool CPU(s) to return back: 2.00 CPU(s)
Total On/Off CoD CPU(s) to de-activate: 0.00 CPU(s)
Total DLPAR CPU(s) to release: 2.00 CPU(s)
============ End ==============
clhmccmd: 30.00 GB of Enterprise Pool CoD have been returned.
clhmccmd: 2 CPU(s) of Enterprise Pool CoD have been returned.
The following resources were released for application controllers App1Controller.
DLPAR memory: 30.00 GB On/Off CoD memory: 0.00 GB Enterprise Pool memory: 30.00 GB.
DLPAR processor: 2.00 CPU(s) On/Off CoD processor: 0.00 CPU(s) Enterprise Pool processor: 2.00 CPU(s)Close session 22937664 at Sun Nov 8 09:12:32 CST 2015
..
During the releasing process, the de-allocation order is EPCoD and then the local server’s free pool. Because EPCoD is shared between different servers, the standby node on other servers always needs this resource to bring the RG online in a takeover scenario.
Resources online at the standby node (ITSO_S2Node1)
In this case, the RG online on standby node does not need to wait for the DLPAR to complete on the primary node because it is an asynchronous process. In this process, PowerHA SystemMirror acquires a corresponding resource for the onlining RG.
 
Note: Before acquiring the process start, the 2C and 30 GB resources were available in the Enterprise Pool, so this kind of resource can also be used by standby node.
Figure 6-40 describes the resource acquisition process on the standby node (ITSO_S2Node1).
Figure 6-40 The acquisition process on standby node
This acquisition process differs from the scenario that is described in 6.9.1, “Bringing two resource groups online” on page 194. The expected resources to add to the LPAR is 1C and 6 GB and the system’s free pool can satisfy it, so it does not need to acquire resources from EPCoD.
Testing scenario summary
The total time of this RG moving is 80 seconds, from 10:53:15 to 10:53:43.
Removing the resource (2C and 30 GB) from the LPAR to a free pool on the primary node costs 257 seconds, from 10:52:51 to 10:57:08, but we are not concerned with this time because it is an asynchronous process.
Example 6-21 shows the hacmp.out information about ITSO_S1Node1.
Example 6-21 The key time stamp in hacmp.out on the primary node (ITSO_S1Node1)
# egrep "EVENT START|EVENT COMPLETED" hacmp.out
Nov 8 10:52:27 EVENT START: external_resource_state_change ITSO_S2Node1
Nov 8 10:52:27 EVENT COMPLETED: external_resource_state_change ITSO_S2Node1 0
Nov 8 10:52:27 EVENT START: rg_move_release ITSO_S1Node1 1
Nov 8 10:52:27 EVENT START: rg_move ITSO_S1Node1 1 RELEASE
Nov 8 10:52:27 EVENT START: stop_server App1Controller
Nov 8 10:52:28 EVENT COMPLETED: stop_server App1Controller 0
Nov 8 10:52:53 EVENT START: release_service_addr
Nov 8 10:52:54 EVENT COMPLETED: release_service_addr 0
Nov 8 10:52:56 EVENT COMPLETED: rg_move ITSO_S1Node1 1 RELEASE 0
Nov 8 10:52:56 EVENT COMPLETED: rg_move_release ITSO_S1Node1 1 0
Nov 8 10:52:58 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:58 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:53:00 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:53:00 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:53:00 EVENT START: rg_move_acquire ITSO_S1Node1 1
Nov 8 10:53:00 EVENT START: rg_move ITSO_S1Node1 1 ACQUIRE
Nov 8 10:53:00 EVENT COMPLETED: rg_move ITSO_S1Node1 1 ACQUIRE 0
Nov 8 10:53:00 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 1 0
Nov 8 10:53:18 EVENT START: rg_move_complete ITSO_S1Node1 1
Nov 8 10:53:19 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 1 0
Nov 8 10:53:50 EVENT START: external_resource_state_change_complete ITSO_S2Node1
Nov 8 10:53:50 EVENT COMPLETED: external_resource_state_change_complete ITSO_S2Node1 0
Example 6-22 shows the async_release.log file on ITSO_S2Node1.
Example 6-22 The asyn_release.log records the DLPAR operation
# egrep "Sun Nov| eval LC_ALL=C ssh " async_release.log
Sun Nov 8 10:52:51 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lssyscfg -r sys -m 9117-MMD*1016AAP -F name 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:54:19 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:54:19 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:55:32 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:55:32 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:56:40 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:56:40 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r proc -o r --procs 2 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:57:08 CST 2015
Sun Nov 8 10:57:08 CST 2015
Example 6-23 shows the hacmp.out information about ITSO_S2Node1.
Example 6-23 The key time stamp in hacmp.out on the standby node (ITSO_S1Node1)
#egrep "EVENT START|EVENT COMPLETED" hacmp.out
Nov 8 10:52:24 EVENT START: rg_move_release ITSO_S1Node1 1
Nov 8 10:52:24 EVENT START: rg_move ITSO_S1Node1 1 RELEASE
Nov 8 10:52:25 EVENT COMPLETED: rg_move ITSO_S1Node1 1 RELEASE 0
Nov 8 10:52:25 EVENT COMPLETED: rg_move_release ITSO_S1Node1 1 0
Nov 8 10:52:55 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:55 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:52:57 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:57 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:52:57 EVENT START: rg_move_acquire ITSO_S1Node1 1
Nov 8 10:52:57 EVENT START: rg_move ITSO_S1Node1 1 ACQUIRE
Nov 8 10:52:57 EVENT START: acquire_takeover_addr
Nov 8 10:52:58 EVENT COMPLETED: acquire_takeover_addr 0
Nov 8 10:53:15 EVENT COMPLETED: rg_move ITSO_S1Node1 1 ACQUIRE 0
Nov 8 10:53:15 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 1 0
Nov 8 10:53:15 EVENT START: rg_move_complete ITSO_S1Node1 1
Nov 8 10:53:43 EVENT START: start_server App1Controller
Nov 8 10:53:43 EVENT COMPLETED: start_server App1Controller 0
Nov 8 10:53:45 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 1 0
Nov 8 10:53:47 EVENT START: external_resource_state_change_complete ITSO_S2Node1
Nov 8 10:53:47 EVENT COMPLETED: external_resource_state_change_complete ITSO_S2Node1 0
6.9.3 Restarting with the current configuration after the primary node crashes
This case introduces the Automatic Release After a Failure (ARAF) process. We simulate a primary node that crashed immediately. We do not describe how the RG is online on standby node; we describe only what PowerHA SystemMirror does after the primary node restarts. Assume that we activate this node with the current configuration, which means that this LPAR still can hold the same amount of resources as before the crash.
As described in 6.7.3, “Automatic resource release process after an operating system crash” on page 187, after the primary node restarts, the /usr/es/sbin/cluster/etc/rc.init script is triggered by /etc/inittab and performs the resource releasing operation.
The process is shown in Figure 6-41.
Figure 6-41 Resource release process in the ARAF process
The process is similar to “Resource group offline at the primary node (ITSO_S1Node1)” on page 202. In this process, PowerHA SystemMirror tries to release all the resources that were held by the two RGs before.
Testing summary
If a resource was not released because of a PowerHA SystemMirror service crash or an AIX operating system crash, PowerHA SystemMirror can do the release operation automatically after this node starts. This operation occurs before you start the PowerHA SystemMirror service through the smitty clstart or the clmgr start cluster commands.
6.10 Example 2: Setting up one Resource Optimized High Availability cluster (with On/Off CoD)
This section describes the setup of one ROHA cluster example.
6.10.1 Requirements
We have two Power 770 D model servers in one Power Enterprise Pool, and each server has an On/Off CoD license. We want to deploy one PowerHA SystemMirror cluster, and include two nodes that are in different servers. We want the PowerHA SystemMirror cluster to manage the server’s free resources, EPCoD mobile resources, and On/Off CoD resources automatically to satisfy the application’s hardware requirement before starting it.
6.10.2 Hardware topology
Figure 6-42 shows the server and LPAR information of example 2.
Figure 6-42 Server and LPAR information
The topology includes the following components for configuration:
•Two Power 770 D model servers that are named P770D-01 and P770D-02.
•One Power Enterprise Pool with four mobile processors and 100 GB of mobile memory.
•Each server has enabled the On/Off CoD feature.
•PowerHA SystemMirror cluster includes two nodes, ITSO_S1Node1 and ITSO_S2Node1.
•P770D-01 has eight inactive CPUs, 140 GB of inactive memory, 0.5 free CPUs, and 5 GB of available memory.
•P770D-02 has 16 inactive CPUs, 225 GB of inactive memory, 2.5 free CPUs, and 10 GB of available memory.
•This topology also includes the profile configuration for each LPAR.
There are two HMCs to manage the EPCoD that are named e16hmc1 and e16hmc3. Here, e16hmc1 is the master and e16hmc3 is the backup. There are two applications in this cluster and related resource requirements.
Available resources in On/Off CoD
In the examples, the resources that we have at the On/Off CoD level are GB.Days or Processor.Days. For example, we can have 600 GB.Days, or 120 Processors.Days in the On/Off CoD pool. The time scope of the activation is determined through a tunable variable: Number of activating days for On/Off CoD requests (for more information, see 6.2.5, “Change/Show Default Cluster Tunable” on page 156).
If set to 30, for example, it means that we want to activate the resources for 30 days, so the tunable allocates 20 GB of memory only, and so we have 20 GB On/Off CoD only, even if we have 600 GB.Days available.
6.10.3 Cluster configuration
The Topology and RG configuration and HMC configuration is the same as shown in 6.8.3, “Cluster configuration” on page 189.
Hardware resource provisioning for application controller
There are two application controllers to be added, as shown in Table 6-31 and Table 6-32.
Table 6-31 Configure HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
Yes
Application Controller Name
AppController1
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
70
Optimal number of processing units
3.5
Optimal number of virtual processors
7
Table 6-32 Configure HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
Yes
Application Controller Name
AppController2
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
80
Optimal number of processing units
4.5
Optimal number of virtual processors
9
Cluster-wide tunables
All the tunables are at the default value, as shown in Table 6-33.
Table 6-33 Configuration of HMC1
Items
Value
DLPAR Always Start RGs
No (default)
Adjust Shared Processor Pool size if required
No (default)
Force synchronous release of DLPAR resources
No (default)
I agree to use On/Off CoD and be billed for extra costs
Yes
Number of activating days for On/Off CoD requests
30 (default)
This configuration requires that you perform a Verify and Synchronize Cluster Configuration after changing the previous configuration.
6.10.4 Showing the Resource Optimized High Availability configuration
The clmgr view report roha command shows the current ROHA data, as shown in Example 6-24.
Example 6-24 Shows Resource Optimized High Availability data with the clmgr view report roha command
# clmgr view report roha
Cluster: ITSO_ROHA_cluster of NSC type --> NSC means No Site Cluster
Cluster tunables --> Following is the cluster tunables
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '1'
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1 --> Information of ITSO_S1Node1 node
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for resource groups
No ROHA provisioning.
Node: ITSO_S2Node1 --> Information of ITSO_S2Node1 node
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130' --> Information of HMCs
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133'
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP' --> Information of P770D-01
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '52' GB processing units '4.00'
Inactive: memory '140' GB processing units '8.00'
Available: memory '5' GB processing units '0.50'
On/Off CoD --> Information of On/Off CoD on P770D-01 server
On/Off CoD memory
State: 'Available'
Available: '9907' GB.days
On/Off CoD processor
State: 'Available'
Available: '9959' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S1Node1'
This 'ITSO_S1Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Managed System 'r1r9m1-9117-MMD-1038B9P' --> Information of P770D-02
Hardware resources of managed system
Installed: memory '320' GB processing units '32.00'
Configurable: memory '95' GB processing units '16.00'
Inactive: memory '225' GB processing units '16.00'
Available: memory '10' GB processing units '2.50'
On/Off CoD --> Information of On/Off CoD on P770D-02 server
On/Off CoD memory
State: 'Available'
Available: '9889' GB.days
On/Off CoD processor
State: 'Available'
Available: '9976' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S2Node1'
This 'ITSO_S2Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Enterprise pool 'DEC_2CEC' --> Information of Enterprise Pool
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB -->Total mobile resource of Pool, not change during resource moving
Available memory: '100' GB -->Available for assign, will change during resource moving
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB --> the number that is assigned from EPCoD to server
Unreturned memory: '0' GB --> the number has been released to EPCoD but not reclaimed, need to reclaimed within a period time
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
6.11 Test scenarios for Example 2 (with On/Off CoD)
Based on the configuration in 6.10, “Example 2: Setting up one Resource Optimized High Availability cluster (with On/Off CoD)” on page 211, this section introduces two testing scenarios:
6.11.1 Bringing two resource groups online
When PowerHA SystemMirror starts cluster services on the primary node (ITSO_S1Node1), the two RGs go online. The procedure that is related to ROHA is shown in Figure 6-43.
Figure 6-43 Acquire resource process of example 2
Section 6.6, “Introduction to resource acquisition” on page 171 introduces four steps for PowerHA SystemMirror to acquire the resources. In this case, the following sections are the detailed descriptions of the four steps.
Query step
PowerHA SystemMirror queries the server, EPCoD, the On/Off CoD, the LPARs, and the current RG information. The data is shown in the yellow tables in Figure 6-43.
For the On/Off CoD resources, we do not display the available resources because there are enough resources in our testing environment:
•P770D-01 has 9959 CPU.days and 9917 GB.days.
•P770D-02 has 9976 CPU.days and 9889 GB.days.
We display the actual amount that is used.
Compute step
In this step, PowerHA SystemMirror computes how many resources you must add through the DLPAR. PowerHA SystemMirror needs 7C and 126 GB. The purple tables show this process (Figure 6-43 on page 216). We take the CPU resources as follows:
•The expected total processor unit number is 0.5 (Min) + 3.5 (RG1 requirement) + 4.5 (RG2 requirement) +0 (running RG requirement (there is no running RG)) = 8.5C.
•Take this value to compare with the LPAR’s profile, which must be less than or equal to the Maximum and more than or equal to the Minimum value.
•If this configuration satisfies the requirement, then take this value minus the current running CPU (8.5 - 1.5 = 7), and this is the number that we want to add to the LPAR through DLPAR.
Identify and acquire step
After the compute step, PowerHA SystemMirror identifies how to satisfy the requirement. For CPU, it gets 4C from EPCoD and 3C from the On/Off CoD. Because the minimum operation unit is 1 for EPCoD and On/Off CoD, and even if there is 0.5 CPU in the server’s free pool, the requirement is 7, you leave it in the free pool.
PowerHA SystemMirror gets the remaining 5 GB of this server, all 100 GB from EPCoD, and 21 GB from the On/Off CoD. The process is shown in the green table in Figure 6-43 on page 216.
 
Note: During this process, PowerHA SystemMirror adds mobile resources from EPCoD to the server’s free pool first, then adds all the free pool’s resources to the LPAR through DLPAR. To describe this clearly, the free pool means the available resources of only one server before adding the EPCoD’s resources to it.
The orange table shows (Figure 6-43 on page 216) the result of this scenario, including the LPAR’s running resources, EPCoD, On/Off CoD, and the server’s resource status.
Tracking the hacmp.out log to know what is happening
From hacmp.out, you know that all the resources (sevenCPUs and 126 memory) costs 117 seconds as a synchronous process, as shown in Example 6-25:
22:44:40 → 22:46:37
Example 6-25 The hacmp.out log shows the resource acquisition of example 2
===== Compute ROHA Memory =====
minimal + optimal + running = total <=> current <=> maximum
8.00 + 150.00 + 0.00 = 158.00 <=> 32.00 <=> 160.00 : => 126.00 GB
============ End ==============
=== Compute ROHA PU(s)/VP(s) ==
minimal + optimal + running = total <=> current <=> maximum
1 + 16 + 0 = 17 <=> 3 <=> 18 : => 14 Virtual Processor(s)
minimal + optimal + running = total <=> current <=> maximum
0.50 + 8.00 + 0.00 = 8.50 <=> 1.50 <=> 9.00 : => 7.00 Processing Unit(s)
============ End ==============
===== Identify ROHA Memory ====
Remaining available memory for partition: 5.00 GB
Total Enterprise Pool memory to allocate: 100.00 GB
Total Enterprise Pool memory to yank: 0.00 GB
Total On/Off CoD memory to activate: 21.00 GB for 30 days
Total DLPAR memory to acquire: 126.00 GB
============ End ==============
=== Identify ROHA Processor ===
Remaining available PU(s) for partition: 0.50 Processing Unit(s)
Total Enterprise Pool CPU(s) to allocate: 4.00 CPU(s)
Total Enterprise Pool CPU(s) to yank: 0.00 CPU(s)
Total On/Off CoD CPU(s) to activate: 3.00 CPU(s) for 30 days
Total DLPAR PU(s)/VP(s) to acquire: 7.00 Processing Unit(s) and 14.00 Virtual Processor(s)
============ End ==============
clhmccmd: 100.00 GB of Enterprise Pool CoD have been allocated.
clhmccmd: 4 CPU(s) of Enterprise Pool CoD have been allocated.
clhmccmd: 21.00 GB of On/Off CoD resources have been activated for 30 days.
clhmccmd: 3 CPU(s) of On/Off CoD resources have been activated for 30 days.
clhmccmd: 126.00 GB of DLPAR resources have been acquired.
clhmccmd: 14 VP(s) or CPU(s) and 7.00 PU(s) of DLPAR resources have been acquired.
The following resources were acquired for application controllers App1Controller App2Controller.
DLPAR memory: 126.00 GB On/Off CoD memory: 21.00 GB Enterprise Pool memory: 100.00 GB.
DLPAR processor: 7.00 PU/14.00 VP On/Off CoD processor: 3.00 CPU(s) Enterprise Pool processor: 4.00 CPU(s)
Resource Optimized High Availability report update
The clmgr view report roha command reports the ROHA data, as shown in Example 6-26.
Example 6-26 Resource Optimized High Availability data after acquiring resources in example 2
# clmgr view report roha
Cluster: ITSO_ROHA_cluster of NSC type
Cluster tunables
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '1'
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '158' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '8.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '17' maximum '18'
ROHA provisioning for 'ONLINE' resource groups
No ROHA provisioning.
ROHA provisioning for 'OFFLINE' resource groups
No 'OFFLINE' resource group.
Node: ITSO_S2Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for 'ONLINE' resource groups
No 'ONLINE' resource group.
ROHA provisioning for 'OFFLINE' resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130'
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133'
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP'
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '173' GB processing units '11.00'
Inactive: memory '19' GB processing units '1.00'
Available: memory '0' GB processing units '0.50'
On/Off CoD
On/Off CoD memory
State: 'Running'
Available: '9277' GB.days
Activated: '21' GB
Left: '630' GB.days
On/Off CoD processor
State: 'Running'
Available: '9869' CPU.days
Activated: '3' CPU(s)
Left: '90' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S1Node1'
This 'ITSO_S1Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
...
 
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '0' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '0'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '100' GB
Unreturned memory: '0' GB
Activated CPU(s): '4' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
The clmgr view report roha command output (Example 6-26 on page 218) has some updates about the resources of P770D-01, Enterprise Pool, and On/Off CoD.
How to calculate the On/Off CoD consumption
In this case, before bringing the two RGs online, the remaining resources in On/Off CoD are shown in Example 6-27.
Example 6-27 Remaining resources in On/Off CoD before resource acquisition
On/Off CoD memory
State: 'Available'
Available: '9907' GB.days
On/Off CoD processor
State: 'Available'
Available: '9959' CPU.days
After the RG is online, the status of the On/Off CoD resource is shown in Example 6-28.
Example 6-28 Status of the memory resources
On/Off CoD memory
State: 'Running'
Available: '9277' GB.days
Activated: '21' GB
Left: '630' GB.days
On/Off CoD processor
State: 'Running'
Available: '9869' CPU.days
Activated: '3' CPU(s)
Left: '90' CPU.days
For processor, PowerHA SystemMirror assigns three processors and the activation day is 30 days, so the total is 90 CPU.Day. (3*30=90), and the remaining available CPU.Day in the On/Off CoD is 9869 (9959 - 90 = 9869).
For memory, PowerHA SystemMirror assigns 21 GB and the activation day is 30 days, so the total is 630 GB.Day. (21*30=630), and the remaining available GB.Day in On/Off CoD is 9277 (9907 - 630 = 9277).
6.11.2 Bringing one resource group offline
This section introduces the process of RG offline. Figure 6-44 shows the overall process.
Figure 6-44 Overall release process of example 2
The process is similar to the one that is shown in 6.9.2, “Moving one resource group to another node” on page 201.
In the release process, the de-allocation order is On/Off CoD, then EPCoD, and then the server’s free pool because you always need to pay an extra cost for the On/Off CoD.
After the release process completes, you can find the detailed information about compute, identify, and release processes in the hacmp.out file, as shown in Example 6-29.
Example 6-29 The hacmp.out log information in the release process of example 2
===== Compute ROHA Memory =====
minimum + running = total <=> current <=> optimal <=> saved
8.00 + 80.00 = 88.00 <=> 158.00 <=> 70.00 <=> 126.00 : => 70.00 GB
============ End ==============
=== Compute ROHA PU(s)/VP(s) ==
minimal + running = total <=> current <=> optimal <=> saved
1 + 9 = 10 <=> 17 <=> 7 <=> 14 : => 7 Virtual Processor(s)
minimal + running = total <=> current <=> optimal <=> saved
0.50 + 4.50 = 5.00 <=> 8.50 <=> 3.50 <=> 7.00 : => 3.50 Processing Unit(s)
============ End ==============
===== Identify ROHA Memory ====
Total Enterprise Pool memory to return back: 49.00 GB
Total On/Off CoD memory to de-activate: 21.00 GB
Total DLPAR memory to release: 70.00 GB
============ End ==============
=== Identify ROHA Processor ===
Total Enterprise Pool CPU(s) to return back: 1.00 CPU(s)
Total On/Off CoD CPU(s) to de-activate: 3.00 CPU(s)
Total DLPAR PU(s)/VP(s) to release: 7.00 Virtual Processor(s) and 3.50 Processing Unit(s)
============ End ==============
clhmccmd: 49.00 GB of Enterprise Pool CoD have been returned.
clhmccmd: 1 CPU(s) of Enterprise Pool CoD have been returned.
The following resources were released for application controllers App1Controller.
DLPAR memory: 70.00 GB On/Off CoD memory: 21.00 GB Enterprise Pool memory: 49.00 GB.
DLPAR processor: 3.50 PU/7.00 VP On/Off CoD processor: 3.00 CPU(s) Enterprise Pool processor: 1.00 CPU(s)
6.12 Hardware Management Console high availability introduction
More than one HMC can be configured for a node, so that if one HMC fails to respond, the ROHA function can switch to the other HMC.
This section describes the mechanism that enables the HMC to switch from one HMC to another HMC.
Suppose that you have, for a given node, three HMCs in the following order: HMC1, HMC2, and HMC3. (These HMCs can be set either at the node level, at the site level, or at the cluster level. What counts at the end is that you have an ordered list of HMCs for a given node).
A given node uses the first HMC in its list, for example, HMC1, and uses it while it works.
HMC1 might fail for different reasons. For example:
1. HMC1 is not reachable by the ping command.
One parameter controls the ping command in the HMC: Timeout on ping (which is set by default to 3 seconds, and you cannot adjust it). If an HMC cannot be pinged after this timeout, you cannot use it through ssh, so switch immediately to another HMC, in this case the HMC following the current one in the list (for example, HMC2).
2. HMC1 is not reachable through SSH:
 – SSH is not properly configured between the node and HMC1, so it is not worth trying to use HMC1, and it is best to switch to another HMC, in this case, the HMC following the current one in the list, for example, HMC2.
 – SSH has temporary conditions that prevent it from responding.
Two parameters controls the ssh command on the HMC:
 – Connect Attempts (which is set by default to 5).
 – Connect Timeout (which is set by default to 5), meaning that after a 25-second delay, the HMC can be considered as not reachable through ssh.
If the HMC is not reachable through ssh, it is not worth trying to perform an hmc command through ssh on it, and the best is to switch at to another HMC. In this case, the HMC following the current one in the list, for example HMC2.
3. The HMC is repeatedly busy.
When the HMC is processing a command, it cannot perform another command at the same time. The command fails with RC=-1 with the HSCL3205 message indicating that the HMC is busy.
The PowerHA SystemMirror ROHA function has a retry mechanism that is controlled by two parameters:
 – RETRY_COUNT, which indicates how many retries must be done.
 – RETRY_DELAY, which indicates how long to wait between retries.
When the HMC is busy, the retry mechanism is used until declaring that the HMC is flooded.
When the HMC is considered flooded, it is not worth using it again, and the best is to switch immediately to another HMC, which is the HMC following the current one in the list, for example, HMC2.
4. The HMC returns an application error. Several cases can occur:
 – One case is when you request an amount of resources that is not available, and the same request is attempted with another smaller amount.
 – A second case is when the command is not understandable by the HMC, which is more like a programming bug. In these cases, the bug must be debugged at test time. In any case, this is not a reason to switch to another HMC.
If you decide to switch to another HMC, consider the next HMC of the list, and use it.
If the first HMC is not usable (HMC1), you are currently using the second HMC in the list (HMC2), which helps prevent the ROHA function from trying again and failing again by using the first HMC (HMC1). You can add (persistence) into the ODM for which HMC is being used (for example, HMC2).
This mechanism enables the ROHA function to skip the failing HMCs, and to use the HMC that works (in this case, HMC2). At the end of the session, the persistence into the ODM is cleared, meaning that the first HMC in the list is restored to its role of HMC1 or the first in the list.
6.12.1 Switching to the backup HMC for the Power Enterprise Pool
For Enterprise Pool operations, querying operations can be run on the master or backup HMC, but changing operations must run on the master HMC. If the master HMC fails, the PowerHA SystemMirror actions are as follows:
•For querying operations, PowerHA SystemMirror tries to switch to the backup HMC to continue the operation, but does not set the backup HMC as the master.
•For changing operations, PowerHA SystemMirror tries to set the backup HMC as the master, and then continues the operation. Example 6-30 shows the command that PowerHA SystemMirror performs to set the backup HMC as the master. This command is triggered by PowerHA SystemMirror automatically.
Example 6-30 Setting the backup HMC as the master
chcodpool -o setmaster -p <pool> --mc backup
There are some prerequisites in PowerHA SystemMirror before switching to the backup HMC when the master HMC fails:
•Configure the master HMC and the backup HMC for your Power Enterprise Pool.
For more information about how to configure the backup HMC for the Power Enterprise Pool, see the IBM Knowledge Center and Power Enterprise Pools on IBM Power Systems, REDP-5101.
•Both HMCs are configured in PowerHA SystemMirror.
•Establish password-less communication between the PowerHA SystemMirror nodes to the two HMCs.
•Ensure reachability (pingable) from PowerHA SystemMirror nodes to the master and backup HMCs.
•Ensure that all of the servers that participate in the pool are connected to the two HMCs.
•Ensure that the participating servers are in either the Standby state or the Operating state.
6.13 Test scenario for HMC fallover
This section shows how PowerHA SystemMirror switches the HMC automatically when the primary HMC fails.
6.13.1 Hardware topology
Figure 6-45 shows the initial status of the hardware topology.
Figure 6-45 Initial status of the hardware topology
The topology includes the following components:
•Two Power 780 D model servers, named P780D_09 and P780D_10.
•One Power Enterprise Pool, which has 64 mobile processors and 2 TB of mobile memory resources.
•There are 64 CPUs and 2 TB of memory that are installed in P780D_09, 32 CPUs, 1 TB of memory that is configured, and another 32 CPUs and 1 TB of memory are in the inactive status. At this time, there are five CPUs and 10 GB of memory available for the DLPAR.
•The PowerHA SystemMirror cluster includes two nodes:
 – P780_09_Lpar1
 – P780_10_Lpar2
•The PowerHA SystemMirror cluster includes one RG (RG2); this RG has one application controller (app2) configured hardware resource provisioning.
•This application needs 20 C and 40 G when it runs.
•There is no On/Off CoD in this testing.
There are two HMCs to manage the EPCoD, named HMCEP1 and HMCEP2.
HMCEP1 is the master and HMCEP2 is the backup, as shown in Example 6-31.
Example 6-31 HMCs that are available
hscroot@HMCEP1:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=0
In the AIX /etc/hosts file, define the resolution between the HMC IP address, and the HMC’s host name, as shown in Example 6-32.
Example 6-32 Define the resolution between the HMC IP and HMC name in /etc/hosts
172.16.50.129 P780_09_Lpar1
172.16.50.130 P780_10_Lpar1
172.16.51.129 testservice1
172.16.51.130 testservice2
172.16.50.253 HMCEP1
172.16.50.254 HMCEP2
Start the PowerHA SystemMirror service on P780_09_Lpar1. During the start, PowerHA SystemMirror acquires resources from the server’s free pool and EPCoD (Figure 6-46).
Figure 6-46 Resource Acquisition process during the start of the PowerHA SystemMirror service
In this process, HMCEP1 acts as the primary HMC and does all the query and resource acquisition operations. Example 6-33 and Example 6-34 on page 228 show the detailed commands that are used in the acquisition step.
Example 6-33 EPCoD operation during resource acquisition (hacmp.out)
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@HMCEP1 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o add -q 23552 2>&1' -->23552 means 23 GB
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@HMCEP1 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r proc -o add -q 14 2>&1'
Example 6-34 DLPAR add operation in the acquire step
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@172.16.50.253 'chhwres -m SVRP7780-09-SN060C0AT -p P780_09_Lpar1 -r mem -o a -q 33024 -w 32 2>&1' -->33024 means 32.25 GB
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@172.16.50.253 'chhwres -m SVRP7780-09-SN060C0AT -p P780_09_Lpar1 -r proc -o a --procs 19 -w 32 2>&1 -->172.16.50.253 is HMCEP1
 
Note: We do not display the DLPAR and EPCoD operations in the query step in the previous examples.
6.13.2 Bringing one resource group offline when the primary HMC fails
After the RG is online, we bring the RG offline. During this process, we shut down HMCEP1 to see how PowerHA SystemMirror handles this situation.
The resource releasing process is shown in Figure 6-47.
Figure 6-47 Bringing the resource group offline process
Section 6.6, “Introduction to resource acquisition” on page 171 introduces the four steps for PowerHA SystemMirror to acquire the resources. The following sections give a detailed description of the four steps.
Query step
In this step, PowerHA SystemMirror must query the server’s data and the EPCoD’s data.
To get the server’s information, PowerHA SystemMirror uses the default primary HMC (172.16.50.253, HMCEP1). At first, HMCEP1 is alive, and the operation succeeds. But after the HMCEP1 shutdown, the operation fails and PowerHA SystemMirror uses 172.16.50.254 as the primary HMC to continue. Example 6-35 shows the takeover process.
Example 6-35 HMC takeover process
+testRG2:clhmccmd[get_local_hmc_list:815] g_hmc_list='172.16.50.253 172.16.50.254'
--> default, the global HMC list is:172.16.50.253 is first, then 172.16.50.254
...
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 172.16.50.253
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=1
+testRG2:clhmccmd[clhmcexec:3514] (( 1 > 0 ))
+testRG2:clhmccmd[clhmcexec:3516] : Cannot contact this HMC. Ask following HMC in list.
--> after checking, confirm that 172.16.50.253 is unaccessible, then to find next HMC in the list
...
+testRG2:clhmccmd[clhmcexec:3510] : Try to ping the HMC at address 172.16.50.254.
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 172.16.50.254
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=0
+testRG2:clhmccmd[clhmcexec:3514] (( 0 > 0 ))
--> 172.16.50.254 is the next, so PowerHA SystemMirror check it
...
+testRG2:clhmccmd[update_hmc_list:3312] g_hmc_list='172.16.50.254 172.16.50.253'
--> it is accessible, change it as first HMC in global HMC list
...
+testRG2:clhmccmd[clhmcexec:3456] loop_hmc_list='172.16.50.254 172.16.50.253'
--> global HMC list has been changed, following operation will use 172.16.50.254
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no [email protected] 'lshmc -v 2>&1'
--> start with 172.16.50.254 to do query operation
...
+testRG2:clhmccmd[clhmcexec:3618] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o Conne
ctionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lscodpool -p 0019 --level sys --filter names=SVRP7780-09-SN060C0AT -F inactive_mem:mobile_mem:unreturne
d_mobile_mem:inactive_procs:mobile_procs:unreturned_mobile_procs 2>&1''
+t
--> using 172.16.50.254 to query EPCoD information
 
Important: The process enables you to query EPCoD information from the backup HMC. However, any change operations must be done on the master HMC.
Compute step
This step does not require an HMC operation. For more information, see Figure 6-47 on page 228.
Identify and acquire step
After the identify step, there are some resources that must be released to EPCoD. Therefore, PowerHA SystemMirror returns the resource back to EPCoD immediately before the resource is removed from the LPAR. This generates an unreturned resource temporarily.
At this time, PowerHA SystemMirror checks whether the master HMC is available. If not, it switches to the backup HMC automatically. Example 6-36 shows the detailed process.
Example 6-36 The EPCoD master and backup HMC switch process
+testRG2:clhmccmd[clhmcexec:3388] cmd='chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o remove -q 23552 --force'
-->PowerHA SystemMirror try to do chcodpool operation
...
+testRG2:clhmccmd[clhmcexec:3401] : If working on an EPCoD Operation, we need master
-->PowerHA SystemMirror want to check whether master HMC is accessible
...
ctionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lscodpool -p 0019 --level pool -F master_mc_name:backup_master_mc_name 2>&1''
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no [email protected] 'lscodpool -p 0019 --level pool -F master_mc_name:backup_master_mc_name 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3415] res=HMCEP1:HMCEP2
-->Current HMC is 172.16.50.254, so PowerHA SystemMirror query current master and backup HMC name from it. At this time, HMCEP1 is master and HMCEP2 is backup.
...
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 HMCEP1
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=1
+testRG2:clhmccmd[clhmcexec:3514] (( 1 > 0 ))
+testRG2:clhmccmd[clhmcexec:3516] : Cannot contact this HMC. Ask following HMC in list.
+testRG2:clhmccmd[clhmcexec:3518] dspmsg scripts.cat -s 38 500 '%1$s: WARNING: unable to ping HMC at address %2$s. ' clhmccmd HMCEP1
-->PowerHA SystemMirror try to ping HMCEP1, but fails
...
+testRG2:clhmccmd[clhmcexec:3510] : Try to ping the HMC at address HMCEP2.
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 HMCEP2
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=0
+testRG2:clhmccmd[clhmcexec:3514] (( 0 > 0 ))
-->PowerHA SystemMirror try to verify HMCEP2 and it is available
...
+testRG2:clhmccmd[clhmcexec:3527] : the hmc is the master_hmc
+testRG2:clhmccmd[clhmcexec:3529] (( g_epcod_modify_operation == 1 && loop_hmc_counter != 1 ))
+testRG2:clhmccmd[clhmcexec:3531] : If not, we need to change master_hmc, we also try to
+testRG2:clhmccmd[clhmcexec:3532] : set a backup_master_hmc
 
+testRG2:clhmccmd[clhmcexec:3536] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o Conne
ctionAttempts=3 -o TCPKeepAlive=no $'hscroot@HMCEP2 'chcodpool -p 0019 -o setmaster --mc this 2>&1''
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -o setmaster --mc this 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3536] out_str=''
+testRG2:clhmccmd[clhmcexec:3537] ssh_rc=0
-->PowerHA SystemMirror set backup HMC(HMCEP2) as master
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -o update -a "backup_master_mc_name=HMCEP1" 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3722] out_str='HSCL90E9 Management console HMCEP1was not found.'
-->PowerHA SystemMirror also try to set HMCEP1 as backup, but it fails because HMCEP1 is shut down at this time
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o remove -q 23552 --force 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
-->PowerHA SystemMirror do the force release for memory resource
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r proc -o remove -q 14 --force 2>&1'
-->PowerHA SystemMirror do the force release for CPU resource
Example 6-37 shows the update that is performed from the EPCoD view.
Example 6-37 EPCoD status change during the takeover operation
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=50,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2024448,unreturned_mobile_mem=0
--> There are 14CPU(64-50) and 23 GB((2048000-2024448)/1024) has assigned to P780D_09.
hscroot@HMCEP2:~> lscodpool -p 0019 --level sys
name=SVRP7780-10-SN061949T,mtms=9179-MHD*061949T,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=32,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1024000,installed_mem=2097152
name=SVRP7780-09-SN060C0AT,mtms=9179-MHD*060C0AT,mobile_procs=14,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=18,installed_procs=64,mobile_mem=23552,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1000448,installed_mem=2097152
--> Show the information from server level report
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=unavailable,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=unavailable,avail_mobile_procs=unavailable,unreturned_mobile_procs=unavailable,mobile_mem=unavailable,avail_mobile_mem=unavailable,unreturned_mobile_mem=unavailable
--> After HMCEP1 shutdown, the EPCoD’s status is changed to 'unavailable'
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP2,master_mc_mtms=V017-f93*ba3e3aa,backup_master_mc_mtms=none,mobile_procs=64,avail_mobile_procs=50,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2024448,unreturned_mobile_mem=0
--> After PowerHA SystemMirror run 'chcodpool -p 0019 -o setmaster --mc this' on HMCEP2, the master HMC is changed and status is changed to 'In compliance'
....
hscroot@HMCEP2:~> lscodpool -p 0019 --level sys
name=SVRP7780-10-SN061949T,mtms=9179-MHD*061949T,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=32,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1024000,installed_mem=2097152
name=SVRP7780-09-SN060C0AT,mtms=9179-MHD*060C0AT,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=14,inactive_procs=18,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=23552,inactive_mem=1000448,installed_mem=2097152
--> After PowerHA SystemMirror forcibly releases resource, unreturned resource is generated
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=Approaching out of compliance (within server grace period),sequence_num=5,master_mc_name=HMCEP2,master_mc_mtms=V017-f93*ba3e3aa,backup_master_mc_mtms=none,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=14,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=23553
--> At this time, the resource has returned to EPCoD and can be used by other servers.
..
When PowerHA SystemMirror completes the previous steps, it raises an asynchronous process to remove the resources from P780_09_Lpar1 by using DLPAR. The resources include 19 CPUs and 32.25 GB of memory.
After the DLPAR operation, the unreturned resource is reclaimed automatically, and the EPCoD status is changed to In compliance, as shown in Example 6-38.
Example 6-38 EPCoD status that is restored after the DLPAR operation completes
hscroot@HMCEP1:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=0
6.13.3 Testing summary
This scenario introduced how PowerHA SystemMirror performs HMC takeovers when the primary HMC fails. This is an automatic process and has no impact on your environment.
6.14 Managing, monitoring, and troubleshooting
This section introduces some tools to manage, monitor, and troubleshoot a ROHA cluster.
6.14.1 The clmgr interface to manage Resource Optimized High Availability
SMIT relies on the clmgr command to perform the configuration that is related to ROHA.
HMC configuration
The following examples show how to configure HMC with the clmgr command.
Query/Add/Modify/Delete
Example 6-39 shows how to query, add, modify, and delete HMC with the clmgr command.
Example 6-39 Query/Add/Modify/Delete HMC with the clmgr command
# clmgr query hmc -h
clmgr query hmc [<HMC>[,<HMC#2>,...]]
 
# clmgr –v query hmc
NAME="r1r9sdmc.austin.ibm.com"
TIMEOUT="-1"
RETRY_COUNT="8"
RETRY_DELAY="-1"
NODES=clio1,clio2
SITES=site1
 
# clmgr add hmc -h
clmgr add hmc <HMC>
[ TIMEOUT={<#>} ]
[ RETRY_COUNT={<#>} ]
[ RETRY_DELAY={<#>} ]
[ NODES=<node>[,<node#2>,...]> ]
[ SITES=<site>[,<site#2>,...]> ]
[ CHECK_HMC={<yes>|<no>} ]
# clmgr modify hmc -h
clmgr modify hmc <HMC>
[ TIMEOUT={<#>} ]
[ RETRY_COUNT={<#>} ]
[ RETRY_DELAY={<#>} ]
[ NODES=<node>[,<node#2>,...]> ]
[ SITES=<site>[,<site#2>,...]> ]
[ CHECK_HMC={<yes>|<no>} ]
 
# clmgr delete hmc -h
clmgr delete hmc {<HMC>[,<HMC#2>,...] | ALL}
Query/Modify a node with the list of associated HMCs
Example 6-40 shows how to query and modify a node with the list of associated HMCs.
Example 6-40 Query/Modify node with a list of associated HMCs with the clmgr command
# clmgr query node -h
clmgr query node {<node>|LOCAL}[,<node#2>,...]
 
# clmgr –v query node
NAME="rar1m31"
…
HMCS="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr modify node -h
clmgr modify node <NODE>
…
[ HMCS=<sorted_hmc_list> ]
Query/Modify a site with the list of associated HMCs
Example 6-41 shows how to query and modify the site with a list of associated HMCs with the clmgr command.
Example 6-41 Query/Modify a site with a list of the associated HMCs with the clmgr command
# clmgr query site -h
clmgr query site [<site> [,<site#2>,...]]
 
# clmgr –v query site
NAME="site1"
…
HMCS="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr modify site -h
clmgr modify site <SITE>
…
[ HMCS =<sorted_hmc_list> ]
Query/Modify a cluster with the default HMC tunables
Example 6-42 on page 235 shows how to query and modify the cluster with the default HMC tunables.
Example 6-42 Query/Modify a cluster with the default HMC tunables with the clmgr command
# clmgr query cluster -h
clmgr query cluster [ ALL | {CORE,SECURITY,SPLIT-MERGE,HMC,ROHA} ]
 
# clmgr query cluster hmc
DEFAULT_HMC_TIMEOUT="10"
DEFAULT_HMC_RETRY_COUNT="5"
DEFAULT_HMC_RETRY_DELAY="10"
DEFAULT_HMCS_LIST="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr manage cluster hmc -h
clmgr manage cluster hmc
[ DEFAULT_HMC_TIMEOUT=# ]
[ DEFAULT_HMC_RETRY_COUNT=# ]
[ DEFAULT_HMC_RETRY_DELAY=# ]
[ DEFAULT_HMCS_LIST=<new_hmcs_list> ]
Hardware resource provisioning
SMIT relies on the clmgr command to list or query current values of the hardware resource provisioning and to add/modify/delete the HACMPserver ODM data structure, as shown in Example 6-43.
Example 6-43 Hardware resource provisioning configuration with the clmgr command
# clmgr query cod -h
clmgr query cod [<APP>[,<APP#2>,...]]
 
# clmgr –v query cod
NAME="appli1_APPCON_A"
USE_DESIRED=No
OPTIMAL_MEM="4"
OPTIMAL_CPU="3"
OPTIMAL_PU="2.5"
OPTIMAL_PV="3.0"
 
 
# clmgr add cod -h
clmgr add cod <APPCTRL>
[ USE_DESIRED =<Yes|No> ]
[ OPTIMAL_MEM=# ]
[ OPTIMAL_CPU=# ]
[ OPTIMAL_PU=#.# ]
[ OPTIMAL_PV=#.# ]
 
# clmgr modify cod -h
clmgr modify cod <APPCTRL>
[ USE_DESIRED =<Yes|No> ]
[ OPTIMAL_MEM=# ]
[ OPTIMAL_CPU=# ]
[ OPTIMAL_PU=#.# ]
[ OPTIMAL_PV=# ]
 
# clmgr delete cod -h
clmgr delete cod {<APPCTRL> | ALL}
Cluster tunables
SMIT relies on the clmgr command to query or modify cluster CoD tunables, as shown in Example 6-44.
Example 6-44 Cluster-wide tunables configuration with the clmgr command
# clmgr query cluster -h
clmgr query cluster [ ALL | {CORE,SECURITY,SPLIT-MERGE,HMC,ROHA} ]
 
# clmgr query cluster roha
ALWAYS_START_RG="no"
ADJUST_SPP_SIZE="yes"
FORCE_SYNC_RELEASE="no"
AGREE_TO_COD_COSTS="no"
COD_ONOFF_DAYS="30"
RESOURCE_ALLOCATION_ORDER="free_pool_first"
 
# clmgr manage cluster roha –h
clmgr manage cluster roha
[ ALWAYS_START_RG={yes|no} ]
[ ADJUST_SPP_SIZE={yes|no} ]
[ FORCE_SYNC_RELEASE={yes|no} ]
[ AGREE_TO_COD_COSTS={yes|no} ]
[ COD_ONOFF_DAYS=<new_number_of_days> ]
 
Important: There is a new variable that is shown in Example 6-44:
RESOURCE_ALLOCATION_ORDER
The resource allocation order specifies the order in which resources are allocated. The resources are released in the reverse order in which they are allocated. The default value for this field is Free Pool First.
Select Free Pool First to acquire resources from the free pool. If the amount of resources in the free pool is insufficient, PowerHA SystemMirror first requests more resources from the Enterprise pool and then from the CoD pool.
Select Enterprise Pool First to acquire the resources from the Enterprise pool. If the amount of resources in the CoD pool is insufficient, PowerHA SystemMirror first requests more resources from the free pool and then from the CoD pool.
6.14.2 Changing the DLPAR and CoD resources dynamically
You can change the DLPAR and CoD resource requirements for application controllers without stopping the cluster services. Synchronize the cluster after making the changes.
The new configuration is not reflected until the next event that causes the application (hence the RG) to be released and reacquired on another node. A change in the resource requirements for CPUs, memory, or both does not cause the recalculation of the DLPAR resources. PowerHA SystemMirror does not stop and restart the application controllers solely for the purpose of making the application provisioning changes.
If another dynamic reconfiguration change causes the RGs to be released and reacquired, the new resource requirements for DLPAR and CoD are used at the end of this dynamic reconfiguration event.
6.14.3 View the Resource Optimized High Availability report
The clmgr view report roha command is intended to query all the ROHA data so that a report and a summary can be presented to the user.
The output of this command includes the following sections:
•CEC name
•LPAR name
•LPAR profile (min, desired, and max)
•LPAR processing mode
•If shared (capped or uncapped, SPP name, and SPP size)
•LPAR current level of resources (mem, cpu, and pu)
•Number and names of AC and optimal level of resources, and the sum of them
•Release mode (sync/async), which is computed at release time
•All On/Off CoD information of the CECs
•All EPCoD information of the CECs
6.14.4 Troubleshooting DLPAR and CoD operations
This section provides a brief troubleshooting of the DLPAR and CoD operations.
Log files
There are several log files that you can use to track the ROHA operation process.
Logs for verification
In the process of Verify and Synchronize Cluster Configuration, there are some log files that are generated in the /var/hacmp/clverify directory. The clverify.log and the ver_collect_dlpar.log files are useful for debugging if the process fails. For example, after performing the process, there is some error information appearing in the console output (/smit.log), as shown in Example 6-45.
Example 6-45 Error information about console or /smit.log
WARNING: At the time of verification, node ITSO_S2Node1 would not have been able to acquire
sufficient resources to run Resource Group(s) RG1 (multiple Resource Groups
in case of node collocation). Please note that the amount of resources and
CoD resources available at the time of verification may be different from
the amount available at the time of an actual acquisition of resources.
Reason : 708.00 GB of memory that is needed will exceed LPAR maximum of 160.00 GB. 12.50 Processing Unit(s) needed will exceed LPAR maximum of 9.00 Processing Unit(s).
ERROR: At the time of verification, no node (out of 2) was able to acquire
sufficient resources to run Resource Group(s) RG1
You can get detailed information to help you identify the errors’ root causes from the clverify.log and the ver_collect_dlpar.log files, as shown in Example 6-46.
Example 6-46 Detailed information in ver_collect_dlpar.log
[ROHALOG:2490918:(19.127)] clmanageroha: ERROR: 708.00 GB of memory that is needed will exceed LPAR maximum of 160.00 GB.
[ROHALOG:2490918:(19.130)] ===== Compute ROHA Memory =====
[ROHALOG:2490918:(19.133)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.137)] 8.00 + 700.00 + 0.00 = 708.00 <=> 32.00 <=> 160.00 : => 0.00 GB
[ROHALOG:2490918:(19.140)] ============ End ==============
[ROHALOG:2490918:(19.207)] clmanageroha: ERROR: 12.50 Processing Unit(s) needed will exceed LPAR maximum of 9.00 Processing Unit(s).
[ROHALOG:2490918:(19.212)] === Compute ROHA PU(s)/VP(s) ==
[ROHALOG:2490918:(19.214)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.217)] 1 + 12 + 0 = 13 <=> 3 <=> 18 : => 0 Virtual Processor(s)
[ROHALOG:2490918:(19.220)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.223)] 0.50 + 12.00 + 0.00 = 12.50 <=> 1.50 <=> 9.00 : => 0.00 Processing Unit(s)
[ROHALOG:2490918:(19.227)] ============ End ==============
[ROHALOG:2490918:(19.231)] INFO: received error code 21.
[ROHALOG:2490918:(19.233)] No or no more reassessment.
[ROHALOG:2490918:(19.241)] An error occurred while performing acquire operation.
PowerHA SystemMirror simulates the resource acquisition process based on the current configuration and generates the log in the ver_collect_dlpar.log file.
Logs for resource group online and offline
During the process of resource online or offline, the hacmp.out and the async_release.log logs are useful for monitoring or debugging. In some RG offline scenarios, the DLPAR remove operation is a synchronous process. In this case, PowerHA SystemMirror generates the DLPAR operation logs in the async_release.log file. In a synchronous process, only hacmp.out is used.
AIX errpt output
Sometimes, the DLPAR operation fails, and AIX generates some errors that are found in the errpt output, as shown in Example 6-47.
Example 6-47 The errpt error report
252D3145 1109140415 T S mem DR failed by reconfig handler
47DCD753 1109140415 T S PROBEVUE DR: memory remove failed by ProbeVue rec
You can identify the root cause of the failure by using this information.
HMC commands
You can use the following commands on the HMC to do some monitor or maintenance. For a detailed description of the commands, see the man page for the HMC.
The lshwres command
This command shows the LPAR minimum, LPAR maximum, the total amount of memory, and the number of CPUs that are currently allocated to the LPAR.
The lssyscfg command
This command verifies that the LPAR node is DLPAR capable.
The chhwres command
This command runs the DLPAR operations on the HMC outside of PowerHA SystemMirror to manually change the LPAR minimum, LPAR minimum, and LPAR required values for the LPAR. This might be necessary if PowerHA SystemMirror issues an error or a warning, during the verification process, if you requested DLPAR and CoD resources in PowerHA SystemMirror.
The lscod command
This command views the system CoD of the current configuration.
The chcod command
This command runs the CoD operations on the HMC outside of PowerHA SystemMirror and manually changes the Trial CoD, On/Off CoD, and so on, of the activated resources. This command is necessary if PowerHA SystemMirror issues an error or a warning during the verification process, or if you want to use DLPAR and On/Off CoD resources in PowerHA SystemMirror.
The lscodpool command
This command shows the system Enterprise Pool current configuration.
The chcodpool command
This command runs the Enterprise Pool CoD operations on the HMC outside of PowerHA SystemMirror and manually changes the Enterprise pool capacity resources. This command is necessary if PowerHA SystemMirror issues an error or a warning during the verification process, or if you want to use DLPAR, On/Off CoD, or EPCoD resources in PowerHA SystemMirror.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset