Resource Optimized High Availability (ROHA)
This chapter describes one feature: Resource Optimized High Availability (ROHA). This feature is one new feature of PowerHA SystemMirror Standard and Enterprise Edition, version 7.2.
In this chapter, we introduce the following topics:
6.1 ROHA concept and terminology
With this feature, PowerHA SystemMirror can manage dynamic LPAR (DLPAR) and Capacity of Demand (CoD) resources. CoD resources are mainly composed of Enterprise Pool CoD resources and On/Off CoD resources.
Enterprise Pool CoD (EPCoD) resource
EPCoD resources are resources that can be freely moved among servers in the same pool where the resources are best used. Physical resources (like CPU or memory) are not really moved between servers, what is moved, in fact, is the privilege to use them. You can grant this privilege to any server of the pool. This enables you to flexibly manage the pool of resources and allocate the resources where they are most needed.
On/Off CoD resource
On/Off CoD resources are preinstalled and inactive (and unpaid for) physical resources for a given server: Processors or memory capacity. On/Off CoD is a type of CoD license enabling temporary activation of processors and memory. PowerHA SystemMirror can dynamically activate these resources and can make them available to the system, so that they are allocated when needed to the LPAR through a DLPAR operation.
Dynamic Logical Partitioning (DLPAR)
DLPAR represents the facilities in some IBM Power Systems that provide the ability to logically attach and detach a managed system’s resources to and from a logical partition’s operating system without restarting.
By integrating with DLPAR and CoD resources, PowerHA SystemMirror ensures that each node can support the application with reasonable performance at a minimum cost. This way, you can tune the capacity of the logical partition flexibly when your application requires more resources, without having to pay for idle capacity until you actually need it (for On/Off CoD), or without keeping allocated resources if you don’t use them (for Enterprise Pool CoD).
You can configure cluster resources so that the logical partition with minimally allocated resources serves as a standby node, and the application is on another LPAR node that has more resources than the standby node. This way, you do not use any additional resources that the frames have until the resources are required by the application.
PowerHA SystemMirror uses the system-connected HMC to perform DLPAR operation and manage CoD resources.
Table 6-1 displays all available types of CoD offering. Only two of them are dynamically managed and controlled by PowerHA SystemMirror: EPCoD and On/Off CoD.
Table 6-1 CoD offering and PowerHA
CoD offering
PowerHA SystemMirror 6.1 Standard and Enterprise Edition
PowerHA SystemMirror 7.1or 7.2
Standard and Enterprise Edition
Enterprise Pool
Memory and Processor
No
Yes, from version 7.2
On/Off CoD (temporary)
Memory
No
Yes, from version 7.1.3SP2
On/Off CoD (temporary)
Processor
Yes
Yes
Utility CoD (temporary)
Memory and Processor
Utility CoD automatically is performed at PHYP/System level. PowerHA cannot play a role in the same
Trial CoD
Memory and Processor
Trial CoD are used if available through DLPAR operation
(At the bottom of this table, describe detail relationship between PowerHA and Trial CoD)
CuoD (permanent)
Memory & Processor
CuoD are used if available through DLPAR operation. PowerHA will not handle this kind of resource directly.
Trial CoD
Trial CoD are temporary resources, but they are not put On or Off to follow dynamic needs. When Trial CoD standard or exception code is entered into HMC, these resources are On at once, and elapsed time starts immediately. The amount of resources granted by Trial CoD directly enters into the available DLPAR resources. It is as if they were configured DLPAR resources.
Therefore, PowerHA SystemMirror can dynamically control the Trial CoD resource after customer manually enter code to activate the resource through HMC.
6.1.1 Environment requirement for ROHA
The following are the requirements to implement ROHA:
•PowerHA SystemMirror 7.2, Standard Edition or Enterprise Edition
•AIX 6.1 TL09 SP5, or AIX 7.1 TL03 SP5, or AIX 7.1 TL4 or AIX 7.2 or later
•HMC requirement
 – To use the Enterprise Pool CoD license, your system must be using HMC 7.8 firmware or later
 – To configure backup HMC for EPCoD as far as possible for high availability requirement
 – For EPCoD User Interface (UI) in HMC, HMC must have a minimum of 2 GB memory
•Hardware requirement for using Enterprise Pool CoD license
 – Power 7+: 9117-MMD(770 D model), 9179-MHD(780 D model), using FW780.10 or later
 – Power 8: 9119-MME (E870), 9119-MHE (E880), using FW820 or later
6.2 New PowerHA SystemMirror SMIT configure panel for ROHA
In order to support the ROHA feature, PowerHA SystemMirror provides some new SMIT menu and clmgr command options. These options include the following functions:
•HMC configuration
•Hardware Resource Provisioning for Application Controller
•Cluster tunables configuration
Figure 6-1 shows the summary of SMIT menu navigation for all new ROHA panels. For the new options of clmgr command, see 6.14.1, “The clmgr interface to manage ROHA” on page 255 for detailed information.
Figure 6-1 All new ROHA panels
6.2.1 Entry point to ROHA
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors). This panel is a menu screen with a title menu option and four item menu options. Only the third item is the entry point to ROHA configuration (Figure 6-2).
           Configure User Applications (Scripts and Monitors)
 
Move cursor to desired item and press Enter.
 
Application Controller Scripts
Application Monitors
Resource Optimized High Availability
Show Cluster Applications
Figure 6-2 Entry point to ROHA menu
Table 6-2 shows the context-sensitive help for the ROHA entry point.
Table 6-2 Context-sensitive help for entry point of ROHA
Name and fast path
context-sensitive help (F1)
Resource Optimized High Availability
# smitty cm_cfg_roha
Choose this option to configure Resource Optimized High Availability (ROHA). ROHA performs dynamic management of hardware resources (memory, cpu) for the account of PowerHA SystemMirror. This dynamic management of resources uses three types of mechanism: DLPAR mechanism, On/Off CoD mechanism, Enterprise Pool CoD mechanism. If resources available on the CEC are not sufficient, and cannot be got through a DLPAR operation, it is possible to fetch into external pools of resources provided by CoD: Either On/Off or Enterprise Pool. On/Off CoD can result in extra costs, and formal agreement from the user is required. The user must configure Hardware Management Consoles (HMC) to contact for actual acquisition/release of resources.
6.2.2 ROHA panel
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability. The next panel is a menu screen with a title menu option and three item menu options. Its fast path is cm_cfg_roha (Figure 6-3).
            Resource Optimized High Availability
 
Move cursor to desired item and press Enter.
 
HMC Configuration
Hardware Resource Provisioning for Application Controller
Change/Show Default Cluster Tunables
Figure 6-3 ROHA panel
Table 6-3 shows the help information for the ROHA panel.
Table 6-3 Context-sensitive help for ROHA panel
Name and fast path
context-sensitive help (F1)
HMC Configuration
# smitty cm_cfg_hmc
Choose this option to configure Hardware Management Console (HMC) used by your cluster configuration, and to optionally associate HMC to your cluster’s nodes. If no HMC associated with a node, PowerHA SystemMirror will use the default cluster configuration.
Change/Show Hardware Resource Provisioning for Application Controller
# smitty cm_cfg_hr_prov
Choose this option to change or show CPU and memory resource requirements for any Application Controller that runs in a cluster that uses DLPAR, CoD, or Enterprise Pool CoD capable nodes, or a combination.
Change/Show Default Cluster Tunables
# smitty cm_cfg_def_cl_tun
Choose this option to modify or view the DLPAR, CoD, and Enterprise Pool CoD configuration parameters.
6.2.3 HMC configuration
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → HMC Configuration. The next panel is a menu screen with a title menu option and seven item menu options. Its fast path is cm_cfg_hmc (Figure 6-4).
            HMC Configuration
 
Move cursor to desired item and press Enter.
 
Add HMC Definition
Change/Show HMC Definition
Remove HMC Definition
Change/Show HMC List for a Node
Change/Show HMC List for a Site
Change/Show Default HMC Tunables
Change/Show Default HMC List
Figure 6-4 HMC configuration menu
Table 6-4 shows the help information for the HMC configuration.
Table 6-4 Context-sensitive help for HMC configuration
Name and fast path
context-sensitive help (F1)
Add HMC Definition
# smitty cm_cfg_add_hmc
Choose this option to add a Hardware Management Console (HMC) and its communication parameters, and add this new HMC to the default list. All the nodes of the cluster will use by default these HMC definitions to perform DLPAR operations, unless you associate a particular HMC to a node.
Change/Show HMC Definition
# smitty cm_cfg_ch_hmc
Choose this option to modify or view a Hardware Management Console (HMC) host name and communication parameters.
Remove HMC Definition
# smitty cm_cfg_rm_hmc
Choose this option to remove a Hardware Management Console (HMC), and then remove it from the default list.
Change/Show HMC List for a Node
# smitty cm_cfg_hmcs_node
Choose this option to modify or view the list of Hardware Management Console (HMC) of a node.
Change/Show HMC List for a Site
# smitty cm_cfg_hmcs_site
Choose this option to modify or view the list of Hardware Management Console (HMC) of a site.
Change/Show Default HMC Tunables
# smitty cm_cfg_def_hmc_tun
Choose this option to modify or view the HMC default communication tunables.
Change/Show Default HMC List
# smitty cm_cfg_def_hmcs
Choose this option to modify or view the default HMC list that is used by default by all nodes of the cluster. Nodes that define their own HMC list will not use this default HMC list.
HMC Add/Change/Remove Definition
 
Note: Before you add HMC, you need to build password-less communication from AIX nodes to the HMC. See 6.4.1, “Consideration before ROHA configuration” on page 183 for detailed steps.
To add HMC, select Add HMC Definition. The next panel is a dialog screen with a title dialog header and several dialog command options. Its fast path is cm_cfg_add_hmc. Each item has a context-sensitive help screen that you access by pressing F1, and can have an associated list (press F4).
Figure 6-5 shows the menu to add the HMC definition and its entry fields.
            Add HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name [] +
DLPAR operations timeout (in minutes) [] #
Number of retries [] #
Delay between retries (in seconds) [] #
Nodes [] +
Sites [] +
Check connectivity between HMC and nodes Yes
Figure 6-5 Add HMC Definition menu
Table 6-5 shows the help and information list for adding the HMC definition.
Table 6-5 Context-sensitive help and Associated List for Add HMC Definition
Name
context-sensitive help (F1)
Associated list (F4)
HMC name
Enter the host name for the Hardware Management Console (HMC). An IP address is also accepted here. Both IPv4 and IPv6 addresses are supported.
Yes (single-selection).
The list is obtained with the following command: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc -a IP
DLPAR operations timeout
(in minutes)
Enter a timeout in minutes on DLPAR commands run on an HMC (-w parameter). This -w parameter only exists on the chhwres command, when allocating or releasing resources. It is adjusted according to the type of resources (for memory, 1 minute per gigabyte is added to this timeout. Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Number of retries
Enter a number of times one HMC command is retried before the HMC is considered as non-responding. The next HMC in the list will be used after this number of retries has failed. Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Delay between retries
(in seconds)
Enter a delay in seconds between two successive retries. Setting no value means that you use the default value, which is defined in the Change/Show Default HMC Tunables panel. When -1 is displayed in this field, it indicates that the default value is used.
None
Nodes
Enter the list of nodes that use this HMC.
Yes (multiple-selection).
A list of nodes to be proposed can be obtained through the following command:
odmget HACMPnode
Sites
Enter the sites that use this HMC. All nodes of the sites will then use this HMC by default, unless the node defines an HMC as its own level.
Yes (multiple-selection).
A list of sites to be proposed can be obtained through the following command:
odmget HACMPsite
Check connectivity between HMC and nodes
Select Yes to check communication links between nodes and HMC.
<Yes>|<No>. The default is Yes.
If DNS (Domain Name Service) is configured in your environment and DNS can do resolution for HMC IP and host name, then you can use F4 to select one HMC to perform the add operation.
Figure 6-6 shows an example of selecting one HMC from the list to perform the add operation.
           HMC name
Move cursor to desired item and press Enter.
e16hmc1 is 9.3.207.130
e16hmc3 is 9.3.207.133
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-6 Select one HMC from HMC list to do add HMC operation
PowerHA SystemMirror also supports entering the HMC IP address to add the HMC. Figure 6-7 shows an example of entering one HMC IP address to add the HMC.
           Add HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name [9.3.207.130] +
DLPAR operations timeout (in minutes) []
Number of retries []
Delay between retries (in seconds) []
Nodes []+
Sites []+
Check connectivity between HMC and nodes Yes
Figure 6-7 Enter one HMC IP address to add HMC
Change/Show HMC Definition
To show or modify an HMC, select Change/Show HMC Definition. The next panel is a selector screen with a selector header that lists all existing HMC names. Its fast path is cm_cfg_ch_hmc (Figure 6-8).
| HMC name
|
| Move cursor to desired item and press Enter.
|
|    e16hmc1
|    e16hmc3
|
| F1=Help F2=Refresh F3=Cancel
| Esc+8=Image Esc+0=Exit Enter=Do
| /=Find n=Find Next
Figure 6-8 Select HMC from list during change or show HMC configuration
Press Enter on an existing HMC to modify it. The next panel is the one presented in Figure 6-9. We cannot change the name of the HMC.
           Change/Show HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name                                           e16hmc1
DLPAR operations timeout (in minutes) [5] #
Number of retries [3] #
Delay between retries (in seconds) [10] #
Nodes                             [ITSO_rar1m3_Node1 ITSO_r1r9m1_Node1] +
Sites [] +
Check connectivity between HMC and nodes Yes
Figure 6-9 Change/Show HMC Definition of SMIT menu
Remove HMC Definition
To delete an HMC, select Remove HMC Definition. The panel as shown in Figure 6-10 is the same selector screen. Press Enter on an existing HMC name to remove it, after confirmation. Its fast path is cm_cfg_rm_hmc.
            HMC name
 
Move cursor to desired item and press Enter.
 
e16hmc1
e16hmc3
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-10 Select one HMC to remove
Figure 6-11 shows the removed HMC definition.
           Remove HMC Definition
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* HMC name                                                 e16hmc1
 
Figure 6-11 Remove one HMC
Change/Show HMC List for a Node
To show or modify the HMC list for a node, select Change/Show HMC List for a Node. The next panel (Figure 6-12) is a selector screen with a selector header that lists all existing nodes. Its fast path is cm_cfg_hmcs_node.
                Select a Node
 
Move cursor to desired item and press Enter.
 
ITSO_rar1m3_Node1
ITSO_r1r9m1_Node1
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next Esc+8=Image
Figure 6-12 Select a Node to change
Press Enter on an existing node to modify it. The next panel (Figure 6-13) is a dialog screen with a title dialog header and two dialog command options.
Note that you cannot add or remove an HMC from this list. You can only reorder (set in the right precedence order) the HMCs used by the node.
            Change/Show HMC List for a Node
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Node name ITSO_rar1m3_Node1
HMC list [e16hmc1 e16hmc3]
Figure 6-13 Change/Show HMC list for a Node
Table 6-6 shows the help information to change or show the HMC list for a node.
Table 6-6 Context-sensitive help for Change or Show HMC list for a Node
Name and fast path
context-sensitive help (F1)
Node name
This is the node name to associate with one or more Hardware Management Consoles (HMCs).
HMC list
Precedence order of the HMCs used by this node. The first in the list is tried first then the second, and so on. You cannot add or remove any HMC. You are only able to modify the order of the already set HMCs.
Change/Show HMC List for a Site
To show or modify the HMC list for a node, select Change/Show HMC List for a Site. The next panel (Figure 6-14) is a selector screen with a selector header that lists all existing sites. Its fast path is cm_cfg_hmcs_site.
                      Select a Site
 
Move cursor to desired item and press Enter.
 
site1
site2
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-14 Select a Site menu when Change/Show HMC List for as Site
Press Enter on an existing site to modify it. The next panel (Figure 6-15) is a dialog screen with a title dialog header and two dialog command options.
          Change/Show HMC List for a Site
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Site Name site1
HMC list [e16hmc1 e16hmc3]
Figure 6-15 Change/Show HMC List for a Site menu
Note that you cannot add or remove an HMC from the list. You can only reorder (set in the right precedence order) the HMCs used by the site. See Table 6-7.
Table 6-7 Site and HMC usage list
Name and fast path
context-sensitive help (F1)
Site name
This is the site name to associate with one or more Hardware Management Consoles (HMCs).
HMC list
Precedence order of the HMCs used by this site. The first in the list is tried first, then the second, and so on. You cannot add or remove any HMC. You are only able to modify the order of the already set HMCs.
Change/Show Default HMC Tunables
To show or modify default HMC communication tunables, select Change/Show Default HMC Tunables. The next panel (Figure 6-16) is a dialog screen with a title dialog header and three dialog command options. Its fast path is cm_cfg_def_hmc_tun. Each item has a context-sensitive help screen (press F1) and can have an associated list (press F4).
            Change/Show Default HMC Tunables
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                                 [Entry Fields]
DLPAR operations timeout (in minutes) [10] #
Number of retries [5] #
Delay between retries (in seconds) [10]
Figure 6-16 Change/Show Default HMC Tunables menu
Change/Show Default HMC List
To show or modify the default HMC list, select Change/Show Default HMC List. The next panel (Figure 6-17) is a dialog screen with a title dialog header and one dialog command option. Its fast path is cm_cfg_def_hmcs. Each item has a context-sensitive help screen (press F1) and can have an associated list (press F4).
            Change/Show Default HMC List
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
HMC list [e16hmc1 e16hmc3]
Figure 6-17 Change/Show Default HMC list menu
6.2.4 Hardware resource provisioning for application controller
To provision hardware, complete the following steps:
1. Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → Hardware Resource Provisioning for Application Controller. The next panel (Figure 6-18) is a menu screen with a title menu option and three item menu options.
            Hardware Resource Provisioning for Application Controller
 
Move cursor to desired item and press Enter.
 
Add Hardware Resource Provisioning to an Application Controller
Change/Show Hardware Resource Provisioning of an Application Controller
Remove Hardware Resource Provisioning from an Application Controller
Figure 6-18 Hardware Resource Provisioning for Application Controller menu
2. Choose one of the following actions:
 – To add an application controller configuration, select Add.
 – To change or show an application controller configuration, select Change/Show.
 – To remove an application controller configuration, select Remove.
In case you choose Add or Change/Show, the following On/Off CoD Agreement is displayed as shown in Figure 6-19. However, this is displayed only if the user has not yet agreed to it. If the user has already agreed to it, it is not displayed.
On/Off CoD Agreement
Figure 6-19 is a dialog screen with a dialog header and one dialog command option.
            On/Off CoD Agreement
Type or select a value for the entry field.
Press Enter AFTER making all desired changes.
[Entry Fields]
Resources Optimized High Availability management No +
can take advantage of On/Off CoD resources.
On/Off CoD use would incur additional costs.
Do you agree to use On/Off CoD and be billed
for extra costs?
Figure 6-19 On/Off CoD Agreement menu
To accept the On/Off CoD Agreement, complete the following steps:
1. Enter Yes to have PowerHA SystemMirror use On/Off Capacity On Demand (On/Off CoD) resources to perform DLPAR operations on your nodes.
2. If you agree to use On/Off CoD, you must ensure that you have entered the On/Off CoD activation code. The On/Off CoD license key needs to be entered into HMC before PowerHA SystemMirror can activate this type of resources.
3. In the following cases, keep the default value:
 – If there is only half Enterprise Pool CoD, keep the default value of No.
 – If there is not Enterprise Pool CoD or On/Off CoD, PowerHA manages only the server’s permanent resources through DLPAR, so also keep the default value.
This option can be modified later in the Change/Show Default Cluster Tunables panel, as shown in Figure 6-22 on page 180.
Add Hardware Resource Provisioning to an Application Controller
The panel shown in Figure 6-20 is a selector screen with a selector header that lists all existing application controllers.
           Select Application Controller
 
Move cursor to desired item and press Enter.
 
App1
   App2
 
F1=Help F2=Refresh F3=Cancel
Esc+8=Image Esc+0=Exit Enter=Do
/=Find n=Find Next
Figure 6-20 Select Application Controller menu
To create a Hardware Resource Provisioning for an Application Controller, the list displays only application controllers that do not already have hardware resource provisioning. See Figure 6-21.
To modify or remove a Hardware Resource Provisioning for an Application Controller, the list displays application controllers that already have hardware resource provisioning.
Press Enter on an existing application controller to modify it. The next panel is a dialog screen with a title dialog header and three dialog command options. Each item has a context-sensitive help screen (press F1) and can have an associated list (press F4).
           Add Hardware Resource Provisioning to an Application Controller
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Application Controller Name App1
Use desired level from the LPAR profile No +
Optimal amount of gigabytes of memory []
Optimal number of dedicated processors [] #
 
Optimal number of processing units []
Optimal number of virtual processors []
Figure 6-21 Add Hardware Resource Provisioning to an Application Controller menu
Table 6-8 shows the help for adding hardware resources.
Table 6-8 Context-sensitive help for add hardware resource provisioning
Name and fast path
context-sensitive help (F1)
Application Controller Name
This is the application controller for which you will configure DLPAR and CoD resource provisioning.
Use desired level from the LPAR profile
There is no default value. You must make one of the following choices:
•Enter Yes if you want the LPAR hosting your node to reach only the level of resources indicated by the desired level of the LPAR’s profile.
By choosing Yes, you trust the desired level of LPAR profile to fit the needs of your application controller.
•Enter No if you prefer to enter exact optimal values for memory, processor (CPU), or both. These optimal values should match the needs of your application controller, and enable you to better control the level of resources to be allocated to your application controller.
•Enter nothing if you do not need to provision any resource for your application controller.
For all application controllers having this tunable set to Yes, the allocation performed lets the LPAR reach the LPAR desired value of the profile.
Suppose you have a mixed configuration, in which some application controllers have this tunable set to Yes, and other application controllers have this tunable set to No with some optimal level of resources specified. In this case, the allocation performed lets the LPAR reach the desired value of the profile added to the optimal values.
Optimal amount of gigabytes of memory
Enter the amount of memory that PowerHA SystemMirror will attempt to acquire to the node before starting this application controller.
This Optimal amount of gigabytes of memory value can be set only if the Used desired level from the LPAR profile value is set to No.
Enter the value in multiples of ¼, ½, ¾, or 1 GB. For example, 1 would represent 1 GB or 1024 MB, 1.25 would represent 1.25 GB or 1280 MB, 1.50 would represent 1.50 GB or 1536 MB, and 1.75 would represent 1.75 GB or 1792 MB.
If this amount of memory is not satisfied, PowerHA SystemMirror takes resource group recovery actions to move the resource group with this application to another node. Alternatively, PowerHA SystemMirror can allocate less memory depending on the Start RG even if resources are insufficient cluster tunable.
Optimal number of dedicated processors
Enter the number of processors that PowerHA SystemMirror will attempt to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Dedicated Processing Mode.
This Optimal number of dedicated processors value can be set only if the Used desired level from the LPAR profile value is set to No.
If this number of CPUs is not satisfied, PowerHA SystemMirror takes resource group recovery actions to move the resource group with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
For more information about how to acquire mobile resources at the resource group onlining stage, see 6.6, “Introduction to resource acquisition” on page 195.
For more information about how to release mobile resources at the resource group offlining stage, see 6.7, “Introduction to release of resources” on page 204.
Optimal number of processing units
Enter the number of processing units that PowerHA SystemMirror will attempt to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Shared Processing Mode.
This Optimal number of processing units value can be set only if the Used desired level from the LPAR profile value is set to No.
Processing units are specified as a decimal number with two decimal places, ranging 0.01 - 255.99.
This value is only used on nodes that support allocation of processing units.
If this amount of CPUs is not satisfied, PowerHA SystemMirror takes resource group recovery actions to move the resource group with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
For more information about how to acquire mobile resources at the resource group onlining stage, see 6.6, “Introduction to resource acquisition” on page 195.
For more information about how to release mobile resources at the resource group offlining stage, see 6.7, “Introduction to release of resources” on page 204.
Optimal number of virtual processors
Enter the number of virtual processors that PowerHA SystemMirror will attempt to allocate to the node before starting this application controller.
This attribute is only for nodes running on LPAR with Shared Processing Mode.
This Optimal number of dedicated or virtual processors value can be set only if the Used desired level from the LPAR profile value is set to No.
If this number of virtual processors is not satisfied, PowerHA SystemMirror takes resource group recovery actions to move the resource group with this application to another node. Alternatively, PowerHA SystemMirror can allocate fewer CPUs depending on the Start RG even if resources are insufficient cluster tunable.
To modify an application controller configuration, select Change/Show. The next panel is the same selector screen as shown in Figure 6-21 on page 177. Press Enter on an existing application controller to modify it. The next panel is the same dialog screen (Figure 6-21 on page 177) as shown previously, (except the title, which is different).
To delete an application controller configuration, select Remove. The next panel is the same selector screen shown previously. Press Enter on an existing application controller to remove it.
If Use desired level from the LPAR profile is set to No, then at least the memory (Optimal amount of gigabytes of memory) or CPU (Optimal number of dedicated or virtual processors) setting is mandatory.
6.2.5 Change/Show Default Cluster Tunable
Start smit sysmirror. Select Cluster Applications and Resources → Resources → Configure User Applications (Scripts and Monitors) → Resource Optimized High Availability → Change/Show Default Cluster Tunables. The next panel (Figure 6-22) is a dialog screen with a title dialog header and seven dialog command options. Each item has a context-sensitive help screen (press F1) and can have an associated list (press F4). Its fast path is cm_cfg_def_cl_tun.
            Change/Show Default Cluster Tunables
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Dynamic LPAR
Start Resource Groups even if resources are No +
insufficient
Adjust Shared Processor Pool size if required No +
Force synchronous release of DLPAR resources No +
On/Off CoD
I agree to use On/Off CoD and be billed for Yes +
extra costs
Number of activating days for On/Off CoD requests [30]
Figure 6-22 Change/Show Default Cluster Tunables menu
Table 6-9 shows the help for the cluster tunables.
Table 6-9 Context-sensitive help for Change/Show Default Cluster Tunables
Name and fast path
context-sensitive help (F1)
Start Resource Groups even if resources are insufficient
Enter Yes to have PowerHA SystemMirror start Resource Groups even if resources are insufficient. This can occur when the total requested resources exceed the LPAR profile’s maximum or the combined available resources. Thus the best-can-do allocation is performed.
Enter No to prevent starting Resources Groups with insufficient resources. Resource Groups can end in an error state if resources are insufficient.
The default is No.
Adjust Shared Processor Pool size if required
Enter Yes to authorize PowerHA SystemMirror to dynamically change the user-defined Shared-Processors Pool boundaries, if necessary. This change can occur only at takeover, and only if CoD resources have been activated for the CEC, so that changing the maximum size of a particular Shared-Processors Pool is not done to the detriment of other Shared-Processors Pools.
The default is No.
Force synchronous release of DLPAR resources
Enter Yes to have PowerHA SystemMirror release CPU and memory resources synchronously. For example, if the client needs to free resources on one side before they can be used on the other side. By default, PowerHA SystemMirror automatically detects the resource release mode by looking if Active and Backup nodes are on the same or different CECs. A leading practice is to have asynchronous release in order not to delay the takeover.
The default is No.
I agree to use On/Off CoD and be billed for extra costs
Enter Yes to have PowerHA SystemMirror use On/Off Capacity On Demand (On/Off CoD) to obtain enough resources to fulfill the optimal amount requested. Using On/Off CoD requires an activation code to be entered on the Hardware Management Console (HMC) and can result in extra costs due to the usage of the On/Off CoD license.
The default is No.
Number of activating days for On/Off CoD requests
 
Enter a number of activating days for On/Off CoD requests. If the requested available resources are insufficient for this duration, then the longest-can-do allocation is performed. We try to allocate the amount of resources requested for the longest duration. To do that we consider the overall resources available: This number is the sum of the On/Off CoD resources already activated but not yet used, and the On/Off CoD resources not yet activated.
The default is 30.
6.3 New PowerHA SystemMirror verification enhancement for ROHA
The ROHA function allows PowerHA SystemMirror to automatically or manually check environment discrepancies. The clverify tool has been improved to check ROHA-related configuration integrity.
Customers will be able to use the verification tool to ensure that their environment is correct with regard to their ROHA setup. Discrepancies will be called out by PowerHA SystemMirror, and the tool assists customers to correct the configuration if possible.
The results will appear in the following files:
•The /var/hacmp/log/clverify.log file
•The /var/hacmp/log/autoverify.log file
The user is actively notified of critical errors. Distinction can be made between errors that are raised during configuration and errors that are raised during cluster synchronization.
As a general principal, any problems that are detected at configuration time should be presented as warnings instead of errors.
Another general principle is that PowerHA SystemMirror checks only what is being configured at configuration time and not the whole configuration. PowerHA SystemMirror checks whole configuration at verification time.
For example, when adding a new HMC, you check only the new HMC (verify that it is pingable, at an appropriate software level, and so on) and not all of the HMCs. Checking the whole configuration can take some time and is done at verify and sync time rather than each individual configuration step.
General verification
Table 6-10 shows the general verification list.
Table 6-10 General verification list
Item
Configuration time
Synchronization
time
Check that all RG active and standby nodes are on different CECs. This enables the asynchronous mode of releasing resources.
Info
Warning
This code cannot run on an IBM Power4.
Error
Error
HMC communication verification
Table 6-11 shows the HMC communication verification list.
Table 6-11 HMC communication verification list
Item
Configuration time
Synchronization
time
Only one HMC is configured per node.
None
Warning
Two HMCs are configured per node.
None
OK
One node is without HMC (if ROHA only).
None
Error
Only one HMC per node can be pinged.
Warning
Warning
Two HMCs per node can be pinged.
OK
OK
One node has a non-pingable HMC.
Warning
Error
Only one HMC with password-less SSH communication exists per node.
Warning
Warning
Two HMCs with password-less SSH communication exist per node.
OK
OK
One node exists with non-SSH accessible HMC.
Warning
Error
Check that all HMCs share the same level (the same version of HMC).
Warning
Warning
Check that all HMCs administer the CEC hosting the current node. It is suggested to configure two HMCs administering the CEC hosting the current node. If not, PowerHA gives a warning message.
Warning
Warning
Check if the HMC level supports FSP Lock Queuing.
Info
Info
Capacity on demand verification
Table 6-12 shows the capacity on demand verification.
Table 6-12 Capacity on demand verification
Item
Configuration Time
Synchronization
Time
Check that all CECs are CoD capable.
Info
Warning
Check if CoD is enabled.
Info
Warning
Power enterprise pool verification
Table 6-13 shows the enterprise pool verification list.
Table 6-13 Power enterprise pool verification
Item
@info
@Sync
Check that all CECs are Enterprise Pool capable.
Info
Info
Determine which HMC is the master, and which HMC is non-master.
Info
Info
Check that the nodes of the cluster are on different pools. This enables the asynchronous mode of releasing resources.
Info
Info
Check that all HMCs have level 7.8 or later.
Info
Warning
Check that the CEC has unlicensed resources.
Info
Warning
Resource provisioning verification
Table 6-14 shows the resource provisioning verification information.
Table 6-14 Resource provisioning verification
Item
@info
@Sync
Check that for one given node the total of optimal memory (of RG on this node) added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that for one given node the total of optimal CPU (of RG on this node) added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that for one given node the total of optimal PU (of RG on this node) added to the profile’s minimum does not exceed the profile’s maximum.
Warning
Error
Check that the total processing units do not break the minimum processing units per virtual processor ratio.
Error
Error
6.4 Planning for one ROHA cluster environment
Before completing the ROHA configuration, read the following considerations.
6.4.1 Consideration before ROHA configuration
This section describes a few considerations to know before a ROHA configuration.
Tips for Enterprise Pool
If you have ordered IBM Power Systems Enterprise Pool license for your servers, and you want to use the resource with your PowerHA SystemMirror cluster, then you must create the Enterprise Pool manually.
Before you create the Enterprise Pool, get the configuration extensible markup language (XML) file from the IBM CoD, and the deactivation code from the IBM CoD project office on the following website:
The configuration XML file is used to enable and generate mobile resources.
The deactivation code is used to deactivate some of the permanent resources to inactive mode. The number is the same independent of how many mobile resources are on the server’s order.
For example, in one order, there are two Power Servers. Each one has 16 static CPUs, 8 mobile CPUs, and 8 inactive CPUs, for a total of 32 CPUs. When you power them on the first time, you can see that each server has 24 permanent CPUs, 16 static CPUs plus 8 mobile CPUs.
After you create the Enterprise Pool with the XML configuration file, you see that there are 16 mobile CPUs generated in the Enterprise Pool, but the previous 8 mobile CPUs are still in permanent status in each server. This results in the server’s status being different from its original order. This will bring some issues in future post-sales activities.
There are two steps to complete the Enterprise Pool implementation:
1. Create the Enterprise Pool with the XML configuration file.
2. Deactivate some permanent resources (the number is the same with mobile resources) to inactive with the deactivation code.
After you finish these two steps, each server has 16 static CPUs and 16 inactive CPUs, and the Enterprise Pool has 16 mobile CPUs. Then the mobile CPUs can be assigned to each of the two servers through the HMC graphical user interface or the command-line interface.
 
Note: These two steps will be combined into one step in the future. As of the time of writing, you need to perform each step separately.
How to get the deactivation code and use it
The following steps explain how to get the deactivation code and how to use it:
1. Send an email to the IBM CoD project office ([email protected]). You need to provide the following information or attach the servers order:
 – Customer name
 – Each server’s system type and serial number
 – Configuration XML file
2. In reply to this note, you will receive from the Capacity on Demand project office a de-activation code for the servers. The de-activation code will lower the number of activated resources to align it with your server order.
 
Note: This de-activation code updates the IBM CoD website after you receive the note. This de-activation code has RPROC and RMEM. RPROC is for reducing processor resources, RMEM is for reducing memory resources.
3. Enter this de-activation code in the corresponding servers through the HMC as shown in Figure 6-23 (shows the menu to Enter CoD Code).
Figure 6-23 Menu to Enter CoD Code
4. After entering the de-activation code, you need to send a listing of the updated VPD (Vital Product Data) output to the Capacity on Demand Project office at [email protected].
Collect the VPD using the HMC command line instruction, as shown in Example 6-1.
Example 6-1 Collecting the VPD information case
Collect the VPD using the HMC command line instruction for every server:
Processor:lscod -m your_system_name -t code -r proc -c mobile
Memory:lscod -m your_system_name -t code -r mem -c mobile
5. With the receipt of the lscod profile, the Project Office will update the CoD database records and close out your request.
For more information about how to use the configuration XML file to create Power Enterprise Pool and some management concept, see the publication Power Enterprise Pools on IBM Power Systems, REDP-5101 on the following website:
Configure redundant HMCs or add EP’s master and backup HMC
In 6.12, “Hardware Management Console (HMC) high availability introduction” on page 244, we introduce HMC high availability design in PowerHA SystemMirror. For the ROHA solution, the HMC is critical, so configuring redundant HMCs is advised.
If there is a Power Enterprise Pool configured, we suggest configuring backup HMC for Enterprise Pool and adding both of them into PowerHA SystemMirror with clmgr add hmc <hmc>’ command or through the SMIT menu. Thus, PowerHA SystemMirror can provide the fallover function if the master HMC fails. 6.12.1, “Switch to the backup HMC for the Power Enterprise Pool” on page 246 introduces some prerequisites when you set up the Power Enterprise Pool.
 
Note: At the time of writing this publication, Power Systems Firmware supports a pair of HMCs to manage one Power Enterprise Pool: One is in master mode, and the other one is in backup mode.
 
Note: At the time of writing this publication, for one Power Systems server, IBM only supports at most two HMCs to manage it.
Verify communication between Enterprise Pool’s HMC IP and AIX LPARs
If you want PowerHA SystemMirror to control Power Systems Enterprise Pool’s mobile resource for resource group automatically, you must be able to ping the HMC’s host name from AIX environment. For example, in our testing environment, the master HMC and backup HMC of Power Enterprise Pool is: e16hmc1 and e16hmc3. You can get the information using the clmgr view report roha command in AIX or using the lscodpool in the HMC command line, as shown in Example 6-2 and Example 6-3.
Example 6-2 Show HMC information with clmgr view report roha through AIX
...
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1' --> Master HMC name of EPCoD
Backup HMC: 'e16hmc3' --> Backup HMC name of EPCoD
Enterprise pool memory
Activated memory: '100' GB
Available memory: '100' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Example 6-3 Show EPCoD HMC information with lscodpool through HMC
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=In compliance,sequence_num=41,master_mc_name=e16hmc1,master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=1,unreturned_mobile_procs=0,mobile_mem=102400,avail_mobile_mem=60416,unreturned_mobile_mem=0
Before PowerHA SystemMirror acquires the resource from EPCoD or releases the resource back to EPCoD, PowerHA tries to check if the HMC is accessible with the ping command. So it is required that AIX can perform the resolution between the IP address and the host name. You can use /etc/hosts or Domain Name System (DNS) or other technology to achieve it. For example, on AIX, run ping e16hmc1 and ping e16hmc3 to check if the resolution works.
If the HMCs are in the DNS configuration, we suggest configuring these HMCs into PowerHA SystemMirror using their names, and not their IPs.
Enter the On/Off CoD code before using the resource
If you purchased the On/Off CoD code and want to use it with PowerHA SystemMirror, you need to enter the code to active it before use it. The menu is shown in Figure 6-23 on page 185.
No restriction about deployment combination with Enterprise Pool
In one PowerHA SystemMirror cluster, there is no restriction for its nodes deployment with Enterprise Pool CoD (EPCoD):
•It supports all the nodes in one server and share mobile resource from one EPCoD.
•It supports the nodes in different servers and share one EPCoD.
•It supports the nodes in different servers and in different EPCoD.
•It supports the nodes in different servers and some of them in EPCoD, and others has no EPCoD.
No restriction about LPAR’s CPU type combination in one cluster
One PowerHA SystemMirror cluster supports:
•All nodes are dedicated processor mode.
•All nodes are shared processor mode.
•Some of them are dedicated processor mode and others are shared.
In Figure 6-24, before the application starts, PowerHA SystemMirror check current LPAR’s processor mode, if it is dedicated, then 2 available CPU is its target, if it is shared mode, then 1.5 available CPUs and 3 available VPs is its target.
Add Hardware Resource Provisioning to an Application Controller
 
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
[Entry Fields]
* Application Controller Name AppController1
 
Use desired level from the LPAR profile No +
Optimal amount of gigabytes of memory [30]
 
Optimal number of dedicated processors [2] #
 
Optimal number of processing units [1.5]
Optimal number of virtual processors [3]
 
Figure 6-24 Mixed CPU type in one PowerHA SystemMirror cluster
Recommendation after changing partition’s LPAR name
If you change one partition’s LPAR name, the profile is changed, but AIX does not recognize this change automatically. You need to shut down the partition and activate it with its profile (AIX IPL process, Initial Program Load), then after restart, the LPAR name information can be changed.
PowerHA SystemMirror gets the LPAR name from the uname -L command’s output and take this name to do DLPAR operations through the HMC.
 
Note: There is one enhancement to support DLPAR name update for AIX commands such as uname -L or lparstat -i. The requirements are as follows:
•Hardware firmware level SC840 or later (for E870 and E880)
•AIX 7.1 TL4 or 7.2 or later
•HMC V8 R8.4.0 (PTF MH01559) with mandatory efix (PTF MH01560)
Build password-less communication from AIX nodes to HMCs
In order for LPARs to communicate with the HMC, they must use SSH. All the LPAR nodes must have SSH correctly set up.
Setting up SSH for password-less communication with the HMC requires that the user run ssh-keygen on each LPAR node to generate a public and private key pair. The public key must then be copied to the HMC’s public authorized keys file. This will allow ssh from the LPAR to contact the HMC without the need for typing in a password. Example 6-4 is one example to set up HMC password-less communication.
Example 6-4 Setting up the HMC password-less communication
# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa):
Created directory '//.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
70:0d:22:c0:28:e9:71:64:81:0f:79:52:53:5a:52:06 root@epvioc3
The key's randomart image is:
…
 
# cd /.ssh/
# ls
id_rsa id_rsa.pub
# export MYKEY='cat /.ssh/id_rsa.pub'
# ssh [email protected] mkauthkeys -a "$MYKEY"
The authenticity of host '172.16.15.42 (172.16.15.42)' can't be established.
RSA key fingerprint is b1:47:c8:ef:f1:82:84:cd:33:c2:57:a1:a0:b2:14:f0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.15.42' (RSA) to the list of known hosts.
Keep Sync turned OFF for the Sync current configuration Capability setting
There is one option in the LPAR if it needs to enable sync of the current configuration. With the ROHA solution, one LPAR’s running CPU and memory size would be resized, if this feature is enabled, the wanted value of profile would be resized too. This will bring confusion to the system administrator. So we suggest disabling this feature in a ROHA environment (Figure 6-25 on page 189).
Figure 6-25 Sync current configuration Capability
Consideration of setting LPAR’s Minimum and Maximum parameters
When you configure an LPAR on the HMC (outside of PowerHA SystemMirror), you provide LPAR minimum and LPAR maximum values for the number of CPUs and amount of memory.
The stated minimum values of the resources must be available when an LPAR node starts. If more resources are available in the free pool on the frame, an LPAR can allocate up to the stated wanted values. During dynamic allocation operations, the system does not allow the values for CPU and memory to go below the minimum or above the maximum amounts specified for the LPAR.
PowerHA SystemMirror obtains the LPAR minimums and LPAR maximums amounts and uses them to allocate and release CPU and memory when application controllers are started and stopped on the LPAR node.
In planning stage, we need to consider how many resources are needed to satisfy all the resource groups online carefully and set LPAR’s minimal and maximum parameter correctly.
Using pre-event and post-event scripts
Existing pre-event and post-event scripts that you maybe using in a cluster with LPARs (before using the CoD integration with PowerHA SystemMirror) might need to be modified or rewritten, if you plan to configure CoD and DLPAR requirements in PowerHA SystemMirror.
Keep in mind the following considerations:
•PowerHA SystemMirror performs all the DLPAR operations before the application controllers are started, and after they are stopped. You might need to rewrite the scripts to account for this.
•Because PowerHA SystemMirror takes care of the resource calculations, requests additional resources from the DLPAR operations and, if allowed, from CUoD, you can get rid of the portions of your scripts that do that.
•PowerHA SystemMirror considers only the free pool on a single frame. If your cluster is configured within one frame, then modifying the scripts as stated above is sufficient.
•However, if a cluster is configured with LPAR nodes that are on two frames, you might still require the portions of the existing pre-event and post-event scripts that deal with dynamically allocating resources from the free pool on one frame to the node on another frame, should the application requires these resources.
About elapsed time of DLPAR operation
When you plan a PowerHA SystemMirror cluster with ROHA feature, DLPAR release time needs to be considered.
While initially bringing up the Resource Group online, PowerHA SystemMirror must wait for all the resources acquisition before it, then can start up user’s application.
While performing a takeover (Fallover to next priority node, for example), PowerHA SystemMirror tries to perform some operations (DLPAR or adjust CoD and EPCoD resource) in parallel the release of resources on source node and the acquisition of resources on target node if user allows it in tunables (the value of Force synchronous release of DLPAR resources is No).
Table 6-15 shows testing result of DLPAR operation, the result maybe different in other environment.
There is one LPAR, its current running CPU resource size is 2C, and the running memory resource size is 8 GB. The DLPAR operation includes add and remove.
Table 6-15 Elapsed time of DLPAR operation
Incremental Value
By DLPAR
Add CPU
(in seconds)
Add Memory
(in seconds)
Remove CPU
(in seconds)
Remove Memory
(in minutes and seconds)
2C and 8 GB
5.5 s
8 s
6 s
88 s (1 m 28 s)
4C and 16 GB
7 s
12 s
9.8 s
149 s (2 m 29 s)
8C and 32 GB
13 s
27 s
23 s
275 s (4 m 35 s)
16C and 64 GB
18 s
34 s
33 s
526 s (8 m 46 s)
32C and 128 GB
24 s
75 s
52 s
1010 s (16 m 50 s)
48C and 192 GB
41 s
179 s
87 s
1480 s (24 m 40 s)
AIX ProbeVue maximum pinned memory setting
ProbeVue is one dynamic tracing facility of AIX. You can use it for both performance analysis and problem debugging. ProbeVue uses the Vue programming language to dynamically specify trace points and provide the actions to run at the specified trace points. This feature is enabled by default. There is one restriction between ProbeVue’s maximum pinned memory and DLPAR remove memory operation:
Max Pinned Memory For ProbeVue tunable would cross the 40% limit of system running memory.
For example, you configured one profile for LPAR with 8 GB (minimum) and 40 GB (wanted), at the first you activate this LPAR, the maximum pinned memory of ProbVue is set to 4 GB (10% of system running memory) as shown in Example 6-5.
From AIX 7.1 TL4 onwards, the tunables are derived based on the available system memory. MAX pinned memory is set to 10% of the system memory. It cannot be adjusted itself when you restart the operating system or adjusting the memory size with the DLPAR operation.
Example 6-5 Current maximum pinned memory for ProbeVue
# probevctrl -l
Probevue Features: on --> ProbeVue is enable at this time
MAX pinned memory for Probevue framework(in MB): 4096 --> this is the value we are discussing
...
Now if you want to remove memory from 40 GB to 8 GB use the following command:
chhwres -r mem -m r1r9m1-9117-MMD-1038B9P -o r -p ITSO_S2Node1 -q 32768
The command fails with the error shown in Example 6-6.
Example 6-6 Error information when you remove memory through DLPAR
hscroot@e16hmc3:~> chhwres -r mem -m r1r9m1-9117-MMD-1038B9P -o r -p ITSO_S2Node1 -q 32768
HSCL2932 The dynamic removal of memory resources failed: The operating system prevented all of the requested memory from being removed. Amount of memory removed: 0 MB of 32768 MB. The detailed output of the OS operation follows:
 
0930-050 The following kernel errors occured during the
DLPAR operation.
 
0930-023 The DR operation could not be supported by one or more kernel extensions.
 
Consult the system error log for more information
....
Please issue the lshwres command to list the memory resources of the partition and to determine whether or not its pending and runtime memory values match. If they do not match, problems with future memory-related operations on the managed system may occur, and it is recommended that the rsthwres command to restore memory resources be issued on the partition to synchronize its pending memory value with its runtime memory value.
From AIX, the error report also generates some error information, as shown in Example 6-7 and Example 6-8.
Example 6-7 AIX error information when remove memory through DLPAR
47DCD753 1109140415 T S PROBEVUE DR: memory remove failed by ProbeVue rec
252D3145 1109140415 T S mem DR failed by reconfig handler
Example 6-8 Detailed information about DR_PVUE_MEM_REM_ERR error
LABEL: DR_PVUE_MEM_REM_ERR
IDENTIFIER: 47DCD753
 
Date/Time: Mon Nov 9 14:04:56 CST 2015
Sequence Number: 676
Machine Id: 00F638B94C00
Node Id: ITSO_S2Node1
Class: S
Type: TEMP
WPAR: Global
Resource Name: PROBEVUE
 
Description
DR: memory remove failed by ProbeVue reconfig handler
 
Probable Causes
Exceeded one or more ProbeVue Configuration Limits or other
 
Failure Causes
Max Pinned Memory For Probevue tunable would cross 40% limit
 
Recommended Actions
Reduce the Max Pinned Memory For Probevue tunable
 
Detail Data
DR Phase Name
PRE
Current System Physical Memory
            42949672960 -->> this is 40GB which is current running memory size
Memory requested to remove
            34359738368 -->> this is 32GB which want to remove
ProbeVue Max Pinned Memory tunable value
4294967296 -->> this is 4GB which is current maximum pinned memory for ProbeVue.
In the ROHA solution, it is possible that PowerHA SystemMirror will remove memory to a low value, like in the procedure of automatic resource release after OS failure, so we have the following comment to avoid this situation:
1. If you want to enable the ProbeVue component, set the maximum pinned memory less or equal to (40% *minimum memory value of one LPAR’s profile). For example, in this case, the minimum memory size is 8 GB, so 40% is 3276.8 MB.
Therefore, we can set the maximum pinned memory size with the command, as shown in Table 6-9.
Example 6-9 Change max_total_mem_siize
# probevctrl -c max_total_mem_size=3276
Attention: The command "/usr/sbin/bosboot -a" must be run for the change to take effect in the next boot.
This means to set it to 3276 MB, which is less than 3276.8 (8 GB*40%). This change will take effect immediately. But if you want this change to take effect after the next boot, you need to run /usr/sbin/bosboot -a before the reboot.
2. If you do not want the ProbeVue component online, you can turn off it with the command shown in Example 6-10.
Example 6-10 Turn off ProveVue
# probevctrl -c trace=off
Attention: The command "/usr/sbin/bosboot -a" must be run for the change to take effect in the next boot.
This change will take effect immediately. But if you want this change to take effect after the next boot, you need to run /usr/sbin/bosboot -a before the reboot.
6.4.2 Configuration steps for ROHA
After finishing all of the preparations and considerations outside of PowerHA SystemMirror, then we do the configuration with PowerHA SystemMirror.
First, you need to configure generic elements for PowerHA SystemMirror cluster:
•Cluster name
•Nodes in Cluster
•CAA repository disk
•Shared VG
•Application Controller
•Service IP
•Resource Group
•Other user-defined contents such as pre-event or post-event
Then start to configure the ROHA-related elements:
 – At least one HMC, two HMC is better
 – Optionally change cluster hmc tunables
 – Optionally change HMC at node or site level
•Optimal resources for each application controller (see 6.2.4, “Hardware resource provisioning for application controller” on page 175)
•Change cluster ROHA tunables optionally (see 6.2.5, “Change/Show Default Cluster Tunable” on page 180)
•Run Verify and Synchronize, review the warning or error message and fix them
•Show ROHA report with the clmgr view report roha command to review
6.5 Resource acquisition and release process introduction
This section introduces the steps of the resource acquisition and release in a ROHA solution.
6.5.1 Steps for allocation and for release
Figure 6-26 shows the steps of allocation and release. During fallover, resources are released on the active node (same as stopping resource groups – in red on the diagram) and resources are acquired on the backup node (same as starting resource groups – in green on the diagram). Figure 6-26 shows the process when CoD Pool and Enterprise Pool are used. On some CECs, none or only one or both of those reservoirs can be used.
Figure 6-26 Allocations and releases steps
For the resource releasing process, in some cases, PowerHA SystemMirror tries to return EPCoD resource before doing the DLPAR remove operation from the LPAR, and this generates unreturned resource on this server. This is an asynchronous process and is helpful to speed up resource group takeover. The unreturned resource will be reclaimed after the DLPAR remove operation is completed.
6.6 Introduction to resource acquisition
Figure 6-27 shows the process of acquisition for memory and processor. Resources are acquired together for a list of applications. It is a four steps process:
1. Query (yellow boxes): Required resources are computed based on the LPAR configuration, and the information provided by PowerHA SystemMirror state (if applications are currently running) and applications. Then, the script reaches out the HMC to get information about available ROHA resources.
2. Compute (purple box): Based on this information, PowerHA SystemMirror figures out the total amount of required resources that are needed on the node, for the list of resource groups that are to be started on the node.
3. Identify (green box): Figures out how to perform this allocation for the node, by looking at each kind of allocations to be made: Which part must come from Enterprise Pool CoD resources, and which part must come from On/Off CoD resources to allocate some supplementary resources to the CEC, and which amount of resources must be allocated from the CEC to the LPAR through a DLPAR operation.
4. Apply (orange boxes): After these decisions are made, the script reaches out the HMC to acquire resources. First, allocate Enterprise Pool CoD resources and activate On/Off CoD resources, and then allocate all DLPAR resources. Each amount of resources is persisted in the HACMPdynresop ODM object for release purposes.
Figure 6-27 Four steps to acquire resource
There are many reasons for success. The script immediately returns if the applications are not configured with optimal resources. The script also exits if there are already enough resources allocated. Finally, the script exits when the entire process of acquisition succeeds.
However, the script can fail and return an error if one of the following situations occurs:
•Maximum LPAR size as indicated in the LPAR profile is exceeded and Start RGs even if resources are insufficient tunable is set to No.
•Shared processor pool size is exceeded and Adjust SPP size if the required tunable is set to No.
•There are not enough free resources on the CEC, or on the Enterprise Pool CoD, or on the On/Off CoD, and the Start RGs even if resources are insufficient tunable is set to No.
•Any one step of the acquisition fails (see the previous four steps). Thus, successful actions previously performed are rolled back, and the node is reset to its initial allocation state as read in the HACMPdynresop ODM object.
In a shared processor partition, more operations must be done. For example, take into account both virtual CPUs and processing units instead of only a number of processors. To activate On/Off CoD resources or allocate Enterprise Pool CoD resources, decimal processing units should be converted to integers and decimal gigabytes of memory should be converted to integers.
On shared processor pool partitions, the maximum pool size can be automatically adjusted, if necessary and if authorized by the user.
6.6.1 Query
In the query step, PowerHA SystemMirror gets the following information:
Getting information from onlining apps
Onlining applications see the ones being brought online. It is achieved by summing values returned by an ODM request to the HACMPserver object containing the applications resources provisioning.
Getting information from running apps
Running applications see the ones currently running on the node. It is achieved by calling the clRGinfo command to obtain all the running applications and summing values returned by an ODM request on all those applications.
Getting LPAR information from HMC
The minimum, maximum, and currently allocated resources for the partition are listed through the HMC command-line interface lshwres.
Getting the DLPAR resource information from HMC
Some people think only the available resource (the query method is shown in Table 6-16) can be used for the DLPAR operation.
Table 6-16 Get server’s available resource from HMC
Memory
lshwres -m <cec> --level sys -r mem -F curr_avail_sys_mem
CPU
lshwres -m <cec> --level sys -r proc -F curr_avail_sys_proc_units
Strictly speaking, it is not correct. Two kinds of cases need to be considered:
•There are stopped partitions on the CEC
A stopped partition still keep its resources because the resources do not appear in the Available of the CEC. As a matter of fact, the resources are available for other LPARs. Therefore, if you have stopped a partition on the CEC, the resource that stopped the partition needs to be available for the DLPAR operation.
Figure 6-28 shows that there is no available CPU resource using the lshwres command to query. But in fact, there is 0.5 CPU which LPAR rar1m34 is holding, and this LPAR is in Not Activated status. The free CPU resource should be 0.5 CPU (0+0.5).
Figure 6-28 Describe the difference between available resource and free resource
•There are uncapped mode partitions in the CEC
In an uncapped shared processor partition, considering only the maximum processor unit is not correct.
Consider the following case (Example 6-11), where one LPAR’s profile includes the following configuration:
Example 6-11 Uncapped mode partition example
Min processor unit: 0.5
Assigned processor unit: 1.5
Maximum processor unit: 3
Min virtual processor: 1
Assigned virtual processor: 6
Maximum virtual processor: 8
This LPAR could get up to six processor units if the workload increases, and if these resources are available in the CEC. Also, this value is above the limit set by the Maximum processor unit, which has a value of 3.
But in any case, allocation beyond the limit of the maximum processor unit is something that is performed at the CEC level, and that cannot be controlled at the PowerHA SystemMirror level.
But it is true that the calculation of available resources could consider what is really being used in the CEC, and should not consider the Maximum processor unit as an intangible maximum. The real maximum comes from the number of Assigned Virtual Processor.
PowerHA SystemMirror supports the uncapped mode, but does not play a direct role in this support, because it is performed at the CEC level. There is no difference in uncapped mode as compared with the capped mode for PowerHA SystemMirror.
Based on the previous considerations, the formula to calculate free resources (memory and processor) for the DLAR operation is shown in Figure 6-29.
Figure 6-29 Formula to calculate free resource of one CEC
 
Note: You read the level of configured resources (configurable_sys_mem in the formula), and you remove from that the level of reserved resources (sys_firmware_mem in the formula), then you end up with the level of resources needed to run one started partition.
Moreover, when computing the free processing units of a CEC, you consider the reserved processing units of any used Shared Processor Pool (the reserved in the formula).
Getting On/Off CoD resource information from the HMC
The available On/Off CoD resources for the CEC is listed through the HMC command-line interface with the lscod command. The state should be Available or Running (a request is ongoing). Table 6-17 shows the commands that PowerHA SystemMirror uses to get On/Off resource information. You do not need to run these commands.
Table 6-17 Get On/Off CoD resources’ status from HMC
Memory
lscod -m <cec> -t cap -c onoff -r mem -F mem_onoff_state:avail_mem_for_onoff
CPU
lscod -m <cec> -t cap -c onoff -r proc -F proc_onoff_state:avail_proc_for_onoff
Getting Power Enterprise Pool resource information from the HMC
The available Enterprise Pool CoD resources for the pool is listed through the HMC command-line interface with the lscodpool command. Table 6-18 shows the commands that PowerHA SystemMirror uses to get the EPCoD information. You do not need to run these commands.
Table 6-18 Get the EPCoD available resource from the HMC
Memory
lscodpool -p <pool> --level pool -F avail_mobile_mem
CPU
lscodpool -p <pool> --level pool -F avail_mobile_procs
 
Note: If the execution of this command fails (either because the link is down or other errors), after the last retry but before trying another HMC, PowerHA SystemMirror changes the master HMC for its Enterprise Pool.
6.6.2 Resource computation
After the query step, PowerHA SystemMirror starts performing computations in order to satisfy PowerHA SystemMirror application controller’s needs. It is likely that some resources have to be allocated from the CEC to the LPAR. The diagram Figure 6-30 on page 200 shows the computation of the amount of resources to be allocated to the partition. This computation is performed for all types of resources, and it takes into account the following:
•The configuration of the partition (minimum, current and maximum amount of resources).
•The optimal resources that are configured for the applications currently running on the partition.
•The optimal resources that are configured for the applications that are being brought online.
In Figure 6-30 on page 200, case 2b is the normal case. The currently allocated resources level matches the blue level, which is the level of resources for the application controllers currently running. PowerHA SystemMirror will add the yellow amount to the blue amount.
But in some cases, where these two levels do not match, we consider having a “start afresh” policy. This policy performs a readjustment of the allocation to the exact needs of the currently running application controllers added to the application controllers that are being brought online (always provides an optimal amount of resources to application controllers). Those alternative cases can occur when the user has manually released (case 2a) or acquired (case 2c) resources.
Figure 6-30 shows the computation policy in the resource acquisition process.
Figure 6-30 Computation policy in the resource acquisition process
•In case 1, PowerHA SystemMirror just keeps current the allocated level to satisfy the needs. This can occur when a partition is at its profile’s wanted level, which is greater than its profile’s minimum.
•In case 2a, the readjustment consists in allocating only the missing part of the application controllers that are being brought online.
•In case 2c, the readjustment consists in allocating the missing part of the application controllers currently running added to the application controllers that are being brought online.
•In case 3, the needed resources cannot be satisfied by this partition. It exceeds the partition profile’s maximum. In that particular case, two behaviors can happen here depending on the Start RGs even if resources are insufficient tunable. If enabled, PowerHA SystemMirror tries to allocate all that can be allocated, raises a warning, and goes on. If disabled, PowerHA SystemMirror stops and returns an error.
In shared processor partitions, both virtual CPUs and processing units are computed. In shared processor partitions that are part of a Shared Processor Pool, computation need is checked against the Shared Processor Pool size. If it is lesser, everything is fine, and the process continues. If it is greater, an Adjust SPP size if required tunable is set to No, and the process stops and return an error. Otherwise, it raises a warning, changes the pools size to the new size, and goes on.
6.6.3 Identify the method of resource allocation
In the resource compute step, the amount of resources that are needed by the LPAR has been computed, so now you need to identify how to achieve the wanted amount. PowerHA SystemMirror considers multiple strategies in the following order:
1. Consider the CEC current free pool for DLPAR operations. This section explains how these available resources are computed.
2. If resources are still insufficient, consider the Enterprise Pool of resources if any.
3. If resources are still insufficient, consider the CoD pool of resources if a license has been activated, and if any On/Off CoD resources are available.
When the right strategy has been chosen, there are two types of resource allocations to be done:
1. Allocation to the CEC: Resources can come from the Enterprise Pool CoD or the On/Off CoD pools.
2. Allocation to the partition: Resources come from the CEC.
Figure 6-31 on page 202 shows the computation for the DLPAR, CoD and Enterprise Pool CoD amount of resources to acquire. The computation is performed for all types of resources. In shared processor partitions, only processing units are computed this way, and takes into account the following:
•The total amount of resources to acquire for the node (computed previously).
•The available amount of DLPAR resources on the CEC.
•The available amount of On/Off CoD resources on the CEC.
•The available amount of Enterprise Pool CoD resources in the pool the CEC belongs to.
Figure 6-31 shows the identified policy in the resource acquisition process.
Figure 6-31 Identify the policy in resource acquisition process
There are four possible cases:
1. There are sufficient DLPAR resources to fulfill the optimal configuration. No EPCoD resources, nor On/Off CoD resources will be allocated to the CEC. A portion of the available DLPAR resources will be allocated to the node.
2. A portion of available Enterprise Pool CoD resources will be allocated to the CEC, and then all DLPAR resources will be allocated. No On/Off CoD resources will be activated.
Alternative case: If there are no available EPCoD resources, a portion of available On/Off CoD resources will be activated instead, and then all DLPAR resources will be allocated.
3. All available Enterprise Pool CoD resources will be allocated to the CEC, then a portion of On/Off CoD resources will be activated, and then all DLPAR resources will be allocated.
Alternative case: If there are no available EPCoD resources, a portion of available On/Off CoD resources will be activated instead, and then all DLPAR resources will be allocated (as in case 2).
4. All available Enterprise Pool CoD resources will be allocated to the CEC, then all On/Off CoD resources will be activated, and then all DLPAR resources will be allocated.
Alternative case: If the cluster has not been configured to automatically start the resource groups even if resources are insufficient, do not allocate nor acquire any resources since it exceeds the available resources for this CEC and exit on error instead.
In shared processor partitions, PowerHA SystemMirror takes into account the minimum ratio of assigned processing units to assigned virtual processors for the partition that is supported by the CEC. In an IBM POWER6® server, the ratio is 0.1 and in an IBM POWER7® server, the ratio is 0.05.
For example, if the current assigned processing unit in the partition is 0.6 and the current assigned virtual processor is 6, and PowerHA SystemMirror acquires virtual processors, it raises an error because it breaks the minimum ratio rule. The same occurs when PowerHA SystemMirror releases the processing units. PowerHA SystemMirror must compare the expected ratio to the configured ratio.
6.6.4 Acquire the resource
After finishing step 6.6.3, “Identify the method of resource allocation” on page 201, PowerHA SystemMirror performs the acquire operation.
Acquire the Power Enterprise Pool resource
The Enterprise Pool CoD resources are allocated through the HMC command-line interface with the chcodpool command. Table 6-19 shows the commands that PowerHA SystemMirror uses to assign EPCoD resource to one server. You do not need to run these commands.
Table 6-19 Acquire the EPCoD mobile resources
Memory
chcodpool -p <pool> -m <system> -o add -r mem -q <mb_of_memory>
CPU
chcodpool -p <pool> -m <system> -o add -r proc -q <cpu>
Acquire the On/Off CoD resource
On/Off CoD resources are activated through the HMC command-line interface with the chcod command. Table 6-20 shows the commands that PowerHA SystemMirror uses to assign the On/Off CoD resource to one server. You do not need to run these commands.
Table 6-20 Acquire On/Off available resources
Memory
chcod -m <cec> -o a -c onoff -r mem -q <mb_of_memory> -d <days>
CPU
chcod -m <cec> -o a -c onoff -r proc -q <cpu> -d <days>
 
Note: For acquiring the Power Enterprise Pool and the On/Off CoD resources, every amount of memory resources are expressed in MB but aligned in GB of memory (for example 1024 or 4096), and every number of processing units is aligned on the whole upper integer.
All Power Enterprise Pool and On/Off CoD resources that are acquired will be located in CEC’s free pool, and these will be added to the target LPAR using DLPAR automatically.
Acquire the DLPAR resource
DLPAR resources are allocated through the HMC command-line interface with the chhwres command. Table 6-21 shows the commands that PowerHA SystemMirror uses to assign resource from the server’s free pool to one LPAR. You do not need to run these commands.
Table 6-21 Assign resource from server’s free pool to target LPAR
Dedicate Memory
chhwres -m <cec> -p <lpar> -o a -r mem -q <mb_of_memory>
Dedicate CPU
chhwres -m <cec> -p <lpar> -o a -r proc --procs <cpu>
Shared CPU
chhwres -m <cec> -p <lpar> -o a -r proc --procs <vp> --proc_units <pu>
For shared processor partitions in a Shared-Processors Pool that is not the default pool, it might be necessary to adjust the maximum processing units of the Shared Processor Pool. It is performed by the operation, as shown in Example 6-12 through the HMC command-line interface by using the chhwres command. The enablement of this adjustment is authorized or not by a tunable.
Example 6-12 shows the command that PowerHA SystemMirror uses to change Shared-Processor Pool’s maximum processing units. You do not need to run this command.
Example 6-12 DLPAR command line from HMC
chhwres -m <cec> -o s -r procpool --poolname <pool> -a max_pool_proc_units=<pu>
6.7 Introduction to release of resources
When the resource groups are stopped, PowerHA SystemMirror computes the amount of resources to be released and is responsible for performing the release of ROHA resources. There are four steps when releasing resources. These steps are also shown in Figure 6-32 on page 205:
1. Query step, appears in yellow. In this step, PowerHA SystemMirror queries all the information that is needed for following the compute, identify, and release steps.
2. Computes step, appears in purple. In this step, PowerHA SystemMirror computes how many resource need to release through DLPAR. In this step, PowerHA SystemMirror uses a “fit to remaining RGs” policy, which consists in computing amounts of resources to be released by taking into account currently allocated resources and total optimal resources that are needed by resource groups remaining on the node. In any case, and as it was done before, PowerHA SystemMirror does not release more than optimal resources for the RGs being released.
3. Identify step, appears in green. In this step, PowerHA SystemMirror identifies how many resources need to be removed from the LPAR, and identify how many resources need to be released to the On/Off CoD and to the Power Enterprise Pool.
4. Remove resources from the LPAR and release resource from CEC to On/Off CoD and Power Enterprise Pool, appears in orange. In this step, PowerHA SystemMirror performs the DLPAR remove operation and after that, releases On/Off CoD resources and EPCoD resources.
Figure 6-32 Four steps to release resources
6.7.1 Query
In the query step, PowerHA SystemMirror gets the information in the following sections for compute.
Getting information from offlining apps
Offlining applications see the one being brought offline. At this step, check that the release of resources is needed. It means that at least one application has been configured with optimal resources.
Getting information from running apps
Running applications see the ones currently running on the node. It is achieved by calling the clRGinfo binary to obtain all the running applications and summing values that are returned by an ODM request on all those applications.
Getting LPAR information from the HMC
The minimum, maximum and currently allocated resources for the partition are listed through the HMC lshwres command.
Get On/Off CoD resource information from the HMC
The active On/Off CoD resources for the CEC is listed through the HMC command-line interface lscod. Table 6-22 shows the commands that PowerHA SystemMirror uses to get On/Off CoD information. You do not need to run these commands.
Table 6-22 Get On/Off active resources in this server from the HMC
Memory
lscod -m <cec> -t cap -c onoff -r mem -F activated_onoff_mem
CPU
lscod -m <cec> -t cap -c onoff -r proc -F activated_onoff_proc
Getting Power Enterprise Pool resource information from the HMC
The allocated Enterprise Pool CoD resources for the pool is listed through the HMC command-line interface lscodpool. Table 6-23 shows the commands that PowerHA SystemMirror uses to get EPCoD information. You do not need to run these commands.
Table 6-23 Get EPCoD resource information
Memory
lscodpool -p <pool> --level pool -F mobile_mem
lscodpool -p <cec> --level sys --filter "names=server name" -F mobile_mem
CPU
lscodpool -p <pool> --level pool -F mobile_procs
lscodpool -p <cec> --level sys --filter "names=server name" -F mobile_procs
Resource computation
The level of resources to be left on the LPAR is computed using the fit to remaining RGs policy. What is above this level will be released, and it takes into account the following information:
1. The configuration of the LPAR (minimum, current and maximum amount of resources).
2. The optimal resources that are configured for the applications currently running on the LPAR. PowerHA SystemMirror tries to fit to the level of remaining RGs running on the node.
3. The optimal amount of resources of the stopping RGs as you do not de-allocate more than this.
Two cases can happen, as shown in Figure 6-33 on page 207:
1. Release resources to a level that enables the remaining applications to run at optimal level. PowerHA SystemMirror applies the fit to remaining RGs policy here to compute and provide the optimal amount of resources to the remaining applications.
2. Do not release any since the level of currently allocated resources is already under the level that is computed by the fit to remaining RGs policy.
Figure 6-33 Resource computation in releasing process
Release resources from LPAR to CEC
DLPAR resources are released through the HMC command-line interface chhwres. Table 6-24 shows the commands that PowerHA SystemMirror uses to release resources from the LPAR. You do not need to run these commands.
Table 6-24 Release resources from LPAR to CEC through the HMC
Dedicate memory
chhwres -m <cec> -p <lpar> -o r -r mem -q <mb_of_memory>
Dedicate CPU
chhwres -m <cec> -p <lpar> -o r -r proc --procs <cpu>
Shared CPU
chhwres -m <cec> -p <lpar> -o r -r proc --procs <vp> --proc_units <pu>
A timeout is given with the -w option and this timeout is set to the configured value at the cluster level (DLPAR operations timeout) added with 1 minute per GB. So for example to release 100 GB, if default timeout value is set to 10 minutes, the timeout will be set to 110 minutes (10 + 100).
For large memory releases, for example instead of making one 100 GB release request, make rather 10 requests of 10 GB release. You can see the logs in the hacmp.out log file.
Identify the resource to release
The diagram as shown in Figure 6-34 shows three cases of DLPAR, CoD, and Enterprise Pool CoD release for memory and processors.
At release, the de-allocation order is reversed, On/Off CoD resources are preferably released, preventing the user from paying for extra costs. Figure 6-34 shows the process.
Figure 6-34 Identifying the source to release
There are three cases in the identify step:
1. There are no On/Off CoD or Enterprise Pool resources used by the CEC. Therefore, no resources need to be released to On/Off CoD or Enterprise Pool.
2. There are On/Off resources allocated to the CEC. Some of the On/Off CoD resources will be deactivated to the limit of what has been previously activated. In this way, some On/Off CoD resources can be left activated on the CEC if they have been activated outside of PowerHA SystemMirror.
3. There are both On/Off CoD resources and Enterprise Pool resources allocated to the CEC. Then, On/Off CoD resources will be deactivated to the limit of what has been previously allocated to the CEC for the node. And then, Enterprise Pool CoD resources will be returned back to the pool to the limit of what has been previously allocated for the CEC by the node. In this way, some On/Off CoD or Enterprise Pool CoD resources can be left activated/allocated if used outside of PowerHA SystemMirror.
Alternative case: If there are no On/Off CoD resources activated on the CEC, only return back Enterprise Pool resources to the pool.
Generate an “unreturned” resource
In this step, if some EPCoD resource is identified, it is possible for PowerHA SystemMirror to release them to EPCoD immediately, and before the DLPAR remove operation even starts.
PowerHA SystemMirror raises an asynchronous process to do the DLPAR remove operation. PowerHA SystemMirror does not need to wait for the DLPAR operation to complete. So PowerHA SystemMirror on standby mode can bring the online resource groups quickly.
This asynchronous process happens only under the following two conditions:
1. If there are only two nodes in the cluster and those two nodes are on different managed systems, or if there are more than two nodes in the cluster and that the operation is a move to target node and that the source node is on another managed system.
2. If you set the Force synchronous release of DLPAR resources as the default, which is No, see 6.2.5, “Change/Show Default Cluster Tunable” on page 180.
About the “unreturned” resource
The unreturned resource is one function of EPCoD. This function enables you to remove Mobile CoD resources from a server that the server cannot reclaim because they are still in use, hence these resources become unreturned resources. From the EPCoD pool point of view, the resource is back and can be assigned to other nodes. This function can allow the standby node to acquire the resource and application to use them while the resource is being released by the primary node.
When an unreturned resource is generated, a grace period timer starts for the unreturned Mobile CoD resources on that server, and EPCoD will be in Approaching out of compliance (within server grace period) status. After the releasing operation completes physically on the primary node, the unreturned resource is reclaimed automatically, and the EPCoD’s status is changed back to In compliance.
 
Note: For detailed information about Enterprise Pool’s status, see the following website:
Release
This section describes the release resource concept.
Deactivate the On/Off CoD resource
CoD resources are deactivated through the HMC command-line interface chcod. PowerHA SystemMirror runs the command automatically.
De-allocate the Enterprise Pool CoD resource
Enterprise Pool CoD resources are returned back to the pool through the HMC command line interface chcodpool. PowerHA SystemMirror runs the command automatically.
6.7.2 Synchronous and asynchronous mode
As release requests take times, PowerHA SystemMirror tries to release DLPAR resources asynchronously. In asynchronous mode, the process of release is run in the background and gives priority back to other tasks.
By default, the release is asynchronous. This default behavior can be changed with a cluster tunable.
But synchronous mode is automatically computed if necessary as follows:
•All nodes of a cluster are on same CEC.
•Otherwise, the backup LPARs of the given list of RGs are on the same CEC.
In following case, if one PowerHA SystemMirror cluster includes two nodes, the two nodes are deployed on different servers and the two servers share one Power Enterprise Pool. In this case, if you are keeping asynchronous mode, you can benefit from the resource group move scenarios because EPCoD’s unreturned resource feature and asynchronous release mode can reduce takeover time.
During resource group offline, operations to release resources to EPCoD pool can be done even if physical resources are not free on the server at that time. The freed resources are added back to the EPCoD pool as available resources immediately, so the backup partition can use these resources to bring the resource group online at once.
6.7.3 Automatic resource release process after an operating system crash
Sometimes, the ROHA resources have not been released by one node before the node failed or crashed. In this kind of cases, an automatic mechanism is implemented to release these resources when the node restarts.
A history of what was allocated for the partition is kept in the AIX ODM object database, and PowerHA SystemMirror uses it to release the same amount of resources at boot time.
 
Note: You do not need to start PowerHA SystemMirror service to activate this process after an operating system reboot as this operation is triggered by the /usr/es/sbin/cluster/etc/rc.init script, which is in the /etc/inittab file.
6.8 Example 1: Setup one ROHA cluster (without On/Off CoD)
This section describes how to set up a ROHA cluster without On/Off CoD.
6.8.1 Requirement
We have two IBM Power 770 D model servers, and these are in one Power Enterprise Pool. We want to deploy one PowerHA SystemMirror cluster with two nodes that are located in different servers. We want the PowerHA SystemMirror cluster to manage the server’s free resources and EPCoD mobile resource to automatically satisfy the application’s hardware requirements before start it.
6.8.2 Hardware topology
Figure 6-35 shows the hardware topology.
Figure 6-35 Hardware topology for example 1
The topology includes the following components for configuration:
•Two Power 770 D model servers, named P770D-01 and P770D-02.
•One Power Enterprise Pool with 4 mobile processor and 100 GB mobile memory resources.
•The PowerHA SystemMirror cluster includes two nodes, ITSO_S1Node1 and ITSO_S2Node1.
•P770D-01 has 4 inactive CPUs, 140 GB inactive memory, 4 free CPUs, and 5 GB free memory.
•P770D-02 has 16 inactive CPUs, 225 GB inactive memory, 4 free CPUs, and 10 GB free memory.
•This topology also includes the profile configuration for each LPAR.
There are two HMCs to manage the EPCoD named e16hmc1 and e16hmc3. Here, e16hmc1 is the master and e16hmc3 is the backup. There are two applications in this cluster and related resource requirement.
6.8.3 Cluster configuration
This section describes the cluster configuration.
Topology and resource group configuration
Table 6-25 shows the cluster’s attributes.
Table 6-25 Cluster’s attributes
 
ITSO_S1Node1
ITSO_S2Node2
Cluster name
ITSO_ROHA_cluster
Cluster type: NSC (No Site Cluster)
Network interface
en0:10.40.1.218
netmask:255.255.254.0
Gateway:10.40.1.1
en0:10.40.0.11
netmask:255.255.254.0
Gateway:10.40.1.1
Network
net_ether_01 (10.40.0.0/23)
CAA
Unicast
primary disk: repdisk1
backup disk: repdisk2
Shared VG
shareVG1:hdisk18
shareVG2:hdisk19
shareVG1:hdisk8
shareVG2:hdisk9
Application controller
App1Controller:
/home/bing/app1start.sh
/home/bing/app1stop.sh
App2Controller:
/home/bing/app2start.sh
/home/bing/app2stop.sh
Service IP
10.40.1.61 ITSO_ROHA_service1
10.40.1.62 ITSO_ROHA_service2
Resource Group
RG1 includes shareVG1, ITSO_ROHA_service1 and App1Controller
RG2 includes shareVG2, ITSO_ROHA_service2 and App2Controller
The node order is:ITSO_S1Node1 ITSO_S2Node1
Startup Policy: Online On Home Node Only
Fallover Policy: Fallover To Next Priority Node In The List
Fallback Policy: Never Fallback
ROHA configuration
The ROHA configuration includes the HMC, hardware resource provisioning, and the cluster-wide tunable configuration.
HMC configuration
There are two HMCs to add as shown in Table 6-26 and Table 6-27 on page 213.
Table 6-26 Configuration of HMC1
Items
Value
HMC name
9.3.207.1301
DLPAR operations timeout (in minutes)
3
Number of retries
2
Delay between retries (in seconds)
5
Nodes
ITSO_S1Node1 ITSO_S2Node1
Sites
 
Check connectivity between HMC and nodes
Yes (default)
1 We suggest entering this item with one HMC name, not IP address. Or select one HMC after press F4 to show HMC list. PowerHA SystemMirror also supports enter an HMC IP address.
Table 6-27 Configure of HMC2
Items
Value
HMC name
9.3.207.1331
DLPAR operations timeout (in minutes)
3
Number of retries
2
Delay between retries (in seconds)
5
Nodes
ITSO_S1Node1 ITSO_S2Node1
Sites
 
Check connectivity between HMC and nodes
Yes (default)
1 We suggest entering one HMC name, not an IP address. Or select one HMC after press F4 to show the HMC list. PowerHA SystemMirror also supports an HMC IP address.
Additionally, in /etc/hosts, there are resolution details between the HMC IP and the HMC host name, as shown in Example 6-13.
Example 6-13 /etc/hosts for example 1 and example 2
10.40.1.218 ITSO_S1Node1
10.40.0.11 ITSO_S2Node1
10.40.1.61 ITSO_ROHA_service1
10.40.1.62 ITSO_ROHA_service2
9.3.207.130 e16hmc1
9.3.207.133 e16hmc3
Hardware Resource Provisioning for Application Controller
There are two application controllers to add as shown in Table 6-28 and Table 6-29 on page 214.
Table 6-28 Configuration for HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
No (default)
Application Controller Name
AppController1
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
30
Optimal number of dedicated processors
2
Table 6-29 Configuration for HMC2
Items
Value
I agree to use On/Off CoD and be billed for extra costs
No (default)
Application Controller Name
AppController2
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
40
Optimal number of dedicated processors
6
Cluster-wide tunables
All the tunables are in default value as shown in Table 6-30.
Table 6-30 Configuration for HMC1
Items
Value
DLPAR Start Resource Groups even if resources are insufficient
No (default)
Adjust Shared Processor Pool size if required
No (default)
Force synchronous release of DLPAR resources
No (default)
I agree to use On/Off CoD and be billed for extra costs
No (default)
Perform the PowerHA SystemMirror Verify and Synchronize Cluster Configuration process after finishing the above configuration.
6.8.4 Show the ROHA configuration
Example 6-14 shows the output of the clmgr view report roha command line.
Example 6-14 Output of the clmgr view report roha command
Cluster: ITSO_ROHA_cluster of NSC type <---NSC means No Site Cluster
Cluster tunables
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '0'
--> don’t use On/Off CoD resource in this case
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP <--this server is P770D-01
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '96'
Processing mode: Dedicated
Processors: minimum '1' desired '2' current '2' maximum '12'
ROHA provisioning for resource groups
No ROHA provisioning.
Node: ITSO_S2Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P <--this server is P770D-02
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '96'
Processing mode: Dedicated
Processors: minimum '1' desired '2' current '2' maximum '12'
ROHA provisioning for resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130' <--this HMC is master
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133' <--this HMC is backup
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP'
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '52' GB processing units '8.00'
Inactive: memory '140' GB processing units '4.00'
Available: memory '5' GB processing units '4.00'
On/Off CoD
--> this server has enabled On/Off CoD, but we don’t use them during resource group bring online or offline scenarios, because we only want to simulate ONLY Enterprise Pool scenarios. Please ignore the On/Off CoD information.
On/Off CoD memory
State: 'Available'
Available: '9927' GB.days
On/Off CoD processor
State: 'Running'
Available: '9944' CPU.days
Activated: '4' CPU(s) <-- this 4CPU is assigned to P770D-01 manually to simulate 4 free processor resource
Left: '20' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC' <-- this is enterprise pool name
Hardware Management Console
9.3.207.130
9.3.207.133
Logical partition 'ITSO_S1Node1'
 
Managed System 'r1r9m1-9117-MMD-1038B9P'
Hardware resources of managed system
Installed: memory '320' GB processing units '32.00'
Configurable: memory '95' GB processing units '16.00'
Inactive: memory '225' GB processing units '16.00'
Available: memory '10' GB processing units '4.00'
On/Off CoD
--> this server has enabled On/Off CoD, but we don’t use them during resource group bring online or offline, because we want to simulate ONLY Enterprise Pool exist scenarios.
On/Off CoD memory
State: 'Available'
Available: '9889' GB.days
On/Off CoD processor
State: 'Available'
Available: '9976' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Logical partition 'ITSO_S2Node1'
This 'ITSO_S2Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Enterprise pool 'DEC_2CEC'
--> shows that there is no EPCoD mobile resource is assigned to any of server
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '100' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
6.9 Test scenarios of Example 1 (without On/Off CoD)
Based on the cluster configuration in 6.5, “Resource acquisition and release process introduction” on page 194, this section introduces several testing scenarios as follows:
6.9.1 Bring two resource groups online
When PowerHA SystemMirror starts cluster service on the primary node (ITSO_S1Node1), the two resource groups will be online. The procedure that is related with ROHA is described in Figure 6-36.
Figure 6-36 Resource acquire procedure to bring two resource groups online
Section 6.6, “Introduction to resource acquisition” on page 195 introduced four steps for PowerHA SystemMirror to acquire resources. In this case, the following section provides the detailed description for the four steps.
Query step
PowerHA SystemMirror queries the server, the EPCoD, the LPARs, and the current resource group information. The data is shown in yellow in Figure 6-36.
Compute step
In this step, PowerHA SystemMirror computes how many resources to be added through DLPAR. It needs 7C and 46 GB. The purple table shows the process in Figure 6-36. We take the CPU resource for example:
•The expected total CPU number is as follows: 1 (Min) + 2 (RG1 require) + 6 (RG2 require) + 0 (running RG require, there is no running RG) = 9C.
•Take this value to compare with LPAR’s profile needs less than or equal to the Maximum and more than or equal to the Minimum value.
•If the requirement is satisfied, and takes this value minus the current running CPU, 9 - 2 = 7, we get the CPU number to add through the DLPAR.
Identify and acquire step
After the compute step, PowerHA SystemMirror identifies how to satisfy the requirement. For CPU, it gets the remaining 4C of this server and 3C from EPCoD. For memory, it gets remaining 5 GB of this server and 41 GB from EPCoD. The process is shown in the green table in Figure 6-36 on page 217. We take the CPU resource for example:
•There are 4 CPUs available in the server’s free pool, so PowerHA SystemMirror reserves them, then needs another 3 CPUs (7-4).
•There are 4 mobile CPUs in the EPCoD pool, so PowerHA SystemMirror assigns the 3 CPUs from EPCoD to this server through the HMC (chcodpool command). At this time, there are 7 CPUs in the free pool, then PowerHA SystemMirror assigns all of them to LPAR (ITSO_S1Node1) through the DLPAR operation (chhwres command).
 
Note: During this process, PowerHA SystemMirror adds mobile resources from EPCoD to the server’s free pool first, then adds all the free pool’s resources to the LPAR through DLPAR. In order to describe the process clearly, the free pool only means the available resources of one server before adding the EPCoD’s resources to it.
The orange table (Figure 6-36 on page 217) shows the result after the resource acquisition, and includes the LPAR’s running resource, EPCoD and the server’s resource status.
Track hacmp.out log to know what is happening
From hacmp.out, we know all the resources (7 CPU and 41 memory) cost 53 seconds, as shown in Example 6-15.
09:11:39 -> 09:12:32
Example 6-15 The hacmp.out log shows the resource acquisition process for example 1
# egrep "ROHALOG|Close session|Open session" /var/hacmp/log/hacmp.out
+RG1 RG2:clmanageroha[roha_session_open:162] roha_session_log 'Open session
Open session 22937664 at Sun Nov 8 09:11:39 CST 2015
INFO: acquisition is always synchronous.
=== HACMProhaparam ODM ====
--> Cluster wide tunables display
ALWAYS_START_RG = 0
ADJUST_SPP_SIZE = 0
FORCE_SYNC_RELEASE = 0
AGREE_TO_COD_COSTS = 0
ONOFF_DAYS = 30
===========================
------------------+----------------+
HMC | Version |
------------------+----------------+
9.3.207.130 | V8R8.3.0.1 |
9.3.207.133 | V8R8.3.0.1 |
------------------+----------------+
------------------+----------------+----------------+
MANAGED SYSTEM | Memory (GB) | Proc Unit(s) |
------------------+----------------+----------------+
Name | rar1m3-9117-MMD-1016AAP | --> Server name
State | Operating |
Region Size | 0.25 | / |
VP/PU Ratio | / | 0.05 |
Installed | 192.00 | 12.00 |
Configurable | 52.00 | 8.00 |
Reserved | 5.00 | / |
Available | 5.00 | 4.00 |
Free (computed) | 5.00 | 4.00 | --> Free pool resource
------------------+----------------+----------------+
------------------+----------------+----------------+
LPAR (dedicated) | Memory (GB) | CPU(s) |
------------------+----------------+----------------+
Name | ITSO_S1Node1 |
State | Running |
Minimum | 8.00 | 1 |
Desired | 32.00 | 2 |
Assigned | 32.00 | 2 |
Maximum | 96.00 | 12 |
------------------+----------------+----------------+
+------------------+----------------+----------------+
| ENTERPRISE POOL | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| Name | DEC_2CEC | --> Enterprise Pool Name
| State | In compliance |
| Master HMC | e16hmc1 |
| Backup HMC | e16hmc3 |
| Available | 100.00 | 4 | --> Available resource
| Unreturned (MS) | 0.00 | 0 |
| Mobile (MS) | 0.00 | 0 |
| Inactive (MS) | 140.00 | 4 | --> Maximum number to add
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| TRIAL COD | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| State | Not Running | Not Running |
| Activated | 0.00 | 0 |
| Days left | 0 | 0 |
| Hours left | 0 | 0 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| ONOFF COD | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| State | Available | Running |
| Activated | 0.00 | 4 | --> just ignore it
| Unreturned | 0.00 | 0 |
| Available | 140.00 | 4 |
| Days available | 9927 | 9944 |
| Days left | 0 | 20 |
| Hours left | 0 | 2 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+
| OTHER | Memory (GB) | CPU(s) |
+------------------+----------------+----------------+
| LPAR (dedicated) | ITSO_S2Node1 |
| State | Running |
| Id | 13 |
| Uuid | 78E8427B-B157-494A-8711-7B8 |
| Minimum | 8.00 | 1 |
| Assigned | 32.00 | 2 |
+------------------+----------------+----------------+
| MANAGED SYSTEM | r1r9m1-9117-MMD-1038B9P |
| State | Operating |
+------------------+----------------+----------------+
| ENTERPRISE POOL | DEC_2CEC |
| Mobile (MS) | 0.00 | 0 |
+------------------+----------------+----------------+
+------------------+----------------+----------------+----------------+---------+
| OPTIMAL APPS | Use Desired | Memory (GB) | CPU(s) |PU(s)/VP(s)
+------------------+----------------+----------------+----------------+---------+
| App1Controller | 0 | 30.00 | 2 | 0.00/0
| App2Controller | 0 | 40.00 | 6 | 0.00/0
+------------------+----------------+----------------+----------------+---------+
| Total | 0 | 70.00 | 8 | 0.00/0
+------------------+----------------+----------------+----------------+---------+
===== HACMPdynresop ODM ====
TIMESTAMP = Sun Nov 8 09:11:43 CST 2015
STATE = start_acquire
MODE = sync
APPLICATIONS = App1Controller App2Controller
RUNNING_APPS = 0
PARTITION = ITSO_S1Node1
MANAGED_SYSTEM = rar1m3-9117-MMD-1016AAP
ENTERPRISE_POOL = DEC_2CEC
PREFERRED_HMC_LIST = 9.3.207.130 9.3.207.133
OTHER_LPAR = ITSO_S2Node1
INIT_SPP_SIZE_MAX = 0
INIT_DLPAR_MEM = 32.00
INIT_DLPAR_PROCS = 2
INIT_DLPAR_PROC_UNITS = 0
INIT_CODPOOL_MEM = 0.00
INIT_CODPOOL_CPU = 0
INIT_ONOFF_MEM = 0.00
INIT_ONOFF_MEM_DAYS = 0
INIT_ONOFF_CPU = 4
INIT_ONOFF_CPU_DAYS = 20
SPP_SIZE_MAX = 0
DLPAR_MEM = 0
DLPAR_PROCS = 0
DLPAR_PROC_UNITS = 0
CODPOOL_MEM = 0
CODPOOL_CPU = 0
ONOFF_MEM = 0
ONOFF_MEM_DAYS = 0
ONOFF_CPU = 0
ONOFF_CPU_DAYS = 0
 
===== Compute ROHA Memory =====
--> compute memory process
minimal + optimal + running = total <=> current <=> maximum
8.00 + 70.00 + 0.00 = 78.00 <=> 32.00 <=> 96.00 : => 46.00 GB
============ End ==============
===== Compute ROHA CPU(s) =====
--> compute CPU process
minimal + optimal + running = total <=> current <=> maximum
1 + 8 + 0 = 9 <=> 2 <=> 12 : => 7 CPU(s)
============ End ==============
===== Identify ROHA Memory ====
--> identify memory process
Remaining available memory for partition: 5.00 GB
Total Enterprise Pool memory to allocate: 41.00 GB
Total Enterprise Pool memory to yank: 0.00 GB
Total On/Off CoD memory to activate: 0.00 GB for 0 days
Total DLPAR memory to acquire: 46.00 GB
============ End ==============
=== Identify ROHA Processor ===
--> identify CPU process
Remaining available PU(s) for partition: 4.00 Processing Unit(s)
Total Enterprise Pool CPU(s) to allocate: 3.00 CPU(s)
Total Enterprise Pool CPU(s) to yank: 0.00 CPU(s)
Total On/Off CoD CPU(s) to activate: 0.00 CPU(s) for 0 days
Total DLPAR CPU(s) to acquire: 7.00 CPU(s)
============ End ==============
--> assign EPCoD resource to server
clhmccmd: 41.00 GB of Enterprise Pool CoD have been allocated.
clhmccmd: 3 CPU(s) of Enterprise Pool CoD have been allocated.
--> assign all resource to LPAR
clhmccmd: 46.00 GB of DLPAR resources have been acquired.
clhmccmd: 7 VP(s) or CPU(s) and 0.00 PU(s) of DLPAR resources have been acquired.
The following resources were acquired for application controllers App1Controller App2Controller.
DLPAR memory: 46.00 GB On/Off CoD memory: 0.00 GB Enterprise Pool memory: 41.00 GB.
DLPAR processor: 7.00 CPU(s) On/Off CoD processor: 0.00 CPU(s) Enterprise Pool processor: 3.00 CPU(s)
INFO: received rc=0.
Success on 1 attempt(s).
===== HACMPdynresop ODM ====
TIMESTAMP = Sun Nov 8 09:12:31 CST 2015
STATE = end_acquire
MODE = 0
APPLICATIONS = 0
RUNNING_APPS = 0
PARTITION = 0
MANAGED_SYSTEM = 0
ENTERPRISE_POOL = 0
PREFERRED_HMC_LIST = 0
OTHER_LPAR = 0
INIT_SPP_SIZE_MAX = 0
INIT_DLPAR_MEM = 0
INIT_DLPAR_PROCS = 0
INIT_DLPAR_PROC_UNITS = 0
INIT_CODPOOL_MEM = 0
INIT_CODPOOL_CPU = 0
INIT_ONOFF_MEM = 0
INIT_ONOFF_MEM_DAYS = 0
INIT_ONOFF_CPU = 0
INIT_ONOFF_CPU_DAYS = 0
SPP_SIZE_MAX = 0
DLPAR_MEM = 46
DLPAR_PROCS = 7
DLPAR_PROC_UNITS = 0
CODPOOL_MEM = 41
CODPOOL_CPU = 3
ONOFF_MEM = 0
ONOFF_MEM_DAYS = 0
ONOFF_CPU = 0
ONOFF_CPU_DAYS = 0
============================
Session_close:313] roha_session_log 'Close session 22937664 at Sun Nov 8 09:12:32 CST 2015'
ROHA report update
The clmgr view report roha command output (Example 6-16) shows updates on the resources of P770D-01 and the Enterprise Pool.
Example 6-16 The update in the ROHA report shows resource acquisition process for example 1
# clmgr view report roha
...
Managed System 'rar1m3-9117-MMD-1016AAP' --> this is P770D-01 server
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '93' GB processing units '11.00'
Inactive: memory '99' GB processing units '1.00'
Available: memory '0' GB processing units '0.00'
...
 
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '59' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '1'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '41' GB
Unreturned memory: '0' GB
Activated CPU(s): '3' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Testing summary
The total time to bring the two resource groups online is 68 s (from 09:11:27 to 9.12:35), and it includes the resource acquisition time, as shown in Example 6-17.
Example 6-17 The hacmp.out log shows the total time
Nov 8 09:11:27 EVENT START: node_up ITSO_S1Node1
Nov 8 09:11:31 EVENT COMPLETED: node_up ITSO_S1Node1 0
Nov 8 09:11:33 EVENT START: rg_move_fence ITSO_S1Node1 2
Nov 8 09:11:33 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 2 0
Nov 8 09:11:33 EVENT START: rg_move_acquire ITSO_S1Node1 2
Nov 8 09:11:33 EVENT START: rg_move ITSO_S1Node1 2 ACQUIRE
Nov 8 09:11:34 EVENT START: acquire_service_addr
Nov 8 09:11:34 EVENT START: acquire_aconn_service en0 net_ether_01
Nov 8 09:11:34 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Nov 8 09:11:35 EVENT START: acquire_aconn_service en0 net_ether_01
Nov 8 09:11:35 EVENT COMPLETED: acquire_aconn_service en0 net_ether_01 0
Nov 8 09:11:35 EVENT COMPLETED: acquire_service_addr 0
Nov 8 09:11:39 EVENT COMPLETED: rg_move ITSO_S1Node1 2 ACQUIRE 0
Nov 8 09:11:39 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 2 0
Nov 8 09:11:39 EVENT START: rg_move_complete ITSO_S1Node1 2
Nov 8 09:12:32 EVENT START: start_server App1Controller
Nov 8 09:12:32 EVENT START: start_server App2Controller
Nov 8 09:12:32 EVENT COMPLETED: start_server App1Controller 0
Nov 8 09:12:32 EVENT COMPLETED: start_server App2Controller 0
Nov 8 09:12:33 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 2 0
Nov 8 09:12:35 EVENT START: node_up_complete ITSO_S1Node1
Nov 8 09:12:35 EVENT COMPLETED: node_up_complete ITSO_S1Node1 0
6.9.2 Move one resource group to another node
There are two resource groups that are running on the primary node (ITSO_S1Node1). Now we want to move one resource group from this node to the standby node (ITSO_S2Node1).
In this case, we split this move into two parts: One is the resource group offline at the primary node, and the other is the resource group online at the standby node.
Resource group offline at primary node (ITSO_S1Node1)
Figure 6-37 describes the offline procedure at the primary node.
Figure 6-37 Resource group offline procedure at the primary node during the resource group move
The following is the description of the offline procedure:
Query step
PowerHA SystemMirror queries the server, EPCoD, the LPARs, and the current resource group information. The data is shown in the yellow table in Figure 6-37.
Compute step
In this step, PowerHA SystemMirror computes how many resources need to remove through the DLPAR. PowerHA SystemMirror needs 2C and 30 GB, purple tables show the process as shown in Figure 6-37:
•In this case, RG1 will be released and RG2 is still running. PowerHA calculates how many resources it can release based on whether RG2 has enough resource to run. So the formula is: 9 (current running) - 1 (Min) - 6 (RG2 still running) = 2C. This means 2 CPUs can be released.
•PowerHA takes into account that sometimes you would adjust your current running resources through a DLPAR operation manually. For example, you added some resources to satisfy another application that was not started with PowerHA. To avoid removing this kind of resource, PowerHA needs to check how many resources it allocated before.
The total number is those that PowerHA will freeze, such that the number is not greater than what was allocated before.
So in this case, PowerHA takes the value in the previous step to compare with the real resources this LPAR allocated before. This value is stored in one ODM object database (HACMPdryresop), and the value is 7. PowerHA SystemMirror select the small one.
 
Identify and release
PowerHA SystemMirror identifies how many resources need to be released to EPCoD and then releases them to EPCoD asynchronously, although the resources are still in use. This process generates an unreturned resource temporarily. Figure 6-38 displays the box shown on the HMC.
Figure 6-38 HMC message shows that there are unreturned resources generated
We can get the unreturned resources using the clmgr view report roha command from the AIX command line, as shown in Example 6-18.
Example 6-18 Display unreturned resources from the AIX command line
# clmgr view report roha
...
Enterprise pool 'DEC_2CEC'
State: 'Approaching out of compliance (within server grace period)'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '89' GB -->the 30GB has been changed to EPCoD available status
Unreturned memory: '30' GB -->the 30GB is marked 'unreturned'
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '3' --> the 2CPU has been changed to EPCoD available status
Unreturned CPU(s): '2' --> the 2CPU is marked 'unreturned'
Used by: 'rar1m3-9117-MMD-1016AAP' -->show unreturned resource from server’s view
Activated memory: '11' GB
Unreturned memory: '30' GB
Activated CPU(s): '1' CPU(s)
Unreturned CPU(s): '2' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
From the HMC command line, you can see the unreturned resource generated, as shown in Example 6-19.
Example 6-19 Show the unreturned resources and the status from the HMC command line
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level sys
name=rar1m3-9117-MMD-1016AAP,mtms=9117-MMD*1016AAP,mobile_procs=1,non_mobile_procs=8,unreturned_mobile_procs=2,inactive_procs=1,installed_procs=12,mobile_mem=11264,non_mobile_mem=53248,unreturned_mobile_mem=30720,inactive_mem=101376,installed_mem=196608
name=r1r9m1-9117-MMD-1038B9P,mtms=9117-MMD*1038B9P,mobile_procs=0,non_mobile_procs=16,unreturned_mobile_procs=0,inactive_procs=16,installed_procs=32,mobile_mem=0,non_mobile_mem=97280,unreturned_mobile_mem=0,inactive_mem=230400,installed_mem=327680
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=Approaching out of compliance (within server grace period),sequence_num=41,master_mc_name=e16hmc1,master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=3,unreturned_mobile_procs=2,mobile_mem=102400,avail_mobile_mem=91136,unreturned_mobile_mem=30720
Meanwhile, PowerHA SystemMirror triggers one asynchronous process to do the DLPAR remove operation, and it removes 2C and 30 GB resources from the LPAR into the server’s free pool. The log is written in the /var/hacmp/log/async_release.log file.
When the DLPAR operation completes, the unreturned resource is reclaimed immediately, and some messages are shown on the HMC (Figure 6-39). The Enterprise Pool’s status is changed back to In compliance.
Figure 6-39 The unreturned resource is reclaimed after the DLPAR operation
You can see the changes from HMC command line, as shown in Example 6-20.
Example 6-20 Show the unreturned resource reclaimed from the HMC command line
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level sys
name=rar1m3-9117-MMD-1016AAP,mtms=9117-MMD*1016AAP,mobile_procs=1,non_mobile_procs=8,unreturned_mobile_procs=0,inactive_procs=3,installed_procs=12,mobile_mem=11264,non_mobile_mem=53248,unreturned_mobile_mem=0,inactive_mem=132096,installed_mem=196608
name=r1r9m1-9117-MMD-1038B9P,mtms=9117-MMD*1038B9P,mobile_procs=0,non_mobile_procs=16,unreturned_mobile_procs=0,inactive_procs=16,installed_procs=32,mobile_mem=0,non_mobile_mem=97280,unreturned_mobile_mem=0,inactive_mem=230400,installed_mem=327680
hscroot@e16hmc1:~> lscodpool -p DEC_2CEC --level pool
name=DEC_2CEC,id=026F,state=In compliance,sequence_num=41,master_mc_name=e16hmc1,
master_mc_mtms=7042-CR5*06K0040,backup_master_mc_name=e16hmc3,backup_master_mc_mtms=7042-CR5*06K0036,mobile_procs=4,avail_mobile_procs=3,unreturned_mobile_procs=0,mobile_mem=102400,avail_mobile_mem=91136,unreturned_mobile_mem=0
 
Note: The Approaching out of compliance status is a normal status in the Enterprise Pool, and it is useful when you need extra resources temporarily. PowerHA SystemMirror’s resource group takeover scenario is one of the cases.
Log information in hacmp.out
The hacmp.out log file records the process of the resource group offlining, as shown in Example 6-21.
Example 6-21 The hacmp.out log file information about the resource group offline process
#egrep "ROHALOG|Close session|Open session" /var/hacmp/log/hacmp.out
...
===== Compute ROHA Memory =====
minimum + running = total <=> current <=> optimal <=> saved
8.00 + 40.00 = 48.00 <=> 78.00 <=> 30.00 <=> 46.00 : => 30.00 GB
============ End ==============
===== Compute ROHA CPU(s) =====
minimal + running = total <=> current <=> optimal <=> saved
1 + 6 = 7 <=> 9 <=> 2 <=> 7 : => 2 CPU(s)
============ End ==============
===== Identify ROHA Memory ====
Total Enterprise Pool memory to return back: 30.00 GB
Total On/Off CoD memory to de-activate: 0.00 GB
Total DLPAR memory to release: 30.00 GB
============ End ==============
=== Identify ROHA Processor ===
Total Enterprise Pool CPU(s) to return back: 2.00 CPU(s)
Total On/Off CoD CPU(s) to de-activate: 0.00 CPU(s)
Total DLPAR CPU(s) to release: 2.00 CPU(s)
============ End ==============
clhmccmd: 30.00 GB of Enterprise Pool CoD have been returned.
clhmccmd: 2 CPU(s) of Enterprise Pool CoD have been returned.
The following resources were released for application controllers App1Controller.
DLPAR memory: 30.00 GB On/Off CoD memory: 0.00 GB Enterprise Pool memory: 30.00 GB.
DLPAR processor: 2.00 CPU(s) On/Off CoD processor: 0.00 CPU(s) Enterprise Pool processor: 2.00 CPU(s)Close session 22937664 at Sun Nov 8 09:12:32 CST 2015
..
During the releasing process, the de-allocation order is EPCoD and then local server’s free pool. Because EPCoD is shared between different servers, the standby node on other server always needs this resource to bring the resource group online in a takeover scenario.
Resource online at standby node (ITSO_S2Node1)
In this case, the resource group online on standby node doesn’t need to wait for DLPAR complete on primary node, it is an asynchronous process. In this process, PowerHA SystemMirror will acquire a corresponding resource for the onlining resource group.
 
Note: Before acquiring process start, the 2C and 30 GB resource was available in the Enterprise Pool, so this kind of resource can also be used by standby node.
Figure 6-40 describes the resource acquire process on standby node (ITSO_S2Node1).
Figure 6-40 The acquisition process on standby node
This acquisition process differs from item 6.9.1, “Bring two resource groups online” on page 217:
The expected resource to add to the LPAR is 1C and 6 GB, the system’s free pool can satisfy it, so it doesn’t need to acquire resource from EPCoD.
Testing scenario summary
The total time of this resource group moving costs 80 seconds, from 10:53:15 to 10:53:43.
The removing resource (2C and 30 GB) from LPAR to a free pool on the primary node costs 257 seconds, from 10:52:51 to 10:57:08, but are not concerned with this time because it is an asynchronous process.
Example 6-22 shows the hacmp.out information on ITSO_S1Node1.
Example 6-22 The key timestamp in hacmp.out on primary node (ITSO_S1Node1)
# egrep "EVENT START|EVENT COMPLETED" hacmp.out
Nov 8 10:52:27 EVENT START: external_resource_state_change ITSO_S2Node1
Nov 8 10:52:27 EVENT COMPLETED: external_resource_state_change ITSO_S2Node1 0
Nov 8 10:52:27 EVENT START: rg_move_release ITSO_S1Node1 1
Nov 8 10:52:27 EVENT START: rg_move ITSO_S1Node1 1 RELEASE
Nov 8 10:52:27 EVENT START: stop_server App1Controller
Nov 8 10:52:28 EVENT COMPLETED: stop_server App1Controller 0
Nov 8 10:52:53 EVENT START: release_service_addr
Nov 8 10:52:54 EVENT COMPLETED: release_service_addr 0
Nov 8 10:52:56 EVENT COMPLETED: rg_move ITSO_S1Node1 1 RELEASE 0
Nov 8 10:52:56 EVENT COMPLETED: rg_move_release ITSO_S1Node1 1 0
Nov 8 10:52:58 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:58 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:53:00 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:53:00 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:53:00 EVENT START: rg_move_acquire ITSO_S1Node1 1
Nov 8 10:53:00 EVENT START: rg_move ITSO_S1Node1 1 ACQUIRE
Nov 8 10:53:00 EVENT COMPLETED: rg_move ITSO_S1Node1 1 ACQUIRE 0
Nov 8 10:53:00 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 1 0
Nov 8 10:53:18 EVENT START: rg_move_complete ITSO_S1Node1 1
Nov 8 10:53:19 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 1 0
Nov 8 10:53:50 EVENT START: external_resource_state_change_complete ITSO_S2Node1
Nov 8 10:53:50 EVENT COMPLETED: external_resource_state_change_complete ITSO_S2Node1 0
Example 6-23 shows the async_release.log on ITSO_S2Node1.
Example 6-23 asyn_release.log records the DLPAR operation
# egrep "Sun Nov| eval LC_ALL=C ssh " async_release.log
Sun Nov 8 10:52:51 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lssyscfg -r sys -m 9117-MMD*1016AAP -F name 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:52:56 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:54:19 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:54:19 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:55:32 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:55:32 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r mem -o r -q 10240 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:56:40 CST 2015
+RG1:clhmccmd[clhmcexec:3624] : Start ssh command at Sun Nov 8 10:56:40 CST 2015
+RG1:clhmccmd[clhmcexec:3625] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'chhwres -m rar1m3-9117-MMD-1016AAP -p ITSO_S1Node1 -r proc -o r --procs 2 -w 30 2>&1''
+RG1:clhmccmd[clhmcexec:3627] : Return from ssh command at Sun Nov 8 10:57:08 CST 2015
Sun Nov 8 10:57:08 CST 2015
Example 6-24 shows the hacmp.out information on ITSO_S2Node1.
Example 6-24 The key timestamp in hacmp.out on standby node (ITSO_S1Node1)
#egrep "EVENT START|EVENT COMPLETED" hacmp.out
Nov 8 10:52:24 EVENT START: rg_move_release ITSO_S1Node1 1
Nov 8 10:52:24 EVENT START: rg_move ITSO_S1Node1 1 RELEASE
Nov 8 10:52:25 EVENT COMPLETED: rg_move ITSO_S1Node1 1 RELEASE 0
Nov 8 10:52:25 EVENT COMPLETED: rg_move_release ITSO_S1Node1 1 0
Nov 8 10:52:55 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:55 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:52:57 EVENT START: rg_move_fence ITSO_S1Node1 1
Nov 8 10:52:57 EVENT COMPLETED: rg_move_fence ITSO_S1Node1 1 0
Nov 8 10:52:57 EVENT START: rg_move_acquire ITSO_S1Node1 1
Nov 8 10:52:57 EVENT START: rg_move ITSO_S1Node1 1 ACQUIRE
Nov 8 10:52:57 EVENT START: acquire_takeover_addr
Nov 8 10:52:58 EVENT COMPLETED: acquire_takeover_addr 0
Nov 8 10:53:15 EVENT COMPLETED: rg_move ITSO_S1Node1 1 ACQUIRE 0
Nov 8 10:53:15 EVENT COMPLETED: rg_move_acquire ITSO_S1Node1 1 0
Nov 8 10:53:15 EVENT START: rg_move_complete ITSO_S1Node1 1
Nov 8 10:53:43 EVENT START: start_server App1Controller
Nov 8 10:53:43 EVENT COMPLETED: start_server App1Controller 0
Nov 8 10:53:45 EVENT COMPLETED: rg_move_complete ITSO_S1Node1 1 0
Nov 8 10:53:47 EVENT START: external_resource_state_change_complete ITSO_S2Node1
Nov 8 10:53:47 EVENT COMPLETED: external_resource_state_change_complete ITSO_S2Node1 0
6.9.3 Primary node crashes and reboots with current configuration
This case introduces Automatic Release After a Failure (ARAF) process. We simulated that the primary node is crashed immediately, and we don’t introduce how resource group is online on standby node. We only describe how PowerHA SystemMirror does after the primary node reboots. We assume that we activate this node with current configuration, that means this LPAR still can hold the same amount of resource as before crash.
As described in 6.7.3, “Automatic resource release process after an operating system crash” on page 210, after the primary node reboot completes, /usr/es/sbin/cluster/etc/rc.init script is triggered by /etc/inittab and will do the resource releasing operation.
The process is shown in Figure 6-41.
Figure 6-41 Resource release process in ARAF process
The process is similar to the “Resource group offline at primary node (ITSO_S1Node1)” on page 224. In this process, PowerHA SystemMirror tries to release all the resources that were held by the two resource groups before.
Testing summary
If some resource was not released because of PowerHA SystemMirror service crash or an AIX operating system crash, PowerHA SystemMirror can do the release operation automatically after this node comes up again. This operation occurs before you start the PowerHA SystemMirror service with the smitty clstart or the clmgr start cluster commands.
6.10 Example 2: Set up one ROHA cluster (with On/Off CoD)
This section describes the setup of one ROHA cluster example.
6.10.1 Requirements
We have two Power 770 D model servers, these are in one Power Enterprise Pool, and each server has an On/Off CoD license. We want to deploy one PowerHA SystemMirror cluster, include two nodes, and these are located in different servers. We want the PowerHA SystemMirror cluster to manage the server’s free resources, EPCoD mobile resources, and On/Off CoD resources automatically to satisfy the application’s hardware requirement before starting it.
6.10.2 Hardware topology
Figure 6-42 shows the server and LPAR information of example 2.
Figure 6-42 Server and LPAR information
The topology includes the following components for configuration:
•Two Power 770 D model servers, named P770D-01 and P770D-02.
•One Power Enterprise Pool, it has 4 mobile processor and 100 GB mobile memory resource.
•Each server has enabled the On/Off CoD feature.
•PowerHA SystemMirror cluster includes two nodes, ITSO_S1Node1 and ITSO_S2Node1.
•P770D-01 has 8 inactive CPUs, 140 GB inactive memory, 0.5 free CPUs, and 5 GB free memory.
•P770D-02 has 16 inactive CPUs, 225 GB inactive memory, 2.5 free CPUs, and 10 GB free memory.
•This also includes the profile configuration for each LPAR.
There are two HMCs to manage the EPCoD, named e16hmc1 and e16hmc3. Here, e16hmc1 is the master and e16hmc3 is the backup. There are two applications in this cluster and related resource requirements.
Available resource in On/Off CoD
In the examples, we must keep in mind that the resources we have at the On/Off CoD level are GB.Days or Processor.Days. For example, we can have in On/Off CoD pool: 600 GB.Days, or 120 Processors.Days. The time scope of the activation is determined through a tunable variable: Number of activating days for On/Off CoD requests (6.2.5, “Change/Show Default Cluster Tunable” on page 180).
If set to 30, for example, it means that we want to activate for 30 days, so it can allocate 20 GB of memory only, and we would say that we have 20 GB On/Off CoD only, even if we have 600 GB.Days available.
6.10.3 Cluster configuration
The Topology and resource group configuration and HMC configuration is the same as 6.8.3, “Cluster configuration” on page 212.
Hardware Resource Provisioning for Application Controller
There are two application controllers to be added, as shown in Table 6-31 and Table 6-32.
Table 6-31 Configure HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
Yes
Application Controller Name
AppController1
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
70
Optimal number of processing units
3.5
Optimal number of virtual processors
7
Table 6-32 Configure HMC1
Items
Value
I agree to use On/Off CoD and be billed for extra costs
Yes
Application Controller Name
AppController2
Use wanted level from the LPAR profile
No
Optimal number of gigabytes of memory
80
Optimal number of processing units
4.5
Optimal number of virtual processors
9
Cluster-wide tunables
All the tunables are at default value, as shown in Table 6-33.
Table 6-33 Configure of HMC1
Items
Value
DLPAR Start Resource Groups even if resources are insufficient
No (default)
Adjust Shared Processor Pool size if required
No (default)
Force synchronous release of DLPAR resources
No (default)
I agree to use On/Off CoD and be billed for extra costs
Yes
Number of activating days for On/Off CoD requests
30 (default)
This requires that you perform a Verify and Synchronize Cluster Configuration after changing the previous configuration.
6.10.4 Showing the ROHA configuration
The clmgr view report roha command shows the current ROHA data, as shown in Example 6-25.
Example 6-25 Shows ROHA data with the clmgr view report roha command
# clmgr view report roha
Cluster: ITSO_ROHA_cluster of NSC type --> NSC means No Site Cluster
Cluster tunables --> Following is the cluster tunalbes
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '1'
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1 --> Information of ITSO_S1Node1 node
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for resource groups
No ROHA provisioning.
Node: ITSO_S2Node1 --> Information of ITSO_S2Node1 node
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130' --> Information of HMCs
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133'
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP' --> Information of P770D-01
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '52' GB processing units '4.00'
Inactive: memory '140' GB processing units '8.00'
Available: memory '5' GB processing units '0.50'
On/Off CoD --> Information of On/Off CoD on P770D-01 server
On/Off CoD memory
State: 'Available'
Available: '9907' GB.days
On/Off CoD processor
State: 'Available'
Available: '9959' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S1Node1'
This 'ITSO_S1Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Managed System 'r1r9m1-9117-MMD-1038B9P' --> Information of P770D-02
Hardware resources of managed system
Installed: memory '320' GB processing units '32.00'
Configurable: memory '95' GB processing units '16.00'
Inactive: memory '225' GB processing units '16.00'
Available: memory '10' GB processing units '2.50'
On/Off CoD --> Information of On/Off CoD on P770D-02 server
On/Off CoD memory
State: 'Available'
Available: '9889' GB.days
On/Off CoD processor
State: 'Available'
Available: '9976' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S2Node1'
This 'ITSO_S2Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
Enterprise pool 'DEC_2CEC' --> Information of Enterprise Pool
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB -->Total mobile resource of Pool, not change during resource moving
Available memory: '100' GB -->Available for assign, will change during resource moving
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '4'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '0' GB --> the number assigned from EPCoD to server
Unreturned memory: '0' GB --> the number has been released to EPCoD but not reclaimed actually, need to reclaimed within a period time
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
6.11 Test scenarios for Example 2 (with On/Off CoD)
Based on the configuration in 6.10, “Example 2: Set up one ROHA cluster (with On/Off CoD)” on page 232, we will introduce several testing scenarios in this section:
6.11.1 Bring two resource groups online
When PowerHA SystemMirror starts cluster services on the primary node (ITSO_S1Node1), the two resource groups go online. The procedure that is related to ROHA is shown in Figure 6-43.
Figure 6-43 Acquire resource process of example 2
Section 6.6, “Introduction to resource acquisition” on page 195 introduces four steps for PowerHA SystemMirror to acquire the resources. In this case, the following are the detail descriptions of the four steps.
Query step
PowerHA SystemMirror queries the server, EPCoD, the On/Off CoD, the LPARs, and the current resource group information. The data is shown in the yellow tables in Figure 6-43.
For the On/Off CoD resources, we do not display the available resources because there are enough resources in our testing environment:
•P770D-01 has 9959 CPU day and 9917 GB day.
•P770D-02 has 9976 CPU day and 9889 GB day.
So we display the actual use amount.
Compute step
In this step, PowerHA SystemMirror computes how many resources need to add through the DLPAR. PowerHA SystemMirror needs 7C and 126 GB, purple tables show this process (Figure 6-43). We take the CPU resources for example:
•The expected total processor unit number is: 0.5 (Min) + 3.5 (RG1 require) + 4.5 (RG2 require) +0 (running RG require, there is no running RG) = 8.5C.
•Take this value to compare with the LPAR’s profile, which needs to be less than or equal to the Maximum and more than or equal to the Minimum value.
•If this satisfies the requirement, then take this value minus the current running CPU (meaning 8.5 - 1.5 = 7), and this is the number that we want to add to the LPAR through DLPAR.
Identify and acquire the step
After the compute step, PowerHA SystemMirror identifies how to satisfy the requirement. For CPU, it gets 4C from EPCoD and 3C from the On/Off CoD. Because for EPCoD and On/Off CoD, the minimum operation unit is 1, so even if there is 0.5 CPU in the server’s free pool, but because the requirement is 7, you leave it in the free pool.
PowerHA SystemMirror gets the remaining 5 GB of this server, all 100 GB from EPCoD and 21 GB from the On/Off CoD. The process is shown in the green table in Figure 6-43 on page 238.
 
Note: During this process, PowerHA SystemMirror adds mobile resources from EPCoD to the server’s free pool first, then adds all the free pool’s resources to the LPAR through DLPAR. In order to describe this clearly, the free pool only means the available resources of one server before adding EPCoD’s resources to it.
The orange table shows (Figure 6-43 on page 238) the result of this scenario, including the LPAR’s running resources, EPCoD, On/Off CoD and the server’s resource status.
Track the hacmp.out log to know what is happening
From hacmp.out, we know that all the resources (7 CPU and 126 memory) costs 117 seconds as a synchronous process as shown in Example 6-26.
22:44:40 → 22:46:37
Example 6-26 The hacmp.out log shows the resource acquisition of example 2
===== Compute ROHA Memory =====
minimal + optimal + running = total <=> current <=> maximum
8.00 + 150.00 + 0.00 = 158.00 <=> 32.00 <=> 160.00 : => 126.00 GB
============ End ==============
=== Compute ROHA PU(s)/VP(s) ==
minimal + optimal + running = total <=> current <=> maximum
1 + 16 + 0 = 17 <=> 3 <=> 18 : => 14 Virtual Processor(s)
minimal + optimal + running = total <=> current <=> maximum
0.50 + 8.00 + 0.00 = 8.50 <=> 1.50 <=> 9.00 : => 7.00 Processing Unit(s)
============ End ==============
===== Identify ROHA Memory ====
Remaining available memory for partition: 5.00 GB
Total Enterprise Pool memory to allocate: 100.00 GB
Total Enterprise Pool memory to yank: 0.00 GB
Total On/Off CoD memory to activate: 21.00 GB for 30 days
Total DLPAR memory to acquire: 126.00 GB
============ End ==============
=== Identify ROHA Processor ===
Remaining available PU(s) for partition: 0.50 Processing Unit(s)
Total Enterprise Pool CPU(s) to allocate: 4.00 CPU(s)
Total Enterprise Pool CPU(s) to yank: 0.00 CPU(s)
Total On/Off CoD CPU(s) to activate: 3.00 CPU(s) for 30 days
Total DLPAR PU(s)/VP(s) to acquire: 7.00 Processing Unit(s) and 14.00 Virtual Processor(s)
============ End ==============
clhmccmd: 100.00 GB of Enterprise Pool CoD have been allocated.
clhmccmd: 4 CPU(s) of Enterprise Pool CoD have been allocated.
clhmccmd: 21.00 GB of On/Off CoD resources have been activated for 30 days.
clhmccmd: 3 CPU(s) of On/Off CoD resources have been activated for 30 days.
clhmccmd: 126.00 GB of DLPAR resources have been acquired.
clhmccmd: 14 VP(s) or CPU(s) and 7.00 PU(s) of DLPAR resources have been acquired.
The following resources were acquired for application controllers App1Controller App2Controller.
DLPAR memory: 126.00 GB On/Off CoD memory: 21.00 GB Enterprise Pool memory: 100.00 GB.
DLPAR processor: 7.00 PU/14.00 VP On/Off CoD processor: 3.00 CPU(s) Enterprise Pool processor: 4.00 CPU(s)
ROHA report update
The clmgr view report roha command reports the ROHA data, as shown in Example 6-27.
Example 6-27 ROHA data after acquire resource in example 2
# clmgr view report roha
Cluster: ITSO_ROHA_cluster of NSC type
Cluster tunables
Dynamic LPAR
Start Resource Groups even if resources are insufficient: '0'
Adjust Shared Processor Pool size if required: '0'
Force synchronous release of DLPAR resources: '0'
On/Off CoD
I agree to use On/Off CoD and be billed for extra costs: '1'
Number of activating days for On/Off CoD requests: '30'
Node: ITSO_S1Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: rar1m3-9117-MMD-1016AAP
LPAR: ITSO_S1Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '158' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '8.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '17' maximum '18'
ROHA provisioning for 'ONLINE' resource groups
No ROHA provisioning.
ROHA provisioning for 'OFFLINE' resource groups
No 'OFFLINE' resource group.
Node: ITSO_S2Node1
HMC(s): 9.3.207.130 9.3.207.133
Managed system: r1r9m1-9117-MMD-1038B9P
LPAR: ITSO_S2Node1
Current profile: 'ITSO_profile'
Memory (GB): minimum '8' desired '32' current '32' maximum '160'
Processing mode: Shared
Shared processor pool: 'DefaultPool'
Processing units: minimum '0.5' desired '1.5' current '1.5' maximum '9.0'
Virtual processors: minimum '1' desired '3' current '3' maximum '18'
ROHA provisioning for 'ONLINE' resource groups
No 'ONLINE' resource group.
ROHA provisioning for 'OFFLINE' resource groups
No ROHA provisioning.
 
Hardware Management Console '9.3.207.130'
Version: 'V8R8.3.0.1'
 
Hardware Management Console '9.3.207.133'
Version: 'V8R8.3.0.1'
 
Managed System 'rar1m3-9117-MMD-1016AAP'
Hardware resources of managed system
Installed: memory '192' GB processing units '12.00'
Configurable: memory '173' GB processing units '11.00'
Inactive: memory '19' GB processing units '1.00'
Available: memory '0' GB processing units '0.50'
On/Off CoD
On/Off CoD memory
State: 'Running'
Available: '9277' GB.days
Activated: '21' GB
Left: '630' GB.days
On/Off CoD processor
State: 'Running'
Available: '9869' CPU.days
Activated: '3' CPU(s)
Left: '90' CPU.days
Yes: 'DEC_2CEC'
Enterprise pool
Yes: 'DEC_2CEC'
Hardware Management Console
9.3.207.130
9.3.207.133
Shared processor pool 'DefaultPool'
Logical partition 'ITSO_S1Node1'
This 'ITSO_S1Node1' partition hosts 'ITSO_S2Node1' node of the NSC cluster 'ITSO_ROHA_cluster'
 
...
 
Enterprise pool 'DEC_2CEC'
State: 'In compliance'
Master HMC: 'e16hmc1'
Backup HMC: 'e16hmc3'
Enterprise pool memory
Activated memory: '100' GB
Available memory: '0' GB
Unreturned memory: '0' GB
Enterprise pool processor
Activated CPU(s): '4'
Available CPU(s): '0'
Unreturned CPU(s): '0'
Used by: 'rar1m3-9117-MMD-1016AAP'
Activated memory: '100' GB
Unreturned memory: '0' GB
Activated CPU(s): '4' CPU(s)
Unreturned CPU(s): '0' CPU(s)
Used by: 'r1r9m1-9117-MMD-1038B9P'
Activated memory: '0' GB
Unreturned memory: '0' GB
Activated CPU(s): '0' CPU(s)
Unreturned CPU(s): '0' CPU(s)
The clmgr view report roha command output (Example 5-8) has some updates about the resources of P770D-01, Enterprise Pool and On/Off CoD.
How to calculate the On/Off CoD consumption
In this case, before bringing the two resource groups online, the remaining resources in On/Off CoD are shown in Example 6-28.
Example 6-28 Remaining resources in On/Off CoD before resource acquisition
On/Off CoD memory
State: 'Available'
Available: '9907' GB.days
On/Off CoD processor
State: 'Available'
Available: '9959' CPU.days
After the resource group is online, the status of the On/Off CoD resource is shown in Example 6-29.
Example 6-29 Status of the memory resources
On/Off CoD memory
State: 'Running'
Available: '9277' GB.days
Activated: '21' GB
Left: '630' GB.days
On/Off CoD processor
State: 'Running'
Available: '9869' CPU.days
Activated: '3' CPU(s)
Left: '90' CPU.days
For processor, PowerHA SystemMirror assigns 3 processors and activate day is 30 days, so the total is 90 CPU.Day. (3*30=90), then the remaining available CPU.Day in the On/Off CoD is 9869 (9959 - 90 = 9869).
For memory, PowerHA SystemMirror assigns 21 GB and activate day is 30 days, so the total is 630 GB.Day. (21*30=630), then the remaining available GB.Day in On/Off CoD is 9277 (9907 - 630 = 9277).
6.11.2 Bring one resource group offline
This section introduces the process of resource group offline. Figure 6-44 shows the overall process.
Figure 6-44 Overall releasing process of example 2
The process is similar to the one shown in 6.9.2, “Move one resource group to another node” on page 223.
In the releasing process, the de-allocation order is On/Off CoD, then EPCoD, and last is the server’s free pool because you always need to pay extra cost for the On/Off CoD.
After the releasing process completes, in the hacmp.out file, you can find the detailed information about compute, identify, and release processes, as shown in Example 6-30.
Example 6-30 The hacmp.out log information in the releasing process of example 2
===== Compute ROHA Memory =====
minimum + running = total <=> current <=> optimal <=> saved
8.00 + 80.00 = 88.00 <=> 158.00 <=> 70.00 <=> 126.00 : => 70.00 GB
============ End ==============
=== Compute ROHA PU(s)/VP(s) ==
minimal + running = total <=> current <=> optimal <=> saved
1 + 9 = 10 <=> 17 <=> 7 <=> 14 : => 7 Virtual Processor(s)
minimal + running = total <=> current <=> optimal <=> saved
0.50 + 4.50 = 5.00 <=> 8.50 <=> 3.50 <=> 7.00 : => 3.50 Processing Unit(s)
============ End ==============
===== Identify ROHA Memory ====
Total Enterprise Pool memory to return back: 49.00 GB
Total On/Off CoD memory to de-activate: 21.00 GB
Total DLPAR memory to release: 70.00 GB
============ End ==============
=== Identify ROHA Processor ===
Total Enterprise Pool CPU(s) to return back: 1.00 CPU(s)
Total On/Off CoD CPU(s) to de-activate: 3.00 CPU(s)
Total DLPAR PU(s)/VP(s) to release: 7.00 Virtual Processor(s) and 3.50 Processing Unit(s)
============ End ==============
clhmccmd: 49.00 GB of Enterprise Pool CoD have been returned.
clhmccmd: 1 CPU(s) of Enterprise Pool CoD have been returned.
The following resources were released for application controllers App1Controller.
DLPAR memory: 70.00 GB On/Off CoD memory: 21.00 GB Enterprise Pool memory: 49.00 GB.
DLPAR processor: 3.50 PU/7.00 VP On/Off CoD processor: 3.00 CPU(s) Enterprise Pool processor: 1.00 CPU(s)
6.12 Hardware Management Console (HMC) high availability introduction
More than one HMC can be configured for a node, so that if one HMC fails to respond, the ROHA functionality can switch to the other HMC.
This section describes the mechanism that enables the HMC to switch from one HMC to another HMC.
Suppose that you have, for a given node, three HMCs in the following order: HMC1, HMC2, and HMC3. (These HMCs can be set either at the node level, at the site level or at the cluster level, as it is explained later in this document. What counts at the end is that you have for a given node an ordered list of HMCs).
All of this means that a given node will use the first HMC in its list, for example HMC1, and use it for as long as it works.
HMC1 could fail for different reasons, for example in the following situations:
1. HMC1 is not reachable via the ping command:
One parameter controls the ping command in the HMC: Timeout on ping (which is set by default to 3 seconds, and you cannot adjust it). If an HMC cannot be pinged after this timeout, you cannot use it through ssh, so switch immediately to another HMC, in this case the HMC following the current one in the list (for example, HMC2).
2. HMC1 is not reachable through SSH:
 – SSH is not properly configured between the node and HMC1, and you can consider that it is not worth trying to use HMC1, and it is best to switch to another HMC. In this case, the HMC following the current one in the list, for example, HMC2.
 – SSH has temporary conditions that prevent it from responding.
Two parameters controls the ssh command on the HMC:
 – Connect Attempts (which is set by default to 5)
 – Connect Timeout (which is set by default to 5), meaning that after a 25-second delay, the HMC can be considered as not reachable through ssh
If the HMC is not reachable through ssh, it is not worth trying to perform an hmc command through ssh on it, and the best is to switch at to another HMC. In this case, the HMC following the current one in the list, for example HMC2.
3. The HMC is repeatedly busy:
When the HMC is processing a command, it cannot perform another command at the same time. The command fails with RC=-1 and with the HSCL3205 message indicating that the HMC is busy.
PowerHA SystemMirror ROHA functionality has a retry mechanism that is controlled with two parameters:
 – RETRY_COUNT, indicating how many retries must be done
 – RETRY_DELAY, indicating how long to wait between retries.
This means that when the HMC is busy, the retry mechanism is used until declaring that the HMC is flooded.
When the HMC is considered flooded, it is not worth using it again, and the best is to switch immediately to another HMC, the HMC following the current one in the list, for example HMC2.
4. The HMC returns an application error. Several cases can occur:
 – One case is for example when you request for an amount of resources which is not available, and the same request is attempted with another smaller amount.
 – A second case is when the command is not understandable by the HMC which is more like a programming bug, and in these cases should be debugged at test time. In any case, this is not a reason to switch to another HMC.
If you decide to switch to another HMC, consider the next HMC of the list, and use it.
Remember to note that the first HMC is not usable (HMC1), and that you are currently using the second HMC in the list (HMC2). This helps to prevent the ROHA functionality from trying again and failing again using the first HMC (HMC1). You can add (persistence) into the ODM which HMC is being used (for example, HMC2).
This mechanism enables the ROHA functionality to skip the failing HMCs, and to use the HMC that works (in this case, HMC2). At the end of the session, the persistence into the ODM is cleared, meaning that the first HMC in the list is restored to its role of HMC1 or the first in the list.
6.12.1 Switch to the backup HMC for the Power Enterprise Pool
For Enterprise Pool operations, querying operations can be run on the master or backup HMC, but changing operations must run on the master HMC. If the master HMC fails, PowerHA SystemMirror’s actions are as follows:
•For querying operations, PowerHA SystemMirror tries to switch to the backup HMC to continue the operation, but does not set the backup HMC as the master.
•For changing operations, PowerHA SystemMirror tries to set the backup HMC as the master, and then continues the operation. Example 6-31 shows the command that PowerHA SystemMirror performs to set the backup HMC as the master. This command is triggered by PowerHA SystemMirror automatically.
Example 6-31 Setting the backup HMC as the master
chcodpool -o setmaster -p <pool> --mc backup
There are some prerequisites in PowerHA SystemMirror before switching to the backup HMC when the master HMC fails:
•Configure the master HMC and the backup HMC for your Power Enterprise Pool.
For more information about how to configure the backup HMC for the Power Enterprise Pool, read the following two references:
•Both HMCs are configured in PowerHA SystemMirror.
•Establish password-less communication between the PowerHA SystemMirror nodes to the two HMCs.
•Ensure reachability (pingable) from PowerHA SystemMirror nodes to the master and backup HMCs.
•Ensure that all of the servers that participate in the pool are connected to the two HMCs.
•Ensure that the participating servers are in either Standby state or the Operating state.
6.13 Test scenario for HMC fallover
This section shows how PowerHA SystemMirror switches the HMC automatically when the primary HMC fails.
6.13.1 Hardware topology
Figure 6-45 shows the initial status of the hardware topology.
Figure 6-45 Initial status of the hardware topology
The topology includes the following components:
•Two Power 780 D model servers, named P780D_09 and P780D_10.
•One Power Enterprise Pool, which has 64 mobile processors and 2 TB of mobile memory resources.
•There are 64 CPUs and 2 TB of memory installed in P780D_09, 32 CPUs, and 1 TB of memory are configured, and another 32 CPUs and 1 TB of memory are in inactive status. At this time, there are 5 CPUs and 10 GB of memory available for the DLPAR.
•The PowerHA SystemMirror cluster includes two nodes:
 – P780_09_Lpar1
 – P780_10_Lpar2
•The PowerHA SystemMirror cluster includes one resource group (RG2), this resource group has one application controller (app2) configured hardware resource provisioning.
•This application needs 20 C and 40 G when it runs.
•There is no On/Off CoD in this testing.
There are two HMCs to manage the EPCoD, named HMCEP1 and HMCEP2.
HMCEP1 is the master and HMCEP2 is the backup, as shown in Example 6-32.
Example 6-32 HMCs available
hscroot@HMCEP1:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=0
In the AIX /etc/hosts file, define the resolution between the HMC IP address, and the HMC’s host name, as shown in Example 6-33.
Example 6-33 Define resolution between HMC IP and HMC name in the /etc/hosts
172.16.50.129 P780_09_Lpar1
172.16.50.130 P780_10_Lpar1
172.16.51.129 testservice1
172.16.51.130 testservice2
172.16.50.253 HMCEP1
172.16.50.254 HMCEP2
Then start the PowerHA SystemMirror service on P780_09_Lpar1. During the start, PowerHA SystemMirror will acquire resources from the server’s free pool and EPCoD (Figure 6-46).
Figure 6-46 Resource Acquisition process during the start of PowerHA SystemMirror service
In this process, HMCEP1 acts as the primary HMC and does all the query and resource acquisition operations. Example 6-34 and Example 6-35 show the detailed commands used in the acquisition step.
Example 6-34 EPCoD operation during resource acquisition (hacmp.out)
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@HMCEP1 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o add -q 23552 2>&1' -->23552 means 23GB
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@HMCEP1 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r proc -o add -q 14 2>&1'
Example 6-35 DLPAR add operation in the acquire step
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@172.16.50.253 'chhwres -m SVRP7780-09-SN060C0AT -p P780_09_Lpar1 -r mem -o a -q 33024 -w 32 2>&1' -->33024 means 32.25GB
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no hscroot@172.16.50.253 'chhwres -m SVRP7780-09-SN060C0AT -p P780_09_Lpar1 -r proc -o a --procs 19 -w 32 2>&1 -->172.16.50.253 is HMCEP1
 
Note: We do not display the DLPAR and EPCoD operations in the query step in the previous examples.
6.13.2 Bring one resource group offline when primary HMC fails
After the resource group is online, we bring the resource group offline. During this process, we shut down HMCEP1 to see how PowerHA SystemMirror handles this situation.
The resource releasing process is shown in Figure 6-47.
Figure 6-47 Bring the resource group offline process
Section 6.6, “Introduction to resource acquisition” on page 195 introduces the four steps for PowerHA SystemMirror to acquire the resources. In this case, the following is the detailed description of the four steps.
Query step
In this step, PowerHA SystemMirror needs to query the server’s data and the EPCoD’s data.
For getting the server’s information, PowerHA SystemMirror uses the default primary HMC (172.16.50.253, HMCEP1). At first, HMCEP1 is alive, the operation succeeds. But after HMCEP1 shutdown, the operation fails and PowerHA SystemMirror uses 172.16.50.254 as the primary HMC to continue. Example 6-36 on page 251 shows the takeover process.
Example 6-36 HMC takeover process
+testRG2:clhmccmd[get_local_hmc_list:815] g_hmc_list='172.16.50.253 172.16.50.254'
--> default, the global HMC list is:172.16.50.253 is first, then 172.16.50.254
...
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 172.16.50.253
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=1
+testRG2:clhmccmd[clhmcexec:3514] (( 1 > 0 ))
+testRG2:clhmccmd[clhmcexec:3516] : Cannot contact this HMC. Ask following HMC in list.
--> after checking, confirm that 172.16.50.253 is unaccessible, then to find next HMC in the list
...
+testRG2:clhmccmd[clhmcexec:3510] : Try to ping the HMC at address 172.16.50.254.
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 172.16.50.254
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=0
+testRG2:clhmccmd[clhmcexec:3514] (( 0 > 0 ))
--> 172.16.50.254 is the next, so PowerHA SystemMirror check it
...
+testRG2:clhmccmd[update_hmc_list:3312] g_hmc_list='172.16.50.254 172.16.50.253'
--> it is accessable, change it as first HMC in globale HMC list
...
+testRG2:clhmccmd[clhmcexec:3456] loop_hmc_list='172.16.50.254 172.16.50.253'
--> global HMC list has been changed, following operation will use 172.16.50.254
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no [email protected] 'lshmc -v 2>&1'
--> start with 172.16.50.254 to do query operation
...
+testRG2:clhmccmd[clhmcexec:3618] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o Conne
ctionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lscodpool -p 0019 --level sys --filter names=SVRP7780-09-SN060C0AT -F inactive_mem:mobile_mem:unreturne
d_mobile_mem:inactive_procs:mobile_procs:unreturned_mobile_procs 2>&1''
+t
--> using 172.16.50.254 to query EPCoD information
 
Important: The process enables you to query EPCoD information from the backup HMC. However, any change operations need to be done on the master HMC.
Compute step
This step does not need to do an HMC operation. See Figure 6-47 on page 250 for the detailed process.
Identify and acquire step
After the identify step, there are some resources that are needed to release to EPCoD. Therefore, PowerHA SystemMirror returns the resource back to EPCoD immediately before the resource is removed from the LPAR. This generates an unreturned resource temporarily.
At this time, PowerHA SystemMirror checks if the master HMC is available. If not, switches to the backup HMC automatically. Example 6-37 shows the detailed process.
Example 6-37 The EPCoD master and backup HMC switch process
+testRG2:clhmccmd[clhmcexec:3388] cmd='chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o remove -q 23552 --force'
-->PowerHA SystemMirror try to do chcodpool operation
...
+testRG2:clhmccmd[clhmcexec:3401] : If working on an EPCoD Operation, we need master
-->PowerHA SystemMirror want to check if master HMC is accessiable
...
ctionAttempts=3 -o TCPKeepAlive=no $'[email protected] 'lscodpool -p 0019 --level pool -F master_mc_name:backup_master_mc_name 2>&1''
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -o TCPKeepAlive=no [email protected] 'lscodpool -p 0019 --level pool -F master_mc_name:backup_master_mc_name 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3415] res=HMCEP1:HMCEP2
-->Current HMC is 172.16.50.254, so PowerHA SystemMirror query current master and backup HMC name from it. At this time, HMCEP1 is master and HMCEP2 is backup.
...
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 HMCEP1
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=1
+testRG2:clhmccmd[clhmcexec:3514] (( 1 > 0 ))
+testRG2:clhmccmd[clhmcexec:3516] : Cannot contact this HMC. Ask following HMC in list.
+testRG2:clhmccmd[clhmcexec:3518] dspmsg scripts.cat -s 38 500 '%1$s: WARNING: unable to ping HMC at address %2$s. ' clhmccmd HMCEP1
-->PowerHA SystemMirror try to ping HMCEP1, but fails
...
+testRG2:clhmccmd[clhmcexec:3510] : Try to ping the HMC at address HMCEP2.
+testRG2:clhmccmd[clhmcexec:3512] ping -c 1 -w 3 HMCEP2
+testRG2:clhmccmd[clhmcexec:3512] 1> /dev/null 2>& 1
+testRG2:clhmccmd[clhmcexec:3512] ping_output=''
+testRG2:clhmccmd[clhmcexec:3513] ping_rc=0
+testRG2:clhmccmd[clhmcexec:3514] (( 0 > 0 ))
-->PowerHA SystemMirror try to verify HMCEP2 and it is available
...
+testRG2:clhmccmd[clhmcexec:3527] : the hmc is the master_hmc
+testRG2:clhmccmd[clhmcexec:3529] (( g_epcod_modify_operation == 1 && loop_hmc_counter != 1 ))
+testRG2:clhmccmd[clhmcexec:3531] : If not, we need to change master_hmc, we also try to
+testRG2:clhmccmd[clhmcexec:3532] : set a backup_master_hmc
 
+testRG2:clhmccmd[clhmcexec:3536] eval LC_ALL=C ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o Conne
ctionAttempts=3 -o TCPKeepAlive=no $'hscroot@HMCEP2 'chcodpool -p 0019 -o setmaster --mc this 2>&1''
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -o setmaster --mc this 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3536] out_str=''
+testRG2:clhmccmd[clhmcexec:3537] ssh_rc=0
-->PowerHA SystemMirror set backup HMC(HMCEP2) as master
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -o update -a "backup_master_mc_name=HMCEP1" 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
+testRG2:clhmccmd[clhmcexec:3722] out_str='HSCL90E9 Management console HMCEP1was not found.'
-->PowerHA SystemMirror also try to set HMCEP1 as backup, but it fails because HMCEP1 is shutdown at this time
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r mem -o remove -q 23552 --force 2>&1'
+testRG2:clhmccmd[clhmcexec:1] LC_ALL=C
-->PowerHA SystemMirror do the force release for memory resource
...
+testRG2:clhmccmd[clhmcexec:1] ssh -o StrictHostKeyChecking=no -o LogLevel=quiet -o AddressFamily=any -o BatchMode=yes -o ConnectTimeout=3 -o ConnectionAttempts=3 -
o TCPKeepAlive=no hscroot@HMCEP2 'chcodpool -p 0019 -m SVRP7780-09-SN060C0AT -r proc -o remove -q 14 --force 2>&1'
-->PowerHA SystemMirror do the force release for CPU resource
Example 6-38 shows the update that is performed from EPCoD’s view.
Example 6-38 EPCoD status change during the takeover operation
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=50,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2024448,unreturned_mobile_mem=0
--> There are 14CPU(64-50) and 23GB((2048000-2024448)/1024) has assigned to P780D_09.
hscroot@HMCEP2:~> lscodpool -p 0019 --level sys
name=SVRP7780-10-SN061949T,mtms=9179-MHD*061949T,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=32,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1024000,installed_mem=2097152
name=SVRP7780-09-SN060C0AT,mtms=9179-MHD*060C0AT,mobile_procs=14,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=18,installed_procs=64,mobile_mem=23552,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1000448,installed_mem=2097152
--> Show the information from server level report
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=unavailable,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=unavailable,avail_mobile_procs=unavailable,unreturned_mobile_procs=unavailable,mobile_mem=unavailable,avail_mobile_mem=unavailable,unreturned_mobile_mem=unavailable
--> After HMCEP1 shutdown, the EPCoD’s status is changed to 'unavailable'
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP2,master_mc_mtms=V017-f93*ba3e3aa,backup_master_mc_mtms=none,mobile_procs=64,avail_mobile_procs=50,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2024448,unreturned_mobile_mem=0
--> After PowerHA SystemMirror run 'chcodpool -p 0019 -o setmaster --mc this' on HMCEP2, the master HMC is changed and status is changed to 'In compliance'
....
hscroot@HMCEP2:~> lscodpool -p 0019 --level sys
name=SVRP7780-10-SN061949T,mtms=9179-MHD*061949T,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=0,inactive_procs=32,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=0,inactive_mem=1024000,installed_mem=2097152
name=SVRP7780-09-SN060C0AT,mtms=9179-MHD*060C0AT,mobile_procs=0,non_mobile_procs=32,unreturned_mobile_procs=14,inactive_procs=18,installed_procs=64,mobile_mem=0,non_mobile_mem=1073152,unreturned_mobile_mem=23552,inactive_mem=1000448,installed_mem=2097152
--> After PowerHA SystemMirror forcely release resource, ureturned resource is generated
...
hscroot@HMCEP2:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=Approaching out of compliance (within server grace period),sequence_num=5,master_mc_name=HMCEP2,master_mc_mtms=V017-f93*ba3e3aa,backup_master_mc_mtms=none,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=14,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=23553
--> At this time, the resource has returned to EPCoD and can be used by other servers.
..
When PowerHA SystemMirror completes the above steps, it raises an asynchronous process to remove the resources from P780_09_Lpar1 using DLPAR. The resources include 19 CPUs and 32.25 GB of memory.
After the DLPAR operation, the unreturned resource is reclaimed automatically, and EPCoD’s status is changed to In compliance, as shown in Example 6-39.
Example 6-39 EPCoD’s status restored after DLPAR operation complete
hscroot@HMCEP1:~> lscodpool -p 0019 --level pool
name=0019,id=0019,state=In compliance,sequence_num=5,master_mc_name=HMCEP1,master_mc_mtms=V017-ffe*d33e8a1,backup_master_mc_name=HMCEP2,backup_master_mc_mtms=V017-f93*ba3e3aa,mobile_procs=64,avail_mobile_procs=64,unreturned_mobile_procs=0,mobile_mem=2048000,avail_mobile_mem=2048000,unreturned_mobile_mem=0
6.13.3 Testing summary
This scenario introduced how PowerHA SystemMirror performs HMC takeovers when the primary HMC fails. This is an automatic process and has no impact to your environment.
6.14 Manage, monitor and troubleshooting
This section introduces some tools to manage, monitor, and troubleshoot a ROHA cluster.
6.14.1 The clmgr interface to manage ROHA
SMIT relies on the clmgr command to perform configuration that is related with ROHA.
HMC configuration
The following examples show how to configure HMC with the clmgr command.
Query/Add/Modify/Delete
Example 6-40 shows how to query, add, modify, and delete HMC with the clmgr command.
Example 6-40 query/add/modify/delete HMC with clmgr command
# clmgr query hmc -h
clmgr query hmc [<HMC>[,<HMC#2>,...]]
 
# clmgr –v query hmc
NAME="r1r9sdmc.austin.ibm.com"
TIMEOUT="-1"
RETRY_COUNT="8"
RETRY_DELAY="-1"
NODES=clio1,clio2
SITES=site1
 
# clmgr add hmc -h
clmgr add hmc <HMC>
[ TIMEOUT={<#>} ]
[ RETRY_COUNT={<#>} ]
[ RETRY_DELAY={<#>} ]
[ NODES=<node>[,<node#2>,...]> ]
[ SITES=<site>[,<site#2>,...]> ]
[ CHECK_HMC={<yes>|<no>} ]
# clmgr modify hmc -h
clmgr modify hmc <HMC>
[ TIMEOUT={<#>} ]
[ RETRY_COUNT={<#>} ]
[ RETRY_DELAY={<#>} ]
[ NODES=<node>[,<node#2>,...]> ]
[ SITES=<site>[,<site#2>,...]> ]
[ CHECK_HMC={<yes>|<no>} ]
 
# clmgr delete hmc -h
clmgr delete hmc {<HMC>[,<HMC#2>,...] | ALL}
Query/Modify a node with the list of associated HMC
Example 6-41 shows how to query and modify a node with the list of associated HMCs.
Example 6-41 query/modify node with a list of associated HMC with the clmgr command
# clmgr query node -h
clmgr query node {<node>|LOCAL}[,<node#2>,...]
 
# clmgr –v query node
NAME="rar1m31"
…
HMCS="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr modify node -h
clmgr modify node <NODE>
…
[ HMCS=<sorted_hmc_list> ]
Query/Modify a site with the list of associated HMC
Example 6-42 shows how to query and modify the site with a list of associated HMCs with the clmgr command.
Example 6-42 query/modify site with a list of associated HMCs with clmgr command
# clmgr query site -h
clmgr query site [<site> [,<site#2>,...]]
 
# clmgr –v query site
NAME="site1"
…
HMCS="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr modify site -h
clmgr modify site <SITE>
…
[ HMCS =<sorted_hmc_list> ]
Query/Modify cluster with default HMC tunables
Example 6-43 shows how to query and modify the cluster with the default HMC tunables.
Example 6-43 query/modify cluser with default HMC tunables with clmgr command
# clmgr query cluster -h
clmgr query cluster [ ALL | {CORE,SECURITY,SPLIT-MERGE,HMC,ROHA} ]
 
# clmgr query cluster hmc
DEFAULT_HMC_TIMEOUT="10"
DEFAULT_HMC_RETRY_COUNT="5"
DEFAULT_HMC_RETRY_DELAY="10"
DEFAULT_HMCS_LIST="r1r9sdmc.austin.ibm.com cuodhmc.austin.ibm.com"
 
# clmgr manage cluster hmc -h
clmgr manage cluster hmc
[ DEFAULT_HMC_TIMEOUT=# ]
[ DEFAULT_HMC_RETRY_COUNT=# ]
[ DEFAULT_HMC_RETRY_DELAY=# ]
[ DEFAULT_HMCS_LIST=<new_hmcs_list> ]
Hardware resource provisioning
SMIT relies on the clmgr command to List or Query current values of the hardware resource provisioning and to Add/Modify/Delete the HACMPserver ODM data structure, as shown in Example 6-44.
Example 6-44 Hardware resource provisioning configuration with the clmgr command
# clmgr query cod -h
clmgr query cod [<APP>[,<APP#2>,...]]
 
# clmgr –v query cod
NAME="appli1_APPCON_A"
USE_DESIRED=No
OPTIMAL_MEM="4"
OPTIMAL_CPU="3"
OPTIMAL_PU="2.5"
OPTIMAL_PV="3.0"
 
 
# clmgr add cod -h
clmgr add cod <APPCTRL>
[ USE_DESIRED =<Yes|No> ]
[ OPTIMAL_MEM=# ]
[ OPTIMAL_CPU=# ]
[ OPTIMAL_PU=#.# ]
[ OPTIMAL_PV=#.# ]
 
# clmgr modify cod -h
clmgr modify cod <APPCTRL>
[ USE_DESIRED =<Yes|No> ]
[ OPTIMAL_MEM=# ]
[ OPTIMAL_CPU=# ]
[ OPTIMAL_PU=#.# ]
[ OPTIMAL_PV=# ]
 
# clmgr delete cod -h
clmgr delete cod {<APPCTRL> | ALL}
Cluster tunables
SMIT relies on the clmgr command to query or modify cluster CoD tunables, as shown in Example 6-45.
Example 6-45 Cluster wide tunables configuration with clmgr command
# clmgr query cluster -h
clmgr query cluster [ ALL | {CORE,SECURITY,SPLIT-MERGE,HMC,ROHA} ]
 
# clmgr query cluster roha
ALWAYS_START_RG="no"
ADJUST_SPP_SIZE="yes"
FORCE_SYNC_RELEASE="no"
AGREE_TO_COD_COSTS="no"
COD_ONOFF_DAYS="30"
SECONDARY_LPARS_ENABLE="no"
SECONDARY_LPARS_POLICY=""
SECONDARY_LPARS_THRESHOLD=""
 
# clmgr manage cluster roha –h
clmgr manage cluster roha
[ ALWAYS_START_RG={yes|no} ]
[ ADJUST_SPP_SIZE={yes|no} ]
[ FORCE_SYNC_RELEASE={yes|no} ]
[ AGREE_TO_COD_COSTS={yes|no} ]
[ COD_ONOFF_DAYS=<new_number_of_days> ]
[ SECONDARY_LPARS_ENABLE={yes|no} ]
[ SECONDARY_LPARS_POLICY={minimize|shutdown} ]
[ SECONDARY_LPARS_THRESHOLD=<priority> ]
6.14.2 Changing the DLPAR and CoD resources dynamically
You can change the DLPAR and CoD resource requirements for application controllers without stopping the cluster services. Remember to synchronize the cluster after making the changes.
The new configuration is not reflected until the next event that causes the application (hence the resource group) to be released and reacquired on another node. In other words, a change in the resource requirements for CPUs, memory or both does not cause the recalculation of the DLPAR resources. PowerHA SystemMirror does not stop and restarts the application controllers solely for the purpose of making the application provisioning changes.
If another dynamic reconfiguration change causes the resource groups to be released and reacquired, the new resource requirements for DLPAR and CoD are used at the end of this dynamic reconfiguration event.
6.14.3 View the ROHA report
The clmgr view report roha command is intended to query all the Resource Optimized High Availability data, so that a report and a summary can be presented to the user.
The output of this command includes the following sections:
•CEC name
•LPAR name
•LPAR profile (min, desired, max)
•LPAR processing mode
•If shared (capped or uncapped, SPP name, SPP size)
•LPAR current level of resources (mem, cpu, pu)
•Number and names of AC and optimal level of resources, and the sum of them
•Release mode (sync/async) which would be computed at release time
•All On/Off CoD information of the CECs
•All EPCoD information of the CECs
There is an example of the report in 6.10.4, “Showing the ROHA configuration” on page 235.
6.14.4 Troubleshooting DLPAR and CoD operations
This section provides a brief troubleshooting of the DLPAR and CoD operations.
Log files
There are several log files use to track ROHA operation process.
Logs for verification
In the process of Verify and Synchronize Cluster Configuration, there are some log files generated in the /var/hacmp/clverify directory. The clverify.log and the ver_collect_dlpar.log files are useful for debugging if the process fails. For example, after performing the process, there is some error information appearing in the console output(/smit.log), as shown in Example 6-46.
Example 6-46 Error information on console or /smit.log
WARNING: At the time of verification, node ITSO_S2Node1 would not have been able to acquire
sufficient resources to run Resource Group(s) RG1 (multiple Resource Groups
in case of node collocation). Please note that the amount of resources and
CoD resources available at the time of verification may be different from
the amount available at the time of an actual acquisition of resources.
Reason : 708.00 GB of memory needed will exceed LPAR maximum of 160.00 GB. 12.50 Processing Unit(s) needed will exceed LPAR maximum of 9.00 Processing Unit(s).
ERROR: At the time of verification, no node (out of 2) was able to acquire
sufficient resources to run Resource Group(s) RG1
You can get detailed information to help you identify errors’ root causes from the clverify.log and the ver_collect_dlpar.log files, as shown in Example 6-47.
Example 6-47 Detailed information in ver_collect_dlpar.log
[ROHALOG:2490918:(19.127)] clmanageroha: ERROR: 708.00 GB of memory needed will exceed LPAR maximum of 160.00 GB.
[ROHALOG:2490918:(19.130)] ===== Compute ROHA Memory =====
[ROHALOG:2490918:(19.133)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.137)] 8.00 + 700.00 + 0.00 = 708.00 <=> 32.00 <=> 160.00 : => 0.00 GB
[ROHALOG:2490918:(19.140)] ============ End ==============
[ROHALOG:2490918:(19.207)] clmanageroha: ERROR: 12.50 Processing Unit(s) needed will exceed LPAR maximum of 9.00 Processing Unit(s).
[ROHALOG:2490918:(19.212)] === Compute ROHA PU(s)/VP(s) ==
[ROHALOG:2490918:(19.214)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.217)] 1 + 12 + 0 = 13 <=> 3 <=> 18 : => 0 Virtual Processor(s)
[ROHALOG:2490918:(19.220)] minimal + optimal + running = total <=> current <=> maximum
[ROHALOG:2490918:(19.223)] 0.50 + 12.00 + 0.00 = 12.50 <=> 1.50 <=> 9.00 : => 0.00 Processing Unit(s)
[ROHALOG:2490918:(19.227)] ============ End ==============
[ROHALOG:2490918:(19.231)] INFO: received error code 21.
[ROHALOG:2490918:(19.233)] No or no more reassessment.
[ROHALOG:2490918:(19.241)] An error occurred while performing acquire operation.
PowerHA SystemMirror simulates the resource acquisition process based on the current configuration and generate the log in the ver_collect_dlpar.log file.
Logs for resource group online and offline
During the process of resource online or offline, the hacmp.out and the async_release.log logs are useful for monitoring or debugging. In some resource group offline scenarios, the DLPAR remove operation is a synchronous process. In this case, PowerHA SystemMirror generates the DLPAR operation logs into the async_release.log file. In a synchronous process, only the hacmp.out is used.
AIX errpt output
Sometimes, the DLPAR operation fails, and AIX generates some errors that are found in the errpt output, as shown in Example 6-48.
Example 6-48 The errpt error report
252D3145 1109140415 T S mem DR failed by reconfig handler
47DCD753 1109140415 T S PROBEVUE DR: memory remove failed by ProbeVue rec
You can identify the root cause of the failure using this information.
HMC commands
You can use the following commands on the HMC to do some monitor or maintenance. See the HMC man manual for a detailed description about the commands.
The lshwres command
This command shows the LPAR minimum, LPAR maximum, the total amount of memory, and the number of CPUs that are currently allocated to the LPAR.
The lssyscfg command
This command verifies that the LPAR node is DLPAR capable.
The chhwres command
This command runs the DLPAR operations on the HMC outside of PowerHA SystemMirror to manually change the LPAR minimum, LPAR minimum and LPAR required values for the LPAR. This might be necessary if PowerHA SystemMirror issues an error or a warning, during the verification process, if you requested to use DLPAR and CoD resources in SystemMirror.
The lscod command
This command views the system CoD of the current configuration.
The chcod command
This command runs the CoD operations on the HMC outside of PowerHA SystemMirror and to manually change the Trial CoD, On/Off CoD, and so on, of the activated resources. This is necessary if PowerHA SystemMirror issues an error or a warning during the verification process, or if you requested to use DLPAR and On/Off CoD resources in PowerHA SystemMirror.
The lscodpool command
This command views the system Enterprise Pool current configuration.
The chcodpool command
This command runs the Enterprise Pool CoD operations on the HMC outside of PowerHA SystemMirror and to manually change the Enterprise pool capacity resources. This is necessary if PowerHA SystemMirror issues an error or a warning during the verification process, or if you requested to use DLPAR, On/Off CoD, or EPCoD resources in PowerHA SystemMirror.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset