Oracle Management Agent

The Oracle Management Agent is responsible for sending metrics and pending alerts for hosts on which it is installed. To provide high availability, the following features should be enabled for the agent:

• Enable the agent to start automatically on OS boot up. This should be configured automatically after running the root.sh script.
• Install the agent on redundant storage. This includes a RAID-configured storage back end.
• Configure restart parameters for the agent watchdog process. The watchdog process is responsible for restarting the agents based on the following environment variables:

• EM_MAX_RETRIES  indicates the maximum number of times the watchdog process will attempt to restart the agent within the EM_MAX_RETRY_WINDOW. The default is set to 3.
• EM_RETRY_WINDOW  indicates the time interval that is used with EM_MAX_RETRIES before trying to start the management agent. The default is set to 600 seconds.

After installing the Enterprise Manager agent, the <AGENT_HOME>/root.sh script should be executed. This will create the following scripts on Linux and some Unix operating systems that control the startup of the management agent.

• /etc/init.d/gcstartup
• /etc/init/gcstartup
• /etc/rc.d/init.d/gcstartup

If these files are not present, make sure the <AGENT_HOME>/root.sh script has been executed.

Loss or corruption of the agent will result in loss of monitoring and metric data uploads for its associated targets. Likewise, if the agent is down, targets will not be able to communicate with the management server, resulting in loss of manageability.

Management Repository

The repository is the persistent store for monitoring data uploaded by agents. It stores metrics and configuration data from all monitored targets. Loss of the repository will result in management server failure. It is recommended to enable database high-availability features such as Real Application Clusters (RAC) or RAC One Node, Automatic Storage Management (ASM) for data file storage (depending on redundancy levels and back-end RAID configuration),and Data Guard for the repository. Additionally, the management service should be configured to use the Single Client Access Name (SCAN) and a nondefault RAC service name if the repository is configured as a RAC database. The following is an example of configuring the management service for use with a RAC database:

emctl config oms -store_repos_details -repos_conndesc "
(DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)
(HOST=emrep-scan.example.com)(PORT=1521) )
(CONNECT_DATA=(SERVER=DEDICATED) (SERVICE_NAME = emrep)))"
-repos_user sysman

Figure 13-2 shows an example of an OMS configured with a RAC repository database using the SCAN name. For additional details on configuring SCAN, see the Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2).

Consider using the Maximum Availability Architecture¹ (MAA) Advisor in Enterprise Manager Cloud Control 12c to configure additional HA components, including the following:

• Enable ARCHIVELOG mode so that consistent hot backups of the database can be performed.
• Enable block checksums to enable Oracle to detect corruption due to physical disks or storage systems. Set the DB_BLOCK_CHECKSUM parameter to TYPICAL or FULL.
• Configure the size of redo log files and groups appropriately. Optimize I/O throughput and performance of the database so that log switches do not occur too frequently. Use of the Redo Log Sizing Advisor (enabled by setting FAST_START_MTTR_TARGET) can help determine the optimal sizes.
• Use a Fast Recovery Area² to store recovery-related files such as redo logs and archive logs. Oracle can then use suggested backup mechanisms that use the Fast Recovery Area.
• Enable Flashback Database to enable faster database recovery using flashback logs.
• Use fast-start fault recovery to control the instance-recovery time. Set the initialization parameter FAST_START_MTTR_TARGET to specify the number of seconds that instance recovery should take.
• Enable database block checking to enable early detection and prevention of data and memory corruption. Set DB_BLOCK_CHECKING to TRUE or FULL.

After configuring the components, you can monitor the status of each by using the High Availability Console, shown in Figure 13-3.

A physical standby database is recommended to provide disaster recovery in case of a failure at the primary site. Data on the primary repository will be kept in sync with the standby database. When configuring a physical standby for the repository database, use similar hardware and resources as the primary site so that there aren’t any performance implications in the event of a failover/switchover. Use Enterprise Manager to create a standby database from the primary database. Determine the network mode that you want to use to synchronize the standby database based on your network bandwidth and recovery objectives:

• Maximum Protection: Synchronous writes to standby
• Maximum Availability: Synchronous writes to standby
• Maximum Performance: Asynchronous writes to standby

Management Service

The management service, or OMS, provides a user interface via the Enterprise Manager console and processes data from agents. A loss of the OMS will result in a complete Enterprise Manager outage: agent uploads, jobs, incidents, and notifications will all stop to function. The Oracle WebLogic Node Manager and the Oracle Process Manager and Notification Server (OPMN) will attempt to restart the management service automatically if it is down. Although this provides some benefit, it will not protect the OMS if the host is down. At a minimum, the OMS and repository should be installed on separate hosts if possible. Multiple OMSs can be deployed behind a server load balancer to provide protection against a single host being down. Also, you can opt to install the OMS on a shared filesystem, which will provide passive failover in case of the loss of a single host. You will look at each of these options for protecting the OMS in further detail.

$ $OMS_HOME/bin/emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2  Copyright (c) 1996, 2012 Oracle
Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem1.example.com
HTTP Console Port          : 7790
HTTPS Console Port         : 7803
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4904
EM Instance Home           : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oem1.example.com:443/em
Upload URL: https://oem1.example.com:4904/empbs/upload
 
WLS Domain Information
Domain Name      : GCDomain
Admin Server Host: oem1.example.com
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem1.example.com
WebTier is Up
Oracle Management Server is Up

emctl secure oms –sysman_pwd <sysman_password> 
       -reg_pwd <agent_reg_password> 
       -host <virtualhostname> 
       -secure_port 4899 
       -slb_port 4899 
       -slb_console_port 443 
       -console 
       –lock_console

$OMS_HOME/bin/emctl secure oms -sysman_pass 
-reg_pwd regpass 
-host slb.example.com 
-secure_port 4904 
-slb_port 4904 
-slb_console_port 443 
-reset 
-console 
-lock_console
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Securing OMS... Started.
Securing OMS... Successful
Restart OMS

emctl stop oms –all
 
emctl start oms

emctl status oms –details

$ $OMS_HOME/bin/emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2  Copyright (c) 1996, 2012 Oracle
Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem1.example.com
HTTP Console Port          : 7790
HTTPS Console Port         : 7803
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4904
EM Instance Home           : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log
SLB or virtual hostname: slb.example.com
HTTPS SLB Upload Port : 4904
HTTPS SLB Console Port : 443
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://slb.example.com:443/em
Upload URL: https://slb.example.com:4904/empbs/upload
 
WLS Domain Information
Domain Name      : GCDomain
Admin Server Host: oem1.example.com
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem1.example.com
WebTier is Up
Oracle Management Server is Up

emctl secure agent –emdWalletSrcUrl
https://slb.example.com:4899/em

$ $AGENT_HOME/core/12.1.0.2.0/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : 12.1.0.2.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u02/app/oracle/agent12c/agent_inst
Agent Binaries    : /u02/app/oracle/agent12c/core/12.1.0.2.0
Agent Process ID  : 19494
Parent Process ID : 19454
Agent URL         : https://oem1.example.com:3872/emd/main/
Repository URL    : https://slb.example.com:4904/empbs/upload
Started at        : 2012-11-03 12:44:10
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2012-11-03 12:47:04
Last attempted upload                        : 2012-11-03 12:47:05
Total Megabytes of XML files uploaded so far : 1.09
Number of XML files pending upload           : 44
Size of XML files pending upload(MB)         : 0.64
Available disk space on upload filesystem    : 44.63%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2012-11-03 12:46:09
Last successful heartbeat to OMS             : 2012-11-03 12:46:09
Next scheduled heartbeat to OMS              : 2012-11-03 12:47:09
 
---------------------------------------------------------------
Agent is Running and Ready

emcli setup –url=
https://slb.example.com/em
–username=sysman

Software Library

The Software Library is used for patching, agent deployment, provisioning, and self-updating. It should be installed on shared storage so that multiple management servers can access it. Any of the following options can be used for shared storage:

• NFS
• OCFS2
• DBFS
• ACFS

Because the Software Library is so critical, it should be located on a highly available file system. This means using a RAID (mirrored and striped) back-end storage system.

The Software Library can be configured by logging into the Enterprise Manager console and choosing Setup Provisioning and Patching Software Library. Choose a location on a shared filesystem that has been mounted on the OMS for the Software Library (see Figure 13-29). Multiple locations can also be added.

Repository Backup

The first step in configuring a backup strategy for the repository database is to ensure that it is in ARCHIVELOG mode. This will allow consistent online backups to be taken and also facilitates point-in-time recovery of the database. Hot backups should be taken regularly using recommended backup strategies. Database high-availability features such as the Fast Recovery Area and flashback database should also be enabled to allow for faster recovery of the database in the event of a failure. Backups can be configured by using the Enterprise Manager Cloud Control console. Consider using Enterprise Manager’s Recommended Backup Strategy option to back up the repository database.

To schedule a database backup using Enterprise Manager, follow these steps:

1. Go to the repository database home page and choose Availability Backup & Recovery Schedule Backup.
2. Log in as a user with SYSDBA credentials.
3. Click the Schedule Oracle-Suggested Backup button to create a fully automated backup strategy based on the destination for the backups (Disk, Tape, and so forth), as shown in Figure 13-30. The Oracle-suggested backup is based on an incrementally updated backup strategy. It will schedule a full backup first, after which incremental backups will be scheduled. Based on the backup, a recovery to any point in a day will be possible.

   

   Figure 13-30 .  Scheduling an Oracle-suggested backup

    Note  If a disk-based backup strategy is used, Oracle recommends using the Fast Recovery Area to enable faster recovery.

4. Select the destination media for the backup, as shown in Figure 13-31.

   

   Figure 13-31 .  Scheduling a backup destination type

5. Enter the location for the backup, as shown in Figure 13-32. In a disk-based backup strategy, the default is to use the Fast Recovery Area if it has been configured.

   

   Figure 13-32 .  Choose the Fast Recovery Area as the disk-based backup location

6. Enter a schedule for the recurring backup, as shown in Figure 13-33.

   

   Figure 13-33 .  Oracle-suggested backup schedule

7. Review the Backup Settings and generated RMAN script (see Figure 13-34). Click Submit Job to schedule the backup.

   

   Figure 13-34 .  Oracle-suggested backup settings

Using Oracle Enterprise Manager Cloud Control to schedule the backup job also enables you to receive notifications about the job status as well as view backup information including run times and time of last backup.

Oracle Management Service Backup

The Oracle Management Service is responsible for communicating with management agents and uploading data to the management repository. It includes the Oracle Fusion Middleware home, the OMS home, and the Web Tier (OHS) Oracle home in addition to plug-in homes. The OMS filesystem should be backed up periodically to preserve any changes such as patches, updates, or configuration changes. Use the exportconfig command to back up the OMS instance home (MW_HOME/gc_inst) including the WebLogic Server, Web Tier, and Administration Server. To create a backup of the instance and all subcomponents, use the following syntax:

emctl exportconfig oms [-sysman_pwd <sysman password>]
     [-dir <backup dir>]     Specify directory to store backup file
     [-oms_only]             Specify OMS-only backup on Admin Server host
     [-keep_host]            Specify to backup hostname if no slb defined
                             (Use this option only if recovery will be done
                              on machine that responds to this hostname)

The following example backs up the OMS configuration to a shared directory. An archive is created with a .bka extension.

$ $OMS_HOME/bin/emctl exportconfig oms -sysman_pwd oracle12c -dir /mnt/backup
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...
Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /mnt/backup/opf_ADMIN_20121104_224806.bka
 
The export file contains sensitive data.
Please ensure that it is kept secure.
 
ExportConfig completed successfully!

In addition to backing up the OMS configuration, a backup of the Oracle inventory and output of the opatch lsinventory –detail command should also be performed.

The following example generates the opatch inventory output and writes it to a shared filesystem.

$OMS_HOME/OPatch/opatch lsinventory –detail > /mnt/backups/opatch_bkup.log

The backup of the OMS can either be done manually or scheduled via the Enterprise Manager Cloud Control Job System.

Management Agent Backup

The management agents are stateless, so no backups are necessary. Instead, a reference gold image of the agent should be created for all platforms being monitored. Any patches and customizations should be included in the gold configuration. In the event that a management agent is lost, it should be installed from the reference gold image.

Repository Recovery

The Cloud Control console will be unavailable in the event that the OMS is down. To restore the repository database, the RMAN command-line utility should be used. With RMAN, you can restore the database by using either a full recovery or a point-in-time recovery (see the Oracle Database Backup and Recovery Guide). In the case of the latter, you may need to resynchronize agents that are out of sync with the repository. To resynchronize the agent by using EMCTL, use the following syntax:

emctl resync repos (-full|-agentlist "agent names")
[-name "resync name"]
[-sysman_pwd "sysman password"]

This command should be executed after the management repository has been restored and before starting the OMS.

The steps required to make Enterprise Manager Cloud Control operational after restoring the repository database will vary depending on whether a full or incomplete recovery is performed as well as whether the recovery was done on the same host or a different host. Table 13-3 summarizes the steps required for the different recovery scenarios. In each scenario, the repository database should be recovered after stopping the OMS.

OMS reconfiguration is required whenever the repository database is recovered on a different host. This stores the new database connection description in the OMS. We use the following command to reconfigure the repository database in the OMS:

emctl config oms -store_repos_details (-repos_host <host> -repos_port <port> -repos_sid
<sid> | -repos_conndesc <connect descriptor>) -repos_user <username> [-repos_pwd <pwd>]

Follow these steps to reconfigure the OMS:

1. Stop all the OMSs.

   emctl stop oms

2. Reconfigure the repository connection string on each of the OMSs.

   emctl config oms -store_repos_details

3. Restart Admin Server and all the OMSs by using the following:

   emctl stop oms -all
   emctl start oms

Recovering the repository on a different host also requires relocating the management repository database target to a different agent running on the new host. This can be done only if an agent already exists on the host and no other database has been discovered by it. Use the following syntax to relocate the target to a new host:

emctl config repos [-sysman_pwd <sysman password>]
      [-agent <new agent>]    Specify new destination agent for repository target
      [-host <new host>]      Specify new hostname for repository target
      [-oh <new oracle home>] Specify new OracleHome for repository target
      [-conn_desc [<jdbc connect descriptor>]]
                              Update Connect Descriptor with value if specified,
                              else from value stored in emoms.properties
      [-ignore_timeskew]      ignores timeskew on agent

The monitoring configuration for the OMS and repository target should also be updated by using this command:

emctl config emrep [-sysman_pwd <sysman password>]
      [-agent <new agent>]    Specify new destination agent for emrep target
      [-conn_desc [<jdbc connect descriptor>]]
                              Update Connect Descriptor with value if specified,
                              else from value stored in emoms.properties
      [-ignore_timeskew]      ignores timeskew on agent

After the database has been recovered and reconfiguration performed, the next step would be to log in to the Enterprise Manager Cloud Control console and verify that all operations have been restored.

Oracle Management Service Recovery

Recovering an OMS requires recovery of both the software homes (Fusion Middleware) and the instance homes (gc_inst). The software homes can either be recovered from a filesystem backup or be reinstalled using the Install Software Only option on the same or a different host using the Enterprise Manager Cloud Control 12c installer. The OMS home must be in the same location as the OMS home being recovered.

All plug-ins that existed in the OMS are required to be installed for the recovery to succeed. The following SQL query should be run as the SYSMAN user:

SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file,
null, 'Media/External', 'https://updates.oracle.com/Orion/Services/download/ '
||aru_file||'?aru='||aru_id||chr(38)||'patch_file='||aru_file) URL
FROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities su
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id
AND su.entity_id = epv.su_entity_id;

The following example output shows the plug-ins, versions, revisions, and URLs. The URL will display if downloaded via Self Update. Otherwise, the status will be Media/External.

SQL>
 
DISPLAY_NAME                    PLUGIN_ID           VERSION      REV_VERSION   URL
------------------------------  ------------------  -----------  ------------  ----------------
Oracle MOS (My Oracle Support)  oracle.sysman.mos   12.1.0.2.0              0  Media/External
Oracle Fusion Middleware        oracle.sysman.emas  12.1.0.3.0              0  Media/External
Oracle Database                 oracle.sysman.db    12.1.0.2.0       20120804  Media/External
Oracle Exadata                  oracle.sysman.xa    12.1.0.3.0              0  Media/External

If any additional plug-ins are listed, they should be downloaded into a single directory;  then rename the extensions from .zip to .opar. Use the Install Software Only option to install the Middleware and OMS Oracle home components if not restoring from a filesystem backup. After the software has been reinstalled or restored, the next step is to install the additional plug-ins (if any). Execute the PluginInstall.sh script located in OMS_HOME/sysman/install by specifying the –PluginLocation flag to select the location where the downloaded plug-ins are kept. In the case of a Software Only installation, all patches previously applied will have to be redone.

After restoring the software homes, the next step would be to restore or re-create the OMS. This is done by using the omsca utility and specifying the path of the backup generated by the emctl exportconfig command.

omsca recover –as –ms -nostart –backup_file <exportconfig file>

The steps required to recover the OMS may vary depending on whether an SLB is in use or multiple management services are configured as well as whether recovery is done on the same host or a different host. The following steps are required for restoring a single OMS on the same host without an SLB. The OMS instance will be recovered by using the OMS configuration backup taken by emctl exportconfig.

1. Clean up the failed host.
2. Verify that the Software Library is available.
3. Restore the software home by restoring a backup of the filesystem or run the installer’s Software Only option. (See the Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide Release 2).
4. Execute omsca in recovery mode and specify the location of the backup file taken previously.
5. Recover agents if necessary.
6. Start the OMS.
7. Verify that the OMS is working.

See the Oracle Enterprise Manager Cloud Control 12c Administrator’s Guide for details on other OMS recovery scenarios.

Management Agent Recovery

The recovery of the management agent requires reinstalling it, preferably from a reference install, or performing a filesystem restore from a previous backup. The agent should be cloned with the existing patches and customizations. The agent should also be installed by using the same port. After it has been reinstalled, a resynchronization should be performed from the Agent Resynchronization page in Enterprise Manager Cloud Control.

Agent recovery in a typical scenario usually follows these steps:

1. Remove the existing agent Oracle home by using the Oracle Universal Installer (OUI) to clean up the inventory.
2. Install a new agent or clone from the reference install, using the same port and path. The filesystem can also be restored from a previous backup.
3. Perform Agent Resynchronization from the Enterprise Manager Cloud Control Agent home page.
4. Reconfigure user-defined metrics if necessary.
5. Verify the status of the agent.

   emctl status agent
   emctl upload agent

Switchover and Failover

Both the management service and repository can be switched over and failed over independently when using level 2, 3, or 4 high-availability configurations. Switchover is usually done during planned maintenance, including operating system and software patching, while a failover normally occurs during unplanned maintenance such as hardware or software failure.

In a level 2 configuration, the switchover and failover of the management service follow the same procedure. As mentioned earlier, this requires relocating the OMS virtual hostname and VIP to another host and then starting the OMS manually or using Clusterware to automate the relocation. In level 2, 3, or 4 configurations that utilize a physical standby database for the Enterprise Manager repository, the database has to be switched over (planned) or failed over (unplanned). The OMS does not require any action in level 3 high availability because multiple management services are involved. The SLB will monitor the management services and detect when one has failed and route traffic to the available OMS.

To switch over the management repository using Data Guard, use the DGMRL command-line utility as the Enterprise Manager console will not be able to switch over the repository. Follow these steps to do a switchover of the management repository with a management service that has not been switched over:

1. From the Enterprise console, choose Availability ä Data Guard Administration.
2. Check that the Redo Apply process is up-to-date. Verify the ApplyLag status is zero as shown in Figure 13-35.

   

   Figure 13-35 .  Data Guard administration showing ApplyLag and TransportLag status

3. Stop the management services on the primary site by running the following command. If multiple management services are being used, this should be done on each OMS.

   emctl stop oms -all

4. Switch over to the standby database by using the DGMGRL command-line utility. This can be done on either the primary or standby site in the case of a switchover. Verify the Data Guard configuration and databases before doing a switchover. The following example illustrates these steps.
   • a.   Verify the Data Guard configuration and databases before doing a switchover:

     $ dgmgrl
     DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production
      
     Copyright (c) 2000, 2009, Oracle. All rights reserved.
      
     Welcome to DGMGRL, type "help" for information.
     DGMGRL> connect sys/<password>
     Connected.
     DGMGRL> show configuration
      
     Configuration - emrep
      
       Protection Mode: MaxPerformance
       Databases:
         emrep  - Primary database
         emrep2 - Physical standby database
      
     Fast-Start Failover: DISABLED
      
     Configuration Status:
     SUCCESS
      
     DGMGRL> show database emrep
      
     Database - emrep
      
       Role:            PRIMARY
       Intended State:  TRANSPORT-ON
       Instance(s):
         emrep
      
     Database Status:
     SUCCESS
      
     DGMGRL> show database emrep2
      
     Database - emrep2
      
       Enterprise Manager Name: emrep_sby
       Role:                    PHYSICAL STANDBY
       Intended State:          APPLY-ON
       Transport Lag:           0 seconds
       Apply Lag:               0 seconds
       Real Time Query:         OFF
       Instance(s):
         emrep2
      
     Database Status:
     SUCCESS

   • b.   Issue the SWITCHOVER command:

     DGMGRL> switchover to emrep2;
     Performing switchover NOW, please wait...
     New primary database "emrep2" is opening...
     Operation requires shutdown of instance "emrep" on database "emrep"
     Shutting down instance "emrep"...
     ORACLE instance shut down.
     Operation requires startup of instance "emrep" on database "emrep"
     Starting instance "emrep"...
     Unable to connect to database
     ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
      
     Failed.
     Warning: You are no longer connected to ORACLE.
      
     Please complete the following steps to finish switchover:
            start up and mount instance "emrep" of database "emrep"

      Note  Additional steps may be required after issuing the switchover command depending on the current state of the Data Guard configuration. In the example shown, the new standby database should be started and mounted manually.

   • c.   Verify the status of the Data Guard configuration to see that the roles have been switched:

     DGMGRL> connect sys/<password>
     Connected.
     DGMGRL> show configuration;
      
     Configuration - emrep
      
       Protection Mode: MaxPerformance
       Databases:
         emrep2 - Primary database
         emrep  - Physical standby database
      
     Fast-Start Failover: DISABLED
      
     Configuration Status:
     SUCCESS
      
     DGMGRL> show database emrep2;
      
     Database - emrep2
      
       Enterprise Manager Name: emrep_sby
       Role:                    PRIMARY
       Intended State:          TRANSPORT-ON
       Instance(s):
         emrep2
      
     Database Status:
     SUCCESS
      
     DGMGRL> show database emrep;
      
     Database - emrep
      
       Role:            PHYSICAL STANDBY
       Intended State:  APPLY-ON
       Transport Lag:   0 seconds
       Apply Lag:       0 seconds
       Real Time Query: OFF
       Instance(s):
         emrep
      
     Database Status:
     SUCCESS

     Here you can see that the role of the database has been successfully changed. The next step is to reconfigure the OMS to use the new primary database.

5. Start the Admin Server if it is not already running.

   emctl start oms –admin_only

6. Configure the OMS with the new primary database repository.

   $OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem3)(PORT=1522)))
   (CONNECT_DATA=(SID=emrep2)))" -repos_user sysman
   Oracle Enterprise Manager Cloud Control 12c Release 2
   Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
   Enter Repository User's Password :

7. Start the OMS.

   emctl start oms

8. Relocate the management repository target. The management agent on an OMS monitors the OMS and management repository targets. After the switchover, the repository target needs to be updated to reflect the switched-over repository database, which is the new primary.

   emctl config emrep –agent <central_agent_name> -conn_desc <conn_desc_of_new_primary>

Failover of the management repository using Data Guard is similar to a switchover, except the command issued using DGMGRL would be failover to <standby_db_name>.

In addition to using the preceding manual steps, you can automate the failover of the OMS and management repository by using Data Guard fast-start failover:

• Fast-start failover determines when a failover to a standby is necessary by using an Observer process that runs on another host independent of both primary and standby servers. If the Observer cannot contact the primary site after a certain configurable period, it will automatically fail over to the standby database.
• After failover occurs, a database event called DB_ROLE_CHANGE is fired.
• The DB_ROLE_CHANGE event causes a trigger to be fired, which in turn starts up the Enterprise Manager Application tier.

A script can be created that will automate these steps. Using the sample script provided in the <OMS_HOME>/sysman/ha directory, create a script that will configure the OMS to point to a new primary database and start up all management services. Listing 13-1 is an example of a script that will start up the standby OMS and reconfigure the OMS with the new primary management repository database that has been switched over in a Data Guard configuration.

Listing 13-1.  Sample Script to Start EM Tier on Standby Site

#!/bin/sh
LOGFILE="/oms_swlib/em/failover/em_failover.log"
OMS_ORACLE_HOME="/u01/app/oracle/Middleware/oms"
CENTRAL_AGENT="oem2.example.com:3872"
SYSMAN_PWD="oracle12c"
  
#log message
echo "###############################" >> $LOGFILE
date >> $LOGFILE
echo $OMS_ORACLE_HOME >> $LOGFILE
id >>  $LOGFILE 2>&1
 
#switch all OMS to point to new primary and startup all OMS
$OMS_ORACLE_HOME/bin/emctl config oms -store_repos_details -repos_conndesc "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem1)(PORT=1521)))
(CONNECT_DATA=(SID=emrep2)))" 
-repos_user sysman –repos_pwd $SYSMAN_PWD >> $LOGFILE 2>&1
$OMS_ORACLE_HOME/bin/emctl sync_opss_policy_store 
 -sysman_pwd oracle12c  >> $LOGFILE 2>&1
$OMS_ORACLE_HOME/bin/emctl stop oms >>  $LOGFILE 2>&1
$OMS_ORACLE_HOME/bin/emctl start oms >>  $LOGFILE 2>&1
 
#relocate Management Services and Repository target
#to be done only once in a multiple OMS setup
#allow time for OMS to be fully initialized
$OMS_ORACLE_HOME/bin/emctl config emrep -agent $CENTRAL_AGENT 
-conn_desc -sysman_pwd $SYSMAN_PWD >> $LOGFILE 2>&1
 
#always return 0 so that dbms scheduler job completes successfully
exit 0

The sample script in Listing 13-2 creates a trigger that will be fired whenever the DB_ROLE_CHANGE event occurs during a switchover or failover operation. This trigger will then call the preceding script to start the Enterprise Manager tier.

Listing 13-2.  Sample Database Role-Change Trigger

--
--
-- Sample database role change trigger
--
--
CREATE OR REPLACE TRIGGER FAILOVER_EM
AFTER DB_ROLE_CHANGE ON DATABASE
DECLARE
    v_db_unique_name varchar2(30);
    v_db_role varchar2(30);
BEGIN
    select upper(VALUE) into v_db_unique_name
    from v$parameter where NAME='db_unique_name';
    select database_role into v_db_role
    from v$database;
  
    if v_db_role = 'PRIMARY' then
  
      -- Submit job to Resync agents with repository
      -- Needed if running in maximum performance mode
      -- and there are chances of data-loss on failover
      -- Uncomment block below if required
      -- begin
      --  SYSMAN.setemusercontext('SYSMAN', SYSMAN.MGMT_USER.OP_SET_IDENTIFIER);
      --  SYSMAN.emd_maintenance.full_repository_resync('AUTO-FAILOVER to '||
          v_db_unique_name||'- '||systimestamp, true);
      --  SYSMAN.setemusercontext('SYSMAN', SYSMAN.MGMT_USER.OP_CLEAR_IDENTIFIER);
      -- end;
  
      -- Start the EM mid-tier
      dbms_scheduler.create_job(
          job_name=>'START_EM',
          job_type=>'executable',
          job_action=> '/oms_swlib/em/failover/' || v_db_unique_name|| '_start_oms.sh',
          enabled=>TRUE
      );
    end if;
EXCEPTION
WHEN OTHERS
THEN
    SYSMAN.mgmt_log.log_error('LOGGING', SYSMAN.MGMT_GLOBAL.UNEXPECTED_ERR,
SYSMAN.MGMT_GLOBAL.UNEXPECTED_ERR_M || 'EM_FAILOVER: ' ||SQLERRM);
END;
/