In this chapter, you will learn to:

• Determine What to Monitor
• Use Alarms

• Designing an Alarm
• Removing Alarms
• Modifying Alarms
• Moving Alarms
• Get Currently Active Alarms

In several of the preceding chapters, we’ve shown you how to automate the deployment and much of the management of your vSphere environment. Thanks to this high level of automation, you avoid a lot of human errors. But even when you have everything running as designed, there can be mishaps. Remember Murphy’s Law! To capture these unforeseen events and to react to them as fast as possible, you need to monitor your vSphere environment at all times.

In this chapter, you’ll determine what you need to monitor and how to employ vCenter alarms in the monitoring process.

Determine What to Monitor

The short answer to the “What should I monitor?” question is, of course, “Everything!” The problem with that statement is that you can never foresee everything. Until you realize that certain events are possible causes of problems, you won’t be able to monitor them.

So, how should you tackle this monitoring thing?

We think you have to adopt a commonsense strategy. Take a couple of steps back and look at your vSphere environment. What is actually there? When you take this approach, it becomes clear that there are two major abstract groups present in every vSphere environment: resources and services.

In the resources group, you have elements like servers, storage, and networks. In the services group, you’ll find elements like hosting platforms, virtual machines, HA, DRS, vMotion, and SvMotion. Each of these two major groups will need to be detailed further to arrive at manageable chunks that you can monitor. Here are some examples of these lower-level elements that you can monitor:

• Server availability
• CPU usage
• Memory usage
• Datastore usage
• Network connectivity
• vMotion and SvMotion functionality

It should be obvious that this list is just an example and that your list is in reality much more extensive. It should also be obvious that the list created for Company A will be different from the list compiled for Company B.

Once you have confirmed your requirements and planned out the elements to monitor, your next step is to use vCenter alarms to alert yourself or other teams in your organization that an issue has occurred.

Use Alarms

As you might have noticed, vCenter comes with a number of built-in alarms. (The list keeps getting longer and better with each release.) On top of that it is quite easy to create a new alarm via the GUI. PowerCLI cmdlets are available for working with existing alarms, but if you want to create your own alarms programmatically, you will have to fall back on the methods available in the SDK.

Alarms, in general, follow a rather simple principle. You start by defining one or more triggers for the alarm. When the alarm fires, the actions associated with the alarm are taken. So, let’s take a look at how to get started with creating alarms.

Designing an Alarm

The Alarm managed object—or, in other words, the object that is used to represent an alarm in a vSphere environment—has the properties described in Table 17-1. Figure 17-1 shows how the various elements of an alarm relate to one another. You’ll need to understand each of these elements if you intend to design useful alarms for your system.

Alarm Actions

The action part of an alarm is where you can define what needs to be done when an alarm is fired. vCenter offers the actions described in Table 17-2. Note that you can define multiple actions on a single alarm.

Alarm Firing Mechanisms

There are, in the current vSphere environment, three types of firing mechanisms available, as described in Table 17-3.

Alarm Expressions

Let’s take a quick tour of the three types of alarm expressions and how to provide values to the important properties. For the expression object in an alarm specification, you have two choices: an AndAlarmExpression or an OrAlarmExpression. Use AndAlarmExpression with AND operators and OrAlarmExpression with OR operators. These are, of course, for expressions with multiple conditions. With a single-trigger alarm, there is no associated single trigger alarm expression, so you can use either AndAlarmExpression or OrAlarmExpression to achieve the same result.

Let’s see how you would create expressions for these three different scenarios:

• Single-trigger expression

  $expression = New-Object VMware.Vim.AndAlarmExpression

• AND operator expression

  $expression = New-Object VMware.Vim.AndAlarmExpression

• OR operator expression

  $expression = New-Object VMware.Vim.OrAlarmExpression

Event Alarms

As we described in Chapter 15, “Reporting and Auditing,” the vSphere environment generates an event for nearly every activity that occurs. Events can range from the creation of a guest, to a failing cluster node, all the way to detection of a guest that doesn’t have the VMware Tools installed. Some extensions, for example the Update Manager, generate their own events. Any event can be used to trigger an alarm. The moment vCenter Server sees an event of the type you specified, it will trigger the corresponding alarm.

The API Reference documentation describes three types of events; we will use this knowledge in Listing 17-1 to examine requirements for an EventAlarmExpression:

• Regular
• EventEx
• ExtendedEvent

We use the term regular to group all the events that are not of the EventEx or ExtendedEvent type. Some samples of such regular events are AccountCreatedEvent, ClusterReconfiguredEvent, and HostIPChangedEvent. More than 400 types of events are included in the regular events group.

The EventEx and ExtendedEvent types contain events that are introduced in vCenter Server by the installation of extensions. The License Manager is such an extension, and it introduces additional events on vCenter Server.

The first step in the creation of event alarms is to determine which specific event to use to define the alarm and which properties you need to provide in the method call to create it. To help you find the data you need to define the alarm event you wish to monitor, we created the Get-AlarmEventId function (see Listing 17-1). It produces a list that contains all the information you need to define the alarm trigger. This information is not well documented in the vSphere Reference, and by using Listing 17-1 you are able, for example, to export the information into a CSV file for easy examination. The CSV file will contain the correct names for the EventType and EventTypeId properties you’ll have to provide when you’re setting up the event alarm.

Listing 17-1: The Get-AlarmEventId function

function Get-AlarmEventId {
  <#
.SYNOPSIS
  Returns the data needed to define an EventAlarmExpression
.DESCRIPTION
  The function will return the required properties that
  are needed to populate the eventType and eventTypeId
  properties 
.INPUTS
None
.OUTPUTS
System.Management.Automation.PSCustomObject
.EXAMPLE
Get-AlarmEventId | Export-Csv "C:Alarm-eventId.csv"
#>
[CmdletBinding()]`
 [OutputType('System.Management.Automation.PSCustomObject')]

Param ()

$commonArgs = "changeTag","computeResource","computeResource.name",
              "datacenter","datacenter.name","ds","ds.name","dvs",
              "dvs.name","fullFormattedMessage","host","host.name",
              "net","net.name","userName","vm","vm.name"

try {

    $evtMgr = Get-View EventManager

    $evtMgr.Description.EventInfo | ForEach-Object {
        if ($_.Key -eq "eventEx"){

           $eventTypeName = $_.FullFormat.Split("|")[0]
           $eventType = "EventEx|ExtendedEvent"
           $eventTypeId = $_.FullFormat.Split("|")[0]
           $attributes = $evtMgr.RetrieveArgumentDescription($eventTypeId)

            if ($attributes){

                $specialAttributes = $attributes | 
                 Where-Object {$commonArgs -notcontains $_.Name} |
                 ForEach-Object {$_.Name + "(" + $_.Type + ")"}

                if ($specialAttributes){

                    $attributes = [string]::Join(',',$specialAttributes)
                }
            }
            $group = "EventEx"
        }
        elseif ($_.Key -eq "ExtendedEvent"){

           $eventType = "ExtendedEvent|EventEx"
           $eventTypeId = $_.FullFormat.Split("|")[0]
           $attributes = $evtMgr.RetrieveArgumentDescription($eventTypeId)

            if ($attributes){

                $specialAttributes = $attributes | 
                 Where-Object {$commonArgs -notcontains $_.Name} |
                 ForEach-Object {$_.Name + "(" + $_.Type + ")"}

                if($specialAttributes){

                    $attributes = [string]::Join(',',$specialAttributes)
                }
            }
            $group = "ExtendedEvent"
        }
        else{

           $eventTypeName = $_.Key
           $eventType = $eventTypeName
           $eventTypeId = "vim.event." + $_.Key
           $attributes = $evtMgr.RetrieveArgumentDescription($eventTypeId)

            if($attributes){

                $specialAttributes = $attributes | 
                 Where-Object {$commonArgs -notcontains $_.Name} | 
                 ForEach-Object {$_.Name + "(" + $_.Type + ")"}

                if($specialAttributes){

                    $attributes = [string]::Join(',',$specialAttributes)
                }
            }
            $group = "regular"
        }

        $Object = [pscustomobject]@{
            
            EventType = $eventType
            EventTypeId = $eventTypeId
            Group = $group
            Description = $_.Description
            Attributes = $attributes
        }
        Write-Output $Object
    }
}
catch [Exception]{
    throw "Unable to retrieve Alarm Event IDs"
}
}

A sample run of the function results in a listing of the EventType and EventTypeId values you’ll need in order to define your trigger. Here’s a sample output that displays all three types: regular, EventEx, and ExtendedEvent:

Get-AlarmEventId | Select-Object -First 3

EventType   : AccountRemovedEvent
EventTypeId : vim.event.AccountRemovedEvent
Group       : regular
Description : Account removed
Attributes  : account(string),group(bool)

EventType   : EventEx|ExtendedEvent
EventTypeId : com.vmware.license.AddLicenseEvent
Group       : EventEx
Description : Added License
Attributes  : eventTypeId(string),message(string),objectId(st...
 objectName(string),severity(string),licenseKey(string)

EventType   : ExtendedEvent|EventEx
EventTypeId : com.vmware.vc.HA.ClusterFailoverActionCompleted...
Group       : ExtendedEvent
Description : vSphere HA completed a failover action
Attributes  : message(string),eventTypeId(string),managedObje...

Now that you have seen illustrations of the three expression types—regular (the standard set of events that come with vSphere), EventEx, and ExtendedEvent—and you know what information needs to be supplied to define the event, let’s create an example.

First, let’s create an EventEx type alarm expression that will fire when the network redundancy for the dvPorts is restored. Both properties in the EventAlarmExpression object would be populated like this:

$expression = New-Object VMware.Vim.EventAlarmExpression
$expression.eventType = "EventEx"
$expression.eventTypeId = `
    "esx.clear.net.dvport.redundancy.restored"

According to the vSphere Reference, you can use the values EventEx or ExtendedEvent for the eventType property. Just make sure that the eventType property is not empty!

Now, let’s look at an example using a regular event. Here we’ll monitor the status of file ownership. The ChangeOwnerOfFileEvent event will fire when the ownership of a file has changed. To specify this event, you will need to populate the properties as follows:

$expression = New-Object -Type VMware.Vim.EventAlarmExpression -Property `
@{
    eventType = "ChangeOwnerOfFileEvent"
    eventTypeId = "vim.event.ChangeOwnerOfFileEvent"
}

Notice that we added the prefix vim.event before the name of the event. For regular events, the prefix is required when you fill in the eventTypeId property. The Get-AlarmEventId function from Listing 17-1 does that for you, and in the output produced by the function, you will find the correct string to assign to the eventTypeId property.

The comparisons property is another important, but optional, property that you can use when creating an event alarm. The operators you can use for these comparisons are documented in the EventAlarmExpressionComparisonOperator enumerator that you can find in the API Reference. (If you’re not familiar with the API Reference, don’t worry. We’ll tell you all about it in Chapter 18, “The SDK.”)

Using the comparisons property, you can test attributes. But how can you find out which attributes there are? Turns out that there is a method, RetrieveArgumentDescription, on the EventManager object which returns all arguments (or attributes) that you can use with an event trigger. Table 17-4 lists the attributes that you can use with any event. The attributes that are marked as having the moid types are, in fact, managed object IDs. Not too handy to use in a comparison, but we included them in the table since the RetrieveArgumentDescription method will return them!

In addition to the common attributes, some events accept additional attributes. Those are listed under the Attributes property in the objects that the Get-AlarmEventId function returns. With the AccountRemovedEvent event, for example, you can use the account attribute to filter out the alarms for test accounts. The operators you can use for these comparisons are documented in the EventAlarmExpressionComparisonOperator enumerator that you can find in the API Reference. The other properties in the EventAlarmExpression object are clearly explained in the vSphere API Reference documentation.

State Alarms

State alarms allow you to trigger an alarm based on the state of a virtual machine, a vSphere Server, or a datastore. In vSphere 6.0, the states listed in Table 17-5 can be used to trigger an alarm.

Metric Alarms

An alarm expression based on a metric has some intricacies. The metricId you use in the expression can be different on different vCenter Servers. For that reason, the New-Alarm function you will see later in this chapter will allow you to specify the metric by its name. That means you can specify cpu.usage.average instead of a number.

Alarm Actions

Alarm actions are many and varied. Some are quite simple to work with, such as group, method, task, or SNMP actions. We’ll provide an overview of those actions. Others, like script or email actions, are more flexible and complex. We’ll cover those in greater detail.

1. Grouping Actions The API Reference indicates that you can have a single AlarmTriggeringAction without a GroupAlarmAction. It is indeed possible to define an alarm with a single action, but the problem comes when you try to edit the settings of that alarm from the vSphere Client; the Edit Settings option is grayed out. The vSphere Web Client has an even worse issue; an error will be generated when you try to edit the alarm. For that reason we advise you to always specify your actions, even if it is only one action, under a GroupAlarmAction where these issues do not occur.

1. Method Action Method actions are only available for HostSystem and VirtualMachine objects. Using a method action, you can call any SDK method that is available on the HostSystem or VirtualMachine object that fired the alarm.
2. Task Action Task actions also are only available for HostSystem and VirtualMachine objects. You can call any of the tasks that are available on the extensions active on your vCenter Server.
3. SNMP Action Simple Network Management Protocol (SNMP) actions are the simplest of all actions. They don’t require any parameters. The SNMP trap will be sent to the SNMP communities you defined in your vCenter Server.

Script Actions

Script actions take one parameter—the command that you want vCenter Server to execute when an alarm is fired. There can be some issues related to how you achieve this, in particular with the security account used to execute the script and the privileges within vCenter Server that are required. We prefer to take the following approach:

• The vCenter Service runs with a domain account. Consequently, the scripts execute through the Security Support Provider Interface (SSPI) and use single sign-on to connect to vCenter Server. Therefore you don’t need to specify separate credentials.
• The domain account that the vCenter Server runs under holds the required privileges within vCenter’s permission model to execute whatever you want to accomplish in the script.

Unlike the code you’ll see next, the actual text for the Configuration field in the Actions section of the Edit Alarm wizard is entered in one line, with no carriage returns or linefeeds. (We had to break the text into three lines here; it’s too long to fit on the page.) For a 64-bit OS, like Windows 2012, it would look like Listing 17-2.

Listing 17-2: An example script action

C:WindowsSystem32cmd.exe /c powershell.exe
  -PSConsolefile C:ScriptsMyConsole.psc1 -noninteractive
  -noprofile -file C:ScriptsAlarm-action-script.ps1

A common method for monitoring processes on a Windows server is to use the Process Explorer tool from Microsoft’s Sysinternals website at http://sysinternals.com. When you use Process Explorer on your vCenter Server, you can see how vCenter Server runs your alarm script. Process Explorer allows you to see the parameters that are used for each of the spawned processes. Open the properties of the cmd.exe and the powershell.exe processes. Process Explorer also allows you to verify when your alarm script is finished. Just check whether the cmd.exe and powershell.exe processes have disappeared below the vpxd.exe process.

In Listing 17-2, notice the use of the console file on the command line. With this file, you can define the snap-in(s) that need to be loaded into the PowerShell session that will run your script.

Script actions can take quite a bit of time to completely execute. This depends on the load of your vCenter Server but also on the number of snap-ins you are loading through the console file. So be patient when you want to check the results of your alarm script.

On the command line, you can pass one or more of the predefined variables that vCenter Server provides. Again, you’ll enter the information on one line with no carriage returns or line breaks (Listing 17-3).

Listing 17-3: An example script action with predefined variables

C:WindowsSystem32cmd.exe /c powershell.exe
  -PSConsolefile C:ScriptsMyConsole.psc1 -noninteractive
  -noprofile -file C:ScriptsAlarm-action-script.ps1
  {alarmName} {targetName}

Note that these parameters are case sensitive. That means that, for example, you can’t use {targetname}. You can find the complete list of parameters in the VMware vSphere 6.0 Documentation Center (http://vmw.re/1FHtTHO). The script can access these parameters as regular PowerShell function parameters. In other words, you can use the $Args variable or you can define a param statement in your script.

Before using your script action in an alarm, it’s a good idea to first test it by running it as an individual piece of code. That way, you can guarantee it is going to run successfully as part of the alarm and there will be no issues, such as the local security policy preventing its execution, or problems with permissions in vCenter Server.

If you want to capture the execution of the alarm script, one method is to pipe all output to the Out-File cmdlet. That way, all output from the script is captured in a file. An alarm script that logs the parameters it receives to a file could look something like this:

$outFile = C:Alarmalarm.log
$Args | Out-File -FilePath $outFile -Append

There is an alternative for accessing information about the alarm that triggered your script. vCenter Server spawns your alarm script with a number of environment variables that contain information about the alarm that triggered the script. You can find the complete list of these environment variables in the VMware vSphere 6.0 Documentation Center (http://vmw.re/1MMautd).

A summary of these and typical values can be found in Table 17-6.

You can use these environment variables in your script through the environment variable provider. A short example is shown in Listing 17-4.

Listing 17-4: Environment variable provider example

$outFile = "C:outfile.txt"
$report = @()
$report += ("Time:`t" + (Get-Date).ToShortTimeString())
$report += ("Alarm:`t" + $env:VMWARE_ALARM_NAME)
$report += ("On:`t" + $env:VMWARE_ALARM_TARGET_NAME)
$report += ("User:`t" + $env:VMWARE_ALARM_EVENT_USERNAME)

$report | Out-File -FilePath $outFile

This script will produce something like this:

Time:  3:21 AM
Alarm:  Change custom field
On:  PC1
User:  TESTadministrator

If you are going to use PowerCLI cmdlets in your script, you will need to connect to a vCenter Server and have PowerCLI installed on that server. As we said earlier, these alarm scripts are spawned from the vCenter service; just make sure that this service runs with a domain account that has the required privileges. That way, you can connect through SSPI and you don’t have to provide credentials.

The environment variables that are available in the alarm script contain a number of ID values. Take, for example, the VMWARE_ALARM_TARGET_ID variable (see Table 17-6). Unfortunately, this is not the kind of ID that can be used in a PowerCLI cmdlet like Get-View. Consequently, it is much better to use the environment variables that end with a name suffix. These contain the name of the vSphere entity that you can use to access (through the PowerCLI cmdlets) the vSphere object in your script. The code in Listing 17-5 shows how to use, for example, the VMWARE_ALARM_TARGET _NAME environment variable to retrieve the virtual machine and datastore.

Listing 17-5: Accessing vSphere objects in an action script

$outFile = "C:outfile.txt"

Connect-VIServer -Server vSphere
$vm = Get-VM -Name $env:VMWARE_ALARM_TARGET_NAME
$ds = Get-Datastore -VM $vm

$report = @()
$report += ("Time:`t" + (Get-Date).ToShortTimeString())
$report += ("Alarm:`t" + $env:VMWARE_ALARM_NAME)
$report += ("On:`t" + $env:VMWARE_ALARM_TARGET_NAME)
$report += ("User:`t" + $env:VMWARE_ALARM_EVENT_USERNAME)
$report += ("DS:`t" + $ds.Name)

$report | Out-File -FilePath $outFile

We used the Out-File cmdlet to track what the script did. You can use that same log file to track any errors your alarm script produces. The script in Listing 17-6 appends the $Error variable to the log file you created in Listing 17-5. Note that for the purpose of illustration, Listing 17-5 generates an error. To force the error, we supplied an incorrect vCenter Server (WrongvSphere).

Listing 17-6: Appending errors to the alarm script log file

$ErrorActionPreference = "SilentlyContinue"
$Error.Clear()

$outFile = "C:outfile.txt"

Set-PowerCLIConfiguration -DefaultVIServerMode Single `
  -Confirm:$false
Connect-VIServer -Server WrongvSphere

$report = @()
$report += ("Server:`t" + $defaultVIServer.Name)

$report | Out-File -FilePath $outFile

“==> Errors” | Out-File -FilePath $outFile -Append
$Error | Out-File -FilePath $outFile -Append

By choosing SilentlyContinue for the $ErrorActionPreference, we made sure that the script would continue even when errors were encountered. At the end of the script, we dumped the $Error variable via the Out-File cmdlet to the external file. Notice that we cleared the $Error variable at the beginning of the script to prevent any prior and irrelevant errors from the current PowerShell session being included in the log file. You can, of course, use different files for your regular output and for the dump of the $Error variable by changing the value of the $outFile variable.

You might wonder why we included the Set-PowerCLIConfiguration cmdlet at the beginning of the script. We needed that code line because the first time you connect to a vSphere Server, the Connect-VIServer will prompt you to specify whether you want to work in single mode or multimode. To avoid this question, PowerCLI provides this cmdlet to set the mode before the Connect-VIServer cmdlet.

In the sample, we used a nonexistent vCenter Server name to force an error. The resulting output file looks something like this:

Server:
==> Errors
Connect-VIServer : 12/5/2015 1:23:16 PM    Connect-VIServer
   Could not resolve the requested VC server.
At C:ScriptsAlarm-action-script.ps1:14 char:17
+ Connect-VIServer <<<<  -Server WrongvSphere
    + CategoryInfo          : ObjectNotFound: (:)
        [Connect-VIServer], ViServer
   ConnectionException
    + FullyQualifiedErrorId :
        Client20_ConnectivityServiceImpl_Reconnect_Name
     ResolutionFailure,VMware.VimAutomation.ViCore.Cmdlets
     .Commands.ConnectVIServer

The fact that we can now see at least the error messages from our alarm script makes it a lot easier to debug the script.

To conclude this section, the complete alarm script is displayed in Listing 17-7. It uses the technique from Listing 17-5 to retrieve additional information about an environment variable through the use of a PowerCLI cmdlet. This alarm could be used when monitoring virtual machines. When triggered, the alarm script will return the vCenter Server name, the time, the alarm name, the alarm event username, the VM, and the datastore that the VM resides on.

Listing 17-7: The complete alarm script

$ErrorActionPreference = "SilentlyContinue"
$Error.Clear()

$outFile = "C:outfile.txt"

Set-PowerCLIConfiguration -DefaultVIServerMode Single `
  -Confirm:$false
Connect-VIServer -Server vSphere

$vm = Get-VM -Name $env:VMWARE_ALARM_TARGET_NAME
$ds = Get-Datastore -VM $vm

$report = @()
$report += ("Server:`t" + $defaultVIServer.Name)
$report += ("Time:`t" + (Get-Date).ToShortTimeString())
$report += ("Alarm:`t" + $env:VMWARE_ALARM_NAME)
$report += ("User:`t" + $env:VMWARE_ALARM_EVENT_USERNAME)
$report += ("VM:`t" + $vm.Name)
$report += ("DS:`t" + $ds.Name)
$report | Out-File -FilePath $outFile

if($Error){
  "==> Errors" | Out-File -FilePath $outFile -Append
  $Error | Out-File -FilePath $outFile -Append
}

The result of this alarm script looks like this:

Server:  vSphere
Time:  2:15 PM
Alarm:  Book: Change custom field
User:  TESTadministrator
VM:  PC1
DS:  DS1

Email Action

Provided you have defined an SMTP server on your vCenter Server, the email action will send an email message when an alarm is fired. You can specify the To: field, the Cc: field, the subject, and the body of the email message.

In the Body property, you can use the same environment variables that we showed you in the “Script Actions” section. vCenter Server will replace these variables with actual values before it sends the email. The list of the available environment variables can be found in Table 17-6 and the full list found at http://vmw.re/1MMautd.

The SNMP Action

SNMP is the simplest of all actions. It doesn’t require any parameters. The SNMP trap will be send to the SNMP communities you defined in your vCenter Server.

A General Alarm Creation Function

An Alarm object is quite complex, and since it necessitates a large numbers of parameters on a function to create an alarm, we decided to provide several specific functions, each with a manageable number of parameters. The diagram in Figure 17-2 shows which function to use to define the different parts of an alarm specification.

The ultimate function, called New-Alarm, uses all the objects you created via the supporting functions and creates the alarm on your vCenter Server. To make the supporting functions easy to use, and consequently New-Alarm itself, we have created a PowerShell module vCenterAlarms containing all the functions you need to create an alarm. It is available for download from this book’s web page at www.wiley.com/go/vmwarevspherepowercli2e.

Once you have downloaded the vCenterAlarms module and placed it in an appropriate place on your workstation, check out Listing 17-8, which demonstrates how to import the module into your PowerShell session and how to list the functions available in the module.

Listing 17-8: Import and List functions in the vCenterAlarms module

Import-Module vCenterAlarms
Get-Command -Module vCenterAlarms -Verb New

Typical output would be

CommandType     Name                           ModuleName   
-----------     ----                           ----------   
Function        New-Alarm                      vCenterAlarms
Function        New-AlarmActionEmail           vCenterAlarms
Function        New-AlarmActionMethod          vCenterAlarms
Function        New-AlarmActionScript          vCenterAlarms
Function        New-AlarmActionSnmp            vCenterAlarms
Function        New-AlarmActionTask            vCenterAlarms
Function        New-AlarmExpressionAnd         vCenterAlarms
Function        New-AlarmExpressionEvent       vCenterAlarms
Function        New-AlarmExpressionMetric      vCenterAlarms
Function        New-AlarmExpressionOr          vCenterAlarms
Function        New-AlarmExpressionState       vCenterAlarms
Function        New-AlarmTransition            vCenterAlarms
Function        New-AlarmTriggerAction         vCenterAlarms

The functions you need to use will depend on the type of alarm you are trying to create. We will illustrate some different possibilities in the following examples.

Listing 17-9 creates an alarm that sends an SNMP trap when one of the hosts has an average CPU usage above 70 percent for longer than 25 seconds. The alarm will be yellow under those circumstances, and if average CPU usage is above 85 percent for longer than 10 seconds, it will be red.

With the help of the New-Alarm* supporting functions you can construct separate building blocks (transition states, actions, expressions). And with the New-Alarm function you bring them all together. Let’s look at those examples.

Listing 17-9: Yellow CPU usage alarm

# The transition state for green to yellow
$trans = @()
$trans += New-AlarmTransition -Start "green" -Final "yellow"

# Send an SNMP trap
$action = New-AlarmActionSnmp

# Combine the transition and the action
$groupactions = @()
$groupactions += New-AlarmTriggerAction -Action $action `
  -Transition $trans

# Fire the alarm on the metric cpu.usage.average
$expression = @()
$expression += New-AlarmExpressionAnd
$expression += New-AlarmExpressionMetric -Metric "cpu.usage.average" `
  -Object "HostSystem" -Operator "isAbove" -YellowValue 7000 `
  -YellowInterval 25 -RedValue 8500 -RedInterval 10

# Create the alarm in the vCenter root
New-Alarm -Entity (Get-Inventory -Name "Datacenters") -Name `
  "Book: host busy" -Description "Host too busy" -Action $groupactions `
  -Expression $expression -Enabled

The next alarm, shown in Listing 17-10, sends an email and an SNMP trap when a guest loses its network connectivity. The alarm is enabled for all guests in datacenter DC1.

Listing 17-10: Guest network connectivity alarm

# The transition state
$trans = @()
$trans += New-AlarmTransition -Start "green" -Final "yellow" `
  -Repeat:$true

$groupactions = @()

# Send an SNMP trap
$action = New-AlarmActionSnmp
$groupactions += New-AlarmTriggerAction -Action $action `
  -Transition $trans

# Send an Email
$action = New-AlarmActionEmail -To "[email protected]" `
  -Subject "Mail subject" -Body "Body text"
$groupactions += New-AlarmTriggerAction -Action $action `
  -Transition $trans

# Fire the alarm on the vmNetworkFailed event
$expression = @()
$expression += New-AlarmExpressionOr
$expression += New-AlarmExpressionEvent -Event
  "VmNetworkFailedEvent" `
  -Object "VirtualMachine" -Status "red"

# Create the alarm in the vCenter root
$dc = Get-Datacenter -Name "DC1"
New-Alarm -Name "Book: guest lost network" -Description
  "VM network lost" `
  -Entity $dc `
  -Action $groupactions -Expression $expression -Enabled

The next example, in Listing 17-11, creates an alarm that will fire when someone changes a custom attribute on a guest. The alarm will run a PowerShell script.

Listing 17-11: Guest custom attribute change alarm

# The transition state
$trans = @()
$trans += New-AlarmTransition -Start "green" -Final "yellow"

$groupactions = @()

# Run a script
$cmd = "C:WindowsSystem32cmd.exe /c powershell.exe" + `
  "-PSConsolefile C:ScriptsMyConsole.psc1 -noninteractive" + `
  "-noprofile -file C:ScriptsCustom-changed.ps1"
$action = New-AlarmActionScript -Path $cmd
$groupactions += New-AlarmTriggerAction -Action $action `
-Transition $trans

# Fire the alarm on the metric cpu.usage.average
$expression = @()
$expression += New-AlarmExpressionOr
$expression += New-AlarmExpressionEvent `
  -Event "CustomFieldValueChangedEvent" `
  -Object "VirtualMachine" -Status "yellow"

# Create the alarm on the datacenter
$dc = Get-Datacenter -Name DC1
New-Alarm -Name "Book: CA changed" `
  -Description "custom attribute changed" `
  -Entity $dc `
  -Action $groupactions -Expression $expression -Enabled

The script that is run through the action looks like the code found in Listing 17-12.

Listing 17-12: Custom-changed.ps1

$ErrorActionPreference = "SilentlyContinue"
$Error.Clear()

$errorFile = "C:error.txt"

$line = "Time:" + (Get-Date).ToShortTimeString()
$line += (",User:" + $env:VMWARE_ALARM_EVENT_USERNAME)
$line += (",On:" + $env:VMWARE_ALARM_TARGET_NAME)
$line += (",Description:" + $env:VMWARE_ALARM_EVENTDESCRIPTION)

$line | Out-File -FilePath "C:CA-changes.txt" -Append

if($Error){
  "==> Errors" | Out-File -FilePath $errorFile -Append
  $Error | Out-File -FilePath $errorFile -Append
}

Removing Alarms

The Remove-AlarmDefinition function (see Listing 17-13) is quite straightforward; it deletes one or more alarms that you pass to the function.

Listing 17-13: Removing an alarm definition

function Remove-AlarmDefinition {
  <#
.SYNOPSIS
Removes one or more alarm definitions
.DESCRIPTION
The function will remove all the alarm definitions whose name matches.
.PARAMETER Name
Datastore Name 
.INPUTS
System.String
.OUTPUTS
None
.EXAMPLE
Remove-AlarmDefinition -Name "Book: My Alarm"
.EXAMPLE
Remove-AlarmDefinition -Name "Book:*"
#>
[CmdletBinding(SupportsShouldProcess,ConfirmImpact="High")]

  Param (

  [parameter(Mandatory=$true)]
  [ValidateNotNullOrEmpty()]
  [string]$Name

  )


$alarmMgr = Get-View AlarmManager
$alarmMgr.GetAlarm($null) | ForEach-Object {

$alarm = Get-View $_

if ($alarm.Info.Name -like $Name){

    if ($PSCmdlet.ShouldProcess($Name)){ 

        $alarm.RemoveAlarm()
    }
}
}
}

The Remove-AlarmDefinition function is rather simple to use. Use the -Name parameter to specify a specific alarm for removal. The -Name parameter can also be a mask, for example Test*, to remove several alarms whose name starts with Test.

Remove-AlarmDefinition -Name "Book: obsolete alarm"
Remove-AlarmDefinition -Name "Test*"

Modifying Alarms

The VMware PowerCLI module does contain a number of alarm-related cmdlets for managing existing alarms, but there is nothing for either creating new ones or removing existing alarms. So you can make changes to existing alarms with the help of these cmdlets. The cmdlets available to you are listed next:

Get-Command *alarm* -Module VMware.VimAutomation.Core

CommandType     Name                        ModuleName
-----------     ----                        ----------
Cmdlet          Get-AlarmAction             VMware.VimAutomation.Core
Cmdlet          Get-AlarmActionTrigger      VMware.VimAutomation.Core
Cmdlet          Get-AlarmDefinition         VMware.VimAutomation.Core
Cmdlet          New-AlarmAction             VMware.VimAutomation.Core
Cmdlet          New-AlarmActionTrigger      VMware.VimAutomation.Core
Cmdlet          Remove-AlarmAction          VMware.VimAutomation.Core
Cmdlet          Remove-AlarmActionTrigger   VMware.VimAutomation.Core
Cmdlet          Set-AlarmDefinition         VMware.VimAutomation.Core

For example, Listing 17-14 demonstrates how to add a new alarm action to the existing alarm Host CPU Usage.

Listing 17-14: Adding a new alarm action to an existing Alarm

Get-AlarmDefinition -Name "Host CPU Usage" | 
  New-AlarmAction -Script -ScriptPath 'c:monitor.bat'

The Get-AlarmDefinition can then be used to examine alarm actions on that alarm:

(Get-AlarmDefinition "Host CPU Usage").ExtensionData.Info.Action.
 Action.Action

Script
------
c:monitor.bat

Moving Alarms

Moving alarms within the same vCenter Server is quite easy, and numerous scripts are available in the blogosphere; just Google or Bing around a bit. You’ll find one example of such a script here:

www.lucd.info/2010/02/20/alarms-moving-them-around/

When you want to move alarm definitions between vCenter Servers that don’t “see” each other, you can’t use a script that reads the alarm definitions from one vCenter Server and then re-creates these alarms on another vCenter Server. The script in this case will need to store the alarm definitions somewhere where they can be picked up when the script is running on the target vCenter Server. Unfortunately, serializing and deserializing PowerShell objects is not too straightforward. That is why we came up with the following alternative. The Get-AlarmScript function takes an object, produced by the Get-Alarm function, and will generate PowerShell code that contains all the necessary statements to create the alarm. This PowerShell code can be saved in a PS1 file and can be transferred to your destination vCenter Server. When you execute the code on the destination vCenter Server, the alarm will be created. The Get-AlarmScript and Get-Alarm functions are included as part of the vCenterAlarms module produced by the authors and referenced earlier in the chapter.

Listing 17-15 shows a sample run using both functions.

Listing 17-15: Generating PowerShell code to move alarms between vCenters

Get-Alarm -Name "Book: test alarm" | Get-AlarmScript |
Set-Content "C:myAlarm.ps1"

The file myAlarm.ps1 file will contain a PS1 script that you can execute on another vCenter Server, and provided the entity on which the original alarm was defined is there, the script will create a new alarm identical to the one you started from. The generated script looks something like Listing 17-16.

Listing 17-16: myAlarm.ps1

#Requires -Version 4.0
#Requires -Modules @{ModuleName="VMware.VimAutomation.Core";`
 ModuleVersion="6.0.0.0"}

$spec = New-Object VMware.Vim.AlarmSpec
$spec.Name = "Book: Change custom field"
$spec.Description = "" 
$spec.ActionFrequency = 0
$spec.Enabled = $True

$action = New-Object VMware.Vim.GroupAlarmAction

$action1 = New-Object VMware.Vim.AlarmTriggeringAction
$action1.Green2yellow = $False
$action1.Red2yellow = $False
$action1.Yellow2green = $False
$action1.Yellow2red = $False
$trans1 = New-Object
 VMware.Vim.AlarmTriggeringActionTransitionSpec
$trans1.StartState = "green"
$trans1.FinalState = "yellow"
$trans1.Repeats = $False

$action1.TransitionSpecs += $trans1

$action.Action += $action1


$spec.Action = $action

$expression = New-Object VMware.Vim.OrAlarmExpression

$expression1 = New-Object VMware.Vim.EventAlarmExpression
$expression1.EventType = CustomFieldValueChangedEvent
$expression1.EventTypeId =
 vim.event.CustomFieldValueChangedEvent
$expression1.ObjectType = VirtualMachine
$expression1.status = yellow

$expression.Expression += $expression1

$spec.Expression = $expression

$setting = New-Object VMware.Vim.AlarmSetting
$setting.reportingFrequency = 0
$setting.toleranceRange = 0

$spec.Setting = $setting

$entity = Get-Inventory -Name DC1

$alarmMgr = Get-View AlarmManager
$alarmMgr.CreateAlarm($entity.Extensiondata.MoRef,$spec)

Get Currently Active Alarms

Now that you have been able to create new alarms, modify existing ones. and remove them, the last thing on the list is to display currently active alarms. Listing 17-17 shows the Get-ActiveAlarm function, which will return active alarms for a given vSphere entity.

Listing 17-17: Retrieving active alarms for an entity

function Get-ActiveAlarm {
  <#
.SYNOPSIS
Returns currently active alarms on an entity
.DESCRIPTION
Returns currently active alarms on an entity  
.INPUTS
System.Management.Automation.PSCustomObject
.OUTPUTS
System.Management.Automation.PSCustomObject
.EXAMPLE
$Folder = Get-Folder "Datacenters"
Get-ActiveAlarm -Entity $Folder
.EXAMPLE
Get-Cluster Cluster01 | Get-ActiveAlarm
#>
[CmdletBinding()]`
 [OutputType('System.Management.Automation.PSCustomObject')]

Param (

    [parameter(Mandatory = $true, ValueFromPipeline = $true)]
    [ValidateNotNullOrEmpty()]
    [PSObject]$Entity    
)

begin {

}

process {

    try {

    foreach ($EntityObject in $Entity){

        foreach ($ActiveAlarm in `
          $EntityObject.ExtensionData.TriggeredAlarmState){

            # --- Get Object Views
            
            $AlarmView = Get-View $ActiveAlarm.Alarm -Property Info.Name
            $EntityView = Get-View $ActiveAlarm.Entity -Property Name 

            # --- Create Output Object
                
            $Object = [pscustomobject]@{                        
                    
                Name = $AlarmView.Info.Name
                Entity = $EntityView.Name
                EntityType = $EntityView.GetType().Name
                Status = $ActiveAlarm.OverallStatus
                Time = $ActiveAlarm.Time
                Acknowledged = $ActiveAlarm.Acknowledged
                AcknowledgedByUser = $ActiveAlarm.AcknowledgedByUser
                ActkowledgedTime = $ActiveAlarm.AcknowledgedTime
            }
        }
               
        Write-Output $Object
    }

    }
    catch [Exception]{

        throw "Unable to get active alarm"
    }
}
end {

}
}

Typical output would be as follows:

Get-Folder "Datacenters" | Get-ActiveAlarm


Name               : Host memory usage
Entity             : vesxi20.sunnydale.local
EntityType         : HostSystem
Status             : yellow
Time               : 11/06/2015 21:57:57
Acknowledged       : True
AcknowledgedByUser : SUNNYDALEjmedd
ActkowledgedTime   : 15/06/2015 22:19:30