Brian Beach1, Steven Armentrout2, Rodney Bozo3 and Emmanuel Tsouris4
(1)
Raleigh, NC, USA
(2)
Mountlake Terrace, WA, USA
(3)
Sterling, VA, USA
(4)
North Bend, WA, USA
This chapter is about architecting your application for high availability. We have covered almost all of the PowerShell commands for EC2, but EC2 is only one of many services that AWS offers. In this chapter, we will examine a few of the services that you can use in concert with EC2 to build a highly available application. These services include Elastic Load Balancers (ELBs), Simple Notification Service (SNS), CloudWatch, Auto Scaling, and Route 53.
We will start by creating a new VPC focused on high availability. This will be a great opportunity to review the material in the prior chapters. Next, we will create an ELB to balance HTTP and HTTPS web traffic across multiple instances. We will configure the ELB to automatically detect errors and remove unhealthy instances. Then, we will use SNS and CloudWatch to create an early warning system that can email us when the application is under stress.
Once that detection system is running, we will use Auto Scaling to automatically scale the application by monitoring load. Auto Scaling will leverage scripted builds to launch and terminate instances throughout the day without human involvement. Finally, we will discuss how Route 53 can be used to extend our application across multiple regions, serving each user from the location nearest them.
This chapter has two exercises. In the first, we consolidate everything we learned in the chapter into one streamlined script. In the second, we create a script to scale up (or resize) an instance. Let’s get started.
Architecting for High Availability
In Chapters 5 and 6, we spent a lot of time discussing VPC with a focus on security. This section focuses on availability. This is not to suggest that we must trade security for high availability. AWS gives you everything you need to achieve both.
We have also discussed regions and availability zones on multiple occasions. Remember that each region includes multiple availability zones connected by high-speed, low-latency links. Each availability zone is a stand-alone data center with distinct power, cooling, and resources. By designing an application to span availability zones, you can build redundancy into your application.
A VPC is limited to a single region, but as shown in Figure 8-1, it can span multiple availability zones. As you already know, a VPC can contain multiple subnets, and each subnet can be in its own availability zone. By spreading our application across availability zones, we can achieve high availability. If once the data centers were to fail, the application could continue running in the other.
Figure 8-1
High availability VPC
Let’s get started by creating the VPC in Figure 8-1. This will be a great opportunity to review much of what we learned in prior chapters.
Let’s assume our application is a simple, single-tier web application with no database. First, we create a new VPC and pick two availability zones in the same region. For example, I am using a private 192.168.0.0 network and the Northern Virginia region. You may have to change the script to use availability zones in your region.
$VPC = New-EC2Vpc -CidrBlock '192.168.0.0/16'
$AvailabilityZone1 = 'us-east-1a'
$AvailabilityZone2 = 'us-east-1b'
Next, we create two subnets in our VPC. Notice that each subnet is using a different availability zone. (If any of this is unfamiliar, go back and review Chapter 5.)
We need to launch at least two instances. This is going to be a web application, so I am using the user data parameter to install and configure IIS. You could use the same method to install your application. (If you have forgotten how to do this, return to Chapter 3.)
At this point we have new VPC with two subnets each in a different availability zone. In addition, we have launched two identical instances. If one of the instances fails, the other will keep running. In fact, even if the entire availability zone failed, the instance in the other zone will keep running. In the next section, we create a load balancer to distribute the load between our two instances.
Managing Elastic Load Balancers
Now that we have multiple instances deployed in multiple data centers, we need a way to distribute requests between them. This is the role of a load balancer. A load balancer receives requests and forwards them to instances in our VPC. The load balancer also monitors the health of the instances and stops sending requests to unhealthy instances automatically. In addition, the load balancer can be configured to terminate SSL and offload the encryption/decryption from the instances acting as web servers.
Elastic Load Balancing (ELB) is actually a family of load balancers including the Classic Load Balancer, Network Load Balancer (NLB), and Application Load Balancer (ALB). I am only going to cover the NLB here, but I want you to be aware that the other variations exist. The functionality of the Classic Load Balancer has been replaced by the ALB and NLB so you can mostly ignore the Classic Load Balancer. The ALB adds support for content-based routing and other enterprise features. You can route traffic to different targets based on host header or URI path.
Figure 8-2 shows our VPC from the prior section with an NLB added. Notice that the NLB is configured in both availability zones. Obviously we need the NLB to be highly available just like the instances we created in the last section. Luckily Amazon does a lot of the heavy lifting for us when we use an NLB. Let’s create one now.
Figure 8-2
VPC with ELB
Preparing the VPC for an ELB
First, we need to create a subnet in each availability zone for the NLB to live in. When we configure the NLB, we tell Amazon to use these subnets. Initially, Amazon will launch an NLB into either one of the subnets. If that availability zone fails, Amazon routes all traffic to the other availability zone. Let’s create two more subnets.
Not all ELBs are Internet facing. You can create an internal ELB that balances traffic between tiers of your application. You can also use PrivateLink to share your application with other AWS accounts.
Now that we have an Internet gateway, we are going to need to configure the route table to use it. One great side effect of using an ELB is that only the ELB needs to be exposed to the Internet. Our instances can live on the private network with no connection to the Internet. Let’s configure a new route table so that only our NLB subnets are public. (If you need to review Internet gateways and route tables, see Chapter 5.)
Now that we have our VPC configured, let’s create an NLB.
Configuring an NLB
Let’s get started by configuring an NLB for HTTP. We will configure HTTPS in the next section. First we create the load balancer. In the following example, I am specifying that I want a Network Load Balancer (as opposed to an Application Load Balancer). I also tell it which subnets the load balancer should run it.
Next, we need to create a target group. A target group is simply a list of instances behind the load balancer. The NLB will balance traffic among the instances in this group. The target group also tracks the health of the instances and takes them out of service if they fail. In the following code, I create a new target group and add our two instances.
Now we just need to connect the NLB to the target group. We do this by defining a listener. In the following example, I use a .Net object to create a listener that forwards all traffic it receives to the target group we created earlier. Note that the ALB supports other actions not covered here including redirection, static responses, and authentication.
It will take a few minutes for the load balancer to launch and complete the initial health checks on the instances to ensure they are healthy. Once it’s ready, you can get the DNS name of the load balancer using the DNSName parameter. You can copy this and paste it into your browser to test the configuration. You can also use a DNS CNAME to map this address to a friendly name (e.g., www.example.com).
$LoadBalancer.DNSName
Now that we have an NLB up and running, let’s look how we can control health checks.
Configuring a Health Check
When we created the preceding target group, we used the default health check configuration. Let’s describe our target group to see that configuration.
Get-ELB2TargetGroup -Name WebTargetGroup
The previous command returns the following results:
TargetGroupName : WebTargetGroup
HealthCheckEnabled : True
HealthCheckIntervalSeconds : 30
HealthCheckPath :
HealthCheckPort : traffic-port
HealthCheckProtocol : TCP
HealthCheckTimeoutSeconds : 10
HealthyThresholdCount : 3
UnhealthyThresholdCount : 3
LoadBalancerArns : {...}
Port : 80
Protocol : TCP
TargetType : instance
VpcId : vpc-123456789012
This is the default health check and it works as follows. Every 30 seconds (HealthCheckInterval) the ELB will attempt to create a TCP (protocol) connection on port 80 (port). If it succeeds three times (HealthyThresholdCount), the instance is healthy. If the connection is not completed within 10 seconds (HealthCheckTimeoutSeconds), the instance is unhealthy. If the instance fails to respond three times (UnhealthyThresholdCount), the ELB will stop forwarding traffic. At this point, the ELB will continue to monitor the instance. If the instance recovers, the ELB will continue to monitor it until it succeeds three times (the HealthyThresholdCount), at which point the ELB will begin forwarding traffic to it again.
Note that the NLB we are using only supports checking the TCP connection. The ALB can do additional checks. For example, you can configure the ALB to request a specific page (e.g., default.htm) using HTTP on port 80 and ensure the web server responds with a 200 status.
If you want, you can check the health of the instances behind the load balancer using Get-ELB2TargetHealth. This command will return a list of instances along with the health of each instance.
At this point our NLB is running and forwarding HTTP requests to our instances. In the next section, we add support for HTTPS.
Configuring an ELB for HTTPS
Most applications today require TLS for at least some portion of the site. As I mentioned earlier, an ELB can be configured to terminate HTTPS. Note that the ELB can also receive an HTTPS request and forward it to the instance without decrypting it, but I did not include an example. Let’s add a new listener to our NLB that terminates HTTPS.
The first step is to create a TLS certificate. You could also import a certificate you already own, but AWS Certificate Manager (ACM) will allow you to easily create free certificates. In the following example, I am creating a certificate for www.example.com. Of course you need to change example.com to a domain that you own.
ACM will validate the certificate by sending an email to the address in the domain registrar (i.e., the email address returned by a whois command). You can also choose to use DNS validation. In this case you will need to prove control over the authoritative DNS for the domain by adding a few CNAME records to the database. Either way, the New-ACMCertificate command will return the ARN of the certificate.
After you have approved your certificate, it will take a few minutes before it is issued. You can check the status with the Get-ACMCertificateDetail command.
Once the certificate is issued, you can create a new listener for HTTPS. This command is similar to the HTTP listener we created earlier. However, we need an additional .Net object to describe the certificate. Also note that I have changed the protocol to TLS and the port to 443.
Obviously you are going to have to create a CNAME record to map your domain (e.g., www.example.com) to the ALB. Now that our load balancer is up and running, let’s spend a minute on CloudWatch.
Monitoring with CloudWatch
Our application is now highly available. If one of the instances becomes unhealthy, the load balancer will remove it from service and send all the traffic to the other instance. While automatic issue resolution is desirable, we still want to know what is happening with our application in the cloud. We need monitoring to alert us when something goes wrong. In this section we will use CloudWatch to create an alert that will email us when CPU utilization exceeds 75% for an extended period of time.
CloudWatch is Amazon’s monitoring solution. CloudWatch can be used to monitor most of the AWS services. In addition, you can create custom metrics using the CloudWatch API. You can configure CloudWatch to take multiple actions when it detects an issue, including sending an email, terminating the instance, launching additional instances, executing a Lambda function, and many other actions.
The first step in creating an email alert is to create a topic with Simple Notification Service (SNS). SNS is a service for sending notifications. It uses a publish-subscribe architecture where many receivers subscribe to notifications that are published using the SNS API. Let’s begin by creating a new topic using the New-SNSTopic command.
$Topic = New-SNSTopic -Name 'MyTopic'
Now that our topic is defined, we want to subscribe to it using email. To create a subscription, use the Connect-SNSNotification command. You will get an email asking you to confirm your email address, and you must accept it before you can receive notifications.
Now that our notification is configured, let’s test it. Remember that SNS is a generic notification service. CloudWatch uses it to send alerts, but you can also use it to send custom notifications. To publish a new message, use the Publish-SNSMessage command. You should receive an email notification with the custom message. For example:
Publish-SNSMessage -TopicArn $Topic -Message "This is a test!"
Now that our notification is configured, we can create an alert. We want to monitor our two instances and receive a notification when CPU utilization exceeds 75% for an extended period of time. The first thing we need to do is define the CloudWatch dimension. A dimension is used to group alerts. In this case we want to group our alerts by instance. Without this dimension we would be measuring the average CPU utilization of all instances in our account. We use a .Net object to create a dimension for the first instance.
CloudWatch is now monitoring our instance. You could create another alarm to monitor the other instance if you want, but I will show an easier way to monitor an entire group of instances in the next section. It will take at least 10 minutes (two periods of 5 minutes) to gather enough data to determine the current state. In the meantime, let’s test our notification by explicitly setting the alarm using the Set-CWAlarmState command.
You should receive an email alarm just like the one you would receive if an instance were in distress. This section has hardly scratched the surface of SNS and CloudWatch. Spend some time reading the documentation about these powerful services. In the next section, we will use Auto Scaling to automatically add and remove instances depending on load.
Using Auto Scaling
Notifications are a great start, but depending on an administrator to respond to alarms is slow. The cloud brings infinite elasticity and with it a whole new way of thinking. Auto Scaling allows us to build an application that automatically responds to changes in demand. Our application can scale out when demand is high and scale in when demand is low. In addition, Auto Scaling can detect issues and replace unhealthy instances.
Figure 8-3 shows the same web application we have been working on throughout this chapter, but the two web instances have been replaced by an Auto Scaling group. The Auto Scaling group is responsible for measuring current load and launching the appropriate number of instances to serve our users.
Figure 8-3
Auto Scaling
The first thing we need to do is terminate the two instances we launched earlier. Going forward, we are going to let the Auto Scaling group launch all of our instances. We don’t want to confuse things by launching instances manually.
Rather than launching instances one at a time, we are going to define a launch configuration and save it for later. The launch configuration is simply a template that the Auto Scaling group will use whenever it needs to launch an instance. Creating a launch configuration is very similar to launching an instance. First, we define the user data script.
Then, we call the New-ASLaunchConfiguration command. New-ASLaunchConfiguration takes all the same parameters as New-EC2Instance and a name used to save the configuration.
With our launch configuration defined, we can create an Auto Scaling group using New-ASAutoScalingGroup. The Auto Scaling group defines how many instances can be launched. DesiredCapacity is the number of instances we think we need, but we also define a min and max that Auto Scaling can work within depending on load. Auto Scaling will ensure that we always have at least the minimum number of instances, but not more than the max.
In addition, we tell the group what subnets to launch instances into, and optionally, which load balancer target group to register with when they start. Note that not all applications will require a load balancer. Some applications will get work from a queue or database table. If you are using a load balancer, you should set HealthCheckType=ELB to adopt the same health check the load balancer is using. By default, Auto Scaling will use instance health – as reported by the hypervisor – to ensure the instance is healthy. You don’t want to end up in a situation where the load balancer has marked all the instances as unhealthy, but the Auto Scaling group is keeping them in service because it thinks they are healthy.
Finally, we can define a HealthCheckGracePeriod and DefaultCoolDown. These last two parameters are really important. HealthCheckGracePeriod defines how long, in seconds, to wait before evaluating the health of a new instance. The default value is 5 minutes, but it can sometimes take longer for a Windows instance to launch and configure itself. If we do not override the defaults, the Auto Scaling group will think the instance is unhealthy and replace it before it finishes configuration. Similarly, DefaultCoolDown defines how long to wait between each Auto Scaling action. Again the default is 5 minutes. If we don’t change this, Auto Scaling will keep launching more and more instances while it waits for the first instance to boot up.
As soon as we run New-ASAutoScalingGroup, the group will begin to launch new instances. You can use the Get-ELBInstanceHealth command to monitor the instances that the group is managing and determine the status of each. You will use this command often while you tune your Auto Scaling rules.
At this point, the Auto Scaling group will launch the desired number of instances and monitor health. If an instance fails, it will be replaced, but we have not defined any Auto Scaling rules so it will not yet respond to changes in load. We use CloudWatch to define the rules just like we did before, but rather than sending a notification, the rule will trigger an Auto Scaling policy.
The first thing we need to do is define a new CloudWatch dimension. In the previous example, we measured the load of an individual instance. In this example, we want to measure the average load of all the instances in our Auto Scaling group. The following dimension will calculate the aggregate over the entire group.
Now we can define a policy to scale up using Write-ASScalingPolicy. This policy simply says to increase the capacity by two instances. Note that you can also override the default cool down to ensure the instance has time to boot before the next scaling occurs.
With the scaling policy defined, we can create a cloud watch alarm to trigger it. This is almost identical to the alarm we created for notification except that the action invokes the scaling policy rather than sending an email.
Write-CWMetricAlarm -AlarmName 'AS75'
-AlarmDescription 'Add capacity when average CPU within the auto scaling group is more than 75%' -Threshold 75
Of course, we also need a policy to remove instances when load diminishes. Otherwise our application will grow and never contract. The policy and alarm are almost identical with a few exceptions. First, the ScalingAdjustment is a negative number to indicate we are removing instances. Second, our alarm is defined as less than 25%.
Once your Auto Scaling group is running, it will work continuously to keep the application properly scaled. In fact, if you manually terminate an instance, it will be replaced within a few minutes. We have launched a lot of infrastructure in the chapter. Let’s delete the Auto Scaling group and load balancer before moving on. First, we set the desired capacity of Auto Scaling group to zero to terminate all the instances.
At this point, we have created a self-healing, Auto Scaling application, but we can go a step further. In the next section, we will look at how Route 53 distribute load across the globe.
Using Route 53
Our application is now architected for high availability, but we could go even further. At the moment, we have achieved high availability by launching redundant instances across availability zones in Northern Virginia. We could also launch a redundant stack in another region. This way the application would continue to work even if an entire region failed. This is where Route 53 comes in.
As seen in Figure 8-4, Route 53 can be used to balance traffic between regions, similar to how an ELB routes traffic between instances. Route 53 is a DNS service and requires that you make AWS your DNS provider. This is a significant commitment you are not likely willing to make to run a few samples from a book. As a result, I have not included any examples in this section, but I wanted you to be aware of Route 53 and how it can help you scale.
Figure 8-4
Route 53
There is another advantage to this architecture beyond high availability. As you know, AWS offers multiple regions around the world. If we deploy our application in many regions, we can serve users from the region closest to them, minimizing latency. This will give the user the best experience.
The advantage of using Amazon’s DNS service is that it offers latency-based routing. Latency-based routing uses geolocation to determine which region is closest to the user and will therefore give them the best experience. In addition, Route 53 can monitor the health of each region and will not route users to a region that is unhealthy.
As we have seen throughout this chapter, AWS offers many services that can be used to monitor and scale an application. In the first exercise, we will pull together everything we learned in this chapter into a single script.
Exercise 8.1: Scaling Out
In this chapter, we learned how to use EC2, VPC, SNS, CloudWatch, Auto Scaling, and Route 53 to create a self-healing application that automatically responds to changes in load. In the process, we took a roundabout approach focused more on exploring each technology than the final solution. In this exercise, we will pull together everything we learned into a single provisioning script that will add an Auto Scaling group to an existing VPC.
First, we need to define the input parameters. This script will add to an existing VPC; therefore, we expect the VPC, subnets (two for the ELBs and two for the application instances), and security groups to be defined already. In addition, the script takes the instance type, AMI, and user data configuration script.
Next, we launch the new load balancer for our application. In this exercise I am only configuring HTTP, but you could easily adapt the script to support HTTPS as described in the chapter.
Then, we create the Auto Scaling group. Here I am specifying two to eight instances with a 10-minute grace period and cool down. This is probably too high, but again we don’t know what the application is; therefore, it is better to err on the high side. If the instance is not up and running within the cool-down period, it will be killed and replaced. This will result in thrashing, where the Auto Scaling continuously kills and replaces instances without giving them time to become effective.
Now we can configure CloudWatch to monitor our application. First, we create a new dimension that aggregates metrics across the entire Auto Scaling group.
That’s all you need to build a self-healing application that automatically responds to changes in load. Auto Scaling is a great solution, but the application must be built with scaling in mind. Some applications are simply not built to scale out. For these applications, you must scale up. In the next section, we create a script to scale up, or move from one instance type to another.
Exercise 8.2: Scaling Up
In this chapter, we created a solution to scale out in response to load. Scaling out refers to adding additional instances in response to demand. Another option is to scale up, or increase the size of an instance. Some systems, such as relational databases, do not scale out easily. These applications must be scaled up.
Luckily AWS has a command for this named Edit-EC2InstanceAttribute. Edit-EC2InstanceAttribute allows you to change many of instance’s attributes including
BlockDeviceMappings
DisableApiTermination
EbsOptimized
Groups
InstanceInitiatedShutdownBehavior
InstanceType
Kernel
Ramdisk
SourceDestCheck
UserData
We are interested in changing the InstanceType. Let’s create a quick script to resize an instance. Our script will take two simple parameters, the instance ID you want to modify and the new instance type.
That’s all there is to it. Once again AWS makes it easy to do something that would be really hard with physical servers in a traditional data center.
Before we wrap up, I want to point out a few limitations of this script:
1.
Your instance must be stopped before you can resize it.
2.
Be really careful with ephemeral storage. Ephemeral disk configurations depend on the instance type and are not always compatible across systems. Be really careful with Elastic Network Interfaces (ENIs) and secondary IP addresses. Again, ENIs and secondary IP configurations differ among instance types.
3.
Be careful with marketplace instances. Marketplace instances cannot be resized as you are licensed for a specific size.
In this exercise we created a script that can be used to resize an instance. In general, scaling out is preferred, but when the application does not support it, we can always scale up.
Summary
In this chapter, we saw the true power of the cloud and AWS. We used VPC to build a highly available application served from two or more active-active data centers. We developed a notification system using SNS and CloudWatch to provide an early warning system that informs the administrator before the application fails.
We also used an ELB to balance traffic across multiple instances and monitor the health of the individual instances. In addition, we used Auto Scaling to monitor load in real time to dynamically resize the application by launching and terminating instances in response to load. Finally, we deployed our application in multiple regions and used Route 53 to automatically route users to the nearest region, minimizing latency.
In the exercises we created scripts to scale out and scale up depending on the application.
It is very easy to overlook the power of what we just did. Very few traditional enterprises can achieve web scale using their own data centers and enterprise solution. But, using the cloud, a single person can build a world-class application from their favorite coffee shop.
This chapter wraps up our discussion on Elastic Compute Cloud (EC2). In the remaining three chapters, we will examine Relational Database Service (RDS), Simple Storage Service (S3), and Identity and Access Management (IAM).
