AWS CloudFormation is a service to make AWS resource creation easy. A simple JSON format text file could give you the power to create application infrastructure with just a few clicks. System administrators and developers can create, update, and manage their AWS resources easily without worrying about human error. In this section, we will leverage the content of the previous sections in this chapter and use CloudFormation to create them and launch instances with the Kubernetes setting automatically.
The unit of CloudFormation is a stack. One stack is created by one CloudFormation template, which is a text file listing AWS resources in the JSON format. Before we launch a CloudFormation stack using the CloudFormation template in the AWS console, let's get a deeper understanding of the tab names on the CloudFormation console:
|
Tab name |
Description |
|---|---|
|
|
Stack profile overview. Name, status and description are listed here |
|
|
The output fields of this stack |
|
|
The resources listed in this stack |
|
|
The events when doing operations in this stack |
|
|
Text file in JSON format |
|
|
The input parameters of this stack |
|
|
AWS tags for the resources |
|
|
Stack policy to use during update. This can prevent you from removing or updating resources accidentally |
One CloudFormation template contains many sections; the descriptions are put in the following sample template:
{
"AWSTemplateFormatVersion":"AWS CloudFormation templateversion date",
"Description":"stack description",
"Metadata":{
# put additional information for this template
},
"Parameters":{
# user-specified the input of your template
},
"Mappings":{
# using for define conditional parameter values and use it in the template
},
"Conditions":{
# use to define whether certain resources are created, configured in a certain condition.
},
"Resources":{
# major section in the template, use to create and configure AWS resources
},
"Outputs":{
# user-specified output
}
}We will use these three major sections:
- Parameters
- Resources
- Outputs
Parameters are the variable you might want to input when creating the stack, Resources are a major section for declaring AWS resource settings, and Outputs are the section you might want to expose to the CloudFormation UI so that it's easy to find the output information from a resource when a template is deployed.
Intrinsic Functions are built-in functions of AWS CloudFormation. They give you the power to link your resources together. It is a common use case that you need to link several resources together, but they'll know each other until runtime. In this case, the intrinsic function could be a perfect match to resolve this. Several intrinsic functions are provided in CloudFormation. In this case, we will use Fn::GetAtt, Fn::GetAZs and Ref.
The following table has their descriptions:
|
Functions |
Description |
Usage |
|---|---|---|
|
|
Retrieve a value of an attribute from a resource |
|
|
|
Return a list of AZs for the region |
|
|
|
Select a value from a list |
|
|
|
Return a value from a logical name or parameter |
|
Instead of launching a bit template with a thousand lines, we'll split it into two: one is for network resources-only, another one is application-only. Both the templates are available on our GitHub repository via https://github.com/kubernetes-cookbook/cloudformation.
Let's review the following infrastructure listed in the Building the Kubernetes infrastructure in AWS section. We will create one VPC with 10.0.0.0/16 with two public subnets and two private subnets in it. Besides these, we will create one Internet Gateway and add related route table rules to a public subnet in order to route the traffic to the outside world. We will also create a NAT Gateway, which is located in the public subnet with one Elastic IP, to ensure a private subnet can get access to the Internet:
How do we do that? At the beginning of the template, we'll define two parameters: one is Prefix and another is CIDRPrefix. Prefix is a prefix used to name the resource we're going to create. CIDRPrefix is two sections of an IP address that we'd like to create; the default is 10.0. We will also set the length constraint to it:
"Parameters":{
"Prefix":{
"Description":"Prefix of resources",
"Type":"String",
"Default":"KubernetesSample",
"MinLength":"1",
"MaxLength":"24",
"ConstraintDescription":"Length is too long"
},
"CIDRPrefix":{
"Description":"Network cidr prefix",
"Type":"String",
"Default":"10.0",
"MinLength":"1",
"MaxLength":"8",
"ConstraintDescription":"Length is too long"
}
}Then, we will start describing the Resources section. For detailed resource types and attributes, we recommend you visit the AWS Documentation via http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html:
"VPC":{
"Type":"AWS::EC2::VPC",
"Properties":{
"CidrBlock":{
"Fn::Join":[
".",
[
{
"Ref":"CIDRPrefix"
},
"0.0/16"
]
]
},
"EnableDnsHostnames":"true",
"Tags":[
{
"Key":"Name",
"Value":{
"Fn::Join":[
".",
[
{
"Ref":"Prefix"
},
"vpc"
]
]
}
},
{
"Key":"EnvName",
"Value":{
"Ref":"Prefix"
}
}
]
}
}Here, we'll create one resource with the logical name VPC and type AWS::EC2::VPC. Please note that the logical name is important and it cannot be duplicated in one template. You could use {"Ref": "VPC"} in any other resource in this template to refer to VPCId. The name of VPC will be $Prefix.vpc with CIDR $CIDRPrefix.0.0/16. The following image is that of a created VPC:
Next, we'll create the first public subnet with CIDR $CIDRPrefix.0.0/24. Note that {Fn::GetAZs:""} will return a list of all the available AZs. We'll use Fn::Select to select the first element with index 0:
"SubnetPublicA":{
"Type":"AWS::EC2::Subnet",
"Properties":{
"VpcId":{
"Ref":"VPC"
},
"CidrBlock":{
"Fn::Join":[
".",
[
{
"Ref":"CIDRPrefix"
},
"0.0/24"
]
]
},
"AvailabilityZone":{
"Fn::Select":[
"0",
{
"Fn::GetAZs":""
}
]
},
"Tags":[
{
"Key":"Name",
"Value":{
"Fn::Join":[
".",
[
{
"Ref":"Prefix"
},
"public",
"subnet",
"A"
]
]
}
},
{
"Key":"EnvName",
"Value":{
"Ref":"Prefix"
}
}
]
}
}The second public subnet and two private subnets are the same as the first one just with a different CIDR $CIDRPrefix.1.0/24. The difference between public and private subnets are whether they're Internet reachable or not. Typically, an instance in a public subnet will have a public IP or an Elastic IP with it that is Internet reachable. However, a private subnet cannot be reachable from the Internet, except using a bastion host or via VPN. The difference in the AWS setting is the routes in route tables. In order to let your instances communicate with the Internet, we should create an Internet Gateway to a public subnet and a NAT Gateway to a private subnet:
"InternetGateway":{
"Type":"AWS::EC2::InternetGateway",
"Properties":{
"Tags":[
{
"Key":"Stack",
"Value":{
"Ref":"AWS::StackId"
}
},
{
"Key":"Name",
"Value":{
"Fn::Join":[
".",
[
{
"Ref":"Prefix"
},
"vpc",
"igw"
]
]
}
},
{
"Key":"EnvName",
"Value":{
"Ref":"Prefix"
}
}
]
}
},
"GatewayAttachment":{
"Type":"AWS::EC2::VPCGatewayAttachment",
"Properties":{
"VpcId":{
"Ref":"VPC"
},
"InternetGatewayId":{
"Ref":"InternetGateway"
}
}
}We will declare one Internet Gateway with the name $Prefix.vpc.igw and the logical name InternetGateway; we will also attach it to VPC. Then, let's create NatGateway. NatGateway needs one EIP by default, so we'll create it first and use the DependsOn function to tell CloudFormation that the NatGateway resource must be created after NatGatewayEIP. Note that there is AllocationId in the properties of NatGateway rather than the Gateway ID. We'll use the intrinsic function Fn::GetAtt to get the attribute AllocationId from the resource NatGatewayEIP:
"NatGatewayEIP":{
"Type":"AWS::EC2::EIP",
"DependsOn":"GatewayAttachment",
"Properties":{
"Domain":"vpc"
}
},
"NatGateway":{
"Type":"AWS::EC2::NatGateway",
"DependsOn":"NatGatewayEIP",
"Properties":{
"AllocationId":{
"Fn::GetAtt":[
"NatGatewayEIP",
"AllocationId"
]
},
"SubnetId":{
"Ref":"SubnetPublicA"
}
}
}Time to create a route table for public subnets:
"RouteTableInternet":{
"Type":"AWS::EC2::RouteTable",
"Properties":{
"VpcId":{
"Ref":"VPC"
},
"Tags":[
{
"Key":"Stack",
"Value":{
"Ref":"AWS::StackId"
}
},
{
"Key":"Name",
"Value":{
"Fn::Join":[
".",
[
{
"Ref":"Prefix"
},
"internet",
"routetable"
]
]
}
},
{
"Key":"EnvName",
"Value":{
"Ref":"Prefix"
}
}
]
}
}What about private subnets? You could use the same declaration; just change the logical name to RouteTableNat. After creating a route table, let's create the routes:
"RouteInternet":{
"Type":"AWS::EC2::Route",
"DependsOn":"GatewayAttachment",
"Properties":{
"RouteTableId":{
"Ref":"RouteTableInternet"
},
"DestinationCidrBlock":"0.0.0.0/0",
"GatewayId":{
"Ref":"InternetGateway"
}
}
}This route is for the route table of a public subnet. It will relocate to the RouteTableInternet table and route the packets to InternetGatway if the destination CIDR is 0.0.0.0/0. Let's take a look at a private subnet route:
"RouteNat":{
"Type":"AWS::EC2::Route",
"DependsOn":"RouteTableNat",
"Properties":{
"RouteTableId":{
"Ref":"RouteTableNat"
},
"DestinationCidrBlock":"0.0.0.0/0",
"NatGatewayId":{
"Ref":"NatGateway"
}
}
}It is pretty much the same with RouteInternet but route the packets to NatGateway if there are any, to 0.0.0.0/0. Wait, what's the relation between subnet and a route table? We didn't see any declaration indicate the rules in a certain subnet. We have to use SubnetRouteTableAssociation to define their relation. The following examples define both public subnet and private subnet; you might also add a second public/private subnet by copying them:
"SubnetRouteTableInternetAssociationA":{
"Type":"AWS::EC2::SubnetRouteTableAssociation",
"Properties":{
"SubnetId":{
"Ref":"SubnetPublicA"
},
"RouteTableId":{
"Ref":"RouteTableInternet"
}
}
},
"SubnetRouteTableNatAssociationA":{
"Type":"AWS::EC2::SubnetRouteTableAssociation",
"Properties":{
"SubnetId":{
"Ref":"SubnetPrivateA"
},
"RouteTableId":{
"Ref":"RouteTableNat"
}
}
}We're done for the network infrastructure. Then, let's launch it from the AWS console. First, just click and launch a stack and select the VPC sample template.
Click on next; you will see the parameters' pages. It has its own default value, but you could change it at the creation/update time of the stack.
After you click on finish, CloudFormation will start creating the resources you claim on the template. It will return Status as CREATE_COMPLETE after completion.
For application management, we'll leverage OpsWorks, which is an application lifecycle management in AWS. Please refer to the previous two sections to know more about OpsWorks and Chef. Here, we'll describe how to automate creating the OpsWorks stack and related resources.
We'll have eight parameters here. Add K8sMasterBaAccount, K8sMasterBaPassword, and EtcdBaPassword as the basic authentication for Kubernetes master and etcd. We will also put the VPC ID and the private subnet ID here as the input, which are created in the previous sample. As parameters, we could use the type AWS::EC2::VPC::Id as a drop-down list in the UI. Please refer to the supported type in the AWS Documentation via http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html:
"Parameters":{
"Prefix":{
"Description":"Prefix of resources",
"Type":"String",
"Default":"KubernetesSample",
"MinLength":"1",
"MaxLength":"24",
"ConstraintDescription":"Length is too long"
},
"PrivateNetworkCIDR":{
"Default":"192.168.0.0/16",
"Description":"Desired Private Network CIDR or Flanneld (must not overrap VPC CIDR)",
"Type":"String",
"MinLength":"9",
"AllowedPattern":"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/\d{1,2}",
"ConstraintDescription":"PrivateNetworkCIDR must be IPv4 format"
},
"VPCId":{
"Description":"VPC Id",
"Type":"AWS::EC2::VPC::Id"
},
"SubnetPrivateIdA":{
"Description":"Private SubnetA",
"Type":"AWS::EC2::Subnet::Id"
},
"SubnetPrivateIdB":{
"Description":"Private SubnetB",
"Default":"subnet-9007ecc9",
"Type":"AWS::EC2::Subnet::Id"
},
"K8sMasterBaAccount":{
"Default":"admin",
"Description":"The account of basic authentication for k8s Master",
"Type":"String",
"MinLength":"1",
"MaxLength":"75",
"AllowedPattern":"[a-zA-Z0-9]*",
"ConstraintDescription":"Account and Password should follow Base64 pattern"
},
"K8sMasterBaPassword":{
"Default":"Passw0rd",
"Description":"The password of basic authentication for k8s Master",
"Type":"String",
"MinLength":"1",
"MaxLength":"75",
"NoEcho":"true",
"AllowedPattern":"[a-zA-Z0-9]*",
"ConstraintDescription":"Account and Password should follow Base64 pattern"
},
"EtcdBaPassword":{
"Default":"Passw0rd",
"Description":"The password of basic authentication for Etcd",
"Type":"String",
"MinLength":"1",
"MaxLength":"71",
"NoEcho":"true",
"AllowedPattern":"[a-zA-Z0-9]*",
"ConstraintDescription":"Password should follow Base64 pattern"
}
}Before we get started with the OpsWorks stack, we need to create two IAM roles for it. One is a service role, which is used to launch instances, attaching ELB, and so on. Another is an instance role, which is to define the permission for what your OpsWorks instances can perform, to access the AWS resources. Here, we won't access any AWS resources from EC2, so we could just create a skeleton. Please note that you'll need to have IAM permission when you launch CloudFormation with IAM creation. Click on the following checkbox when launching the stack:
In the SecurityGroup section, we will define each ingress and egress to a set of machines. We'll take the Kubernetes master as an example. Since we put ELB in front of the master and ELB to retain the flexibility for future HA settings. Using ELB, we'll need to create a security group to ELB and point the ingress of the Kubernetes master that can be in touch with 8080 and 6443 from the ELB security group. Following is the example of the security group for the Kubernetes master; it opens 80 and 8080 to the outside world:
"SecurityGroupELBKubMaster":{
"Type":"AWS::EC2::SecurityGroup",
"Properties":{
"GroupDescription":{
"Ref":"Prefix"
},
"SecurityGroupIngress":[
{
"IpProtocol":"tcp",
"FromPort":"80",
"ToPort":"80",
"CidrIp":"0.0.0.0/0"
},
{
"IpProtocol":"tcp",
"FromPort":"8080",
"ToPort":"8080",
"SourceSecurityGroupId":{
"Ref":"SecurityGroupKubNode"
}
}
],
"VpcId":{
"Ref":"VPCId"
},
"Tags":[
{
"Key":"Application",
"Value":{
"Ref":"AWS::StackId"
}
},
{
"Key":"Name",
"Value":{
"Fn::Join":[
"-",
[
{
"Ref":"Prefix"
},
"SGElbKubMaster"
]
]
}
}
]
}
},Here is the example of the Kubernetes master instance set. It allows you to receive traffic from 8080 and 6443 from its ELB. We will open the SSH port to use the kubectl command:
"SecurityGroupKubMaster":{
"Type":"AWS::EC2::SecurityGroup",
"Properties":{
"GroupDescription":{
"Ref":"Prefix"
},
"SecurityGroupIngress":[
{
"IpProtocol":"tcp",
"FromPort":"22",
"ToPort":"22",
"CidrIp":"0.0.0.0/0"
},
{
"IpProtocol":"tcp",
"FromPort":"8080",
"ToPort":"8080",
"SourceSecurityGroupId":{
"Ref":"SecurityGroupELBKubMaster"
}
},
{
"IpProtocol":"tcp",
"FromPort":"6443",
"ToPort":"6443",
"SourceSecurityGroupId":{
"Ref":"SecurityGroupELBKubMaster"
}
}
],
"VpcId":{
"Ref":"VPCId"
},
"Tags":[
{
"Key":"Application",
"Value":{
"Ref":"AWS::StackId"
}
},
{
"Key":"Name",
"Value":{
"Fn::Join":[
"-",
[
{
"Ref":"Prefix"
},
"SG-KubMaster"
]
]
}
}
]
}
} Please refer to the examples from the book about the security group setting of etcd and node. Next, we'll start creating the OpsWorks stack. CustomJson acts as the input of the Chef recipe. If there is anything that Chef doesn't know at the beginning, you will have to pass the parameters into CustomJson:
"OpsWorksStack":{
"Type":"AWS::OpsWorks::Stack",
"Properties":{
"DefaultInstanceProfileArn":{
"Fn::GetAtt":[
"RootInstanceProfile",
"Arn"
]
},
"CustomJson":{
"kubernetes":{
"cluster_cidr":{
"Ref":"PrivateNetworkCIDR"
},
"version":"1.1.3",
"master_url":{
"Fn::GetAtt":[
"ELBKubMaster",
"DNSName"
]
}
},
"ba":{
"account":{
"Ref":"K8sMasterBaAccount"
},
"password":{
"Ref":"K8sMasterBaPassword"
},
"uid":1234
},
"etcd":{
"password":{
"Ref":"EtcdBaPassword"
},
"elb_url":{
"Fn::GetAtt":[
"ELBEtcd",
"DNSName"
]
}
},
"opsworks_berkshelf":{
"debug":true
}
},
"ConfigurationManager":{
"Name":"Chef",
"Version":"11.10"
},
"UseCustomCookbooks":"true",
"UseOpsworksSecurityGroups":"false",
"CustomCookbooksSource":{
"Type":"git",
"Url":"https://github.com/kubernetes-cookbook/opsworks-recipes.git"
},
"ChefConfiguration":{
"ManageBerkshelf":"true"
},
"DefaultOs":"Red Hat Enterprise Linux 7",
"DefaultSubnetId":{
"Ref":"SubnetPrivateIdA"
},
"Name":{
"Ref":"Prefix"
},
"ServiceRoleArn":{
"Fn::GetAtt":[
"OpsWorksServiceRole",
"Arn"
]
},
"VpcId":{
"Ref":"VPCId"
}
}
},After creating the stack, we can start creating each layer. Take the Kubernetes master as an example:
"OpsWorksLayerKubMaster":{
"Type":"AWS::OpsWorks::Layer",
"Properties":{
"Name":"Kubernetes Master",
"Shortname":"kube-master",
"AutoAssignElasticIps":"false",
"AutoAssignPublicIps":"false",
"CustomSecurityGroupIds":[
{
"Ref":"SecurityGroupKubMaster"
}
],
"EnableAutoHealing":"false",
"StackId":{
"Ref":"OpsWorksStack"
},
"Type":"custom",
"CustomRecipes":{
"Setup":[
"kubernetes-rhel::flanneld",
"kubernetes-rhel::repo-setup",
"kubernetes-rhel::master-setup"
],
"Deploy":[
"kubernetes-rhel::master-run"
]
}
}
},The run list of Chef in this layer is ["kubernetes-rhel::flanneld", "kubernetes-rhel::repo-setup", "kubernetes-rhel::master-setup"] and ["kubernetes-rhel::master-run"] at the deployment stage. For the run list of etcd, we'll use ["kubernetes-rhel::etcd", "kubernetes-rhel::etcd-auth"] to perform etcd provisioning and authentication setting. For the Kubernetes nodes, we'll use ["kubernetes-rhel::flanneld", "kubernetes-rhel::docker-engine", "kubernetes-rhel::repo-setup", "kubernetes-rhel::node-setup"] as a run list at the setup stage and ["kubernetes-rhel::node-run"] at the deployment stage.
After setting up the layer, we can create ELB and attach it to the stack. The target of health check for the instance is HTTP:8080/version. It will then receive traffic from the port 80 and redirect it to the 6443 port in the master instances, and receive traffic from 8080 to the instance port 8080:
"ELBKubMaster":{
"DependsOn":"SecurityGroupELBKubMaster",
"Type":"AWS::ElasticLoadBalancing::LoadBalancer",
"Properties":{
"LoadBalancerName":{
"Fn::Join":[
"-",
[
{
"Ref":"Prefix"
},
"Kub"
]
]
},
"Scheme":"internal",
"Listeners":[
{
"LoadBalancerPort":"80",
"InstancePort":"6443",
"Protocol":"HTTP",
"InstanceProtocol":"HTTPS"
},
{
"LoadBalancerPort":"8080",
"InstancePort":"8080",
"Protocol":"HTTP",
"InstanceProtocol":"HTTP"
}
],
"HealthCheck":{
"Target":"HTTP:8080/version",
"HealthyThreshold":"2",
"UnhealthyThreshold":"10",
"Interval":"10",
"Timeout":"5"
},
"Subnets":[
{
"Ref":"SubnetPrivateIdA"
},
{
"Ref":"SubnetPrivateIdB"
}
],
"SecurityGroups":[
{
"Fn::GetAtt":[
"SecurityGroupELBKubMaster",
"GroupId"
]
}
]
}
} After creating the master ELB, let's attach it to the OpsWorks stack:
"OpsWorksELBAttachKubMaster":{
"Type":"AWS::OpsWorks::ElasticLoadBalancerAttachment",
"Properties":{
"ElasticLoadBalancerName":{
"Ref":"ELBKubMaster"
},
"LayerId":{
"Ref":"OpsWorksLayerKubMaster"
}
}
}That's it! The ELB of etcd is the same setting, but listen to HTTP:4001/version as a health check and redirect 80 traffic from the outside to the instance port 4001. For a detailed example, please refer to our code reference. After launching the second sample template, you should be able to see the OpsWorks stacks, layers, security groups, IAM, and ELBs. If you want to launch by default with CloudFormation, just add the resource type with AWS::OpsWorks::Instance, specify the spec, and you are all set.
In this recipe, we got an understanding on how to write and deploy an AWS CloudFormation template. Please check out the following recipes as well:
- The Exploring architecture recipe in Chapter 1, Building Your Own Kubernetes
- Building the Kubernetes infrastructure in AWS
- Managing applications using AWS OpsWorks
- Auto-deploying Kubernetes through Chef recipes