THE AWS CERTIFIED DEVELOPER – ASSOCIATE EXAM TOPICS COVERED IN THIS CHAPTER MAY INCLUDE, BUT ARE NOT LIMITED TO, THE FOLLOWING:
In the previous chapter, you learned about deploying code packages to AWS Elastic Beanstalk. This is a great way to migrate existing applications to highly available, fault-tolerant infrastructure. As your experience with Amazon Web Services (AWS) deployment increases over time, you may find a need to customize your deployment workflow further than what is supported within a single service. AWS provides a number of deployment services designed for flexibility, empowering customers with complex infrastructure and application deployment requirements.
This chapter introduces the AWS “Code” services. These services are responsible for creating the foundation of a repeatable application, infrastructure, and configuration deployment process. As each service is explained, you will see how they fit into an “enterprise as code” philosophy. You use this approach with each aspect of an enterprise to deploy, configure, and maintain over time via versioned code. (This includes the process to deploy code.) The primary components of an enterprise as code are application, infrastructure, and configuration, though you can take advantage of many more, such as monitoring, compliance, and audit practices.
The AWS “Code” services lay the foundation to deploy different parts of an enterprise starting from a source repository. You start with AWS CodePipeline to create a continuous integration/continuous deployment pipeline (CI/CD) that integrates various sources, tests, deployments, or other components. AWS CodePipeline implements AWS CodeCommit as a source in that it acts as the initialization point of your deployment process. AWS CodeBuild allows you to pull code and packages from various sources to create publishable build artifacts. Lastly, AWS CodeDeploy allows you to deploy compiled artifacts to infrastructure in your environment. AWS CodePipeline is not limited to deploying application code; it can also be used to provision, configure, and manage infrastructure.
In a fully realized enterprise as code, a single commit to a source repository can kick off processes, such as those shown in Figure 7.1.
Organizations can realize a number of benefits from automating the process of testing and preparing software changes. First, there is reduced manual effort required to ensure code changes are tested prior to release. By automating tests, they are consistently run against every change made to a code repository.
Second, developers are no longer tasked with completing steps other than checking in code changes. After the change has been pushed to a source repository, initiation of the build/test process automatically begins. This allows the developers to focus on what they do best: develop software.
Third, the fact that changes are tested immediately after check-in ensures that more bugs are caught earlier in the development process. If bugs are not caught soon, the effort and cost to remediate the errors increases the further they make it in the release process.
Lastly, continuous delivery ensures that quality changes are delivered faster. This increases quality with decreased time to market. So, before you start considering storage options, take time to evaluate your data and decide which of these dimensions your data falls under. This will help you decide what type of storage is best for your data.
AWS CodePipeline is a continuous integration and continuous delivery service for fast and reliable application and infrastructure updates. AWS CodePipeline builds, tests, and deploys your code every time there is a code change, based on the release process models you define. This enables you to deliver features and updates rapidly and reliably. You can easily build an end-to-end solution with prebuilt plugins for popular third-party services like GitHub, or you can integrate your own custom plugins into any stage of your release process. With AWS CodePipeline, you pay only for what you use. There are no up-front fees or long-term commitments.
AWS CodePipeline is the underpinning of CI/CD processes in AWS. Because you define your delivery workflow as a set of stages and actions, multiple changes can be run simultaneously through the same set of processing steps every time. In Figure 7.2, the developer team is responsible for committing changes to a source repository. AWS CodePipeline automatically detects and moves into the source stage. The code change (revision) passes to the build stage, where changes are built into a package or product ready for deployment. A staging deployment is done where users can manually review the functionality that the changes introduce or modify. Before final production release, an authorized user provides a manual approval. After production release, further code changes can reliably pass through the same pipeline.
AWS CodePipeline provides a number of built-in integrations to other AWS services, such as AWS CloudFormation, AWS CodeBuild, AWS CodeCommit, AWS CodeDeploy, Amazon Elastic Container Service (ECS), Elastic Beanstalk, AWS Lambda, AWS OpsWorks Stacks, and Amazon Simple Storage Service (Amazon S3). Some partner tools include GitHub (https://github.com) and Jenkins (https://jenkins.io). Customers also have the ability to create their own integrations, which provides a great degree of flexibility.
You define workflow steps through a visual editor within the AWS Management Console or via a JavaScript Object Notation (JSON) structure for use in the AWS CLI or AWS SDKs. Access to create and manage release workflows is controlled by AWS Identity and Access Management (IAM). You can grant users fine-grained permissions, controlling what actions they can perform and on which workflows.
AWS CodePipeline provides a dashboard where you can review real-time progress of revisions, attempt to retry failed actions, and review version information about revisions that pass through the pipeline.
There are a number of different components that make up AWS CodePipeline and the workflows (pipelines) created by customers. Figure 7.3 displays the AWS CodePipeline concepts.
A pipeline is the overall workflow that defines what transformations software changes will undergo.
You cannot change the name of a pipeline. If you would like to change the name, you must create a new pipeline.
A revision is the work item that passes through a pipeline. It can be a change to your source code or data stored in AWS CodeCommit or GitHub or a change to the version of an archive in Amazon S3. A pipeline can have multiple revisions flowing through it at the same time, but a single stage can process one revision at a time. A revision is immediately picked up by a source action when a change is detected in the source itself (such as a commit to an AWS CodeCommit repository).
If you use Amazon S3 as a source action, you must enable versioning on the bucket.
Details of the most recent revision to pass through a stage are kept within the stage itself and are accessible from the console or AWS CLI. To see the last revision that was passed through a source stage, for example, you can select the revision details at the bottom of the stage, as shown in Figure 7.4.
Depending on the source type (Amazon S3, AWS CodeCommit, or GitHub), additional information will be accessible from the revision details pane (such as a link to the commit on https://github.com), as shown in Figure 7.5.
A stage is a group of one or more actions. Each stage must have a unique name. Should any one action in a stage fail, the entire stage fails for this revision.
An action defines the work to perform on the revision. You can configure pipeline actions to run in series or in parallel. If all actions in a stage complete successfully for a revision, it passes to the next stage in the pipeline. However, if one action fails in the stage, the revision will not pass further through the pipeline. At this point, the stage that contains the failed action can be retried for the same revision. Otherwise, a new revision is able to pass through the stage.
A pipeline must have two or more stages. The first stage includes one or more source actions only. Only the first stage may include source actions.
Every action in the same stage must have a unique name.
The source action defines the location where you store and update source files. Modifications to files in a source repository or archive trigger deployments to a pipeline. AWS CodePipeline supports these sources for your pipeline:
A single pipeline can contain multiple source actions. If a change is detected in one of the sources, all source actions will be invoked.
To use GitHub as a source provider for AWS CodePipeline, you must authenticate to GitHub when you create a pipeline. You provide GitHub credentials to authorize AWS CodePipeline to connect to GitHub to list and view repositories accessible by the authenticating account. For this link, AWS recommends that you create a service account user so that the lifecycle of personal accounts is not tied to the link between AWS CodePipeline and GitHub.
After you authenticate GitHub, a link is created between AWS CodePipeline for this AWS region and GitHub. This allows IAM users to list repositories and branches accessible by the authenticated GitHub user.
You use a build action to define tasks such as compiling source code, running unit tests, and performing other tasks that produce output artifacts for later use in your pipeline. For example, you can use a build stage to import large assets that are not part of a source bundle into the artifact to deploy it to Amazon Elastic Compute Cloud (Amazon EC2) instances. AWS CodePipeline supports the integrations for the following build actions:
You can use test actions to run various tests against source and compiled code, such as lint or syntax tests on source code, and unit tests on compiled, running applications. AWS CodePipeline supports the following test integrations:
The deploy action is responsible for taking compiled or prepared assets and installing them on instances, on-premises servers, serverless functions, or deploying and updating infrastructure using AWS CloudFormation templates. The following services are supported as deploy actions:
An approval action is a manual gate that controls whether a revision can proceed to the next stage in a pipeline. Further progress by a revision is halted until a manual approval by an IAM user or IAM role occurs.
Specifically, the codepipeline:PutApprovalResult action must be included in the IAM policy.
Upon approval, AWS CodePipeline approves the revision to proceed to the next stage in the pipeline. However, if the revision is not approved (rejected or the approval expires), the change halts and will stop progress through the pipeline. The purpose of this action is to allow manual review of the code or other quality assurance tasks prior to moving further down the pipeline.
Approval actions cannot occur within source stages.
You must approve actions manually within seven days; otherwise, AWS CodePipeline rejects the code. When an approval action rejects, the outcome is equivalent to when the stage fails. You can retire the action, which initiates the approval process again. Approval actions provide several options that you can use to provide additional information about what you choose to approve.
Publish approval notifications Amazon Simple Notification Service (Amazon SNS) sends notices to one or more targets that approval is pending.
Specify a Universal Resource Locator (URL) for review You can include a URL in the approval action notification, for example, to review a website published to a fleet of test instances.
Enter comments for approvers You can add additional comments in the notifications for the reviewer’s reference.
You can customize the invoke action within AWS CodePipeline if you leverage the power and flexibility of AWS Lambda. Invoke actions execute AWS Lambda functions, which allows arbitrary code to be run as part of the pipeline execution. Uses for custom actions in your pipeline can include the following:
When you deploy changes to multiple AWS Elastic Beanstalk environments, for example, you can use AWS Lambda to invoke a stage to swap the environment CNAMEs (SwapEnvironmentCNAMEs). This effectively implements blue/green deployments via AWS CodePipeline.
Artifacts are actions that act on a file or set of files. Artifacts can pass between actions and stages in a pipeline to provide a final result or version of the files. For example, an artifact that passes from a build action would deploy to Amazon EC2 during a deploy action.
Multiple actions in a single pipeline cannot output artifacts with the same name.
Every stage makes use of the Amazon S3 artifact bucket that you define when you create the pipeline. Depending on the type of action(s) in the stage, AWS CodePipeline will package the output artifact. For example, the output artifact of a source action would be an archive (.zip) containing the repository contents, which would then act as the input artifact to a build action.
For an artifact to transition between stages successfully, you must provide unique input and output artifact names. In Figure 7.6, the output artifact name for the source action must match the input artifact for the corresponding build action.
Transitions connect stages in a pipeline and define which stages should transition to one another. When all actions in a stage complete successfully, the revision passes to the next stage(s) in the pipeline.
You can manually disable transitions, which stops all revisions in the pipeline once they complete the preceding stage (successfully or unsuccessfully). Once you enable the transition again, the most recent successful revision resumes. Other previous successful revisions will not resume through the pipeline at this time. This concept also applies to stages that are not yet available by the time the next revision completes. If more than one revision completes while the next stage is unavailable, they will be batched. This means that the most current revision will continue through the pipeline once the next stage becomes available.
Approval actions halt further progress through a pipeline until an authorized IAM user or IAM rule approves the transition. You can use approvals to review changes manually before final release into production, or as a code review step.
Figure 7.7 shows a pipeline with three stages: Source, Staging, and LambdaStage. The Source stage contains a source action referencing an Amazon S3 bucket. The source action has already completed and passed the source artifact to Staging. In Staging, the deploy action deploys the source artifact to Amazon EC2 with AWS CodeDeploy. If this action completes successfully, the LambdaStage stage begins, which also deploys to Amazon EC2 via AWS CodeDeploy.
Table 7.1 lists the AWS CodePipeline service limits.
Table 7.1 AWS CodePipeline Service Limits
Limit | Value |
Pipelines per region | US East (N. Virginia) (us-east-1): 40 US West (Oregon) (us-west-2): 60 EU (Ireland) (eu-west-1): 60 Other supported regions: 20 |
Stages per pipeline | Minimum: 2 Maximum: 10 |
Actions per stage | Minimum: 1 Maximum: 20 |
Parallel actions per stage | Maximum: 10 |
Sequential actions per stage | Maximum: 10 |
Maximum artifact size | Amazon S3 source: 2 GB AWS CodeCommit source: 1 GB GitHub source: 1 GB |
When you deploy to AWS CloudFormation, the maximum artifact size is 256 MB.
The remainder of this section will focus on the tasks you need to build and execute a simple pipeline and how to outline the requirements to build cross-account pipelines. This concept is particularly important for organizations that have multiple AWS accounts, especially when you separate environments across accounts (such as Account A for development, Account B for Quality Assurance [QA], and Account C for production), as AWS CodePipeline will need access to resources in each account to automate deployments successfully.
Before you start the next steps, make sure that you have an IAM user with an access key and secret access key and that the user has sufficient AWS CodePipeline permissions.
It is best to name your pipeline something meaningful, such as Dev_S3_Bucket. After you select a source provider (Amazon S3, AWS CodeCommit, or GitHub), you must enter a full object path. This corresponds to the .zip archive that will be tracked for changes.
When you select Amazon S3, AWS CodePipeline creates an Amazon CloudWatch Events rule, IAM role, and AWS CloudTrail trail. These are the default methods that notify AWS CodePipeline of changes to the source archive. You can also use AWS CodePipeline to check regularly for changes. This, however, will provide a slower update experience.
You select AWS CodeBuild, Jenkins, or Solano CI for a build provider.
The Jenkins build provider requires you to install the AWS CodePipeline plugin for Jenkins on the server.
The Solano CI build provider requires authentication to GitHub with a valid user. After authenticating to GitHub, you must authenticate to Solano CI.
If you do not select a build provider, you must select a deployment provider (if you select a build provider, the deployment step is optional). This option is useful if you desire the pipeline execution to be a finished build artifact, such as the case with custom media transcoding with AWS CodeBuild. The available providers for the deployment stage are Amazon ECS, AWS CloudFormation, AWS CodeDeploy, AWS Elastic Beanstalk, and AWS OpsWorks Stacks.
AWS Elastic Beanstalk allows customers to automate deployment of application archives to one or more Amazon EC2 instances. It also handles health checks, load balancing, log gathering, and other important tasks automatically. Since it requires a bundled application archive to upload to instances for deployment, it is a natural fit for AWS CodePipeline, which provides artifacts as archives. To deploy to AWS Elastic Beanstalk from AWS CodePipeline, simply provide the application and environment name.
For deployment to AWS Elastic Beanstalk, the maximum application archive size is 512 MB. The deployment artifact must not exceed this size, or the deployment will fail.
You select a service role for AWS CodePipeline to access AWS resources within your account. You can select an existing IAM role or create a new role.
You can only select IAM roles with a trust policy that allows AWS CodePipeline to assume them.
After you create a pipeline, the first stage updates the source repository or archive, and then the pipeline will automatically begin execution. To rerun the pipeline for the most recent revision, select Release Change in the AWS CodePipeline console, or invoke the aws codepipeline start-pipeline-execution AWS CLI command.
aws codepipeline start-pipeline-execution --name SamplePipeline
If a pipeline action fails for any reason, you can retry that action on the same revision in the console or use the aws codepipeline retry-stage-execution AWS CLI command. However, there are certain situations where a failed action may become ineligible for retries.
In some architectures, environments may be spread across two or more AWS accounts. You can implement a single CI/CD workflow with AWS CodePipeline that interacts with resources in multiple AWS accounts.
If an organization has separate accounts for development, test, and production workloads, you can leverage one pipeline to deploy to resources in all three. To do so, you must create and shard several components between accounts.
A source action of Amazon S3 cannot reference buckets in accounts other than the pipeline account.
In the following steps, the account that contains the pipeline will be the pipeline account. The account to deploy resources will be the target account.
If you deploy revisions to Amazon EC2 instances (as with AWS CodeDeploy), you apply a policy to the instance role that allows access to the Amazon S3 bucket that the AWS CodePipeline uses in the pipeline account. Additionally, the instance role must also have a policy that allows access to the AWS KMS key.
AWS CodeCommit is a fully managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories. AWS CodeCommit eliminates the need to operate your own source control system or worry about scaling its infrastructure. You can use AWS CodeCommit to store anything securely, from source code to binaries, and it works seamlessly with your existing Git tools.
Before any activities can occur to deploy applications, you must first have a location where you can store and version application code in a reliable fashion. AWS CodeCommit is a cloud-based, highly available, and redundant version control service. AWS CodeCommit leverages the Git framework, and it is fully compatible with existing tooling. There are a number of benefits to this service, such as the following:
However, the biggest benefit of AWS CodeCommit is built-in integration with multiple other AWS services, like AWS CodePipeline. With these integrations, AWS CodeCommit acts as the initial step to automate application code releases.
This section details the concepts behind AWS CodeCommit.
When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you request. AWS uses the security credentials to authenticate and authorize your requests.
HTTPS connectivity to a Git-based repository requires a username and password, which pass to the repository as part of a request. To use AWS CodeCommit with HTTPS credentials, you must first add them to an IAM user with sufficient permissions to interact with the repository. To create Git credentials for your IAM user, you open the IAM console, and select the user who will need to authenticate to the AWS CodeCommit repository via HTTPS.
AWS generates security credentials for the usernames and passwords, and they cannot be set to custom values.
Make sure to download or copy the credentials because the password will be lost after you close the success window.
After you configure your Git CLI/application to use the repository’s HTTPS endpoint and the username/password, you will have access to the AWS CodeCommit repository.
With SSH authentication, there is no need to install the AWS CLI to connect to your repository. However, you perform some additional configuration tasks.
To configure SSH authentication to AWS CodeCommit repositories, follow these steps:
Host git-codecommit.*.amazonaws.com
User YOUR_SSH_KEY_ID
IdentityFile YOUR_PRIVATE_KEY_FILE
# Format: ssh git-codecommit.[REGION_CODE].amazonaws.com
ssh git-codecommit.us-east-1.amazonaws.com
The previous HTTPS and SSH authentication methods both rely on additional credentials aside from IAM access/secret keys. It is also possible to authenticate to AWS CodeCommit with IAM credentials and the AWS CodeCommit credential helper. The credential helper translates IAM credentials to those that AWS CodeCommit can use to perform Git actions, such as to clone a repository or merge a pull request. To configure the credential helper on your workstation, do the following:
git config --global
credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true
Once complete, HTTPS interactions with the AWS CodeCommit repository should work as expected.
The credential helper authentication method is the only one available for root and federated IAM Users.
AWS CodeCommit integrates automatically with any development tools that support IAM credentials. Additionally, after you set up HTTPS Git credentials, you are able to use any tools that support this authentication mechanism instead. Examples of supported integrated development environment (IDE) include the following:
A repository (repo) is the foundation of AWS CodeCommit. This is the location where you store source code files, track revisions, and merge contributions (commits). When you create a repository, it will contain an empty master branch by default. To configure additional branches and commit code changes, you connect the repository to a local workstation where changes can be made before you upload or push them.
Repository names must be unique within an individual AWS account; however, you can change them without re-creating the repository. When you change a repository name, you need to update any local copies of the repository to have their remote point to the new HTTPS or SSH URL with the git remote add command.
AWS CodeCommit supports triggers via Amazon SNS, which you can use to leverage other AWS services for post-commit actions, such as firing a webhook with AWS Lambda after a commit is pushed to a development branch. To implement this, AWS CodeCommit uses AWS CloudWatch Events. You create event rules that trigger for each of the event types that you select in AWS CodeCommit. Event types that will fire notifications include the following:
If you change the name of the repository through the AWS CLI or SDK, the notifications cease to function. (This behavior is not present when you change names in the AWS Management Console.) To restore lost notifications, delete the settings and configure them a second time.
Repository triggers are not the same as notifications, as the events that fire each differ greatly. Use repository triggers to send notifications to Amazon SNS or AWS Lambda during these events:
Triggers are similar in functionality to webhooks used by other Git providers, like GitHub. You can use triggers to perform automated tasks such as to start external builds, to notify administrators of code pushes, or to perform unit tests. There are some restrictions on how to configure triggers.
In some situations, the repository that contains the application source code may be located in a separate AWS account from the IAM user/role attempting to access it. In these situations, there are several steps that you must perform in the repository account and the user account.
Once these steps are complete, the IAM user will first need to assume the cross-account role before you attempt to clone or otherwise access the repository. You adjust the AWS credentials file ~/.aws/config (Linux/macOS) or drive:Usersusername.awsconfig (Windows). A profile will be added to this config file that specifies the cross-account role to assume.
[profile MyCrossAccountProfile]
region = US East (Ohio)
role_arn=arn:aws:iam:111122223333:role/MyCrossAccountContributorRole
source_profile=default
Lastly, you need to modify the AWS CLI credential helper so that you use MyCrossAccountProfile.
git config --global credential.helper
'!aws codecommit credential-helper --profile MyCrossAccountProfile $@'
From this point, the IAM user in the user account will be able to clone and interact with the repository in the repository account.
A file is a piece of data that is subject to version control by AWS CodeCommit. AWS CodeCommit tracks any modifications made to this file on a per-line level. You use the Git client to push changes in a file to the repository, where it tracks against other changes in previous commits.
Pull requests are the primary vehicle on which you review and merge code changes between branches. Unlike branch merging, pull requests allow multiple users to comment on changes before they merge with the destination branch. The typical workflow of a pull request is as follows:
Commits are point-in-time changes to contents of files in a repository. A commit is not a new copy of the file, but it is instead a way to track changes in the line(s) in a file, by whom, and when. When you push a commit to the repository, AWS CodeCommit tracks the following file changes:
Commits to a repository in AWS CodeCommit can be made in one of two ways. The most common workflow is to use the Git CLI and update the repository using git push. The AWS CLI supports the aws codecommit put-file action, which allows you to update a file on the repository with a local copy and specify a branch, parent commit, and message.
aws codecommit put-file --repository-name MyDemoRepo
--branch-name feature-branch
--file-content file://MyDirectory/ExampleFile.txt
--file-path /solutions/ExampleFile.txt
--parent-commit-id 11112222EXAMPLE
--name "Developer"
--email [email protected]
--commit-message "Fixed a bug”
The AWS CodeCommit console supports viewing differences between commits. To view differences between a commit and its parent, open the Commits pane on the repository dashboard and then select the commit ID, as shown in Figure 7.10.
After doing so, you can view changes in this commit either side by side (Split view) or in the same pane (Unified view), as shown in Figure 7.11.
You can also view differences between arbitrary commit IDs in the same repository. In the Compare window of the repository dashboard, you can choose two commit IDs for comparison, as shown in Figure 7.12.
After you select two commit IDs, click the Compare button. This will provide a similar split or unified view of changes.
Branches are ways to separate and organize groups of commits. This allows developers to organize work in a meaningful fashion, separating changes into logical groups based on the feature or bug-fix being developed. For example, as you can see in Figure 7.13, a single repository may have branches for each environment: dev, test, and prod. Or, individual features and bug fixes can have separate branches.
A default branch is the base when you clone the repository. When you clone a repository to your local machine, the default branch (such as “master” or “prod”) clones. You cannot delete the default branch until a new branch is set as default, or you will delete the entire repository.
You can change the default branch for a repository, but first the new default branch must exist in the remote repository.
This section details how to migrate a Git repository, unversioned files, or another repository type, and it is important when you migrate a high volume of or large files.
You use an AWS CodeCommit to migrate from a Git repository, as shown in Figure 7.14.
The first step is to create the AWS CodeCommit repository (via either the AWS Management Console or the AWS CLI or AWS SDK). After you create the repository, clone the project to a local workstation. To push this repository to AWS CodeCommit, set the repository’s remote to the AWS CodeCommit repository’s HTTPS or SSH URL.
git push
https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyClonedRepository
--all
If you need to push any tags to the new repository, run the following code:
git push
https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyClonedRepository
--tags
You can migrate any local or unversioned content to AWS CodeCommit in a similar manner, as if the content exists in another Git-based repository service. The primary difference is that you set up a new repository instead of cloning an existing one to migrate. Refer to Figure 7.15.
First create the AWS CodeCommit repository (either via the AWS Management Console or via the AWS CLI or AWS SDK). Next, create a local directory with the files to migrate, and run git init from the command line or terminal in that directory. This will initialize the directory to work with Git so that any file changes are tracked. After the directory initializes, run git add . to add all current files to Git. Run git commit -m 'Initial Commit' to generate a commit. Lastly, push the commit to AWS CodeCommit with git push https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyFirstRepo --all.
For large repositories, you can migrate in incremental steps and push many smaller files. This prevents any network issues that may cause the entire push to fail. If any smaller commit fails, it is a trivial matter to restart it when you compare it to a single, monolithic commit.
Additionally, when you push large repositories, AWS recommends that you use SSH over HTTPS, as there is a chance that the HTTPS connection may terminate because of various network or firewall issues.
AWS CodeCommit enforces the service limits in Table 7.2. An asterisk (*) indicates limits that require you to submit a request to AWS Support to increase the limits.
Table 7.2 AWS CodeCommit Service Limits
Limit | Value |
Repositories per account* | 1,000 |
References per push | 4,000 |
Triggers per repository | 10 |
Git blob size | 2 GB |
You can use AWS CodeCommit as a source action in your pipeline. This allows you to utilize a highly available, redundant version control system as the initialization point of your CI/CD pipeline.
When you select AWS CodeCommit as the source provider, you must provide a repository name and branch. If you use AWS CodeCommit, it creates an Amazon CloudWatch Events rule and an IAM role to monitor the repository and branch for changes, as shown in Figure 7.16.
One issue that can arise is if you store large binary files. Because of the system Git uses to track file changes, Git creates a new copy of every modification to a binary file within the repository. Over time, this can cause repositories to grow rapidly in size. Instead of storing binary files in AWS CodeCommit, add an additional Amazon S3 source action to the pipeline. If you store large binary files in Amazon S3, you can reduce the overall cost and development time because of the reduction of time it takes to push/pull commits. Since Amazon S3 already supports versioning (and requires it for use with AWS CodePipeline), changes to binary objects will still be tracked so that rollbacks are straightforward.
AWS CodeBuild is a fully managed build service that compiles source code, runs tests, and produces software packages that are ready to deploy. With AWS CodeBuild, you do not need to provision, manage, and scale your own build servers. AWS CodeBuild scales continuously and processes multiple builds concurrently, so your builds do not wait in a queue. AWS CodeBuild has prepackaged build environments, or you can create custom build environments that use your own build tools. With AWS CodeBuild, AWS charges by the minute for the compute resources you use.
AWS CodeBuild enables you to define the build environment to perform build tasks and the actual tasks that it will perform. AWS CodeBuild comes with prepackaged build environments for most common workloads and build tools (Apache Maven, Grade, and others), and it allows you to create custom environments for any custom tools or processes.
AWS CodePipeline includes built-in integration with AWS CodeBuild, which can act as a provider for any build or test actions in your pipeline, as shown in Figure 7.17.
AWS CodeBuild initiates build tasks inside a build project, which defines the environmental settings, build steps to perform, and any output artifacts. The build container’s operating system, runtime, and build tools make up the build environment.
Build projects define all aspects of a build. This includes the environment in which to perform builds, any tools to include in the environment, the actual build steps to perform, and outputs to save.
When you create a build project, you first select a source provider. AWS CodeBuild supports AWS CodeCommit, Amazon S3, GitHub, and BitBucket as source providers. When you use GitHub or BitBucket, a separate authentication flow will be invoked. This allows access to the source repository from AWS CodeBuild. GitHub source repositories also support webhooks to trigger builds automatically any time you push a commit to a specific repository and branch.
After AWS CodeBuild successfully connects to the source repository or location, you select a build environment. AWS CodeBuild provides preconfigured build environments for some operating systems, runtimes, and runtime versions, such as Ubuntu with Java 9.
Next, you will configure the build specification. This can be done in one of two ways. You can insert build commands in the console or specify a buildspec.yml file in your source code. Both options are valid, but if you use a buildspec.yml file, you will see additional configuration options.
If your build creates artifacts you would like to use in later steps of your pipeline/process, you can specify output artifacts to save to Amazon S3. Otherwise, you can choose not to save any artifacts. You will need to specify individual filename(s) for AWS CodeBuild to save on your behalf.
AWS CodeBuild supports caching, which you can configure in the next step. Caching saves some components of the build environment to reduce the time to create environments when you submit build jobs.
Every build project requires an IAM service role that is accessible by AWS CodeBuild. When you create new projects, you can automatically create a service role that you restrict to this project only. You can update service roles to work with up to 10 build projects at a time.
Lastly, you can configure AWS CodeBuild to create build environments with connectivity to an Amazon Virtual Private Cloud (Amazon VPC) in your account. To do so, specify the Amazon VPC ID, subnets, and security groups to assign to the build environment. You can configure other settings when you create the build, such as to run the Docker daemon in privileged mode to build Docker images.
After you set the build project properties, you can select the compute type (memory and vCPU settings), any environment variables to pass to the build container, and tags to apply to the project.
When you set environment variables, they will be visible in plain text in the AWS CodeBuild console and AWS CLI or SDK. If there is sensitive information that you would like to pass to build jobs, consider using the AWS Systems Manager Parameter Store. This will require the build project’s IAM role to have permissions to access the parameter store.
The buildspec.yml file can provide the build specification to your build projects in the AWS CodeBuild console, the AWS CLI, or the AWS SDK when you create the build project, or as part of your source repository in a YAML-formatted buildspec.yml file. You can supply only one build specification to a build project. A build specification’s format is as follows:
version: 0.2
env:
variables:
key: "value"
parameter-store:
key: "value"
phases:
install:
commands:
- command
pre_build:
commands:
- command
build:
commands:
- command
post_build:
commands:
- command
artifacts:
files:
- location
discard-paths: yes
base-directory: location
cache:
paths:
- path
AWS supports multiple build specification versions; however, AWS recommends you use the latest version whenever possible.
You can add optional environment variables to build jobs. Any key/value pairs that you provide in the variables section are available as environment variables in plain text.
Any environment variables that you define here will overwrite those you define elsewhere in the build project, such as those in the container itself or by Docker.
The parameter-store mapping specifies parameters to query in AWS Systems Manager Parameter Store.
The phases mapping specifies commands to run at each stage of the build job. When you specify build settings in the AWS CodeBuild console, AWS CLI, or AWS SDK, you are not able to separate commands into phases. With a build specifications file, you can separate commands into phases.
install Commands to execute during installation of the build environment.
pre_build Commands to be run before the build begins.
build Commands to be run during the build.
post_build Commands to be run after the build completes.
If a command fails in any stage, subsequent stages will not run.
The artifacts mapping specifies where AWS CodeBuild will place output artifacts, if any. This is required only if your build job produces actual outputs. For example, unit tests would not produce output artifacts for later use in a pipeline. The files list specifies individual files in the build environment that will act as output artifacts. You can specify individual files, directories, or recursive directories. You can use discard-paths and base-directory to specify a different directory structure to package output artifacts.
If you configure caching for the build project, the cache map specifies which files to upload to Amazon S3 for use in subsequent builds.
This example sets the JAVA_HOME and LOGIN_PASSWORD environment variables (the latter is retrieved from AWS Systems Manager Parameter Store), installs updates in the build environment, runs a Maven installation, and saves the .jar output to Amazon S3 as a build artifact. For future builds, the content of the /root/.m2 directory (and any subdirectories) is cached to Amazon S3.
version: 0.2
env:
variables:
JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64"
parameter-store:
LOGIN_PASSWORD: "dockerLoginPassword"
phases:
install:
commands:
- echo Entered the install phase...
- apt-get update -y
- apt-get install -y maven
pre_build:
commands:
- echo Entered the pre_build phase...
- docker login –u User –p $LOGIN_PASSWORD
build:
commands:
- echo Entered the build phase...
- echo Build started on 'date'
- mvn install
post_build:
commands:
- echo Entered the post_build phase...
- echo Build completed on 'date'
artifacts:
files:
- target/messageUtil-1.0.jar
discard-paths: yes
cache:
paths:
- '/root/.m2/**/*'
A build environment is a Docker image with a preconfigured operating system, programming language runtime, and any other tools that AWS CodeBuild uses to perform build tasks and communicate with the service, along with other metadata for the environment, such as the compute settings. AWS CodeBuild maintains its own repository of preconfigured build environments. If these environments do not meet your requirements, you can use public Docker Hub images. Alternatively, you can use container images in Amazon Elastic Container Registry (Amazon ECR).
AWS CodeBuild provides build environments for Ubuntu and Amazon Linux operating systems, and it supports the following:
Not all programming languages support both Ubuntu and Amazon Linux build environments.
Table 7.3 lists the memory, virtual central processing unit (vCPU), and disk space configurations for build environments.
Table 7.3 Compute Configurations for Build Environments
Compute Type | Memory | vCPUs | Disk Space |
BUILD_GENERAL1_SMALL | 3 GB | 2 | 64 GB |
BUILD_GENERAL1_MEDIUM | 7 GB | 4 | 128 GB |
BUILD_GENERAL1_LARGE | 15 GB | 8 | 128 GB |
AWS CodeBuild provides several environment variables by default, such as AWS_REGION, CODEBUILD_BUILD_ID, and HOME.
When you create your own environment variables, AWS CodeBuild reserves the CODEBUILD_ prefix.
When you initiate a build, AWS CodeBuild copies the input artifact(s) into the build environment. AWS CodeBuild uses the build specification to run the build process, which includes any steps to perform and outputs to provide after the build completes. Build logs are made available to Amazon CloudWatch Logs for real-time monitoring.
When you run builds manually in the AWS CodeBuild console, AWS CLI, or AWS SDK, you have the option to change several properties before you run a build job.
AWS CodeBuild enforces service limits in Table 7.4. An asterisk (*) indicates that you can increase limits if you submit a request to AWS Support.
Table 7.4 AWS CodeBuild Service Limits
Limit | Value |
Build projects per region per account* | 1,000 |
Build timeout | 8 hours |
Concurrently running builds* | 20 |
AWS CodePipeline enables you to build jobs for both build and test actions. Both action types require exactly one input artifact and may return zero or one output artifacts. When you create a build or test actions in your pipeline with your build projects, the only input that you require is the build project name. The AWS CodePipeline console also has the option to create new build projects when you create the action, as shown in Figure 7.18.
AWS CodeDeploy is a service that automates software deployments to a variety of compute services, such as Amazon EC2, AWS Lambda, and instances running on-premises. AWS CodeDeploy makes it easier for you to release new features rapidly, helps you avoid downtime through application deployment, and handles the complexity to update your applications. You can use AWS CodeDeploy to automate software deployments and eliminate the need for error-prone manual operations. The service scales to match your deployment needs, from a single AWS Lambda function to thousands of Amazon EC2 instances.
AWS CodeDeploy standardizes and automates deployments of any types of content or configuration to Amazon EC2 instances, on-premises servers, or AWS Lambda functions. Because of its flexibility, it is not restricted to deploy only application code, and it can perform various administrative tasks that are part of your deployment process. Additionally, you can create custom deployment configurations tailored to your specific infrastructure needs.
You can install and enable NGINX as part of a deployment of configuration files to reverse proxy instances. The service itself does not involve any changes to your current source code, and it only requires you to install a lightweight agent on any managed instances or on-premises servers.
Should deployments fail in your environment, you can configure AWS CodeDeploy with a predetermined failure tolerance. Once this tolerance is breached, deployment will automatically roll back to the last version that works.
You can automate deployment of AWS CodeDeploy with AWS Lambda functions through traffic switching. When updates to functions deploy, AWS CodeDeploy will create new versions of each updated function and gradually route requests from the previous version to the updated function. AWS Lambda functions also support custom deployment configurations, which can specify the rate and percentage of traffic to switch.
When you deploy to Amazon EC2 on-premises instances, a revision occurs.
A revision is an artifact that contains both application files to deploy and an AppSpec configuration file. Application files can include compiled libraries, configuration files, installation packages, static media, and other content. The AppSpec file specifies what steps AWS CodeDeploy will follow when it performs deployments of an individual revision.
A revision must contain any source files and scripts to execute on the target instance inside a root directory. Within this root directory, the appspec.yml file must exist at the topmost level and not in any subfolders.
/tmp/ or c: emp (root folder)
|--content (subfolder)
| |--myTextFile.txt
| |--mySourceFile.rb
| |--myExecutableFile.exe
| |--myInstallerFile.msi
| |--myPackage.rpm
| |--myImageFile.png
|--scripts (subfolder)
| |--myShellScript.sh
| |--myBatchScript.bat
| |--myPowerShellScript.ps1
|--appspec.yml
When you deploy to AWS Lambda, a revision contains only the AppSpec file. It contains information about the functions to deploy, as well as the steps to validate that the deployment was successful.
In either case, when a code revision is ready to deploy, you package it into an archive file and store it in one of these three repositories:
When you use GitHub or Bitbucket, the source code does not need to be a .zip archive, as AWS CodeDeploy will package the repository contents on your behalf. Amazon S3, however, requires a .zip archive file.
AWS Lambda deployments support only Amazon S3 buckets as a source repository.
A deployment is the process of copying content and executing scripts on instances in your deployment group. To accomplish this, AWS CodeDeploy performs the tasks outlined in the AppSpec configuration file. For both Amazon EC2 on-premises instances and AWS Lambda functions, the deployment succeeds or fails based on whether individual AppSpec tasks complete successfully. There are two types of deployments supported by AWS CodeDeploy: in-place and blue/green.
In in-place deployments, revisions deploy to new infrastructure instead of an existing one. After deployment completes successfully, the new infrastructure gradually replaces old code in a phased rollout. After all traffic routes to the new infrastructure, you can keep the old code for review or discard it.
On-premises instances do not support blue/green deployments.
When you deploy to AWS Lambda functions, blue/green deployments publish new versions of each function, after which traffic shifting routes requests to the new function versions according to the deployment configuration that you define.
You can stop deployments via the AWS CodeDeploy console or AWS CLI. If you stop deployments to Amazon EC2 on-premises instances, this can result in some deployment groups being left in an undesired deployment state. For example, when you deploy to instances in an Auto Scaling group, if you stop the deployment, it may result in some instances having different application versions. In situations where this occurs, you can configure the application to roll back to the last valid deployment automatically. To do this, you submit a new deployment to the instances with the previous revision, and they appear as a new deployment in the console.
Some instances that fail the most recent deployment may still have scripts run or files placed that are part of the failed deployment. If you configure automatic rollbacks, AWS CodeDeploy will attempt to remove any successfully created files.
AWS CodeDeploy achieves automatic rollbacks by redeploying the last working revision to any instances in the deployment group (this will generate a new deployment ID). If you do not configure automatic rollbacks for the application, you can perform a manual rollback by redeploying a previous revision as a new deployment. This will accomplish the same result as an automatic rollback.
During the rollback process, AWS CodeDeploy will attempt to remove any file(s) that were created on the instance during the failed deployment. A record of the created files is kept in the location on your instances.
The AWS CodeDeploy agent that runs on the instance will reference this cleanup file as a record of what files were created during the last deployment.
By default, AWS CodeDeploy will not overwrite any files that were not created as part of a deployment. You can override this setting for new deployments.
AWS CodeDeploy tracks cleanup files; however, script executions are not tracked. Any configuration or modification to the instance that is done by scripts run on your instance cannot be rolled back automatically by AWS CodeDeploy. As an administrator, you will be responsible for implementing logic in your deployment scripts to ensure that the desired state is reached during deployments and rollbacks.
If you would like to test whether a revision will successfully deploy to an instance you are able to access, you can use the codedeploy-local command in the AWS CodeDeploy agent. This command will search the execution path for an AppSpec file and any content to deploy. If this is found, the agent will attempt a deployment on the instance and provide feedback on the results. This provides a useful alternative to executing the full workflow when you want simply to validate the deployment package.
The following example command attempts to perform a local deployment of an archive file located in Amazon S3:
codedeploy-local --bundle-location s3://mybucket/bundle.tgz --type tgz
The codedeploy-local command requires the AWS CodeDeploy agent that you install on the instance or on-premises server where you execute the command.
A deployment group designates the Amazon EC2 on-premises instances that a revision deploys. When you deploy to AWS Lambda functions, this specifies what functions will deploy new versions. Deployment groups also specify alarms that trigger automatic rollbacks after a specified number or percentage of instances, or functions fail their deployment.
For Amazon EC2 on-premises deployments, you can add instances to a deployment group based on tag name/value pairs or Amazon EC2 Auto Scaling group names. An individual application can have one or more deployment groups defined. This allows you to separate groups of instances into environments so that changes can be progressively rolled out and tested before going to production. You can identify instances by individual tags or tag groups. If an instance matches one or more tags in a tag group, it is associated with the deployment group. If you would like to require that an instance match multiple tags, each tag must be in a separate tag group. A single deployment group supports up to 10 tags in up to three tag groups.
In Figure 7.19, if tags Environment, Region, and Type are present in tag groups 1, 2, and 3 respectively, then instances must have at least one tag in each tag group to identify with the deployment group.
When you create deployment groups, you can also configure the following:
Amazon SNS notifications Any recipients that subscribe to the topic will receive notifications when deployment events occur. You must create the topic before you configure this notification, and the AWS CodeDeploy service role must have permission to publish messages to the topic.
Amazon CloudWatch alarms You can configure alarms to trigger cancellation and rollback of deployments whenever the metric has passed a certain threshold. For example, you could configure an alarm to trigger when CPU utilization exceeds a certain percentage for instances in an AWS Auto Scaling group. If this alarm triggers, the deployment automatically rolls back. For AWS Lambda deployments, you can configure alarms to monitor function invocation errors.
Automatic rollbacks You can configure rollbacks to initiate automatically when a deployment fails or based on Amazon CloudWatch alarms. To test deployments, you can disable automatic rollbacks when you create a new deployment.
You can host instances for an Amazon EC2 on-premises deployment group in either an AWS account or your own data center. To configure an on-premises instance to work with AWS CodeDeploy, you must complete several tasks. Before you begin, you need to ensure that the instance has the ability to communicate with AWS CodeDeploy service endpoints over HTTPS (port 443). You will also need to create an IAM user that the instance assumes and has permissions to interact with AWS CodeDeploy.
aws deploy register --instance-name AssetTag12010298EX
--iam-user-arn arn:aws:iam::8039EXAMPLE:user/CodeDeployUser-OnPrem
--tags Key=Name,Value=CodeDeloyDemo-OnPrem
--region us-west-2
aws deploy install --override-config
--config-file /tmp/codedeploy.onpremises.yml
--region us-west-2
After you complete the previous steps, the instance will be available for deployments to the deployment group(s).
When you deploy to Amazon EC2 Auto Scaling groups, AWS CodeDeploy will automatically run the latest successful deployment on any new instances created when the group scales out. If the deployment fails on an instance, it updates to maintain the count of healthy instances. For this reason, AWS does not recommend that you associate the same Auto Scaling group with multiple deployment groups (for example, you want to deploy multiple applications to the same Auto Scaling group). If both deployment groups perform a deployment at roughly the same time and the first deployment fails on the new instance, it terminates by AWS CodeDeploy. The second deployment, unaware that the instance terminated, will not fail until the deployment times out (the default timeout value is 1 hour). Instead, you should combine your application deployments into one or consider the use of multiple Auto Scaling groups with smaller instance types.
You use deployment configurations to drive how quickly Amazon EC2 on-premises instances update by AWS CodeDeploy. You can configure deployments to deploy to all instances in a deployment group at once or subgroups of instances at a time, or you can create an entire new group of instances (blue/green deployment). A deployment configuration also specifies the fault tolerance of deployments, so you can roll back changes if a specified number or percentage of instances or functions in your deployment group fail to complete their deployments and signal success back to AWS CodeDeploy.
When you deploy to Amazon EC2 on-premises instances, you can configure either in-place or blue/green deployments.
In-Place deployments These deployments recycle currently running instances and deploy revisions on existing instances.
Blue/Green deployments These deployments replace currently running instances with sets of newly created instances.
In both scenarios, you can specify wait times between groups of deployed instances (batches). Additionally, if you register the deployment group with an elastic load balancer, newly deployed instances also register with the load balancer and are subject to its health checks.
The deployment configuration specifies success criteria for deployments, such as the minimum number of healthy instances that must pass health checks during the deployment process. This is done to maintain required availability during application updates. AWS CodeDeploy provides three built-in deployment configurations.
For in-place deployments, AWS CodeDeploy will attempt to deploy to all instances in the deployment group at the same time. The success criteria for this deployment configuration requires that at least once instance succeed for the deployment to be successful. If all instances fail the deployment, then the deployment itself fails.
For blue/green deployments, AWS CodeDeploy will attempt to deploy to the entire set of replacement instances at the same time and follows the same success criteria as in-place deployments. Once deployment to the replacement instances succeeds (at least one instance deploys successfully), traffic routes to all replacement instances at the same time. The deployment fails only if all traffic routing to replacement instances fails.
For in-place deployments, up to half of the instances in the deployment group deploy at the same time (rounded down). Success criteria for this deployment configuration requires that at least half of the instances (rounded up) deploy successfully.
Blue/green deployments use the same rules for the replacement environment, with the exception that the deployment will fail if less than half of the instances in the replacement environment successfully handle rerouted traffic.
For in-place and blue/green deployments, this is the most stringent of the built-in deployment configurations, as it requires all instances to deploy the new application revision successfully, with the exception of the final instance in the deployment. For deployment groups with only one instance, the instance must complete successfully for the deployment to complete.
For blue/green deployments, the same rule applies for traffic routing. If all but the last instance registers successfully, the deployment is successful (with the exception of single-instance environments, where it must register without error).
For in-place deployments, AWS CodeDeploy will attempt to deploy to all instances in the deployment group at the same time. The success criteria for this deployment configuration requires that at least one instance succeed for the deployment to be successful. If all instances fail the deployment, then the deployment itself fails.
For blue/green deployments, AWS CodeDeploy will attempt to deploy to the entire set of replacement instances at the same time and follows the same success criteria as in-place deployments. Once deployment to the replacement instances succeeds (at least one instance deploys successfully), traffic routes to all replacement instances at the same time. The deployment fails only if all traffic routing to replacement instances fails.
For in-place deployments, up to half of the instances in the deployment group deploy at the same time (rounded down). Success criteria for this deployment configuration requires that at least half of the instances (rounded up) deploy successfully.
Blue/green deployments use the same rules for the replacement environment, with the exception that the deployment will fail if less than half of the instances in the replacement environment successfully handle rerouted traffic.
This is the most stringent of the built-in deployment configurations, as it requires that all instances successfully deploy the new application revision (both in-place and blue/green deployments), with the exception of the final instance in the deployment. For deployment groups with only one instance, the instance must complete successfully for the deployment to complete.
For blue/green deployments, the same rule applies for traffic routing. If all but the last instance registers successfully, the deployment is successful (with the exception of single-instance environments, where it must register without error).
AWS CodeDeploy handles updates to AWS Lambda functions differently than to Amazon EC2 or on-premises instances. When you deploy to AWS Lambda, the deployment configuration specifies the traffic switching policy to follow, which stipulates how quickly to route requests from the original function versions to the new versions. You can configure AWS CodeDeploy to deploy instances only in a blue/green fashion. AWS Lambda does not support in-place deployments. This is because AWS CodeDeploy will deploy updates to new functions.
AWS CodeDeploy supports three methods for handling traffic switching in an AWS Lambda environment.
Traffic shifts in two percentage-based increments. The first increment routes to the new function version, and it is monitored for the number of minutes you define. After this time period, the remainder of traffic routes to the new version if the initial increment of request executes.
AWS CodeDeploy provides a number of built-in canary-based deployment configurations, such as CodeDeployDefault.LambdaCanary10Percent15Minutes. If you use this deployment configuration, 10 percent of traffic shifts in the first increment and is monitored for 15 minutes. After this time period, the 90 percent of traffic that remains shifts to the new function version. You can create additional configurations as needed.
Traffic can be shifted in a number of percentage-based increments, with a set number of minutes between each increment. During the waiting period between each increment, the requests routed to the new function versions must complete successfully for the deployment to continue.
AWS CodeDeploy provides a number of built-in linear deployment configurations, such as CodeDeployDefault.LambdaLinear10PercentEvery1Minute. With this configuration, 10 percent of traffic is routed to the new function version every minute, until all traffic is routed after 10 minutes.
All traffic is shifted at once to the new function versions.
An application is a logical grouping of a deployment group, revision, and deployment configuration. This serves as a reference to the entire set of objects needed to complete a deployment to your instances or functions.
The AppSpec configuration file is a JSON or YAML file that manages deployments on instances or functions in your environment. The actual format and purpose of an AppSpec file differs between Amazon EC2/on-premises and AWS Lambda deployments.
For Amazon EC2 on-premises deployments, the AppSpec file must be YAML formatted and follow the YAML specifications for spacing and indentation. You place the AppSpec file (appspec.yml) in the root of the revision’s source code directory structure (it cannot be in a subfolder).
When you deploy to Amazon EC2 on-premises instances, the AppSpec file defines the following:
The AppSpec file specifies scripts to execute at each stage of the deployment lifecycle. These scripts must exist in the revision for AWS CodeDeploy to call them successfully; however, they can call any other scripts, commands, or tools present on the instance. The AWS CodeDeploy agent uses the hooks section of the AppSpec file to reference which scripts must execute at specific times in the deployment lifecycle. When the deployment is at the specified stage (such as ApplicationStop), the AWS CodeDeploy agent will execute any scripts in that stage in the hooks section of the AppSpec file. All scripts must return an exit code of 0 to be successful.
For any files to place on the instance, the AWS CodeDeploy agent refers to the files section of the AppSpec file, where a mapping of files and directories in the revision dictates where on the instance these files reside and with what permissions. Here’s an example of an appspec.yml file:
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/WordPress
hooks:
BeforeInstall:
- location: scripts/install_dependencies.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/change_permissions.sh
timeout: 300
runas: root
ApplicationStart:
- location: scripts/start_server.sh
- location: scripts/create_test_db.sh
timeout: 300
runas: root
ApplicationStop:
- location: scripts/stop_server.sh
timeout: 300
runas: root
In the previous example, the following events occur during deployment:
The high-level structure of an Amazon EC2 on-premises AppSpec file is as follows:
version: 0.0
os: operating-system-name
files:
source-destination-files-mappings
permissions:
permissions-specifications
hooks:
deployment-lifecycle-event-mappings
version Currently the only supported version number is 0.0.
os The os section defines the target operating system of the deployment group. Either windows or linux (Amazon Linux, Ubuntu, or Red Hat Enterprise Linux) is supported.
files The files section defines the mapping of revision files and their location to deploy on-instance during the install lifecycle event. This section is not required if no files are being copied from the revision to your instance. The files section supports a list of source/ destination pairs.
files:
- source: source-file-location
destination: destination-file-location
The source key refers to a file or a directory’s local path within the revision (use / for all files in the revision). If source refers to a file, the file copies to destination, specified as the fully qualified path on the instance. If source refers to a directory, the directory contents copy to the instance.
permissions For any deployed files or directories, the permissions section specifies the permissions to apply to files and directories on the target instance. You can also apply permissions to files on the instance by AWS CodeDeploy using the files directive of the AppSpec configuration.
permissions:
- object: object-specification
pattern: pattern-specification
except: exception-specification
owner: owner-account-name
group: group-name
mode: mode-specification
acls:
- acls-specification
context:
user: user-specification
type: type-specification
range: range-specification
type:
- object-type
Each object specification includes a set of files or directories to which the permissions will apply. You can select files based on a pattern expression and ignore them with a comma-delimited list in the except property. The owner, group, and mode properties correspond to their Linux equivalents. You can apply access control lists with the acls property, providing a list of user/group permissions assignments (such as u:ec2-user:rw). The context property is reserved for SELinux-enabled instances. This property corresponds to a set of context labels to apply to objects. Lastly, you use the type property to specify to which types of objects (file or directory) the specified permissions will apply.
Windows instances do not support permissions.
hooks The hooks section specifies the scripts to run at each lifecycle event and under what user context to execute them.
One or more scripts can execute for each lifecycle hook.
ApplicationStop Before the application revision downloads to the instance, this lifecycle event can stop any running services on the instance that would be affected by the update. It is important to note that, since the revision has not yet been downloaded, the scripts execute from the previous revision. Because of this, the ApplicationStop hook does not run on the first deployment to an instance.
DownloadBundle The AWS CodeDeploy agent uses this lifecycle event to copy application revision files to a temporary location on the instance.
Linux /opt/codedeploy-agent/deployment-root/[deployment-group-id]/[deployment-id]/deployment-archive
WindowsC:ProgramDataAmazonCodeDeploy[deployment-group-id][deployment-id]deployment-archive
This event cannot run custom scripts, as it is reserved for the AWS CodeDeploy agent.
BeforeInstall Use this event for any pre-installation tasks, such as to clear log files or to create backups.
Install This event is reserved for the AWS CodeDeploy agent.
AfterInstall Use this event for any post-installation tasks, such as to modify the application configuration.
ApplicationStart Use this event to start any services that were stopped during the ApplicationStop event.
ValidateService Use this event to verify deployment completed successfully.
If your deployment group is registered with a load balancer, additional lifecycle events become available. These can be used to control certain behaviors as the instance is registered or deregistered from the load balancer.
BeforeBlockTraffic Use this event to run tasks before the instance is deregistered from the load balancer.
BlockTraffic This event is reserved for the AWS CodeDeploy agent.
AfterBlockTraffic Use this event to run tasks after the instance is deregistered from the load balancer.
BeforeAllowTraffic Similar in concept to BeforeBlockTraffic, this event occurs before instances register with the load balancer.
AllowTraffic This event is reserved for the AWS CodeDeploy agent.
AfterAllowTraffic Similar in concept to AfterBlockTraffic, this event occurs after instances register with the load balancer.
hooks:
deployment-lifecycle-event-name:
- location: script-location
timeout: timeout-in-seconds
runas: user-name
In the hooks section, the lifecycle name must match one of the previous event names, which are not reserved for the AWS CodeDeploy agent. The location property refers to the relative path in the revision archive where the script is located. You can configure an optional timeout to limit how long a script can run before it is considered failed. (Note that this does not stop the script’s execution.) The maximum script duration is 1 hour (3,600 seconds) for each lifecycle event. Lastly, the runas property can specify the user to execute the script. This user must exist on the instance and cannot require a password.
Figure 7.20 displays lifecycle hooks and their availability for in-place deployments with and without a load balancer.
Figure 7.21 displays lifecycle hooks and their availability for blue/green deployments.
When you deploy to AWS Lambda functions, the AppSpec file can be in JSON or YAML format, and it specifies the function versions to deploy as well as other functions to execute for validation testing.
AWS Lambda deployments do not use the AWS CodeDeploy agent.
The high-level structure of an AWS Lambda deployment AppSpec file is as follows:
version: 0.0
resources:
lambda-function-specifications
hooks:
deployment-lifecycle-event-mappings
version Currently the only supported version number is 0.0.
resources The resources section defines the AWS Lambda functions to deploy.
resources:
- name-of-function-to-deploy:
type: "AWS::Lambda::Function"
properties:
name: name-of-lambda-function-to-deploy
alias: alias-of-lambda-function-to-deploy
currentversion: lambda-function-version-traffic-currently-points-to
targetversion: lambda-function-version-to-shift-traffic-to
Name each function in the resources list both as the list item name and in the name property. The alias property specifies the function alias, which maps from the version specified in currentversion to the version specified in targetversion after the update deploys.
hooks The hooks section specifies the additional AWS Lambda functions to run at specific stages of the deployment lifecycle to validate success. The following lifecycle events support hooks in AWS Lambda deployments:
BeforeAllowTraffic For running any tasks prior to traffic shifting taking place
AfterAllowTraffic For any tasks after all traffic shifting has completed
hooks:
- BeforeAllowTraffic: BeforeAllowTrafficHookFunctionName
- AfterAllowTraffic: AfterAllowTrafficHookFunctionName
Figure 7.22 displays the lifecycle hook availability for AWS Lambda deployments.
AWS CodeDeploy reserves the Start, AllowTraffic, and End lifecycle events.
For any functions in the hooks section, the function is responsible for notifying AWS CodeDeploy of success or failure with the PutLifecycleEventHookExecutionStatus call API from within your validation function. Here’s an example for Node.js:
CodeDeploy the prepared validation test results.
codedeploy.putLifecycleEventHookExecutionStatus(params, function(err, data) {
if (err) {
// Validation failed.
callback('Validation test failed');
} else {
// Validation succeeded.
callback(null, 'Validation test succeeded');
}
});
The AWS CodeDeploy agent is responsible for driving and validating deployments on Amazon EC2 on-premises instances. The agent currently supports Amazon Linux (Amazon EC2 only), Ubuntu Server, Microsoft Windows Server, and Red Hat Enterprise Linux, and it is available as an open source repository on GitHub (https://github.com/aws/ aws-codedeploy-agent).
When the agent installs, a codedeployagent.yml configuration file copies to the instances. You can use this file to adjust the behavior of the AWS CodeDeploy agent on instances throughout various deployments. This configuration file is stored in /etc/codedeploy-agent/conf on Linux instances and C:ProgramDataAmazonAWS CodeDeploy on Windows Server instances.
The most common settings are as follows:
max_revisions Use this to configure how many application revisions to archive on an instance. If you are experiencing storage limitations on your instances, turn this value down and release some storage space consumed by the agent.
root_dir Use this to change the default storage location for revisions, scripts, and archives.
verbose Set this to true to enable verbose logging output for debugging purposes.
proxy_url For environments that use an HTTP proxy, this specifies the URL and credentials to authenticate to the proxy and connect to the AWS CodeDeploy service.
AWS CodeDeploy enforces the service limits, as shown in Table 7.5. An asterisk (*) indicates limits that you can increase with a request to AWS Support.
Table 7.5 AWS CodeDeploy Service Limits
Limit | Value |
Applications per account per region | 100 |
Allowed revision file types | .zip, .tar, .tar, and.gz |
Concurrent deployments per deployment group | 1 |
Concurrent deployments per account | 100 |
Maximum deployment lifecycle event duration | 3600 seconds |
Custom deployment configurations per account | 25 |
Deployment groups per application* | 100 |
Tags per deployment group | 10 |
Auto Scaling groups per deployment group | 10 |
Instances per deployment | 500 |
AWS CodeDeploy can integrate automatically with AWS CodePipeline as a deployment action to deploy changes to Amazon EC2 on-premises instances or AWS Lambda functions. You can configure applications, deployment groups, and deployments directly in the AWS CodePipeline console when you create or edit a pipeline, or you can do this ahead of time with the AWS CodeDeploy console or the AWS CLI or AWS SDK.
After you define the deployment provider, application name, and deployment group in the AWS CodePipeline console, the pipeline will automatically configure to pass a pipeline artifact to AWS CodeDeploy for deployment to the specified application/group, as shown in Figure 7.23.
AWS CodeDeploy monitors the progress of any revisions to deploy and report success or failure to AWS CodePipeline.
In this chapter, you learned about these deployment services:
AWS CodePipeline drives application deployments starting with a source repository (AWS CodeCommit), performing builds with AWS CodeBuild, and finally deploying to Amazon EC2 instance or AWS Lambda functions using AWS CodeDeploy. You can use AWS CloudFormation to provision and manage infrastructure in your environment. By integrating this with AWS CodePipeline, you can automate the entire process of creating development, testing, and production environments into a fully hands-off process. In a fully realized enterprise as code, a single commit to a source repository can kick off processes such as those shown in Figure 7.1.
Know the difference between continuous integration, continuous delivery, and continuous deployment. Continuous integration is the practice where all code changes merge into a repository. Continuous delivery is the practice where all code changes are prepared for release. Continuous deployment is the practice where all code is prepared for release and automatically released to production environments.
Know the basics of AWS CodePipeline. AWS CodePipeline contains the steps in the continuous integration and deployment pipeline (CI/CD) workflow, driving automation between different tasks after assets have been committed to a repository or saved in a bucket. AWS CodePipeline uses stages, which correspond to different steps in a workflow. Within each stage, different actions can perform tasks in series or in parallel. Transitions between stages can be automatic or require manual approval by an authorized user.
Understand how revisions can move through a pipeline. Revisions move automatically between stages in a pipeline, provided that all actions in the preceding stage complete. If a manual approval is required, the revision will not proceed until an authorized user allows it to do so. When two changes are pushed to a source repository in a short time span, the latest of the two changes will proceed through the pipeline.
Know the different pipeline actions that are available. A pipeline stage can include one or more actions: build, test, deploy, and invoke. You can also create custom actions.
Know how to deploy a cross-account pipeline. The account containing the pipeline must create a KMS key that can be used by both AWS CodePipeline and the other account. The pipeline account must also specify a bucket policy on the assets bucket that the pipeline uses, which allows the second account to access assets. The AWS CodePipeline service IAM role must include a policy that allows it to assume a role in the second account. The second account must have a role that can be assumed by the pipeline account, which allows the pipeline account to deploy resources and access the assets bucket.
Know the basic concepts of AWS CodeCommit. AWS CodeCommit is a Git-based repository service. It is fully compatible with existing Git tooling. AWS CodeCommit provides various benefits, such as encryption in transit and at rest; automatic scaling to handle increases in activity; access control using IAM users, roles, and policies; and HTTPS/SSH connectivity. AWS CodeCommit supports normal Git workflows, such as pull requests.
Know how to use the credential helper to connect to repositories. It is possible to connect to AWS CodeCommit repositories using IAM credentials. The AWS CodeCommit credential helper translates an IAM access key and secret access key into valid Git credentials. This requires the AWS CLI and a Git configuration file that specifies the credential helper.
Understand the different strategies for migrating to AWS CodeCommit. You can migrate an existing Git repository by cloning to your local workstation and adding a new remote, pointing to the AWS CodeCommit repository you create. You can push the repository contents to the new remote. You can migrate unversioned content in a similar manner; however, you must create a new local Git repository (instead of cloning an existing one). Large repositories can be migrated incrementally because large pushes may fail because of network issues.
Know the basics of AWS CodeBuild. AWS CodeBuild allows you to perform long-running build tasks repeatedly and reliably without having to manage the underlying infrastructure. You are responsible only for specifying the build environment settings and the actual tasks to perform.
Know the basics of AWS CodeDeploy. AWS CodeDeploy standardizes and automates deployments to Amazon EC2 instances, on-premises servers, and AWS Lambda functions. Deployments can include application/static files, configuration tasks, or arbitrary scripts to execute. For Amazon EC2 on-premises deployments, a lightweight agent is required.
Understand how AWS CodeDeploy works with Amazon EC2 Auto Scaling groups. When you deploy to Amazon EC2 Auto Scaling groups, AWS CodeDeploy will automatically run the last successful deployment on any new instances that you add to the group. If the deployment fails on the instance, it will be terminated and replaced (to maintain the desired count of healthy instances). If two deployment groups for separate AWS CodeDeploy applications specify the same Auto Scaling group, issues can occur. If both applications deploy at roughly the same time and one fails, the instance will be terminated before success/failure can be reported for the second application deployment. This will result in AWS CodeDeploy waiting until the timeout period expires before taking any further action.
This exercise demonstrates how to use AWS CodeCommit to submit and merge pull requests to a repository.
The pull request has been merged with the master branch, which can be confirmed by viewing the source code of the markdown file in the master branch.
This exercise demonstrates how to use AWS CodeDeploy to perform an in-place deployment to Amazon EC2 instances in your account.
For the compute platform type, select EC2 On-premises.
Deployment type In-place
Environment configuration Amazon EC2 instances
Tag group Create a tag group that is easy to identify, such as a “Name” for the key, and “CodeDeployInstance” as the value.
Load balancer Clear the Enable load balancing check box.
Make sure to specify the tag value chosen in the previous step.
Sample application bundles for each operating system can be found using the following links:
Windows Server https://docs.aws.amazon.com/codedeploy/latest/userguide/tutorials-windows.html
Amazon Linux or Red Hat Enterprise Linux (RHEL) https://docs.aws.amazon.com/codedeploy/latest/userguide/tutorials-wordpress.html
This exercise demonstrates how to use AWS CodeBuild to perform builds and the compilation of artifacts prior to deployment to Amazon EC2 instances.
Project name Provide a name of your choice.
Source Use Amazon S3.
Bucket Provide the name of the bucket you created.
S3 object key Provide the name of one of the objects you uploaded.
Environment image Select the Managed Image type.
Operating system Use Ubuntu.
Runtime Use Python.
Runtime version Select a version of your choice.
Service role Select New Service Role.
Role name Provide a name for your service role.
Build specifications Select Insert Build Commands.
Build commands Select Switch To Editor and enter the following. Replace the Amazon S3 object paths with paths to the objects you uploaded to your bucket.
version: 0.2
phases:
build:
commands:
- aws s3 cp s3://yourbucket/file1 /tmp/file1
- aws s3 cp s3://yourbucket/file2 /tmp/file2
artifacts:
files:
- /tmp/file1
- /tmp/file2
Artifact Type Use Amazon S3.
Bucket name Select your Amazon S3 bucket.
Artifacts packaging Select Zip.
You have two AWS CodeDeploy applications that deploy to the same Amazon EC2 Auto Scaling group. The first deploys an e-commerce app, while the second deploys custom administration software. You are attempting to deploy an update to one application but cannot do so because another deployment is already in progress. You do not see any instances undergoing deployment at this time. What could be the cause of this?
If you specify a hook script in the ApplicationStop lifecycle event of an AWS CodeDeploy appspec.yml, will it run on the first deployment to your instance(s)?
If a single pipeline contains multiple sources, such as an AWS CodeCommit repository and an Amazon S3 archive, under what circumstances will the pipeline be triggered?
If you want to implement a deployment pipeline that deploys both source files and large binary objects to instance(s), how would you best achieve this while taking cost into consideration?
Your team is building a deployment pipeline to a sensitive application in your environment using AWS CodeDeploy. The application consists of an Amazon EC2 Auto Scaling group of instances behind an Elastic Load Balancing load balancer. The nature of the application requires 100 percent availability for both successful and failed deployments. The development team want to deploy changes multiple times per day. How would this be achieved at the lowest cost and with the fastest deployments?
What would cause an access denied error when attempting to download an archive file from Amazon S3 during a pipeline execution?
How do you output build artifacts from AWS CodeBuild to AWS CodePipeline?
What would be the most secure means of providing secrets to an AWS CodeBuild environment?
In which of the pipeline actions can you execute AWS Lambda functions?
In what ways can pipeline actions be ordered in a stage? (Select TWO.)
If you would like to delete an AWS CloudFormation stack before you deploy a new one in your pipeline, what would be the correct set of actions?
How can you connect to an AWS CodeCommit repository without Git credentials?
Of the following, which event cannot be used to generate notifications to an Amazon Simple Notification Service (SNS) topic from AWS CodeCommit without using a trigger?
Which pipeline actions support AWS CodeBuild projects? (Select TWO.)
Can data passed to build projects using environment variables be encrypted or protected?
What is the only deployment type supported by on-premises instances?
If your AWS CodeDeploy configuration includes creation of a file, nginx.conf, but the file already exists on the server (prior to the use of AWS CodeDeploy), what is the default behavior that will occur during deployment?
How does AWS Lambda support in-place deployments?
What is the minimum number of stages required by a pipeline in AWS CodePipeline?
If an instance is running low on storage, and you find that there are a large number of deployment revisions stored by AWS CodeDeploy, what can be done to free up this space permanently?
Review Questions