Follow Mandatory Cost Tagging

An effective tagging strategy gives you improved visibility and monitoring, helps you create accurate chargeback and showback models, and extract more granular and precise insights into usage and spending by applications and teams. The following tag categories help you achieve these goals:

Environment Distinguishes among development, test, and production infrastructure. Specifying an environment tag reduces analysis time, post-processing, and the need to maintain a separate mapping file of production versus nonproduction accounts.

Application ID Identifies resources that are related to a specific application for easy tracking of spending changes and that turn off at the end of projects.

Automation Opt-In/Opt-Out Indicates whether a resource should be included in an automated activity such as starting, stopping, or resizing instances.

Cost Center/Business Unit Identifies the cost center or business unit associated with a resource, typically for cost allocation and tracking.

Owner Used to identify who is responsible for the resource. This is typically the technical owner. If needed, you can add a separate business owner tag. You can specify the owner as an email address. Using email addresses supports automated notifications to both the technical and business owners as required (for example, if the resource is a candidate for elasticity or right sizing).

Tag on Creation

You can make tagging a part of your build process and automate it with AWS management tools, such as AWS Elastic Beanstalk and AWS OpsWorks.

The following AWS CLI sample adds two tags, CostCenter and environment, for an Amazon Machine Image (AMI) and an instance:

aws ec2 create-tags --resources ami-1a2b3c4d i-1234567890abcdef0 --tags Key=CostCenter,Value=123   Key=environment,Value=Production

You can execute management tasks at scale by listing resources with specific tags and then executing the appropriate actions. For example, you can list all the resources with the tag and value of environment:test; then, for each of the resources, delete or terminate the resource. This is useful for automating shutdown or removal of a test environment at the end of the working day. Running reports on tagged and, more importantly, untagged resources enables greater compliance with internal cost management policies.

Enforce Tag Use

Using AWS Identity and Access Management (IAM) policies, you can enforce tag use to gain precise control over access to resources, ownership, and accurate cost allocation.

The following example policy allows a user to create an Amazon Elastic Block Store (Amazon EBS) volume only if the user applies the tags (Costcenter and environment) that are defined in the policy using the qualifier ForAllValues. If the user applies any tag that is not included in the policy, the action is denied. To enforce case sensitivity, use the condition aws:TagKeys as follows:

Effect: Allow
Action: 'ec2:CreateVolume'
Resource: 'arn:aws:ec2:us-east-1:123456789012:volume/*'
Condition:
    StringEquals:
        'aws:RequestTag/costcenter': '115'
        'aws:RequestTag/environment': prod
    'ForAllValues:StringEquals':
        'aws:TagKeys':
            - Costcenter
            - environment

Tagging Tools

The following tools help you manage your tags:

• AWS Tag Editor—Finds resources with search criteria (including missing and misspelled tags) and enables you to edit tags from the AWS Management Console
• AWS Config—Identifies resources that do not comply with tagging policies
• Capital One’s Cloud Custodian (open source)—Ensures tagging compliance and remediation

Delete Unnecessary EBS Volumes

Stopping an Amazon Elastic Compute Cloud (Amazon EC2) instance leaves any attached Amazon Elastic Block Store (Amazon EBS) volumes operational. You continue to incur charges for these volumes until you delete them.

Stop Unused Instances

Stop instances used in development and production during hours when these instances are not in use and then start them again when their capacity is needed. Assuming a 50-hour workweek, you can save 70 percent of costs by automatically stopping dev/test/production instances during nonbusiness hours.

Delete Idle Resources

Consider the following best practices to reduce costs associated with AWS idle resources, such as unattached Amazon EBS volumes and unused Elastic IP addresses:

• The easiest way to reduce operational costs is to turn off instances that are no longer being used. If you find instances that have been idle for more than two weeks, it’s safe to stop or even terminate them.
• Terminating an instance, however, automatically deletes attached EBS volumes and requires effort to re-provision if the instance is needed again. If you decide to delete an EBS volume, consider storing a snapshot of the volume so that it can be restored later if needed.
• Spin up instances to test new ideas. If the ideas work, keep the instance for further refinement. If not, spin it down.
• An Elastic IP address does not incur charges as long as it is associated with an Amazon EC2 instance. If an Elastic IP address is not used, you can avoid charges by releasing the IP address. After you release an IP address, you cannot provision that same Elastic IP address again.

Update Outdated Resources

As AWS releases new services and features, it is a best practice to review your existing architectural decisions to ensure that they remain cost effective and stay evergreen. As your requirements change, be aggressive in decommissioning resources, components, and workloads that you no longer require.

Delete Unused Keys

Each customer master key (CMK) that you create in AWS Key Management Service (AWS KMS), regardless of whether you use it with KMS-generated key material or key material imported by you, incurs a cost you until you delete it. Before deleting a CMK, you might want to know how many ciphertexts were encrypted under that key. Knowing how a CMK was used in the past might help you decide whether you will need it in the future by using AWS CloudTrail usage logs. After you are sure that you want to delete a CMK in AWS KMS, schedule the key deletion.

Delete Old Snapshots

If your architecture suggests a backup policy that takes EBS volume snapshots daily or weekly, then you will quickly accumulate snapshots. To reduce storage costs, check for “stale” snapshots—ones that are more than 30 days old—and delete them. Deleting a snapshot has no effect on the volume. You can use the AWS Management Console or AWS Command Line Interface (AWS CLI) for this purpose.

Amazon Elastic Cloud Compute

Amazon Elastic Cloud Compute (Amazon EC2) provides a wide selection of instances, which gives you flexibility to right size CPU and memory needs for your compute resources to match capacity needs at the lowest cost. Following are the different options for CPU, memory, and network resources:

General purpose (includes A1, T2, M3, and M4 instance types) A1 instances deliver significant cost savings and are ideally suited for scale-out workloads, such as web servers, containerized microservices, caching fleets, and distributed data stores. T2 instances are a low-cost option that provides a small amount of CPU resources that can be increased in short bursts when additional cycles are available. They are well suited for lower throughput applications, such as administrative applications or low-traffic websites. M3 and M4 instances provide a balance of CPU, memory, and network resources, and they are ideal for running small and midsize databases, more memory-intensive data processing tasks, caching fleets, and backend servers.

Compute optimized (includes the C3 and C4 instance types) This family has a higher ratio of virtual CPUs to memory than the other families and the lowest cost per virtual CPU of all of the Amazon EC2 instance types. Consider compute-optimized instances first if you are running CPU-bound, scale-out applications, such as front-end fleets for high-traffic websites, on-demand batch processing, distributed analytics, web servers, video encoding, and high-performance science and engineering applications.

Memory optimized (includes the X1, R3, and R4 instance types) Designed for memory-intensive applications, these instances have the lowest cost per GiB of RAM of all Amazon EC2 instance types. Use these instances if your application is memory-bound.

Storage optimized (includes the I3 and D2 instance types) Optimized to deliver tens of thousands of low-latency, random input/output operations per second (IOPS) to applications. Storage-optimized instances are best for large deployments of NoSQL databases. I3 instances are designed for I/O-intensive workloads and equipped with super-efficient NVMe SSD storage. These instances can deliver up to 3.3 million IOPS in 4-KB blocks and up to 16 GB per second of sequential disk throughput. D2 or dense storage instances are designed for workloads that require high sequential read and write access to large datasets such as Hadoop, distributed computing, massively parallel processing data warehousing, and log-processing applications.

Accelerated computing (includes the P2, G3, and F1 instance types) Provides access to hardware-based compute accelerators, such as graphics processing units (GPUs) or field programmable gate arrays (FPGAs). Accelerated-computing instances enable more parallelism for higher throughput on compute-intensive workloads.

Amazon Relational Database Service

Similar to Amazon EC2 instances, Amazon Relational Database Service (Amazon RDS) provides options to choose from database instances that are optimized for memory, performance, and I/O.

Standard performance (includes the M3 and M4 instance types) Designed for general-purpose database workloads that do not run many in-memory functions. This family has the most options for provisioning increased IOPS.

Burstable performance (includes T2 instance types) For workloads that require burstable performance capacity.

Memory optimized (includes the R3 and R4 instance types) Optimized for in-memory functions and big data analysis.

Convertible Reserved Instances

Convertible Reserved Instances are provided for a one-year or three-year term, and they enable conversion to different families, new pricing, different instance sizes, different platforms, or tenancy during the period. Use Convertible Reserved Instances when you are uncertain about instance needs in the future, but you are still committed to using Amazon EC2 instances for a three-year term in exchange for a significant discount.

Suppose that you own an Amazon EC2 Reserved Instance for a c4.8xlarge for three years. This Reserved Instance applies to any usage of a Linux/Unix c4 instance with shared tenancy in the same region as the Reserved Instance, such as 1 c4.8xlarge instance, 2 c4.4xlarge instances, or 16 c4.large instances, during this term. This adds flexibility to match the new needs of your workloads:

• There are no limits to how many times you perform an exchange, as long as the target Convertible Reserved Instance is of an equal or higher value than the Convertible Reserved Instances that you are exchanging.
• Exchanging Convertible Reserved Instances is free of charge, but you might need to pay a true-up cost if the value is lower than the value of the Reserved Instances for which you’re exchanging. For example, you can convert C3 Reserved Instances to C4 Reserved Instances to take advantage of a newer instance type, or you can convert C4 Reserved Instances to M4 Reserved Instances if your application requires more memory. You can also use Convertible Reserved Instances to take advantage of Amazon EC2 price reductions over time.

Reserved Instance Marketplace

Use the Reserved Instance Marketplace to sell your unused Reserved Instances and buy Reserved Instances from other AWS customers. As your needs change throughout the course of your term, the AWS Marketplace provides an option to buy Reserved Instances for shorter terms and with a wider selection of prices.

Using Termination Notices

If you need to save state, upload final log files, or remove Spot Instances from an Elastic Load Balancing load balancer before interruption, you can use termination notices, which are issued 2 minutes before interruption.

If your instance is marked for termination, the termination notice is stored in the instance’s metadata 2 minutes before its termination time. The notice is accessible at http://169.254.169.254/latest/meta-data/spot/termination-time. The notice includes the time when the shutdown signal will be sent to the instance’s operating system.

Relevant applications on Spot Instances should poll for the termination notice at 5-second intervals, which gives the application almost the entire 2 minutes to complete any needed processing before the instance is terminated and taken back by AWS.

Using Persistent Requests

You can set your request to remain open so that a new instance is launched in its place when the instance is interrupted. You can also have your Amazon EBS–backed instance stopped upon interruption and restarted when Spot has capacity at your preferred price.

Using Block Durations

You can also launch Spot Instances with a fixed duration (Spot blocks, 1–6 hours), which are not interrupted as the result of changes in the Spot price. Spot blocks can provide savings of up to 50 percent.

You submit a Spot Instance request and use the new BlockDuration parameter to specify the number of hours that you want your instances to run, along with the maximum price that you are willing to pay.

You can submit a request of this type by running the following command:

$ aws ec2 request-spot-instances 
      block-duration-minutes 360 
      instance-count 2 
      spot-price "0.25"
:

Alternatively, you can call the RequestSpotInstances function.

Minimizing the Impact of Interruptions

Because the Spot service can terminate Spot Instances without warning, it is important to build your applications in a way that allows you to make progress even if your application is interrupted. There are many ways to accomplish this, including the following:

Adding checkpoints Add checkpoints that save your work periodically, for example, to an Amazon EBS volume. Another approach is to launch your instances from Amazon EBS–backed AMI.

Splitting up the work By using Amazon Simple Queue Service (Amazon SQS), you can queue up work increments and track work that has already been done.

Dynamic Scaling

The dynamic scaling capabilities of Amazon EC2 Auto Scaling refers to the functionality that automatically increases or decreases capacity based on load or other metrics. For example, if your CPU spikes above 80 percent (and you have an alarm set up), Amazon EC2 Auto Scaling can add a new instance dynamically, reducing the need to provision Amazon EC2 capacity manually in advance. Alternatively, you could set a target value by using the new Request Count Per Target metric from Application Load Balancer, a load balancing option for the Elastic Load Balancing service. Amazon EC2 Auto Scaling will then automatically adjust the number of Amazon EC2 instances as needed to maintain your target.

Scheduled Scaling

Scaling based on a schedule allows you to scale your application ahead of known load changes, such as the start of business hours, thus ensuring that resources are available when users arrive, or in typical development or test environments that run only during defined business hours or periods of time.

You can use APIs to scale the size of resources within an environment (vertical scaling). For example, you could scale up a production system by changing the instance size or class. This can be achieved by stopping and starting the instance and selecting the different instance size or class. You can also apply this technique to other resources, such as EBS volumes, which can be modified to increase size, adjust performance (IOPS), or change the volume type while in use.

Fleet Management

Fleet management refers to the functionality that automatically replaces unhealthy instances in your application, maintains your fleet at the desired capacity, and balances instances across Availability Zones. Amazon EC2 Auto Scaling fleet management ensures that your application is able to receive traffic and that the instances themselves are working properly. When AWS Auto Scaling detects a failed health check, it can replace the instance automatically.

Instances Purchasing Options

With Amazon EC2 Auto Scaling, you can provision and automatically scale instances across purchase options, Availability Zones, and instance families in a single application to optimize scale, performance, and cost. You can include Spot Instances with On-Demand and Reserved Instances in a single AWS Auto Scaling group to save up to 90 percent on compute. You have the option to define the desired split between On-Demand and Spot capacity, select which instance types work for your application, and specify preferences for how Amazon EC2 Auto Scaling should distribute the AWS Auto Scaling group capacity within each purchasing model.

Golden Images

A golden image is a snapshot of a particular state of a resource, such as an Amazon EC2 instance, Amazon EBS volumes, and an Amazon RDS DB instance. You can customize an Amazon EC2 instance and then save its configuration by creating an Amazon Machine Image (AMI). You can launch as many instances from the AMI as you need, and they will all include those customizations. A golden image results in faster start times and removes dependencies to configuration services or third-party repositories. This is important in auto-scaled environments in which you want to be able to launch additional resources in response to changes in demand quickly and reliably.

The predictive scaling feature uses machine learning algorithms to detect changes in daily and weekly patterns, automatically adjusting their forecasts. This removes the need for the manual adjustment of AWS Auto Scaling parameters as cyclicality changes over time, making AWS Auto Scaling simpler to configure, and provides more accurate capacity provisioning. Predictive scaling results in lower cost and more responsive applications.

Storage Management Tools/Features

The following sections detail some of the tools that help to determine when to transition data to another storage class.

Use Amazon S3 Select

Amazon S3 Select enables applications to retrieve only a subset of data from an object by using simple SQL expressions. By using Amazon S3 Select to retrieve only the data needed by your application, you can achieve drastic performance increases—in many cases, you can get as much as a 400 percent improvement.

Following is a Python sample code snippet that shows how to retrieve columns from an object containing data in CSV format. This code snippet retrieves the city and airport code, where country name is similar to “United States.” If you have column headers and you set the FileHeaderInfo to Use, you can identify columns by name in the SQL expression.

result = s3.select_object_content(      
        Bucket=example-bucket-us-west-2',    
        Key='sample-data/airportCodes.csv',    
        ExpressionType='SQL',      
        Expression="select s.city, s.code from s3object s where"   
                "s."Country (Name)" like '%United States%'",  
        InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}},  
        OutputSerialization = {'CSV': {}},    
:

Use Amazon Glacier Select

Amazon Glacier Select unlocks an opportunity to query your archived data easily. With Glacier Select, you can filter directly against an Amazon S3 Glacier object by using standard SQL statements.

It works like any other retrieval job, except for having an additional set of parameters (SelectParameters) that you can pass in an initiate job request.

The following is an example of a Python code snippet that shows how to pass an SQL expression under SelectParameters:

jobParameters = {          
    "Type": "select", "ArchiveId": "ID",      
    "Tier": "Expedited",        
    "SelectParameters": {        
        "InputSerialization": {"csv": {}},      
        "ExpressionType": "SQL",        
        "Expression": "SELECT * FROM archive WHERE _5='498960'",  
        "OutputSerialization": {        
            "csv": {}        
        }          
    }            

With both Amazon S3 Select and Glacier Select, you can lower your costs and uncover more insights from your data, regardless of which storage tier it is in.

Check Configuration

• To achieve the best performance consistently, launch instances as EBS optimized. For instances that are not EBS-optimized by default, you can enable EBS optimization when you launch the instances or enable EBS optimization after the instances are running.

• To enable this feature, you can use either the Amazon EC2 console or AWS Tools. For AWS CLI, use ebs-optimized with the command run-instances to enable EBS optimization when launching and with the command modify-instance-attribute to enable EBS optimization for a running instance.

• Choose an EBS-optimized instance that provides more dedicated EBS throughput than your application needs; otherwise, the Amazon EBS to Amazon EC2 connection becomes a performance bottleneck.
• New EBS volumes receive their maximum performance the moment that they are available and do not require initialization. However, storage blocks on volumes that were restored from snapshots must be initialized before you can access the block. This preliminary action takes time and can cause a significant increase in the latency of an I/O operation the first time each block is accessed.
• To achieve a higher level of performance for a file system than you can provision on a single volume, create a RAID 0 (zero) array. Consider using RAID 0 when I/O performance is more important than fault tolerance. For example, you could use it with a heavily used database where data replication is already set up separately.

Use Monitoring Tools

AWS offers tools that help you optimize block storage.

Delete Unattached Amazon EBS Volumes

To find unattached EBS volumes, look for volumes that are available, which indicates that they are not attached to an Amazon EC2 instance. You can also look at network throughput and IOPS to determine whether there has been any volume activity over the previous two weeks, or you can look up the last time the EBS volume was attached. If the volume is in a nonproduction environment, hasn’t been used in weeks, or hasn’t been attached in a month, there is a good chance that you can delete it.

Before deleting a volume, store an Amazon EBS snapshot (a backup copy of an EBS volume) so that the volume can be quickly restored later if needed.

Resize or Change the EBS Volume Type

Identify volumes that are underutilized and downsize them or change the volume type. Monitor the read/write access of EBS volumes to determine whether throughput is low. If you have a current-generation EBS volume attached to a current-generation Amazon EC2 instance type, you can use the elastic volumes feature to change the size or volume type or (for an SSD io1 volume) adjust IOPS performance without detaching the volume.

Follow these tips:

• For General Purpose SSD gp2 volumes, optimize for capacity so that you’re paying only for what you use.
• With Provisioned IOPS SSD io1 volumes, pay close attention to IOPS utilization rather than throughput, since you pay for IOPS directly. Provision 10–20 percent above maximum IOPS utilization.
• You can save by reducing Provisioned IOPS or by switching from a Provisioned IOPS SSD io1 volume type to a General Purpose SSD gp2 volume type.
• If the volume is 500 GB or larger, consider converting to a Cold HDD sc1 volume to save on your storage rate.

Delete Stale Amazon EBS Snapshots

If you have a backup policy that takes EBS volume snapshots daily or weekly, you will quickly accumulate snapshots. Check for stale snapshots that are more than 30 days old and delete them to reduce storage costs. Deleting a snapshot has no effect on the volume.

Amazon ElastiCache

Amazon ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory cache in the cloud. It supports two open-source, in-memory caching engines: Memcached and Redis.

• The Memcached caching engine is popular for database query results caching, session caching, webpage caching, API caching, and caching of objects such as images, files, and metadata. Memcached is also a great choice to store and manage session data for internet-scale applications in cases wherein persistence is not critical.
• Redis caching engine is a great choice for implementing a highly available in-memory cache to decrease data access latency, increase throughput, and ease the load off your relational or NoSQL database and application. Redis has disk persistence built in, and you can use it for long-lived data.

Lazy loading is a good caching strategy whereby you populate the cache only when an object is requested by the application, keeping the cache size manageable. Apply a lazy caching strategy anywhere in your application where you have data that is going to be read often but written infrequently. In a typical web or mobile app, for example, a user’s profile rarely changes but is accessed throughout the application.

Amazon DynamoDB Accelerator (DAX)

Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available, in-memory cache for Amazon DynamoDB. This feature delivers performance improvements from milliseconds to microseconds, for high throughput. DAX adds in-memory acceleration to your DynamoDB tables without requiring you to manage cache invalidation, data population, or clusters.

DAX is ideal for applications that require the fastest possible response time read operations but that are also cost-sensitive and require repeated reads against a large set of data. For example, consider an ecommerce system that has a one-day sale on a popular product that would sharply increase the demand or a long-running analysis of regional weather data that could temporarily consume all of the read capacity in a DynamoDB table. Naturally, these would negatively impact other applications that must access the same data.

Designing Partition Keys

The more distinct partition key values that your workload accesses, the more those requests are spread across the partitioned space. In general, you use your provisioned throughput more efficiently as the ratio of partition key values accessed to the total number of partition key values increases.

Table 16.1 provides a comparison of the provisioned throughput efficiency of some common partition key schemas.

If a single table has only a small number of partition key values, consider distributing your write operations across more distinct partition key values. Structure the primary key elements to avoid one “hot” (heavily requested) partition key value that slows the overall performance.

For example, consider a table with a composite primary key. The partition key represents the item’s creation date, rounded to the nearest day. The sort key is an item identifier. On a given day, say 2014-07-09, all of the new items are written to that single partition key value (and corresponding physical partition).

If the table fits entirely into a single partition (considering growth of your data over time) and if your application’s read and write throughput requirements don’t exceed the read and write capabilities of a single partition, your application won’t encounter any unexpected throttling because of partitioning.

Implementing Write Sharding

One better way to distribute writes across a partition key space in DynamoDB is to expand the space. One strategy for distributing loads more evenly across a partition key space is to add a random number or a calculated hash suffix to the end of the partition key values. Then you can randomize the writes across the larger space. A randomizing strategy can greatly improve write throughput.

For example, in the case of a partition key that represents today’s date in the Order table, suppose that each item has an accessible OrderId attribute and that you most often need to find items by OrderId in addition to date. Before your application writes the item to the table, it could calculate a hash suffix based on the OrderId (similar to OrderId, modulo 200, + 1) and append it to the partition key date. The calculation might generate a number between 1 and 200 that is fairly evenly distributed, similar to what the random strategy produces.

With this strategy, the writes are spread evenly across the partition key values and thus across the physical partitions. You can easily perform a GetItem operation for a particular item and date because you can calculate the partition key value for a specific OrderId value.

Upload Data Efficiently

Typically, when you load data from other data sources, Amazon DynamoDB partitions your table data on multiple servers. You get better performance if you upload data to all the allocated servers simultaneously.

For example, suppose that you want to upload user messages to a DynamoDB table that uses a composite primary key with UserID as the partition key and MessageID as the sort key.

You can distribute your upload work by using the sort key to load one item from each partition key value, then another item from each partition key value, and so on.

Every upload in this sequence uses a different partition key value, keeping more DynamoDB servers busy simultaneously and improving your throughput performance.

AWS Trusted Advisor

AWS Trusted Advisor is an online tool that provides you with real-time guidance to help you provision your resources following AWS best practices.

Whether you’re establishing new workflows or developing applications, or as part of ongoing improvements, take advantage of the recommendations provided by Trusted Advisor on a regular basis. By reviewing the recommendations, you can look for opportunities to save money.

Here are some Trusted Advisor checks that help you determine how to reduce your bill:

• Low utilization of Amazon EC2 instances
• Idle resources, such as load balancers and Amazon RDS DB instances
• Underutilized Amazon EBS volumes and Amazon Redshift clusters
• Unassociated Elastic IP addresses
• Optimization, lease expiration—Amazon Reserved Instances
• Inefficiently configured Amazon Route 53 latency record sets

AWS Cost Explorer

Use the AWS Cost Explorer tool to dive deeper into your cost and usage data to identify trends, pinpoint cost drivers, and detect anomalies. It includes Amazon EC2 usage reports, which let you analyze the cost and usage of your Amazon EC2 instances over the last 13 months. You can analyze your cost and usage data in aggregate (such as total costs and usage across all accounts) down to granular details (for example, m2.2xlarge costs within the Dev account tagged “project: GuardDuty”).

AWS Cost Explorer built-in reports include the following:

Monthly Costs by AWS Service Allows you to visualize the costs and usage associated with your top five cost-accruing AWS services and gives you a detailed breakdown on all services in the table view. The reports let you adjust the time range to view historical data going back up to 12 months to gain an understanding of your cost trends.

Amazon EC2 Monthly Cost and Usage Lets you view all AWS costs over the past two months, in addition to your current month-to-date costs. From there, you can drill down into the costs and usage associated with particular linked accounts, regions, tags, and more.

Monthly Costs by Linked Account Allows you to view the distribution of costs across your organization.

Monthly Running Costs Provides an overview of all running costs over the past three months and provides forecasted numbers for the coming month with a corresponding confidence interval. This report gives you good insight into how your costs are trending and helps you plan ahead.

AWS Cost Explorer Reserved Instance Reports include the following:

RI Utilization Report Visualize the degree to which you are using your existing resources and identify opportunities to improve your Reserved Instance cost efficiencies. The report shows how much you saved by using Reserved Instances, how much you overspent on Reserved Instances, and your net savings from purchasing Reserved Instances during the selected time range. This helps you to determine whether you have purchased too many Reserved Instances.

RI Coverage Report Discover how much of your overall instance usage is covered by Reserved Instances so that you can make informed decisions about when to purchase or modify a Reserved Instance to ensure maximum coverage. These show how much you spent on On-Demand Instances and how much you might have saved had you purchased more reservations. The report enables you to determine whether you have under-purchased Reserved Instances.

AWS Cost Explorer API

Use AWS Cost Explorer API to query your cost and usage data programmatically (using AWS CLI or AWS SDKs). You can query for aggregated data such as total monthly costs or total daily usage. You can also query for granular data, such as the number of daily write operations for Amazon DynamoDB database tables in your production environment. All of the AWS SDKs greatly simplify the process of signing requests and save you a significant amount of time when compared with using the AWS Cost Explorer API.

You can access your Amazon EC2 Reserved Instance purchase recommendations programmatically through the AWS Cost Explorer API. Recommendations for Reserved Instance purchases are calculated based on your past usage and indicate opportunities for potential cost savings.

The following example retrieves recommendations for Partial Upfront Amazon EC2 instances with a three-year term based on the last 60 days of Amazon EC2 usage.

Here’s the AWS CLI command:

aws ce get-reservation-purchase-recommendation --service "Amazon Redshift" --lookback-period-in-days SIXTY_DAYS --term-in-years THREE_YEARS --payment-option PARTIAL_UPFRONT

Here’s the output:

{
   "Recommendations": [],
   "Metadata": {
       "GenerationTimestamp": "2018-08-08T15:20:57Z",
       "RecommendationId": "00d59dde-a1ad-473f-8ff2-iexample3330b"
   }

AWS Budgets

With AWS Budgets, you can set custom budgets that alert you when your costs or usage exceed (or are forecasted to exceed) your budgeted amount. You can also use AWS Budgets to set Reserved Instance utilization or coverage targets and receive alerts when your utilization drops below the threshold you define. Reserved Instance alerts support Amazon EC2, Amazon RDS, Amazon Redshift, and Amazon ElastiCache reservations.

Budgets can be tracked at the monthly, quarterly, or yearly level, and you can customize the start and end dates. You can further refine your budget to track costs associated with multiple dimensions, such as AWS service, linked account, tag, and others. You can send budget alerts through email or Amazon Simple Notification Service (Amazon SNS) topic. For example, you can set notifications that alert you if you accrue 80, 90, and 100 percent of your actual budgeted costs in addition to a notification that alerts you if you are forecasted to exceed your budget.

AWS Cost and Usage Report

The AWS Cost and Usage Report tracks your AWS usage and provides estimated charges associated with that usage. You can configure this report to present the data hourly or daily. It is updated at least once a day until it is finalized at the end of the billing period. The AWS Cost and Usage report gives you the most granular insight possible into your costs and usage, and it is the source of truth for the billing pipeline. It can be used to develop advanced custom metrics using business intelligence, data analytics, and third-party cost optimization tools.

The AWS Cost and Usage report is delivered automatically to an S3 bucket that you specify, and it can be downloaded directly from there (standard Amazon S3 storage rates apply). It can also be ingested into Amazon Redshift or uploaded to Amazon QuickSight.

Amazon CloudWatch

Amazon CloudWatch is a monitoring service for AWS Cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics and log files, set alarms, and automatically react to changes in your AWS resources. You can create an alarm to perform one or more of the following actions based on the value of the metric:

• Automatically stop or terminate Amazon EC2 instances that have gone unused or underutilized for too long
• Stop your instance if it has an EBS volume as its root device

For example, you may run development or test instances and occasionally forget to shut them off. You can create an alarm that is triggered when the average CPU utilization percentage has been lower than 10 percent for 24 hours, signaling that it is idle and no longer in use. You can create a group of alarms that first sends an email notification to developers whose instance has been underutilized for 8 hours and then terminates that instance if its utilization has not improved after 24 hours.

Amazon CloudWatch Events deliver a near real-time stream of system events that describe changes in AWS resources. Using simple rules, you can route each type of event to one or more targets, such as Lambda functions, Amazon Kinesis streams, and Amazon SNS topics.

AWS Cost Optimization Monitor

AWS Cost Optimization Monitor is an automated reference deployment solution that processes detailed billing reports to provide granular metrics that you can search, analyze, and visualize in a customizable dashboard. The solution uploads detailed billing report data automatically to Amazon Elasticsearch Service (Amazon ES) for analysis and leverages its built-in support for Kibana, enabling you to visualize the first batch of data as soon as it’s processed.

The default dashboard is configured to show specific cost and usage metrics. All of these metrics, as listed here, were selected based on best practices observed across AWS customers:

• Amazon EC2 Instances Running per Hour
• Total Cost
• Cost by Tag Key: Name
• Cost by Amazon EC2 Instance Type
• Amazon EC2 Elasticity
• Amazon EC2 Hours per Dollar Invested

Cost Optimization: Amazon EC2 Right Sizing

Amazon EC2 Right Sizing is an automated AWS reference deployment solution that uses managed services to perform a right-sizing analysis and offer detailed recommendations for more cost-effective instances. The solution analyzes two weeks of utilization data to provide detailed recommendations for right sizing your Amazon EC2 instances.

Amazon CloudWatch

Amazon CloudWatch is essential to performance efficiency, which provides system-wide visibility into resource utilization, application performance, and operational health.

You can create an alarm to monitor any Amazon CloudWatch metric in your account. For example, you can create alarms on an Amazon EC2 instance CPU utilization, Elastic Load Balancing request latency, Amazon DynamoDB table throughput, or Amazon SQS queue length.

In the following example, AWS CLI is used to create an alarm to send an Amazon SNS email message when CPU utilization exceeds 70 percent:

aws cloudwatch put-metric-alarm --alarm-name cpu-mon --alarm-description "Alarm when CPU exceeds 70 percent" --metric-name CPUUtilization --namespace AWS/EC2 --statistic Average --period 300 --threshold 70 --comparison-operator  GreaterThanThreshold  --dimensions "Name=InstanceId,Value=i-12345678" --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:111122223333:MyTopic --unit Percent

Here are a few examples of when and how alarms are sent:

• Sends an email message using Amazon SNS when the average CPU use of an Amazon EC2 instance exceeds a specified threshold for consecutive specified periods
• Sends an email when an instance exceeds 10 GB of outbound network traffic per day
• Stops an instance and sends a text message (SMS) when outbound traffic exceeds 1 GB per hour
• Stops an instance when memory utilization reaches or exceeds 90 percent so that application logs can be retrieved for troubleshooting

AWS Trusted Advisor

AWS Trusted Advisor inspects your AWS environment and makes recommendations that help to improve the speed and responsiveness of your applications.

The following are a few Trusted Advisor checks to improve the performance of your service. The Trusted Advisor checks your service limits, ensuring that you take advantage of provisioned throughput, and monitors for overutilized instances:

• Amazon EC2 instances that are consistently at high utilization can indicate optimized, steady performance, but this check can also indicate that an application does not have enough resources.
• Provisioned IOPS (SSD) volumes that are attached to an Amazon EC2 instance that is not Amazon EBS–optimized. Amazon EBS volumes are designed to deliver the expected performance only when they are attached to an EBS-optimized instance.
• Amazon EC2 security groups with a large number of rules.
• Amazon EC2 instances that have a large number of security group rules.
• Amazon EBS magnetic volumes (standard) that are potentially overutilized and might benefit from a more efficient configuration.
• CloudFront distributions for alternate domain names with incorrectly configured DNS settings.
• Some HTTP request headers, such as Date or User-Agent, significantly reduce the cache hit ratio. This increases the load on your origin and reduces performance because CloudFront must forward more requests to your origin.