Structured Query Language

Structured query language (SQL) is the primary interface that you use to communicate with relational databases. The standard American National Standards Institute (ANSI) SQL is supported by all popular relational database engines. Some of these engines have extensions to ANSI SQL to support functionality that is specific to that engine. You use SQL to add, update, or delete data rows; to retrieve subsets of data for transaction processing and analytics applications; and to manage all aspects of the database.

Data Integrity

Data integrity is the overall completeness, accuracy, and consistency of data. Relational databases use a set of constraints to enforce data integrity in the database. These include primary keys, foreign keys, NOT NULL constraints, unique constraint, default constraints, and check constraints. These integrity constraints help enforce business rules in the tables to ensure the accuracy and reliability of the data. In addition, most relational databases enable you to embed custom code triggers that execute based on an action on the database.

Transactions

A database transaction is one or more SQL statements that execute as a sequence of operations to form a single logical unit of work. Transactions provide an all-or-nothing proposition, meaning that the entire transaction must complete as a single unit and be written to the database, or none of the individual components of the transaction will continue. In relational database terminology, a transaction results in a COMMIT or a ROLLBACK. Each transaction is treated in a coherent and reliable way, independent of other transactions.

ACID Compliance

All database transactions must be atomic, consistent, isolated, and durable (ACID)–compliant or be atomic, consistent, isolated, and durable to ensure data integrity.

Atomicity Atomicity requires that the transaction as a whole executes successfully, or if a part of the transaction fails, then the entire transaction is invalid.

Consistency Consistency mandates that the data written to the database as part of the transaction must adhere to all defined rules and restrictions, including constraints, cascades, and triggers.

Isolation Isolation is critical to achieving concurrency control, and it makes sure that each transaction is independent unto itself.

Durability Durability requires that all of the changes made to the database be permanent when a transaction is successfully completed.

Amazon Relational Database Service

With Amazon Relational Database Service (Amazon RDS), you can set up, operate, and scale a relational database in the AWS Cloud. It provides cost-efficient, resizable capacity for open-standard relational database engines. Amazon RDS is easy to administer, and you do not need to install the database software. Amazon RDS manages time-consuming database administration tasks, which frees you up to focus on your applications and business. For example, Amazon RDS automatically patches the database software and backs up your database. The Amazon RDS managed relational database service works with the popular database engines depicted in Figure 4.1.

Amazon RDS assumes many of the difficult or tedious management tasks of a relational database:

Procurement, configuration, and backup tasks

• When you buy a server, you get a central processing unit (CPU), memory, storage, and input/output operations per second (IOPS) all bundled together. With Amazon RDS, these are split apart so that you can scale them independently and allocate your resources as you need them.
• Amazon RDS manages backups, software patches, automatic failure detection, and recovery.
• You can configure automated backups or manually create your own backup snapshot and use these backups to restore a database. The Amazon RDS restore process works reliably and efficiently.
• You can use familiar database products: MySQL, MariaDB, PostgreSQL, Oracle, Microsoft SQL Server, and the MySQL- and PostgreSQL-compatible Amazon Aurora DB engine.

Security and availability

• You can enable the encryption option for your Amazon RDS DB instance.
• You can get high availability with a primary instance and a synchronous secondary instance that you can fail over to when problems occur. You can also use MySQL, MariaDB, or PostgreSQL read replicas to increase read scaling.
• In addition to the security in your database package, you can use AWS Identity and Access Management (IAM) to define users, and permissions help control who can access your Amazon RDS databases. You can also help protect your databases by storing them in a virtual private cloud (VPC).
• To deliver a managed service experience, Amazon RDS does not provide shell access to DB instances, and it restricts access to certain system procedures and tables that require advanced permissions.

When you host databases on Amazon RDS, AWS is responsible for the items in Figure 4.2.

Relational Database Engines on Amazon RDS

Amazon RDS provides six familiar database engines: Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL, and MariaDB. Because Amazon RDS is a managed service, you gain a number of benefits and features built right into the Amazon RDS service. These features include, but are not limited to, the following:

• Automatic software patching
• Easy vertical scaling
• Easy storage scaling
• Read replicas
• Automatic backups
• Database snapshots
• Multi-AZ deployments
• Encryption
• IAM DB authentication
• Monitoring and metrics with Amazon CloudWatch

To create an Amazon RDS instance, you can run the following command from the AWS CLI:

aws rds create-db-instance 
--db-instance-class db.t2.micro 
--allocated-storage 30 
--db-instance-identifier my-cool-rds-db --engine mysql 
--master-username masteruser --master-user-password masterpassword1!

Depending on the configurations chosen, the database can take several minutes before it is active and ready for use. You can monitor the Amazon RDS Databases console to view the status. When the status states Available, it is ready to be used, as shown in Figure 4.3.

Automatic Software Patching

Periodically, Amazon RDS performs maintenance on Amazon RDS resources. Maintenance mostly involves patching the Amazon RDS database underlying operating system (OS) or database engine version. Because this is a managed service, Amazon RDS handles the patching for you.

When you create an Amazon RDS database instance, you can define a maintenance window. A maintenance window is where you can define a period of time when you want to apply any updates or downtime to your database instance. You also can enable the automatic minor version upgrade feature, which automatically applies any new minor versions of the database as they are released (see Figure 4.4).

You can select a maintenance window by using the AWS Management Console, AWS CLI, or Amazon RDS API. After selecting the maintenance window, the Amazon RDS instance is upgraded (if upgrades are available) during that time window. You can also modify the maintenance window by running the following AWS CLI command:

aws rds modify-db-instance --db-instance-identifer your-db-instance-identifer --preferred-maintenance-window Mon:07:00-Mon:07:30

aws rds modify-db-instance --db-instance-identifer your-db-instance-identifer --db-instance-class db.t2.medium

aws rds modify-db-instance --db-instance-identifer your-db-instance-identifer --allocated-storage 50 --storage-type io1 --iops 3000

aws rds create-db-instance-read-replica --db-instance-identifier your-db-instance-identifier --source-db-instance-identifier your-source-db

Backing Up Data with Amazon RDS

Amazon RDS has two different ways of backing up data of your database instance: automated backups and database snapshots (DB snapshots).

Database Snapshots (Manual)

Unlike automated backups, database snapshots with Amazon RDS are user-initiated and enable you to back up your database instance in a known state at any time. You can also restore to that specific snapshot at any time.

Similar to the other Amazon RDS features, you can create the snapshots through the AWS Management Console, with the CreateDBSnapshot API, or with the AWS CLI.

With DB snapshots, the backups are kept until you explicitly delete them; therefore, before removing any Amazon RDS instance, take a final snapshot before removing it. Regardless of the backup taken, storage I/O may be briefly suspended while the backup process initializes (typically a few seconds), and you may experience a brief period of elevated latency. A way to avoid these types of suspensions is to deploy in a Multi-AZ configuration. With such a deployment, the backup is taken from the standby instead of the primary database.

To create a snapshot of the database, from the Amazon RDS Databases console, under Actions, select the Take snapshot option (see Figure 4.5). After a snapshot is taken, you can view all of your snaps from the Snapshots console.

Multi-AZ Deployments

By using Amazon RDS, you can run in a Multi-AZ configuration. In a Multi-AZ configuration, you have a primary and a standby DB instance. Updates to the primary database replicate synchronously to the standby replica in a different Availability Zone. The primary benefit of Multi-AZ is realized during certain types of planned maintenance, or in the unlikely event of a DB instance failure or an Availability Zone failure. Amazon RDS automatically fails over to the standby so that you can resume your workload as soon as the standby is promoted to the primary. This means that you can reduce your downtime in the event of a failure.

Because Amazon RDS is a managed service, Amazon RDS handles the fail to the standby. When there is a DB instance failure, Amazon RDS automatically promotes the standby to the primary—you will not interact with the standby directly. In other words, you will receive one endpoint connection for the Amazon RDS cluster, and Amazon RDS handles the failover.

Amazon RDS Multi-AZ configuration provides the following benefits:

• Automatic failover; no administration required
• Increased durability in the unlikely event of component failure
• Increased availability in the unlikely event of an Availability Zone failure
• Increased availability for planned maintenance (automated backups; I/O activity is no longer suspended)

To create an Amazon RDS instance in a Multi-AZ configuration, you must specify a subnet group that has two different Availability Zones specified. You can specify a Multi-AZ configuration by using AWS CLI by adding the --multi-az flag to the AWS CLI command, as follows:

aws rds create-db-instance 
--db-instance-class db.t2.micro 
--allocated-storage 30 
--db-instance-identifier multi-az-rds-db --engine mysql 
--master-username masteruser 
--master-user-password masterpassword1! 
--multi-az

Encryption

For encryption at rest, Amazon RDS uses the AWS Key Management Service (AWS KMS) for AES-256 encryption. You can use a default master key or specify your own for the Amazon RDS DB instance. Encryption is one of the few options that must be configured when the DB instance is created. You cannot modify an Amazon RDS database to enable encryption. You can, however, create a DB snapshot and then restore to an encrypted DB instance or cluster.

Amazon RDS supports using the Transparent Data Encryption (TDE) for Oracle and SQL Server. For more information on TDE with Oracle and Microsoft SQL Server, see the following:

• Microsoft SQL Server Transparent Data Encryption Support at:

  https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer .Options.TDE.html

• Options for Oracle DB Instances:

  https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle .Options.html#Appendix.Oracle.Options.AdvSecurity

At the time of this writing, the following Amazon RDS DB instance types are not supported for encryption at rest:

• Db.m1.small
• Db.m1.medium
• Db.m1.large
• Db.m1.xlarge
• Db.m2.xlarge
• Db.m2.2xlarge
• Db.m2.4xlarge
• Db.t2.micro

For encryption in transit, Amazon RDS generates an SSL certificate for each database instance that can be used to connect your application and the Amazon RDS instance. However, encryption is a compute-intensive operation that increases the latency of your database connection. For more information, see the documentation for the specific database engine.

IAM DB Authentication

You can authenticate to your DB instance by using IAM. By using IAM, you can manage access to your database resources centrally instead of storing the user credentials in each database. The IAM feature also encrypts network traffic to and from the database by using SSL.

IAM DB authentication is supported only for MySQL and PostgreSQL. At the time of this writing, the following MySQL versions are supported:

• MySQL 5.6.34 or later
• MySQL 5.7.16 or later

  There’s no support for the following:

• IAM DB Authentication for MySQL 5.5 or MySQL 8.0
• db.t2.small and db.m1.small instances

  The following PostgreSQL versions are supported:

• PostgreSQL versions 10.6 or later
• PostgreSQL 9.6.11 or later
• PostgreSQL 9.5.15 or later

To enable IAM DB authentication for your Amazon RDS instance, run the following command:

aws rds modify-db-instance --db-instance-identifier my-rds-db --enable-iam-database-authentication --apply-immediately

Because downtime is associated with this action, you can enable this feature during the next maintenance window. You can do so by changing the last parameter to --no-apply-immediately.

Monitoring with Amazon CloudWatch

Use Amazon CloudWatch to monitor your database tier. You can create alarms to notify database administrators when there is a failure.

By default, CloudWatch provides some built-in metrics for Amazon RDS with a granularity of 5 minutes (600 seconds). If you want to gather metrics in a smaller window of granularity, such as 1 second, enable enhanced monitoring, which is similar to how you enable these features in Amazon EC2.

To view all the Amazon RDS metrics that are provided through CloudWatch, select the Monitoring tab from the Amazon RDS console (see Figure 4.6).

Amazon RDS integrates with CloudWatch to send it the following database logs:

• Audit log
• Error log
• General log
• Slow query log

From the Amazon RDS console, select the Logs & events tab to view and download the specified logs, as shown in Figure 4.7.

For more information on CloudWatch and its capabilities across other AWS services, see Chapter 15, “Monitoring and Troubleshooting.”

Amazon Aurora

Amazon Aurora is a MySQL- and PostgreSQL-compatible relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases.

Aurora is part of the managed database service Amazon RDS.

Amazon Aurora DB Clusters

Aurora is a drop-in replacement for MySQL and PostgreSQL relational databases. It is built for the cloud, and it combines the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. You can use the code, tools, and applications that you use today with your existing MySQL and PostgreSQL databases with Aurora.

The integration of Aurora with Amazon RDS means that time-consuming administration tasks, such as hardware provisioning, database setup, patching, and backups, are automated.

Aurora features a distributed, fault-tolerant, self-healing storage system that automatically scales up to 64 TiB per database instance. (In comparison, other Amazon RDS options allow for a maximum of 32 TiB.) Aurora delivers high performance and availability with up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon Simple Storage Service (Amazon S3), and replication across three Availability Zones. When you create an Aurora instance, you create a DB cluster. A DB cluster consists of one or more DB instances and a cluster volume that manages the data for those instances. An Aurora cluster volume is a virtual database storage volume that spans multiple Availability Zones, and each Availability Zone has a copy of the DB cluster data.

An Aurora DB cluster has two types of DB instances:

Primary Instance Supports read and write operations and performs all of the data modifications to the cluster volume. Each Aurora DB cluster has one primary instance.

Amazon Aurora Replica Supports read-only operations. Each Aurora DB cluster can have up to 15 Amazon Aurora Replicas in addition to the primary instance. Multiple Aurora Replicas distribute the read workload, and if you locate Aurora Replicas in separate Availability Zones, you can also increase database availability.

Figure 4.8 illustrates the relationship between the cluster volume, the primary instance, and Aurora Replicas in an Aurora DB cluster.

As you can see from Figure 4.8, this architecture is vastly different from the other Amazon RDS databases. Aurora is engineered and architected for the cloud. The primary difference is that there is a separate storage layer, called the cluster volume, which is spread across multiple Availability Zones in a single AWS Region. This means that the durability of your data is increased.

Additionally, Aurora has one primary instance that writes across the cluster volume. This means that Aurora replicas can be spun up quickly, because they don’t have to copy and store their own storage layer; they connect to it. Because the cluster volume is separated in this architecture, the cluster volume can grow automatically as your data increases. This is in contrast to how other Amazon RDS databases are built, whereby you must define the allocated storage in advance.

Best Practices for Running Databases on AWS

The following are best practices for working with Amazon RDS:

Follow Amazon RDS basic operational guidelines. The Amazon RDS Service Level Agreement requires that you follow these guidelines:

• Monitor your memory, CPU, and storage usage. Amazon CloudWatch can notify you when usage patterns change or when you approach the capacity of your deployment so that you can maintain system performance and availability.
• Scale up your DB instance when you approach storage capacity limits. Have some buffer in storage and memory to accommodate unforeseen increases in demand from your applications.
• Enable automatic backups, and set the backup window to occur during the daily low in write IOPS.
• If your database workload requires more I/O than you have provisioned, recovery after a failover or database failure will be slow. To increase the I/O capacity of a DB instance, do any or all of the following:
  • Migrate to a DB instance class with high I/O capacity.
  • Convert from standard storage either to General Purpose or Provisioned IOPS storage, depending on how much of an increase you need. If you convert to Provisioned IOPS storage, make sure that you also use a DB instance class that is optimized for Provisioned IOPS.
  • If you are already using Provisioned IOPS storage, provision additional throughput capacity.
• If your client application is caching the Domain Name Service (DNS) data of your DB instances, set a time-to-live (TTL) value of less than 30 seconds. Because the underlying IP address of a DB instance can change after a failover, caching the DNS data for an extended time can lead to connection failures if your application tries to connect to an IP address that no longer is in service.
• Test failover for your DB instance to understand how long the process takes for your use case and to ensure that the application that accesses your DB instance can automatically connect to the new DB instance after failover.

Allocate sufficient RAM to the DB instance. An Amazon RDS performance best practice is to allocate enough RAM so that your working set resides almost completely in memory. Check the ReadIOPS metric by using CloudWatch while the DB instance is under load to view the working set. The value of ReadIOPS should be small and stable. Scale up the DB instance class until ReadIOPS no longer drops dramatically after a scaling operation or when ReadIOPS is reduced to a small amount.

Implement Amazon RDS security. Use IAM accounts to control access to Amazon RDS API actions, especially actions that create, modify, or delete Amazon RDS resources, such as DB instances, security groups, option groups, or parameter groups, and actions that perform common administrative actions, such as backing up and restoring DB instances, or configuring Provisioned IOPS storage.

• Assign an individual IAM account to each person who manages Amazon RDS resources. Do not use an AWS account user to manage Amazon RDS resources; create an IAM user for everyone, including yourself.
• Grant each user the minimum set of permissions required to perform his or her duties.
• Use IAM groups to manage permissions effectively for multiple users.
• Rotate your IAM credentials regularly.

Use the AWS Management Console, the AWS CLI, or the Amazon RDS API to change the password for your master user. If you use another tool, such as a SQL client, to change the master user password, it might result in permissions being revoked for the user unintentionally.

Use enhanced monitoring to identify OS issues. Amazon RDS provides metrics in real time for the OS on which your DB instance runs. You can view the metrics for your DB instance by using the console or consume the Enhanced Monitoring JSON output from CloudWatch Logs in a monitoring system of your choice. Enhanced Monitoring is available for the following database engines:

• MariaDB
• Microsoft SQL Server
• MySQL version 5.5 or later
• Oracle
• PostgreSQL

Enhanced Monitoring is available for all DB instance classes except for db.m1.small. Enhanced Monitoring is available in all regions except for AWS GovCloud (US).

Use metrics to identify performance issues. To identify performance issues caused by insufficient resources and other common bottlenecks, you can monitor the metrics available for your Amazon RDS DB instance.

Monitor performance metrics regularly to see the average, maximum, and minimum values for a variety of time ranges. If you do so, you can identify when performance is degraded. You can also set CloudWatch alarms for particular metric thresholds.

To troubleshoot performance issues, it’s important to understand the baseline performance of the system. When you set up a new DB instance and get it running with a typical workload, you should capture the average, maximum, and minimum values of all the performance metrics at a number of different intervals (for example, 1 hour, 24 hours, 1 week, or 2 weeks) to get an idea of what is normal. It helps to get comparisons for both peak and off-peak hours of operation. You can then use this information to identify when performance is dropping below standard levels.

Tune queries. One of the best ways to improve DB instance performance is to tune your most commonly used and most resource-intensive queries to make them less expensive to run.

A common aspect of query tuning is creating effective indexes. You can use the Database Engine Tuning Advisor to get potential index improvements for your DB instance.

Use DB parameter groups. AWS recommends that you apply changes to the DB parameter group on a test DB instance before you apply parameter group changes to your production DB instances. Improperly setting DB engine parameters in a DB parameter group can have unintended adverse effects, including degraded performance and system instability. Always exercise caution when modifying DB engine parameters, and back up your DB instance before modifying a DB parameter group.

Use read replicas. Use read replicas to relieve pressure on your master node with additional read capacity. You can bring your data closer to applications in different regions and promote a read replica to a master for faster recovery in the event of a disaster.

When to Use a NoSQL Database

NoSQL databases are a great fit for many big data, mobile, and web applications that require greater scale and higher responsiveness than traditional relational databases. Because of simpler data structures and horizontal scaling, NoSQL databases typically respond faster and are easier to scale than relational databases.

Comparison of SQL and NoSQL Databases

Many developers are familiar with SQL databases but might be new to working with NoSQL databases. Relational database management systems (RDBMS) and nonrelational (NoSQL) databases have different strengths and weaknesses. In a RDBMS, data can be queried flexibly, but queries are relatively expensive and do not scale well in high-traffic situations. In a NoSQL database, you can query data efficiently in a limited number of ways. Table 4.3 shows a comparison of different characteristics of SQL and NoSQL databases.

The storage format for SQL versus NoSQL databases also differs. As shown in Figure 4.9, SQL databases are often stored in a row and column format, whereas NoSQL databases, such as Amazon DynamoDB, have a key-value format that could be in a JSON format, as shown in this example.

NoSQL Database Types

There are four types of NoSQL databases: columnar, document, graph, and in-memory key-value. Generally, these databases differ in how the data is stored, accessed, and structured, and they are optimized for different use cases and applications.

Columnar databases Columnar databases are optimized for reading and writing columns of data as opposed to rows of data. Column-oriented storage for database tables is an important factor in analytic query performance because it drastically reduces the overall disk I/O requirements and reduces the amount of data that you must load from disk.

Document databases Document databases are designed to store semi-structured data as documents, typically in JSON or XML format. Unlike traditional relational databases, the schema for each NoSQL document can vary, giving you more flexibility in organizing and storing application data and reducing storage required for optional values.

Graph databases Graph databases store vertices and directed links called edges. Graph databases can be built on both SQL and NoSQL databases. Vertices and edges can each have properties associated with them.

In-memory key-value stores In-memory key-value stores are NoSQL databases optimized for read-heavy application workloads (such as social networking, gaming, media sharing, and Q&A portals) or compute-intensive workloads (such as a recommendation engine). In-memory caching improves application performance by storing critical pieces of data in memory for low-latency access.

Core Components of Amazon DynamoDB

In DynamoDB, tables, items, and attributes are the core components with which you work. A table is a collection of items, and each item is a collection of attributes. DynamoDB uses partition keys to identify uniquely each item in a table. Secondary indexes can be used to provide more querying flexibility. You can use DynamoDB Streams to capture data modification events in DynamoDB tables.

Figure 4.10 shows the DynamoDB data model, including a table, items, attributes, a required partition key, an optional sort key, and an example of data being stored in partitions.

Attributes

Each item is composed of one or more attributes. Attributes in DynamoDB are similar in many ways to fields or columns in other database systems. An attribute is a fundamental data element, something that does not need to be broken down any further. You can think of an attribute as similar to columns in a relational database. For example, an item in a People table contains attributes called PersonID, Last Name, First Name, and so on.

Figure 4.11 shows a table named People with items and attributes. Each block represents an item, and within those blocks you have attributes that define the overall item:

• Each item in the table has a unique identifier, a primary key, or a partition key that distinguishes the item from all of the others in the table. The primary key consists of one attribute (PersonID).
• Other than the primary key, the People table is schemaless, which means that you do not have to define the attributes or their data types beforehand. Each item can have its own distinct attributes. This is where the contrast begins to show between NoSQL and SQL. In SQL, you would have to define a schema for each person, and every person would need to have the same data points or attributes. As you can see in Figure 4.11, with NoSQL and DynamoDB, each person can have different attributes.
• Most of the attributes are scalar, so they can have only one value. Strings and numbers are common examples of scalars.
• Some of the items have a nested attribute (Address). DynamoDB supports nested attributes up to 32 levels deep.

Primary Key

When you create a table, at a minimum, you are required to specify the table name and primary key of the table. The primary key uniquely identifies each item in the table. No two items can have the same key within a table.

DynamoDB supports two different kinds of primary keys: partition key and partition key and sort key.

Partition key (hash key) A simple primary key, composed of one attribute, is known as the partition key. DynamoDB uses the partition key’s value as an input to an internal hash function. The output from the hash function determines the partition (physical storage internal to DynamoDB) in which the item is stored.

In a table that has only a partition key, no two items can have the same partition key value. For example, in the People table, with a simple primary key of PersonID, you cannot have two items with PersonID of 000-07-1075.

The partition key of an item is also known as its hash attribute. The term hash attribute derives from the use of an internal hash function in DynamoDB that evenly distributes data items across partitions based on their partition key values.

Each primary key attribute must be a scalar (meaning that it can hold only a single value). The only data types allowed for primary key attributes are string, number, or binary. There are no such restrictions for other, nonkey attributes.

Partition key and sort key (range attribute) A composite primary key is composed of two attributes: partition key and the sort key.

The sort key of an item is also known as its range attribute. The term range attribute derives from the way that DynamoDB stores items with the same partition key physically close together, in sorted order, by the sort key value.

The partition key acts the same as the sort key, but in addition to also using a sort key, the items with the same partition key are stored together, in sorted order, by sort key value.

In a table that has a partition key and a sort key, it’s possible for two items to have the same partition key value, but those two items must have different sort key values. You cannot have two items in the table that have identical partition key and sort key values.

For example, if you have a Music table with a composite primary key (Artist and SongTitle), you can access any item in the Music table directly if you provide the Artist and SongTitle values for that item.

A composite primary key gives you additional flexibility when querying data. For example, if you provide only the value for Artist, DynamoDB retrieves all of the songs by that artist. To retrieve only a subset of songs by a particular artist, you can provide a value for Artist with a range of values for SongTitle.

As a developer, the attribute you choose for your application has important implications. If there is little differentiation among partition keys, all of your data is stored together in the same physical location.

Figure 4.12 shows an example of these two types of keys. In the SensorLocation table, the primary key is the SensorId attribute. This means that every item (or row) in this table has a unique SensorId, meaning that each sensor has exactly one location or latitude and longitude value.

Conversely, the SensorReadings table has a partition key and a sort key. The SensorId attribute is the partition key and the Time attribute is the sort key, which combined make it a composite key. For each SensorId, there may be multiple items corresponding to sensor readings at different times. The combination of SensorId and Time uniquely identifies items in the table. This design enables you to query the table for all readings related to a particular sensor.

Secondary Indexes

If you want to perform queries on attributes that are not part of the table’s primary key, you can create a secondary index. By using a secondary index, you can query the data in the table by using an alternate key, in addition to querying against the primary key. DynamoDB does not require that you use indexes, but doing so may give you more flexibility when querying your data depending on your application and table design.

After you create a secondary index on a table, you can then read data from the index in much the same way as you do from the table. DynamoDB automatically creates indexes based on the primary key of a table and automatically updates all indexes whenever a table changes.

A secondary index contains the following:

• Primary key attributes
• Alternate key attributes
• (Optional) A subset of other attributes from the base table (projected attributes)

DynamoDB provides fast access to the items in a table by specifying primary key values. However, many applications might benefit from having one or more secondary (or alternate) keys available. This allows efficient access to data with attributes other than the primary key.

DynamoDB supports two types of secondary indexes: local secondary indexes and global secondary indexes. You can define up to five global secondary indexes and five local secondary indexes per table.

Local Secondary Index

A local secondary index is an index that has the same partition key as the base table, but a different sort key (see Figure 4.13). It is “local” in the sense that every partition of a local secondary index is scoped to a base table partition that has the same partition key value. You can construct only one while creating the table, but you cannot add, remove, or modify it later.

Global Secondary Index

A global secondary index is an index with a partition key and a sort key that can be different from those on the base table (see Figure 4.14). It is considered “global” because queries on the index can span all of the data in the base table across all partitions. You can create one during table creation, and you can add, remove, or modify it later.

 You can create a global secondary index, not a local secondary index, after table creation.

For example, by using a Music table, you can query data items by Artist (partition key) or by Artist and SongTitle (partition key and sort key). Suppose that you also wanted to query the data by Genre and Album Title. To do this, you could create a global secondary index on Genre and AlbumTitle and then query the index in much the same way as you’d query the Music table.

Figure 4.15 shows the example Music table with a new index called GenreAlbumTitle. In the index, Genre is the partition key, and AlbumTitle is the sort key.

Note the following about the GenreAlbumTitle index:

• Every index belongs to a table, which is called the base table for the index. In the preceding example, Music is the base table for the GenreAlbumTitle index.
• DynamoDB maintains indexes automatically. When you add, update, or delete an item in the base table, DynamoDB adds, updates, or deletes the corresponding item in any indexes that belong to that table.
• When you create an index, you specify which attributes will be copied, or projected, from the base table to the index. At a minimum, DynamoDB projects the key attributes from the base table into the index. This is the case with GenreAlbumTitle, wherein only the key attributes from the Music table are projected into the index.

You can query the GenreAlbumTitle index to find all albums of a particular genre (for example, all Hard Rock albums). You can also query the index to find all albums within a particular genre that have certain album titles (for example, all Heavy Metal albums with titles that start with the letter M).

Amazon DynamoDB Streams

Amazon DynamoDB Streams is an optional feature that captures data modification events in DynamoDB tables. The data about these events appears in the stream in near real time and in the order that the events occurred.

Each event is represented by a stream record. If you enable a stream on a table, DynamoDB Streams writes a stream record whenever one of the following events occurs:

• A new item is added to the table—The stream captures an image of the entire item, including all of its attributes.
• An item is updated—The stream captures the “before” and “after” images of any attributes that were modified in the item.
• An item is deleted from the table—The stream captures an image of the entire item before it was deleted.

Each stream record also contains the name of the table, the event timestamp, and other metadata. Stream records have a lifetime of 24 hours; after that, they are automatically removed from the stream.

Figure 4.16 shows how you can use DynamoDB Streams together with AWS Lambda to create a trigger—code that executes automatically whenever an event of interest appears in a stream. For example, consider a Customers table that contains customer information for a company. Suppose that you want to send a “welcome” email to each new customer. You could enable a stream on that table and then associate the stream with a Lambda function. The Lambda function would execute whenever a new stream record appears, but only process new items added to the Customers table. For any item that has an EmailAddress attribute, the Lambda function could invoke Amazon Simple Email Service (Amazon SES) to send an email to that address.

In addition to triggers, DynamoDB Streams enables other powerful solutions that developers can create, such as the following:

• Data replication within and across AWS regions
• Materialized views of data in DynamoDB tables
• Data analysis by using Amazon Kinesis materialized views

Read Consistency

DynamoDB replicates data among multiple Availability Zones in a region. When your application writes data to a DynamoDB table and receives an HTTP 200 response (OK), all copies of the data are updated. The data is eventually consistent across all storage locations, usually within 1 second or less. DynamoDB supports both eventually consistent and strongly consistent reads.

Read and Write Throughput

When you create a table or index in DynamoDB, you must specify your capacity requirements for read and write activity. By defining your throughput capacity in advance, DynamoDB can reserve the necessary resources to meet the read and write activity your application requires, while ensuring consistent, low-latency performance. Specify your required throughput value by setting the ProvisionedThroughput parameter when you create or update a table.

You specify throughput capacity in terms of read capacity units and write capacity units:

• One read capacity unit (RCU) represents one strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size. If you need to read an item that is larger than 4 KB, DynamoDB will need to consume additional read capacity units. The total number of read capacity units required depends on the item size and whether you want an eventually consistent or strongly consistent read.
• One write capacity unit (WCU) represents one write per second for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB must consume additional write capacity units. The total number of write capacity units required depends on the item size.

For example, suppose that you create a table with five read capacity units and five write capacity units. With these settings, your application could do the following:

• Perform strongly consistent reads of up to 20 KB per second (4 KB × 5 read capacity units).
• Perform eventually consistent reads of up to 40 KB per second (twice as much read throughput).
• Write up to 5 KB per second (1 KB × 5 write capacity units).

If your application reads or writes larger items (up to the DynamoDB maximum item size of 400 KB), it consumes more capacity units.

If your read or write requests exceed the throughput settings for a table, DynamoDB can throttle that request. DynamoDB can also throttle read requests excess for an index. Throttling prevents your application from consuming too many capacity units. When a request is throttled, it fails with an HTTP 400 code (Bad Request) and a ProvisionedThroughputExceededException. The AWS SDKs have built-in support for retrying throttled requests, so you do not need to write this logic yourself.

DynamoDB provides the following mechanisms for managing throughput as it changes:

Amazon DynamoDB Auto Scaling DynamoDB automatic scaling actively manages throughput capacity for tables and global secondary indexes. With automatic scaling, you define a range (upper and lower limits) for read and write capacity units. You also define a target utilization percentage within that range. DynamoDB auto scaling seeks to maintain your target utilization, even as your application workload increases or decreases.

Provisioned throughput If you aren’t using DynamoDB auto scaling, you have to define your throughput requirements manually. As discussed, with this setting you may run into a ProvisionedThroughputExceededException if you are throttled. But you can change your throughput with a few clicks.

Reserved capacity You can purchase reserved capacity in advance, where you pay a one-time upfront fee and commit to a minimum usage level over a period of time. You may realize significant cost savings compared to on-demand provisioned throughput settings.

On-demand It can be difficult to plan capacity, especially if you aren’t collecting metrics or perhaps are developing a new application and you aren’t sure what type of performance you require. With On-Demand mode, your DynamoDB table will automatically scale up or down to any previously reached traffic level. If a workload’s traffic level reaches a new peak, DynamoDB rapidly adapts to accommodate the workload. As a developer, focus on making improvements to your application and offload scaling activities to AWS.

Retrieving Data from DynamoDB

Two primary methods are used to retrieve data from DynamoDB: Query and Scan.

import boto3
import json
import decimal
 
# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)
 
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
 
table = dynamodb.Table('Music')
 
print("A query with DynamoDB")
 
response = table.query(
    KeyConditionExpression=Key('Artist').eq('Sam Samuel')
)
 
for i in response['Items']:
    print(i['SongTitle'], "-", i['Genre'], i['Price'])

// Return all of the data in the index
import boto3
import json
import decimal
 
# Create the DynamoDB Resource
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
 
# Use the Music Table
table = dynamodb.Table('Music')
 
# Helper class to convert a DynamoDB decimal/item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)
 
# Specify some filters for the scan
# Here we are stating that the Price must be between 12 - 30
fe = Key('Price').between(12, 30)
pe = "#g, Price"
# Expression Attribute Names for Projection Expression only.
ean = { "#g": "Genre", }
 
#
response_scan = table.scan(
    FilterExpression=fe,
    ProjectionExpression=pe,
    ExpressionAttributeNames=ean
    )
 
# Print all the items
for i in response_scan['Items']:
    print(json.dumps(i, cls=DecimalEncoder))
while 'LastEvaluatedKey' in response:
    response = table.scan(
        ProjectionExpression=pe,
        FilterExpression=fe,
        ExpressionAttributeNames= ean,
        ExclusiveStartKey=response['LastEvaluatedKey']
        )
    for i in response['Items']:
        print(json.dumps(i)

Global Tables

Global tables build upon the DynamoDB global footprint to provide you with a fully managed, multiregion, and multimaster database that provides fast, local, read-and-write performance for massively scaled, global applications. DynamoDB performs all the necessary tasks to create identical tables in these regions and propagate ongoing data changes to all of them. Figure 4.17 shows an example of how global tables can work with a global application and globally dispersed users.

A global table is a collection of one or more DynamoDB tables, all owned by a single AWS account, identified as replica tables. A replica table (or replica, for short) is a single DynamoDB table that functions as a part of a global table. Each replica stores the same set of data items. Any given global table can have only one replica table per region, and every replica has the same table name and the same primary key schema. Changes made in one replica are recorded in a stream and propagated to other replicas, as shown in Figure 4.18.

If your application requires strongly consistent reads, then it must perform all of its strongly consistent reads and writes in the same region. DynamoDB does not support strongly consistent reads across AWS Regions.

Conflicts can arise if applications update the same item in different regions at about the same time (concurrent updates). To ensure eventual consistency, DynamoDB global tables use a “last writer wins” reconciliation between concurrent updates whereby DynamoDB makes a best effort to determine the last writer. With this conflict resolution mechanism, all replicas agree on the latest update and converge toward a state in which they all have identical data.

To create a DynamoDB global table, perform the following steps:

1. Create an ordinary DynamoDB table, with DynamoDB Streams enabled, in an AWS Region.
2. Repeat step 1 for every other AWS Region where you want to replicate your data.
3. Define a DynamoDB global table based on the tables that you have created.

The AWS Management Console automates these tasks so that you can create a global table quickly and easily.

Object Persistence Model

The DynamoDB object persistence model enables you to map client-side classes to DynamoDB tables. The instances of these classes (objects) map to items in a DynamoDB table.

You can use the object persistence programming interface to connect to DynamoDB; perform create, read, update, and delete operations (CRUD); execute queries; and implement optimistic locking with a version number. Figure 4.19 shows an example of using the object persistence model to map client-side objects in DynamoDB.

Support for the object persistence model is available in the Java and .NET SDKs.

Amazon DynamoDB Local

DynamoDB Local is the downloadable version of DynamoDB that lets you write and test applications by using the Amazon DynamoDB API without accessing the DynamoDB web service. Instead, the database is self-contained on your computer. When you’re ready to deploy your application in production, you can make a few minor changes to the code so that it uses the DynamoDB web service.

Having this local version helps you save on provisioned throughput, data storage, and data transfer fees. In addition, you don’t need an internet connection while you’re developing your application.

IAM and Fine-Grained Access Control

You can use AWS IAM to grant or restrict access to DynamoDB resources and API actions. For example, you could allow a user to execute the GetItem operation on a Books table. DynamoDB also supports fine-grained access control so that you can control access to individual data items and attributes. This means that perhaps you have a Users table and you want the specific user to have access only to his or her data. You can accomplish this with fine-grained access control. Use a condition inside an IAM policy with the dynamodb:LeadingKeys property.

By using LeadingKeys, you can limit the user so that they can access only the items where the partition key matches the userID. In the following example in the Users table, you want to restrict who can view the profile information to only the user to which the data or profile information belongs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LeadingKeysExample",
            "Effect": "Allow",
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:BatchGetItem",
                "dynamodb:Query",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:DeleteItem",
                "dynamodb:BatchWriteItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:accountnumber:table/UserProfiles"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "dynamodb:LeadingKeys": [
                        "${www.amazon.com:user_id}"
              
                    ],
                    "dynamodb:Attributes": [
                        "UserId",
                        "FirstName",
                        "LastName",
                        "Email",
                        "Birthday"
                    ]
                },
                "StringEqualsIfExists": {
                    "dynamodb:Select": "SPECIFIC_ATTRIBUTES"
                }
            }
        }
    ]
}

As you can see in the IAM policy, only the specific user is allowed to access a subset of the total attributes that are defined in the Attributes section of the policy. Furthermore, the SELECT statement specifies that the application must provide a list of specific attributes to act upon, preventing the application from requesting all attributes.

Backup and Restore

You can create on-demand backups and enable point-in-time recovery for your DynamoDB tables.

On-demand backups create full backups of your tables or restore them on-demand at any time. These actions execute with zero impact on table performance or availability and without consuming any provisioned throughput on the table.

Point-in-time recovery helps protect your DynamoDB tables from accidental write or delete operations. For example, suppose that a test script accidentally writes to a production DynamoDB table. With point-in-time recovery, you can restore that table to any point in time during the last 35 days. DynamoDB maintains incremental backups of your table. These operations will not affect performance or latency.

Encryption with Amazon DynamoDB

DynamoDB offers fully managed encryption at rest, and it is enabled by default. DynamoDB uses AWS KMS for encrypting the objects at rest. By default, DynamoDB uses the AWS-owned customer master key (CMK); however, you can also specify your own AWS KMS CMK key that you have created. For more information on AWS KMS, see Chapter 5, “Encryption on AWS.”

Amazon DynamoDB Best Practices

Now that you understand what DynamoDB is and how you can use it to create a scalable database for your application, review some best practices for using DynamoDB.

Data Warehouse Benefits

Benefits of using a data warehouse include the following:

• Better decision-making
• Consolidation of data from many sources
• Data quality, consistency, and accuracy
• Historical intelligence
• Analytics processing that is separate from transactional databases, improving the performance of both systems

The data warehousing landscape has changed dramatically in recent years with the emergence of cloud-based services that offer high performance, simple deployment, near-infinite scaling, and easy administration at a fraction of the cost of on-premises solutions.

Comparison of Data Warehouses and Databases

A data warehouse is specially designed for data analytics, which involves reading large amounts of data to understand relationships and trends across the data. A database is used to capture and store data, such as recording details of a transaction. Table 4.6 is useful in comparing the characteristics of data warehouses and databases.

Comparison of Data Warehouses and Data Lakes

Unlike a data warehouse, a data lake, as described in Chapter 3, “Hello, Storage,” is a centralized repository for all data, including structured and unstructured. A data warehouse uses a predefined schema that is optimized for analytics. In a data lake, the schema is not defined, enabling additional types of analytics, such as big data analytics, full text search, real-time analytics, and machine learning. Table 4.7 compares the characteristics of a data warehouse and a data lake.

Comparison of Data Warehouses and Data Marts

A data mart is a data warehouse that serves the needs of a specific team or business unit, such as finance, marketing, or sales. It is smaller, is more focused, and may contain summaries of data that best serve its community of users. Table 4.8 compares the characteristics of a data warehouse and a data mart.

Architecture

An Amazon Redshift data warehouse is a collection of computing resources called nodes, which are organized into a group called a cluster. Each cluster runs an Amazon Redshift engine and contains one or more databases. After you provision your cluster, you can upload your dataset and then perform data analysis queries. Each cluster has a leader node and one or more compute nodes, and you have a choice of a hardware platform for your cluster.

Node Slices

A compute node is partitioned into slices. Each slice is allocated a portion of the node’s memory and disk space, where it processes a portion of the workload assigned to the node. The leader node manages distributing data to the slices and allocates the workload for any queries or other database operations to the slices. The slices then work in parallel to complete the operation. The node size of the cluster determines the number of slices per node.

Figure 4.20 shows the Amazon Redshift data warehouse architecture, including the client applications, JDBC and ODBC connections, leader node, compute nodes, and node slices.

Table Design

Each database within an Amazon Redshift cluster can support many tables. Like most SQL-based databases, you can create a table using the CREATE TABLE command. This command specifies the name of the table, the columns, and their data types. This command also supports specifying compression encodings, distribution strategy, and sort keys in Amazon Redshift.

Loading Data

Loading large datasets can take a long time and consume many computing resources. How your data is loaded can also affect query performance. You can reduce these impacts by using COPY commands, bulk inserts, and staging tables when loading data into Amazon Redshift.

Querying Data

You can query Amazon Redshift tables by using standard SQL commands, such as using SELECT statements, to query and join tables. For complex queries, you are able to analyze the query plan to choose better optimizations for your specific access patterns.

For large clusters supporting many users, you can configure workload management (WLM) to queue and prioritize queries.

Snapshots

Amazon Redshift supports snapshots, similar to Amazon RDS. You can create automated and manual snapshots, which are stored in Amazon S3 by using an encrypted Secure Socket Layer (SSL) connection. If you need to restore from a snapshot, Amazon Redshift creates a new cluster and imports data from the snapshot that you specify.

When you restore from a snapshot, Amazon Redshift creates a new cluster and makes it available before all of the data is loaded so that you can begin querying the new cluster immediately. Amazon Redshift will stream data on demand from the snapshot in response to active queries and load all the remaining data in the background.

Achieving proper durability for a database requires more effort and more attention. Even when using Amazon Elastic Block Store (Amazon EBS) volumes, take snapshots on a frozen file system to be consistent. Also, restoring a database might require additional operations other than restoring a volume from a snapshot and attaching it to an Amazon EC2 instance.

Security

Securing your Amazon Redshift cluster is similar to securing other databases running in the AWS Cloud. To meet your needs, you will use a combination of IAM policies, security groups, and encryption to secure the cluster.

Amazon Redshift Spectrum

Amazon Redshift also includes Redshift Spectrum, allowing you to run SQL queries directly against exabytes of unstructured data in Amazon S3. No loading or transformation is required.

You can use many open data formats, including Apache Avro, CSV, Grok, Ion, JSON, Optimized Row Columnar (ORC), Apache Parquet, RCFile, RegexSerDe, SequenceFile, TextFile, and TSV. Redshift Spectrum automatically scales query compute capacity based on the data being retrieved, so queries against Amazon S3 run fast, regardless of dataset size.

To use Redshift Spectrum, you need an Amazon Redshift cluster and a SQL client that’s connected to your cluster so that you can execute SQL commands. The cluster and the data files in Amazon S3 must be in the same AWS Region.

Benefits of Caching

A cache provides high-throughput, low-latency access to commonly accessed application data by storing the data in memory. Caching can improve the speed of your application. Caching reduces the response latency, which improves a user’s experience with your application.

Time-consuming database queries and complex queries often create bottlenecks in applications. In read-intensive applications, caching can provide large performance gains by reducing application processing time and database access time. Write-intensive applications typically do not see as great a benefit to caching. However, even write-intensive applications normally have a read/write ratio greater than 1, which implies that read caching can be beneficial. In summary, the benefits of caching include the following:

• Improve application performance
• Reduce database cost
• Reduce load on the backend database tier
• Facilitate predictable performance
• Eliminate database hotspots
• Increase read throughput (IOPS)

The following types of information or applications can often benefit from caching:

• Results of database queries
• Results of intensive calculations
• Results of remote API calls
• Compute-intensive workloads that manipulate large datasets, such as high-performance computing simulations and recommendation engines

Consider caching your data if the following conditions apply:

• It is slow or expensive to acquire when compared to cache retrieval.
• It is accessed with sufficient frequency.
• Your data or information for your application is relatively static
• Your data or information for your application is rapidly changing and staleness is not significant.

Caching Strategies

You can implement different caching strategies for your application. Two primary methods are lazy loading and write through. A cache hit occurs when the cache contains the information requested. A cache miss occurs when the cache does not contain the information requested.

Lazy loading Lazy loading is a caching strategy that loads data into the cache only when necessary. When your application requests data, it first makes the request to the cache. If the data exists in the cache (a cache hit), it is retrieved; but if it does not or has expired (a cache miss), then the data is retrieved from your data store and then stored in the cache. The advantage of lazy loading is that only the requested data is cached. The disadvantage is that there is a cache miss penalty resulting in three trips:

1. The application requests data from the cache.
2. If there is a cache miss, you must query the database.
3. After data retrieval, the cache is updated.

Write through The write-through strategy adds data or updates in the cache whenever data is written to the database. The advantage of write through is that the data in the cache is never stale. The disadvantage is that there is a write penalty because every write involves two trips: a write to the cache and a write to the database. Another disadvantage is that because most data is never read in many applications, the data or information that is stored in the cluster is never used. This storage incurs a cost for space and overhead due to the duplicate data. In addition, if your data is updated frequently, the cache may be updating often, causing cache churn.

Benefits of In-Memory Data Stores

The strict performance requirements imposed by real-time applications mandate more efficient databases. Traditional databases rely on disk-based storage. A single user action may consist of multiple database calls. As they accumulate, latency increases. However, by accessing data in memory, in-memory data stores provide higher throughput and lower latency. In fact, in-memory data stores can be one to two orders of magnitude faster than disk-based databases.

As a NoSQL data store, an in-memory data store does not share the architectural limitations found in traditional relational databases. NoSQL data stores are built to be scalable. Traditional relational databases use a rigid table-based architecture. Some NoSQL data stores use a key-value store and therefore don’t enforce a structure on the data. This enables scalability and makes it easier to grow, partition, or shard data as data stores grow. When consumed as a cloud-based service, an in-memory data store also provides availability and cost benefits. On-demand access allows organizations to scale their applications as needed in response to demand spikes and at a lower cost than disk-based stores. Using managed cloud services also eliminates the need to administer infrastructure. Database hotspots are reduced, and performance becomes more predictable. Some cloud-based services also offer the benefit of high availability with replicas and support for multiple Availability Zones.

Benefits of Distributed Cache

A caching layer helps further drive throughput for read-heavy applications. A caching layer is a high-speed storage layer that stores a subset of data. When a read request is sent, the caching layer checks to determine whether it has the answer. If it doesn’t, the request is sent on to the database. Meeting read requests through the caching layer in this manner is more efficient and delivers higher performance than what can be had from a traditional database alone.

It is also more cost-effective. A single node of in-memory cache can deliver the same read throughput as several database nodes. Instead of provisioning additional instances of your traditional database to accommodate a demand spike, you can drive more throughput by adding one node of distributed cache, replacing several database nodes. The caching layer saves you money because you’re paying for one node instead of multiple database nodes, and you get the added benefit of dramatically faster performance for reads.

Redis

Redis is an increasingly popular open-source, key-value store that supports more advanced data structures, such as sorted sets, hashes, and lists. Unlike Memcached, Redis has disk persistence built in, meaning that you can use it for long-lived data. Redis also supports replication, which can be used to achieve Multi-AZ redundancy, similar to Amazon RDS.

Memcached

Memcached is a widely adopted in-memory key store. It is historically the gold standard of web caching. ElastiCache is protocol-compliant with Memcached, and it is designed to work with popular tools that you use today with existing Memcached environments. Memcached is also multithreaded, meaning that it makes good use of larger Amazon EC2 instance sizes with multiple cores.

Comparison of Memcached and Redis

Although both Memcached and Redis appear similar on the surface, in that they are both in-memory key stores, they are quite different in practice. Because of the replication and persistence features of Redis, ElastiCache manages Redis more as a relational database. Redis ElastiCache clusters are managed as stateful entities that include failover, similar to how Amazon RDS manages database failover.

Conversely, because Memcached is designed as a pure caching solution with no persistence, ElastiCache manages Memcached nodes as a pool that can grow and shrink, similar to an Amazon EC2 Auto Scaling group. Individual nodes are expendable, and ElastiCache provides additional capabilities here, such as automatic node replacement and Auto Discovery.

Consider the following requirements when deciding between Memcached and Redis.

Use Memcached if you require one or more of the following:

• Object caching is your primary goal, for example, to offload your database.
• You are interested in as simple a caching model as possible.
• You plan to run large cache nodes and require multithreaded performance with the use of multiple cores.
• You want to scale your cache horizontally as you grow.

Use Redis if you require one or more of the following:

• You are looking for more advanced data types, such as lists, hashes, and sets.
• Sorting and ranking datasets in memory help you, such as with leaderboards.
• Your application requires publish and subscribe (pub/sub) capabilities.
• Persistence of your key store is important.
• You want to run in multiple Availability Zones (Multi-AZ) with failover.
• You want transactional support, which lets you execute a group of commands as an isolated and atomic operation.