Best Practices for Designing and Using Partition Keys

When DynamoDB stores data, it divides table items into multiple physical partitions primarily based on the partition key values, and it distributes the table data accordingly. The primary key that uniquely identifies each item in a DynamoDB table can be either simple (partition key only) or composite (partition key combined with a sort key). Partition key values determine the logical partitions in which a table’s data is stored, which affects the table’s underlying physical partitions. Efficient partition key design keeps your workload spread evenly across these partitions.

A single physical DynamoDB partition supports a maximum of 3,000 read-capacity units (RCUs) or 1,000 write-capacity units (WCUs). Provisioned I/O capacity for the table is divided evenly among all physical partitions. Therefore, design your partition keys to spread I/O requests as evenly as possible across the table’s partitions to prevent “hot spots” that use provisioned I/O capacity inefficiently.

Designing Partition Keys to Distribute Even Workloads

The partition key portion of a table’s primary key determines the logical partitions in which a table’s data is stored, and it affects the underlying physical partitions. Provisioned I/O capacity for the table is divided evenly among these physical partitions, but a partition key design that does not distribute I/O requests evenly can create “hot” partitions that use your provisioned I/O capacity inefficiently and result in throttling.

The optimal usage of a table’s provisioned throughput depends on both the workload patterns of individual items and the partition key design. This does not mean that you must access all partition key values to achieve an efficient throughput level or even that the percentage of accessed partition key values must be high. It does mean that the more distinct partition key values that your workload accesses, the more those requests are spread across the partitioned space. You will use your provisioned throughput more efficiently as the ratio of partition key values accessed to the total number of partition key values increases.

Table 14.1 provides a comparison of the provisioned throughput efficiency of some common partition key schemas.

If a single table has only a small number of partition key values, consider distributing your write operations across more distinct partition key values, and structure the primary key elements to avoid one “hot” (heavily requested) partition key value that slows overall performance.

Consider a table with a composite primary key. The partition key represents the item’s creation date, rounded to the nearest day. The sort key is an item identifier. On a given day, all new items are written to that single partition key value and corresponding physical partition.

If the table fits entirely into a single partition (considering the growth of your data over time), and if your application’s read and write throughput requirements do not exceed the read and write capabilities of a single partition, your application does not encounter any unexpected throttling because of partitioning.

However, if you anticipate your table scaling beyond a single partition, architect your application so that it can use more of the table’s full provisioned throughput.

Random Suffixes in Shards

To distribute loads more evenly across a partition key space, add a random number to the end of the partition key values and then randomize the writes across the larger space. For example, if a partition key represents today’s date, choose a random number from 1 through 200, and add it as a suffix to the date. This yields partition key values such as 2018-07-09.1, 2014-07-09.2, and so on, through 2018-07-09.200. Because you are randomizing the partition key, the writes to the table on each day are spread evenly across multiple partitions. This results in better parallelism and higher overall throughput.

To read all of the items for a given day, you would have to query the items for all of the suffixes and then merge the results. For example, first issue a Query request for the partition key value 2018-07-09.1, then another Query for 2018-07-09.2, and so on, through 2018-07-09.200. When complete, your application then merges the results from all Query requests.

Calculated Suffixes in Shards

A random strategy can improve write throughput, but it is difficult to read a specific item because you do not know which suffix value was written to the item. To make it easier to read individual items, instead of using a random number to distribute the items among partitions, use a number that you can calculate based on what you want to query.

Consider the previous example, where a table uses today’s date in the partition key. Now suppose that each item has an accessible OrderId attribute and that you most often need to find items by OrderId in addition to date. Before your application writes the item to the table, it can calculate a hash suffix based on the OrderId, append it to the partition key date, and generate numbers from 1 through 200 that evenly distribute, similar to what the random strategy produces. You can use a simple calculation, such as the product of the UTF-8 code point values, for the characters in the OrderId, modulo 200, + 1. The partition key value would then be the date concatenated with the calculation result.

With this strategy, the writes are spread evenly across the partition key values and across the physical partitions. You can easily perform a GetItem operation for a particular item and date because you can calculate the partition key value for a specific OrderId value.

To read all of the items for a given day, you must query each of the 2018-07-09.N keys (where N is 1 through 200), and your application then merges the results. With this strategy, you avoid a single “hot” partition key value taking the entire workload.

Items

Each table contains zero or more items. An item is a group of attributes that is uniquely identifiable among all other entities in the table. For example, in a People table, each item represents a person, and in a Cars table, each item represents one vehicle. Items in DynamoDB are similar to rows, records, or tables in other database systems. However, in DynamoDB, there is no limit to the number of items that you can store in a table.

Attributes

Each item in a table is composed of one or more attributes. An attribute is a fundamental data element, something that does not need to be broken down any further. For example, an item in a People table contains attributes called PersonID, LastName, FirstName, and so on. For a Department table, an item may have attributes such as DepartmentID, Name, Manager, and so on. Attributes in DynamoDB are similar in many ways to fields or columns in other database systems.

The naming rules for DynamoDB tables are as follows:

• All names must be encoded using UTF-8 and be case-sensitive.
• Table names must be between 3 and 255 characters long and can contain only the following characters:
  • a–z
  • A–Z
  • 0–9
  • _ (underscore)
  • – (dash)
  • . (period)
• Attribute names must be between 1 and 255 characters long.
• Each item in the table has a unique identifier, or primary key, which distinguishes the item from all others in the table. In a People table, the primary key consists of one attribute, PersonID.
• Other than the primary key, the People table is schema-less, meaning that you are not required to define the attributes or their data types beforehand. Each item can have its own distinct attributes.
• Most of the attributes are scalar, meaning that they can have only one value. Strings and numbers are common scalars.
• Some of the items have a nested attribute. For example, in a People table, the Address attribute may have nested attributes such as Street, City, and PostalCode. DynamoDB supports nested attributes up to 32 levels deep.

Data Types

DynamoDB supports several data types for attributes within a table.

MyFavoriteThings: ["Thriller", "Purple Rain", 1983, 2]

{
  Location: "Labrynth",
  MagicStaff: 1,
  MagicRings: [
   "The One Ring",
   {
      "ElevenKings: { Quantity : 3},
      "DwarfLords: { Quantity  : 7},
      "MortalMen:  {  Quantity  : 9}
    }
  ]
}

Throughput Capacity for Reads and Writes in Tables and Indexes

When you create a table or index in DynamoDB, you must configure your capacity requirements for read and write activity. If you define the throughput capacity in advance, DynamoDB can reserve the necessary resources to meet the read and write activity that your application requires, while it ensures consistent, low-latency performance.

Throughput capacity is specified in terms of read capacity units or write capacity units:

• One read capacity unit represents one strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size. If you need to read an item larger than 4 KB, DynamoDB must consume additional read capacity units. The total number of read capacity units required depends on both the item size and whether you want an eventually consistent read or strongly consistent read.
• One write capacity unit represents one write per second for an item up to 1 KB in size. If you need to write an item larger than 1 KB, DynamoDB must consume additional write capacity units. The total number of write capacity units required depends on the item size.

• For example, if you create a table with five read capacity units and five write capacity units, your application could do the following:
  • Perform strongly consistent reads of up to 20 KB per second (4 KB × 5 read capacity units)
  • Perform eventually consistent reads of up to 40 KB per second (twice as much read throughput)
  • Write up to 5 KB per second (1 KB × 5 write capacity units)

If your application reads or writes larger items (up to the DynamoDB maximum item size of 400 KB), it will consume more capacity units.

If your read or write requests exceed the throughput settings for a table, DynamoDB can throttle that request. DynamoDB can also throttle read requests exceeds for an index. Throttling prevents your application from consuming too many capacity units. When a request is throttled, it fails with HTTP 400 code (Bad Request) and a ProvisionedThroughputExceededException. The AWS SDKs have built-in support for retrying throttled requests, so you do not need to write this logic yourself.

You can use the AWS Management Console to monitor your provisioned and actual throughput and modify your throughput settings if necessary.

DynamoDB provides the following mechanisms for managing throughput:

• DynamoDB automatic scaling
• Provisioned throughput
• Reserved capacity
• AWS Lambda triggers in DynamoDB streams

Setting Initial Throughput Settings

Every application has different requirements for reading and writing from a database. When you determine the initial throughput settings for a DynamoDB table, take the following attributes into consideration:

Item sizes Some items are small enough that they can be read or written by using a single capacity unit. Larger items require multiple capacity units. By estimating the sizes of the items that will be in your table, you can configure accurate settings for your table’s provisioned throughput.

Expected read and write request rates In addition to item size, estimate the number of reads and writes to perform per second.

Read consistency requirements Read capacity units are based on strongly consistent read operations, which consume twice as many database resources as eventually consistent reads. Determine whether your application requires strongly consistent reads, or whether it can relax this requirement and perform eventually consistent reads instead.

Item Sizes and Capacity Unit Consumption

Before you choose read and write capacity settings for your table, understand your data and how your application will access it. These inputs help you determine your table’s overall storage and throughput needs and how much throughput capacity your application will require. Except for the primary key, DynamoDB tables are schemaless, so the items in a table can all have different attributes, sizes, and data types. The total size of an item is the sum of the lengths of its attribute names and values. You can use the following guidelines to estimate attribute sizes:

• Strings are Unicode with UTF-8 binary encoding. The size of a string is as follows:

  (length of attribute name) + (number of UTF-8-encoded bytes)

• Numbers are variable length, with up to 38 significant digits. Leading and trailing zeroes are trimmed.
• The size of a number is approximately as follows:

  (length of attribute name) + (1 byte per two significant digits) + (1 byte)

• A binary value must be encoded in base64 format before it can be sent to DynamoDB, but the value’s raw byte length is used for calculating size. The size of a binary attribute is as follows:

  (length of attribute name) + (number of raw bytes)

• The size of a null attribute or a Boolean attribute is as follows:

  (length of attribute name) + (1 byte)

• An attribute of type List or Map requires 3 bytes of overhead, regardless of its contents. The size of a List or Map is as follows:

  (length of attribute name) + sum (size of nested elements) + (3 bytes).

• The size of an empty List or Map is as follows:

  (length of attribute name) + (3 bytes)

Capacity Unit Consumption for Reads

The following describes how read operations for DynamoDB consume read capacity units:

GetItem Reads a single item from a table. To determine the number of capacity units GetItem will consume, take the item size and round it up to the next 4 KB boundary. If you specified a strongly consistent read, this is the number of capacity units required. For an eventually consistent read (the default), take this number and divide it by 2.

• For example, if you read an item that is 3.5 KB, DynamoDB rounds the item size to 4 KB. If you read an item of 10 KB, DynamoDB rounds the item size to 12 KB.

BatchGetItem Reads up to 100 items, from one or more tables. DynamoDB processes each item in the batch as an individual GetItem request, so DynamoDB first rounds up the size of each item to the next 4-KB boundary and then calculates the total size. The result is not necessarily the same as the total size of all the items.

• For example, if BatchGetItem reads a 1.5-KB item and a 6.5-KB item, DynamoDB calculates the size as 12 KB (4 KB + 8 KB), not 8 KB (1.5 KB + 6.5 KB).

Query Reads multiple items that have the same partition key value. All of the items returned are treated as a single read operation, whereby DynamoDB computes the total size of all items and then rounds up to the next 4-KB boundary.

• For example, suppose that your query returns 10 items whose combined size is 40.8 KB. Amazon DynamoDB rounds the item size for the operation to 44 KB. If a query returns 1,500 items of 64 bytes each, the cumulative size is 96 KB.

Scan Reads all items in a table. DynamoDB considers the size of the items that are evaluated, not the size of the items returned by the scan.

If you perform a read operation on an item that does not exist, DynamoDB will still consume provisioned read throughput. A request for a strongly consistent read consumes one read capacity unit, whereas a request for an eventually consistent read consumes 0.5 of a read capacity unit.

For any operation that returns items, request a subset of attributes to retrieve. However, doing so has no impact on the item size calculations. In addition, Query and Scan can return item counts instead of attribute values. Getting the count of items uses the same quantity of read capacity units and is subject to the same item size calculations, because DynamoDB has to read each item to increment the count:

Read operations and read consistency The preceding calculations assumed requests for strongly consistent reads. For a request for eventually consistent reads, the operation consumes only half of the capacity units. For example, of an eventually consistent read, if the total item size is 80 KB, the operation consumes only 10 capacity units.

Read consistency for Scan A Scan operation performs eventually consistent reads, by default. This means that the Scan results might not reflect changes as the result of recently completed PutItem or UpdateItem operations. If you require strongly consistent reads, when the Scan begins, set the ConsistentRead parameter to true in the Scan request. This ensures that all of the write operations that completed before the Scan began are included in the Scan response. Setting ConsistentRead to true can be useful in table backup or replication scenarios. With DynamoDB streams, to obtain a consistent copy of the data in the table, first use Scan with ConsistentRead set to true. During the Scan, DynamoDB streams record any additional write activity that occurs on the table. After the Scan completes, apply the write activity from the stream to the table.

Capacity Unit Consumption for Writes

The following describes how DynamoDB write operations consume write capacity units:

PutItem Writes a single item to a table. If an item with the same primary key exists in the table, the operation replaces the item. For calculating provisioned throughput consumption, the item size that matters is the larger of the two.

UpdateItem Modifies a single item in the table. DynamoDB considers the size of the item as it appears before and after the update. The provisioned throughput consumed reflects the larger of these item sizes. Even if you update only a subset of the item’s attributes, UpdateItem will consume the full amount of provisioned throughput (the larger of the “before” and “after” item sizes).

DeleteItem Removes a single item from a table. The provisioned throughput consumption is based on the size of the deleted item.

BatchWriteItem Writes up to 25 items to one or more tables. DynamoDB processes each item in the batch as an individual PutItem or DeleteItem request (updates are not supported). DynamoDB first rounds up the size of each item to the next 1-KB boundary and then calculates the total size. The result is not necessarily the same as the total size of all the items. For example, if BatchWriteItem writes a 500-byte item and a 3.5-KB item, DynamoDB calculates the size as 5 KB (1 KB + 4 KB), not 4 KB (500 bytes + 3.5 KB).

For PutItem, UpdateItem, and DeleteItem operations, DynamoDB rounds the item size up to the next 1 KB. If you put or delete an item of 1.6 KB, DynamoDB rounds the item size up to 2 KB.

PutItem, UpdateItem, and DeleteItem allow conditional writes, whereby you configure an expression that must evaluate to true for the operation to succeed. If the expression evaluates to false, DynamoDB consumes write capacity units from the table.

For an existing item, the number of write capacity units consumed depends on the size of the new item. For example, a failed conditional write of a 1-KB item would consume one write capacity unit. If the new item were twice that size, the failed conditional write would consume two write capacity units.

If a ConditionExpression evaluates to false during a conditional write, DynamoDB will consume write capacity from the table based on the following conditions:

• If the item does not currently exist in the table, DynamoDB consumes one write capacity unit.
• If the item does exist, then the number of write capacity units consumed depends on the size of the item. For example, a failed conditional write of a 1-KB item would consume one write capacity unit. If the item were twice that size, the failed conditional write would consume two write capacity units.

A failed conditional write returns a ConditionalCheckFailedException. When this occurs, you do not receive any information in the response about the write capacity that was consumed. However, you can view the ConsumedWriteCapacityUnits metric for the table in Amazon CloudWatch.

To return the number of write capacity units consumed during a conditional write, use the ReturnConsumedCapacity parameter with any of the following attributes:

Total Returns the total number of write capacity units consumed.

Indexes Returns the total number of write capacity units consumed with subtotals for the table and any secondary indexes that were affected by the operation.

None No write capacity details are returned (default).

Creating Data

The following data plane operations enable you to perform create actions on data in a table:

PutItem Writes a single item to a table. You must configure the primary key attributes, but you do not have to configure other attributes.

BatchWriteItem Writes up to 25 items to a table. This is more efficient than multiple PutItem commands because your application needs only a single network round trip to write the items. You can also use BatchWriteItem to delete multiple items from one or more tables.

Performing Batch Operations

DynamoDB provides the BatchGetItem and BatchWriteItem operations for applications that need to read or write multiple items. Use these operations to reduce the number of network round trips from your application to DynamoDB. In addition, DynamoDB performs the individual read or write operations in parallel. Your applications benefit from this parallelism without having to manage concurrency or threading.

The batch operations are wrappers around multiple read or write requests. If a BatchGetItem request contains five items, DynamoDB performs five GetItem operations on your behalf. Similarly, if a BatchWriteItem request contains two put requests and four delete requests, DynamoDB performs two PutItem and four DeleteItem requests.

In general, a batch operation does not fail unless all requests in that batch fail. If you perform a BatchGetItem operation, but one of the individual GetItem requests in the batch fails, the BatchGetItem returns the keys and data from the GetItem request that failed. The other GetItem requests in the batch are not affected.

BatchGetItem A single BatchGetItem operation can contain up to 100 individual GetItem requests and can retrieve up to 16 MB of data. In addition, a BatchGetItem operation can retrieve items from multiple tables.

BatchWriteItem The BatchWriteItem operation can contain up to 25 individual PutItem and DeleteItem requests and can write up to 16 MB of data. The maximum size of an individual item is 400 KB. In addition, a BatchWriteItem operation can put or delete items in multiple tables.

Reading Data

The following data plane operations enable you to perform read actions on data in a table:

GetItem Retrieves a single item from a table. You must configure the primary key for the item that you want. You can retrieve the entire item or only a subset of its attributes.

BatchGetItem Retrieves up to 100 items from one or more tables. This is more efficient than calling GetItem multiple times because your application needs only a single network round trip to read the items.

Query Retrieves all items that have a specific partition key. You must configure the partition key value. You can retrieve entire items or only a subset of their attributes. You can apply a condition to the sort key values so that you retrieve only a subset of the data that has the same partition key.

Scan Retrieves all the items in the table or index. You can retrieve entire items or only a subset of their attributes. You can use a filter condition to return only the values that you want and discard the rest.

Updating Data

UpdateItem modifies one or more attributes in an item. You must configure the primary key for the item that you want to modify. You can add new attributes and modify or remove existing attributes. You can also perform conditional updates so that the update is successful only when a user-defined condition is met. You can also implement an atomic counter, which increments or decrements a numeric attribute without interfering with other write requests.

Deleting Data

The following data plane operations enable you to perform delete actions on data in a table:

DeleteItem Deletes a single item from a table. You must configure the primary key for the item that you want to delete.

BatchDeleteItem Deletes up to 25 items from one or more tables. This is more efficient than multiple DeleteItem calls, because your application needs only a single network round trip. You can also use BatchWriteItem to add multiple items to one or more tables.

DeleteItem

The DeleteItem deletes a single item in a table by primary key. You can perform a conditional delete operation that deletes the item if it exists, or if it has an expected attribute value.

ReturnValues: ALL_OLD

• If you delete an existing item, ALL_OLD returns the entire item as it appeared before you deleted it.
• If you delete a nonexistent item, ALL_OLD does not return any data.

Global Secondary Indexes

Some applications may need to perform many kinds of queries, using a variety of different attributes as query criteria. To support these requirements, you can create one or more global secondary indexes and then issue query requests against these indexes.

To illustrate, Figure 14.2 displays the GameScores table, which tracks users and scores for a mobile gaming application. Each item in GameScores has a partition key (UserId) and a sort key (GameTitle). Figure 14.2 shows the organization of the items.

To write a leaderboard application to display top scores for each game, you could generate a query that specifies the key attributes (UserId and GameTitle). While this would be efficient for the application to retrieve data from GameScores based on GameTitle only, it would need to use a Scan operation. As you add more items to the table, Scan operations of all the data becomes slow and inefficient, making it difficult to answer questions based on Figure 14.2, such as the following:

• What is the top score ever recorded for the game Meteor Blasters?
• Which user had the highest score for Galaxy Invaders?
• What was the highest ratio of wins versus losses?

To better implement queries on non-key attributes, create a global secondary index. A global secondary index contains a selection of attributes from the base table, but you organize them by a primary key that is different from that of the table. The index key does not require any of the key attributes from the table, nor does it require the same key schema as a table.

Every global secondary index must have a partition key and can have an optional sort key. The index key schema can be different from the base table schema. You could have a table with a simple primary key (partition key) and create a global secondary index with a composite primary key (partition key and sort key) or vice versa. The index key attributes can consist of any top-level string, number, or binary attributes from the base table but not other scalar types, document types, and set types.

Attribute Projections

A projection is the set of attributes the secondary index copies from a table. While the partition key and sort key of the table project into the index, you can also project other attributes to support your application’s Query requirements. When you query an index, DynamoDB accesses any attribute in the projection as if those attributes were in a table of their own.

When you create a secondary index, configure the attributes that project into the index. DynamoDB provides the following options:

KEYS_ONLY Each item in the index consists only of the table partition key and sort key values, plus the index key values, and this results in the smallest possible secondary index.

INCLUDE Each item in the index consists only of the table partition key and sort key values plus the index key values, and it includes other non-key attributes that you configure.

ALL Includes all attributes from the source table, including other non-key attributes that you configure. Because the table data is duplicated in the index, an ALL projection results in the largest possible secondary index.

When you choose the attributes to project into a global secondary index, consider the provisioned throughput costs and the storage costs:

• Before accessing a few attributes with the lowest possible latency, consider projecting only those attributes into a global secondary index. The smaller the index, the less it costs to store it and the lower your write costs will be.
• If your application will frequently access non-key attributes, consider projecting those attributes into a global secondary index. The additional storage costs for the global secondary index offset the cost of performing frequent table scans.
• When you’re accessing most of the non-key attributes frequently, project these attributes, or even the entire base table, into a global secondary index. This provides maximum flexibility; however, your storage cost would increase or even double.
• If your application needs to query a table infrequently but must perform many writes or updates against the data in the table, consider projecting KEYS_ONLY. The global secondary index would be of minimal size but would still be available for query activity.

Querying a Global Secondary Index

Use the Query operation to access one or more items in a global secondary index. The query must specify the name of the base table, the name of the index, the attributes the query results return, and any query conditions that you want to apply. DynamoDB can return the results in ascending or descending order.

Consider the following example in which a query requests game data for a leaderboard application:

{
    "TableName": "GameScores",
    "IndexName": "GameTitleIndex",
    "KeyConditionExpression": "GameTitle = :v_title",
    "ExpressionAttributeValues": {
        ":v_title": {"S": "Meteor Blasters"}
    },
    "ProjectionExpression": "UserId, TopScore",
    "ScanIndexForward": false
}

In this query, the following actions occur:

• DynamoDB accesses GameTitleIndex, using the GameTitle partition key to locate the index items for Meteor Blasters. All index items with this partition key are next to each other for rapid retrieval.
• Within this game, DynamoDB uses the index to access the UserID and TopScore for this game.
• The query results return in descending order, as the ScanIndexForward parameter is set to false.

Scanning a Global Secondary Index

You can use the Scan operation to retrieve the data from a global secondary index. Provide the base table name and the index name in the request. With a Scan operation, DynamoDB reads the data in the index and returns it to the application. You can also request only some of the data and to discard the residual data. To do this, use the FilterExpression parameter of the Scan operation.

Synchronizing Data between Tables and Global Secondary Indexes

DynamoDB automatically synchronizes each global secondary index with its base table. When an application writes or deletes items in a table, any global secondary indexes on that table update asynchronously by using an eventually consistent model. Though applications seldom write directly to an index, understand the following the implications of how DynamoDB maintains these indexes:

• When you create a global secondary index, you configure one or more index key attributes and their data types.
• When you write an item to the base table, the data types for those attributes must match the index key schema’s data types.
• When you put or delete items in a table, the global secondary indexes on that table update in an eventually consistent fashion.

Considerations for Provisioned Throughput of Global Secondary Indexes

When you create a global secondary index, you must configure read and write capacity units for the workload that you expect on that index. The provisioned throughput settings of a global secondary index are separate from those of its base table. A Query operation on a global secondary index consumes read capacity units from the index, not the base table.

When you put, update, or delete items in a table, the global secondary indexes on that table are updated. These index updates consume write capacity units from the index, not from the base table.

To view the provisioned throughput settings for a global secondary index, use the DescribeTable operation, and detailed information about the table’s global secondary indexes return.

Read Capacity Units

Global secondary indexes support eventually consistent reads, each of which consume one-half of a read capacity unit. For example, a single global secondary index query can retrieve up to 8 KB (2 × 4 KB) per read capacity unit. For global secondary index queries, DynamoDB calculates the provisioned read activity in the same way that it does for queries against tables, except that the calculation is based on the sizes of the index entries instead of the size of the item in the base table. The number of read capacity units is the sum of all projected attribute sizes across all returned items; the result is then rounded up to the next 4-KB boundary.

The maximum size of the results returned by a Query operation is 1 MB; this includes the sizes of all of the attribute names and values across all returned items.

For example, if a global secondary index contains items with 2,000 bytes of data and a query returns 8 items, then the total size of the matching items is 2,000 bytes × 8 items = 16,000 bytes; this is then rounded up to the nearest 4-KB boundary. Because global secondary index queries are eventually consistent, the total cost is 0.5 × (16 KB/4 KB), or two read capacity units.

Write Capacity Units

When you add, update, or delete an item in a table and a global secondary index is affected by this, then the global secondary index consumes provisioned write capacity units for the operation. The total provisioned throughput cost for a write consists of the sum of the write capacity units consumed by writing to the base table and those consumed by updating the global secondary indexes. If a write to a table does not require a global secondary index update, then no write capacity is consumed from the index.

For a table write to succeed, the provisioned throughput settings for the table and all of its global secondary indexes must have enough write capacity to accommodate the write; otherwise, the write to the table will throttle.

Factors Affecting Cost of Writes

The cost of writing an item to a global secondary index depends on the following factors:

• If you write a new item to the table that defines an indexed attribute or you update an existing item to define a previously undefined indexed attribute, one write operation is required to put the item into the index.
• If an update to the table changes the value of an indexed key attribute (from A to B), two writes are required—one to delete the previous item from the index and another write to put the new item into the index.
• If an item was present in the index, but a write to the table caused the indexed attribute to be deleted, one write is required to delete the old item projection from the index.
• If an item is not present in the index before or after the item is updated, there is no additional write cost for the index.
• If an update to the table changes the value of only projected attributes in the index key schema but does not change the value of any indexed key attribute, then one write is required to update the values of the projected attributes into the index.

All of these factors assume that the size of each item in the index is less than or equal to the 1-KB item size for calculating write capacity units. Larger index entries require additional write capacity units. Minimize your write costs by considering which attributes your queries must return and projecting only those attributes into the index.

Considerations for Storing Global Secondary Indexes

When an application writes an item to a table, DynamoDB automatically copies the correct subset of attributes to any global secondary indexes in which those attributes should appear. Your account is charged for storing the item in the base table and also for storing attributes in any global secondary indexes on that table.

The amount of space used by an index item is the sum of the following:

• Size in bytes of the base table primary key (partition key and sort key)
• Size in bytes of the index key attribute
• Size in bytes of the projected attributes (if any)
• 100 bytes of overhead per index item

To estimate the storage requirements for a global secondary index, estimate the average size of an item in the index and then multiply by the number of items in the base table that have the global secondary index key attributes.

If a table contains an item for which a particular attribute is not defined but that attribute is defined as an index partition key or sort key, DynamoDB does not write any data for that item to the index.

Managing Global Secondary Indexes

Global secondary indexes require you to create, describe, modify, delete, and detect index key violations.

Creating a Table with Global Secondary Indexes

To create a table with one or more global secondary indexes, use the CreateTable operation with the GlobalSecondaryIndexes parameter. For maximum query flexibility, create up to five global secondary indexes per table. Specify one attribute to act as the index partition key. You can specify another attribute for the index sort key. It is not necessary for either of these key attributes to be the same as a key attribute in the table.

Each index key attribute must be a scalar of type string, number, or binary, and cannot be a document or a set. You can project attributes of any data type into a global secondary index, including scalars, documents, and sets. You must also provide ProvisionedThroughput settings for the index, consisting of ReadCapacityUnits and WriteCapacityUnits. These provisioned throughput settings are separate from those of the table but behave in similar ways.

Viewing the Status of Global Secondary Indexes on a Table

To view the status of all the global secondary indexes on a table, use the DescribeTable operation. The GlobalSecondaryIndexes portion of the response shows all indexes on the table, along with the current status of each (IndexStatus).

The IndexStatus for a global secondary index is as follows:

Creating Index is currently being created, and it is not yet available for use.

Active Index is ready for use, and the application can perform Query operations on the index.

Updating Provisioned throughput settings of the index are being changed.

Deleting Index is currently being deleted, and it can no longer be used.

When DynamoDB has finished building a global secondary index, the index status changes from Creating to Active.

Adding a Global Secondary Index to an Existing Table

To add a global secondary index to an existing table, use the UpdateTable operation with the GlobalSecondaryIndexUpdates parameter, and provide the following information:

• An index name, which must be unique among all of the indexes on the table.
• The key schema of the index. Configure one attribute for the index partition key. You can configure another attribute for the index sort key. It is not necessary for either of these key attributes to be the same as a key attribute in the table. The data types for each schema attribute must be scalar: string, number, or binary.
• The attributes to project from the table into the index include the following:

  KEYS_ONLY Each item in the index consists of only the table partition key and sort key values, plus the index key values.

  INCLUDE In addition to the attributes described in KEYS_ONLY, the secondary index includes other non-key attributes that you configure.

  ALL The index includes all attributes from the source table.

• The provisioned throughput settings for the index, consisting of ReadCapacityUnits and WriteCapacityUnits. These provisioned throughput settings are separate from those of the table.

Resource Allocation

DynamoDB allocates the compute and storage resources to build the index. During the resource allocation phase, the IndexStatus attribute is CREATING and the Backfilling attribute is false. Use the DescribeTable operation to retrieve the status of a table and all of its secondary indexes.

While the index is in the resource allocation phase, you cannot delete its parent table, nor can you modify the provisioned throughput of the index or the table. You cannot add or delete other indexes on the table; however, you can modify the provisioned throughput of these other indexes.

Backfilling

For each item in the table, DynamoDB determines which set of attributes to write to the index based on its projection (KEYS_ONLY, INCLUDE, or ALL). It then writes these attributes to the index. During the backfill phase, DynamoDB tracks items that you add, delete, or update in the table and the attributes in the index.

During the backfilling phase, the IndexStatus attribute is CREATING and the Backfilling attribute is true. Use the DescribeTable operation to retrieve the status of a table and all of its secondary indexes.

While the index is backfilling, you cannot delete its parent table. However, you can still modify the provisioned throughput of the table and any of its global secondary indexes.

When the index build is complete, its status changes to Active. You are not able to query or scan the index until it is Active.

Detecting and Correcting Index Key Violations

Throughout the backfill phase of the global secondary index creation, DynamoDB examines each item in the table to determine whether it is eligible for inclusion in the index, because noneligible items cause index key violations. In these cases, the items remain in the table, but the index will not have a corresponding entry for that item.

An index key violation occurs if:

• There is a data type mismatch between an attribute value and the index key schema data type. For example, in Figure 14.1, if one of the items in the GameScores table had a TopScore value of type “string,” and you add a global secondary index with a number-type partition key of TopScore, the item from the table would violate the index key.
• An attribute value from the table exceeds the maximum length for an index key attribute. The maximum length of a partition key is 2,048 bytes, and the maximum length of a sort key is 1,024 bytes. If any of the corresponding attribute values in the table exceed these limits, the item from the table violates the index key.

If an index key violation occurs, the backfill phase continues without interruption; however, any violating items are not included in the index. After the backfill phase completes, all writes to items that violate the new index’s key schema will be rejected.

Deleting a Global Secondary Index from a Table

You use the UpdateTable operation to delete a global secondary index. While the global secondary index is being deleted, there is no effect on any read or write activity in the parent table, and you can still modify the provisioned throughput on other indexes. You can delete only one global secondary index per UpdateTable operation.

Local Secondary Indexes

Some applications query data by using only the base table’s primary key; however, there may be situations where an alternate sort key would be helpful. To give your application a choice of sort keys, create one or more local secondary indexes on a table and issue Query or Scan requests against these indexes.

For example, Figure 14.3 is useful for an application such as discussion forums. The figure shows how the items in the table would be organized.

DynamoDB stores all items with the same partition key value contiguously. In this example, given a particular ForumName, a Query operation could immediately locate the threads for that forum. Within a group of items with the same partition key value, the items are sorted by sort key value. If the sort key (Subject) is also provided in the Query operation, DynamoDB can narrow the results that are returned, such as returning the threads in the S3 forum that have a Subject beginning with the letter a.

Requests may require more complex data-access patterns, such as the following:

• Which forum threads receive the most views and replies?
• Which thread in a particular forum contains the largest number of messages?
• How many threads were posted in a particular forum, within a particular time period?

To answer these questions, the Query action would not be sufficient. Instead, you must scan the entire table. For a table with millions of items, this would consume a large amount of provisioned read throughput and time. However, you can configure one or more local secondary indexes on non-key attributes, such as Replies or LastPostDateTime.

A local secondary index maintains an alternate sort key for a given partition key value. A local secondary index also contains a copy of some, or all, of the attributes from its base table. You configure which attributes project into the local secondary index when you create the table. The data in a local secondary index is organized by the same partition key as the base table but with a different sort key. This enables you to access data items efficiently across this different dimension. For greater Query or Scan flexibility, create up to five local secondary indexes per table.

If an application locates the threads that have been posted within the last three months but lacks a local secondary index, the application must scan the entire thread table and discard any posts that were not listed within the specified time frame. With a local secondary index, a Query operation could use LastPostDateTime as a sort key and find the data quickly.

Figure 14.4 shows a local secondary index named LastPostIndex. The partition key is the same as that of the Thread table (see Figure 14.3), but the sort key is LastPostDateTime.

Every local secondary index must meet the following conditions:

• The partition key is the same as that of its base table.
• The sort key consists of exactly one scalar attribute.
• The sort key of the base table projects into the index, where it acts as a non-key attribute.

In Figure 14.4, the partition key is ForumName, and the sort key of the local secondary index is LastPostDateTime. In addition, the sort key value from the base table (Subject) projects into the index, but it is not a part of the index key. If an application needs a list that is based on ForumName and LastPostDateTime, it can issue a Query request against LastPostIndex. The query results sort by LastPostDateTime and can return in ascending or descending order. The query can also apply key conditions, such as returning only items that have a LastPostDateTime within a particular time span.

Every local secondary index automatically contains the partition and sort keys from its base table, so you can project non-key attributes into the index. When you query the index, DynamoDB can retrieve these projected attributes efficiently. When you query a local secondary index, the Query operation can also retrieve attributes that do not project into the index. DynamoDB automatically collects these attributes from the base table but at a greater latency and with higher provisioned throughput costs.

Creating a Local Secondary Index

To create one or more local secondary indexes on a table, use the LocalSecondaryIndexes parameter of the CreateTable operation. You create local secondary indexes when you create the table. When you delete a table, any local secondary indexes on that table are also deleted. Configure one non-key attribute to act as the sort key of the local secondary index. The local secondary indexes attribute is scalar and includes string, number, binary, document types, and set types. You can project attributes of any data type into a local secondary index.

Querying a Local Secondary Index

In a DynamoDB table, the combined partition key value and sort key value for each item must be unique. However, in a local secondary index, the sort key value does not need to be unique for a given partition key value. If there are multiple items in the local secondary index that have the same sort key value, a Query operation returns all items with the same partition key value. In the response, the items that the query locates do not return in any particular order.

You can query a local secondary index using eventually consistent or strongly consistent reads. To configure which type of consistency you want, use the ConsistentRead parameter of the Query operation. A strongly consistent read from a local secondary index returns the latest updated values. If the Query operation must collect additional attributes from the base table, those attributes will be consistent with respect to the index.

Scanning a Local Secondary Index

You can use the Scan function to retrieve all data from a local secondary index. Provide the base table name and the index name in the request. With a Scan function, DynamoDB reads the data in the index and returns it to the application. You can also scan for specific data to return and discard the other data using the FilterExpression parameter of the Scan API.

Item Writes and Local Secondary Indexes

DynamoDB automatically keeps all local secondary indexes synchronized with their respective base tables. Applications seldom write directly to an index. However, understand the implications of how DynamoDB maintains these indexes.

When you create a local secondary index, configure an attribute to serve as the sort key for the index and configure a data type for that attribute. Whenever you write an item to the base table, if the item defines an index key attribute, its type must match the index key schema’s data type.

There is no requirement for a one-to-one relationship between the items in a base table and the items in a local secondary index. This behavior can be advantageous for many applications, because a table with many local secondary indexes incurs higher costs for write activity than tables with fewer indexes.

Provisioned Throughput for Local Secondary Indexes

When you create a table in DynamoDB, you provision read and write capacity units for the table’s expected workload, which includes read and write activity on the table’s local secondary indexes.

Storage for Local Secondary Indexes

When an application writes an item to a table, DynamoDB automatically copies the correct subset of attributes to any local secondary indexes in which those attributes should appear. Your account is charged for storing the item in the base table and also for storing attributes in any local secondary indexes on that table.

The amount of space used by an index item is the sum of the following elements:

• Size in bytes of the base table primary key (partition and sort key)
• Size in bytes of the index key attribute
• Size in bytes of the projected attributes (if any)
• 100 bytes of overhead per index item

To estimate the storage requirements for a local secondary index, estimate the average size of an item in the index and then multiply by the number of items in the base table.

If a table contains an item where a particular attribute is not defined but that attribute is defined as an index sort key, then DynamoDB does not write any data for that item to the index.

DynamoDB Cross-Region Replication

You can create tables that automatically replicate across two or more AWS Regions with full support for multi-master writes. Using cross-region replication, you can build fast, massively scaled applications for a global user base without having to manage the replication process.

DynamoDB Stream Endpoints

AWS maintains separate endpoints for DynamoDB and DynamoDB Streams. To work with database tables and indexes, your application must access a DynamoDB endpoint. To read and process DynamoDB Streams records, your application must access a DynamoDB Streams endpoint in the same AWS Region.

Figure 14.5 shows the DynamoDB endpoint flow.

The naming convention for DynamoDB Streams endpoints is streams.dynamodb.<region>.amazonaws.com. For example, if you use the endpoint dynamodb.us-west-2.amazonaws.com to access DynamoDB, use the endpoint streams .dynamodb.us-west-2.amazonaws.com to access DynamoDB Streams.

The AWS SDKs provide separate clients for DynamoDB and DynamoDB Streams. Depending on your requirements, your application can access a DynamoDB endpoint, a DynamoDB Streams endpoint, or both at the same time. To connect to both endpoints, your application must instantiate two clients: one for DynamoDB and one for DynamoDB Streams.

Enabling a Stream

You can enable a stream on a new table when you create it, enable or disable a stream on an existing table, or change the settings of a stream. DynamoDB Streams operates asynchronously, so there is no performance impact on a table if you enable a stream.

You can also use the CreateTable or UpdateTable APIs to enable or modify a stream. The StreamSpecification parameter determines how the stream is configured:

• StreamEnabled Specifies whether a stream for the table is enabled (true) or disabled (false)
• StreamViewType Specifies the information that will be written to the stream whenever data in the table is modified:
  • KEYS_ONLY Only the key attributes of the modified item
  • NEW_IMAGE The entire item as it appears after it was modified
  • OLD_IMAGE The entire item as it appeared before it was modified
  • NEW_AND_OLD_IMAGES Both the new and the old images of the item

You can enable or disable a stream at any time. However, if you attempt to enable a stream on a table that already has a stream, you will receive a ResourceInUseException. If you attempt to disable a stream on a table that does not have a stream, you will receive a ValidationException.

When you set StreamEnabled to true, DynamoDB creates a new stream with a unique stream descriptor. If you disable and then re-enable a stream on the table, a new stream is created with a different stream descriptor.

The Amazon Resource Name (ARN) uniquely identifies every stream. The following is an example of defining an ARN for a stream on a DynamoDB table named TestTable:

arn:aws:dynamodb:us-west-2:111122223333:table/TestTable/stream/ 2015-05-11T21:21:33.291

To determine the latest stream descriptor for a table, issue a DynamoDB DescribeTable request and look for the LatestStreamArn element in the response.

Reading and Processing a Stream

To read and process a stream, your application must connect to a DynamoDB Streams endpoint and issue API requests. A stream consists of stream records. Each stream record represents a single data modification in the DynamoDB table to which the stream belongs. Each stream record is assigned a sequence number, reflecting the order in which the record was published to the stream.

Stream records are organized into groups called shards. Each shard acts as a container for multiple stream records and contains information required for accessing and iterating through these records. The stream records within a shard are removed automatically after 24 hours. If you disable a stream, any shards that are open are closed.

Shards are ephemeral, meaning that they can be both created and deleted automatically as necessary. Any shard can automatically split into multiple new shards, and a shard may split in response to high levels of write activity on its parent table to enable applications to process records from multiple shards in parallel.

It is equally possible for a parent shard to have only one child shard. Because shards have a parent-and-children lineage, an application must always process a parent shard before it processes a child shard. This ensures that the stream records process in the correct order.

If you use the DynamoDB Streams Kinesis Adapter, this processing is handled for you. Your application processes the shards and stream records in the correct order, and it automatically handles new or expired shards and shards that split while the application is running.

Figure 14.6 shows the relationship between a stream, shards in the stream, and stream records in the shards.

To access a stream and process the stream records, do the following:

1. Identify the unique ARN of the stream that you want to access.
2. Determine which shard or shards in the stream contain the stream records of interest.
3. Access the shard or shards and retrieve the stream records that you want.

Read Capacity for DynamoDB Streams

DynamoDB is available in multiple AWS Regions around the world. Each region is independent and isolated from other AWS Regions. For example, a table called People in the us-east-2 Region and a table named People in the us-west-2 Region are two entirely separate tables.

Every AWS Region consists of multiple, distinct locations called Availability Zones. Each Availability Zone is isolated from failures in other Availability Zones and provides inexpensive, low-latency network connectivity to other Availability Zones in the same region. This allows rapid replication of your data among multiple Availability Zones in a region.

When an application writes data to a DynamoDB table and receives an HTTP 200 response (OK), all copies of the data are updated. The data is eventually consistent across all storage locations, usually within one second or less.

Eventually consistent reads DynamoDB supports eventually consistent reads and strongly consistent reads. When you read data from a DynamoDB table, the response may not reflect the results of a recently completed write operation and may include some stale data. If you repeat your read request after a short period, the response returns the latest data.

Strongly consistent reads DynamoDB uses eventually consistent reads unless you specify otherwise. Read operations, such as GetItem, query, and Scan, provide a ConsistentRead parameter. If you set this parameter to true, DynamoDB uses strongly consistent reads during the operation. When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. A strongly consistent read may not be available if there is a network delay or outage.

DynamoDB Streams API

The DynamoDB Streams API provides the following operations:

ListStreams Returns a list of stream descriptors for the current account and endpoint, or you can request only the stream descriptors for a particular table name.

DescribeStream Returns information about a stream, such as its ARN, and where your application can begin to read the first few stream records. The output includes a list of shards associated with the stream, including the shard IDs.

GetShardIterator Returns a shard iterator, which describes a location within a shard, to retrieve the records from the stream. You can request that the iterator provide access to the oldest point, the newest point, or a particular point in the stream.

GetRecords Retrieves one or more stream records by using a given shard iterator. Provides the shard iterator returned from a GetShardIterator request.

Data Retention Limit for DynamoDB Streams

All data in DynamoDB Streams is subject to a 24-hour lifetime. You can retrieve and analyze the last 24 hours of activity for any given table; however, data older than 24 hours is susceptible to trimming (removal) at any moment.

If you disable a stream on a table, the data in the stream continues to be readable for 24 hours. After this time, the data expires, and the stream records are deleted automatically. There is no mechanism for manually deleting an existing stream; you must wait until the retention limit expires (24 hours) and all of the stream records are deleted.

AWS Lambda Triggers in DynamoDB Streams

DynamoDB integrates with AWS Lambda, so you can create triggers (code that executes automatically) that automatically respond to events in DynamoDB Streams. With triggers, you can build applications that react to data modifications in DynamoDB tables.

If you enable DynamoDB Streams on a table, you can associate the stream Amazon ARN with a Lambda function that you write. Immediately after an item in the table is modified, a new record appears in the table’s stream. Lambda polls the stream and invokes your Lambda function synchronously when it detects new stream records.

The Lambda function can perform any actions that you configure, such as sending a notification or initiating a workflow. For instance, you can write a Lambda function to copy each stream record to persistent storage, such as Amazon Simple Storage Service (Amazon S3), to create a permanent audit trail of write activity in your table.

Query so you can create triggers that automatically respond to events in DynamoDB Streams. With triggers, you can build applications that react to data modifications in DynamoDB tables.

If you enable DynamoDB Streams on a table, you can associate the stream ARN with a Lambda function that you write. Immediately after an item in the table is modified, a new record appears in the table’s stream. Lambda polls the stream and invokes your Lambda function synchronously when it detects new stream records.

How DynamoDB Auto Scaling Works

The steps in Figure 14.8 summarize the automatic scaling process:

1. Create an Application Auto Scaling policy for your DynamoDB table.
2. DynamoDB publishes consumed capacity metrics to Amazon CloudWatch.
3. If the table’s consumed capacity exceeds your target utilization (or falls below the target) for a specific length of time, CloudWatch triggers an alarm. You can view the alarm on the AWS Management Console and receive notifications using Amazon Simple Notification Service (Amazon SNS).
4. The CloudWatch alarm invokes Application Auto Scaling to evaluate your scaling policy.
5. Application Auto Scaling issues an UpdateTable request to adjust your table’s provisioned throughput.
6. DynamoDB processes the UpdateTable request, increasing or decreasing the table’s provisioned throughput capacity dynamically so that it approaches your target utilization.

DynamoDB automatic scaling modifies provisioned throughput settings only when the actual workload stays elevated or depressed for a sustained period of several minutes. The Application Auto Scaling target tracking algorithm seeks to keep the target utilization at or near your chosen value over the long term. Sudden, short-duration spikes of activity are accommodated by the table’s built-in burst capacity.

Burst Capacity

DynamoDB provides some flexibility in your per-partition throughput provisioning by providing burst capacity. Whenever you are not fully using a partition’s throughput, Amazon DynamoDB reserves a portion of that unused capacity for later bursts of throughput to handle usage spikes.

DynamoDB currently retains up to 5 minutes (300 seconds) of unused read and write capacity. During an occasional burst of read or write activity, these extra capacity units can be consumed quickly—even faster than the per-second provisioned throughput capacity that you have defined for your table. However, do not rely on burst capacity being available at all times, as DynamoDB can also consume burst capacity for background maintenance and other tasks without prior notice.

To enable DynamoDB automatic scaling, you create a scaling policy. This scaling policy specifies the table or global secondary index that you want to manage, which capacity type to manage (read or write capacity), the upper and lower boundaries for the provisioned throughput settings, and your target utilization.

When you create a scaling policy, Application Auto Scaling creates a pair of CloudWatch alarms on your behalf. Each pair represents your upper and lower boundaries for provisioned throughput settings. These CloudWatch alarms are triggered when the table’s actual utilization deviates from your target utilization for a sustained period of time.

When one of the CloudWatch alarms is triggered, Amazon SNS sends you a notification (if you have enabled it). The CloudWatch alarm then invokes Application Auto Scaling, which notifies DynamoDB to adjust the table’s provisioned capacity upward or downward, as appropriate.

Considerations for DynamoDB Auto Scaling

Before you begin using DynamoDB automatic scaling, be aware of the following:

• DynamoDB automatic scaling can increase read capacity or write capacity as often as necessary in accordance with your automatic scaling policy. All DynamoDB limits remain in effect.
• DynamoDB automatic scaling does not prevent you from manually modifying provisioned throughput settings. These manual adjustments do not affect any existing CloudWatch alarms that are related to DynamoDB automatic scaling.
• If you enable DynamoDB automatic scaling for a table that has one or more global secondary indexes, AWS highly recommends that you also apply automatic scaling uniformly to those indexes. You can apply this by choosing Apply same settings to global secondary indexes in the AWS Management Console.

Provisioned Throughput for DynamoDB Auto Scaling

If you are not using DynamoDB automatic scaling, you must manually define your throughput requirements. Provisioned throughput is the maximum amount of capacity that an application can consume from a table or index. If your application exceeds your provisioned throughput settings, it is subject to request throttling.

Data Distribution: Partition Key

If your table has a simple primary key (partition key only), DynamoDB stores and retrieves each item based on its partition key value. To write an item to the table, DynamoDB uses the value of the partition key as input to an internal hash function. The output value from the hash function determines the partition in which the item will be stored. To read an item from the table, you must configure the partition key value for the item. DynamoDB uses this value as input to its hash function, yielding the partition in which the item can be found.

Figure 14.9 shows a table named Pets, which spans multiple partitions. The table’s primary key is AnimalType (only this key attribute is shown). DynamoDB uses its hash function to determine where to store a new item, in this case based on the hash value of the string Dog. The items are not stored in sorted order. Each item’s location is determined by the hash value of its partition key.

Data Distribution: Partition Key and Sort Key

If the table has a composite primary key (partition key and sort key), DynamoDB calculates the hash value of the partition key in the same way, but it stores the items with the same partition key value physically close together, ordered by sort key value.

To write an item to the table, DynamoDB calculates the hash value of the partition key to determine which partition should contain the item. In that partition, there could be several items with the same partition key value, so DynamoDB stores the item among the others with the same partition key in ascending order by sort key.

To read an item from the table, configure both the partition key value and sort key value. DynamoDB calculates the partition key’s hash value, yielding the partition in which the item can be found.

You can read multiple items from the table in a single Query operation, if the desired items have the same partition key value. DynamoDB returns all items with that partition key value. You can apply a condition to the sort key that only returns items within a certain range of values.

Tag Restrictions

Each tag consists of a key and a value, both of which you define. Each DynamoDB table can have only one tag with the same key, so if you attempt to add an existing tag (the same key), the existing tag value updates to the new value.

The following restrictions apply:

• Tag keys and values are case-sensitive.
• The maximum key length is 128 Unicode characters, and the maximum value length is 256 Unicode characters. The allowed character types are letters, white space, and numbers, plus the following special characters: + - = . _ : /.
• The maximum number of tags per resource is 50.
• The AWS-assigned tag names and values are automatically assigned the aws: prefix, which you cannot manually assign.
• AWS-assigned tag names do not count toward the tag limit of 50.
• User-assigned tag names have the prefix user: in the cost allocation report.
• You cannot tag a resource at the same time that you create it.

Enabling Time to Live

When you enable TTL on a table, a background job checks the TTL attribute of items to determine whether they are expired. TTL compares the current time in epoch time format to the time stored in the Time to Live attribute of an item. If the epoch time value stored in the attribute is less than the current time, the item is marked as expired and later deleted.

DynamoDB deletes expired items on a best-effort basis to ensure availability of throughput for other data operations. DynamoDB typically deletes expired items within 48 hours of expiration. The exact duration within which an item is deleted after expiration is specific to the nature of the workload and the size of the table.

Items that have expired but not deleted still show up in reads, queries, and scans. These items can be updated, and successful updates to change or remove the expiration attribute will be honored. As items are deleted, they are immediately removed from local secondary and global secondary indexes in the same eventually consistent way as a standard delete operation.

Before Using Time to Live

Before you enable TTL on a table, consider the following:

• Make sure that any existing timestamp values in the specified Time to Live attribute are correct and in the right format.
• Items with an expiration time greater than five years in the past are not deleted.
• If data recovery is a concern, back up your table.
• For a 24-hour recovery window, use DynamoDB Streams.
• For a full backup, use AWS Data Pipeline.
• Use the AWS CloudFormation to set TTL when you create a DynamoDB table.
• Use Identity and Access Management (IAM) policies to prevent unauthorized updates to the TTL attribute or configuration of the TTL feature. If you allow access to only specified actions in your existing IAM policies, ensure that your policies update to allow DynamoDB:UpdateTimeToLive for roles that need to enable or disable TTL on tables.
• Consider whether you must complete any post-processing of deleted items. The stream’s records of TTL deletes are marked, and you use AWS Lambda function to monitor the records.

When your program sends a request, DynamoDB attempts to process it. If the request is successful, DynamoDB returns an HTTP success status code (200 OK), along with the results from the requested operation.

If the request is unsuccessful, DynamoDB returns an error. Each error has three components:

• An HTTP status code (such as 400)
• An exception name (such as ResourceNameNotFound)
• An error message (such as Requested resource not found: Table: tablename not found)

The AWS SDKs resolve propagating errors in your application so that you can take appropriate action. For example, in a Java program, write try-catch logic to handle a ResourceNotFoundException.

Error Retries and Exponential Backoff

Numerous components on a network, such as DNS servers, switches, load balancers, and others, can return error responses anywhere in the life of a given request. The usual technique for dealing with these error responses in a networked environment is to implement retries in the client application. This technique increases the reliability of the application and reduces operational costs for the developer.

As each AWS SDK automatically implements retry logic, you can modify the retry parameters to suit your needs. For example, consider a Java application that requires a fail-fast strategy with no retries allowed in case of an error. With the AWS SDK for Java, you could use the ClientConfiguration class and provide a maxErrorRetry value of 0 to turn off the retries.

If you are not using an AWS SDK, attempt to retry original requests that receive server errors (5xx). However, client errors (4xx), other than a ThrottlingException or a ProvisionedThroughputExceededException, indicate the need to revise the request itself or to correct the problem before trying again.

In addition to simple retries, each AWS SDK implements the exponential backoff algorithm for better flow control. The concept behind exponential backoff is to use progressively longer waits between retries for consecutive error responses. For example, you can set the wait to up to 50 milliseconds before the first retry, up to 100 milliseconds before the second retry, up to 200 milliseconds before third retry, and so on.

However, if the request has not succeeded after a minute, the request size may exceed your provisioned throughput and not the request rate. Set the maximum number of retries to stop at around one minute. If the request is not successful, investigate your provisioned throughput options.

Item Attributes

You can work with any attribute in an expression, even if it is deeply nested within multiple lists and maps.

Top-Level Attributes

If an attribute is not embedded within another attribute, the attribute is top level. Top-level attributes include the following:

• Id
• Title
• Description
• BicycleType
• Brand
• Price
• Color
• ProductCategory
• InStock
• QuantityOnHand
• RelatedItems
• Pictures
• ProductReviews
• Comment
• Safety.Warning

All of the top-level attributes are scalars, except for Color (list), RelatedItems (list), Pictures (map), and ProductReviews (map).

Nested Attributes

A nested attribute is embedded within another attribute. To access a nested attribute, you use dereference operators:

• [n] for list elements
• .(dot) for map elements

Expressions

In DynamoDB, you can use expressions to denote the attributes that you want to read from an item. To indicate any conditions that must be met (conditional update) and to indicate how the attributes are to be updated, you can also use expressions when writing an item.

Item Projection Expressions

To read data from a table, use operations such as GetItem, Query, or Scan. DynamoDB returns all of the item attributes by default. To acquire select attributes, use a projection expression.

A projection expression is a string that identifies the attributes that you want to collect. To retrieve a single attribute, specify its name. For multiple attributes, the names must be comma-separated.

The following are examples of projection expressions:

• Single top-level attribute:
  • Title
• Three top-level attributes; DynamoDB retrieves the entire Color set:
  • Title, Price, Color
• Four top-level attributes’ DynamoDB returns the entire contents of RelatedItems and ProductReviews:
  • Title, Description, RelatedItems, ProductReviews

Expression Attribute Names

An expression attribute name is a placeholder that you use in an expression as an alternative to an actual attribute name. An expression attribute name must begin with a # and be followed by one or more alphanumeric characters. There are several situations in which you use expression attribute names.

Reserved words On certain occasions, you might need to write an expression containing an attribute name that conflicts with a DynamoDB reserved word. Refer to https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ReservedWords.html.

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "Comment"

Attribute names containing dots In an expression, a dot (.) is interpreted as a separator character in a document path. However, DynamoDB also enables you to use a dot character as part of an attribute name, which can be ambiguous. To illustrate, suppose that you want to retrieve the Safety.Warning attribute from a table.

To work around this, replace Comment with an expression attribute name, such as #c. The # (pound sign) is required, and it indicates that this is a placeholder for an attribute name. Suppose that you want to access Safety.Warning by using a projection-expression:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "Safety.Warning"

DynamoDB returns an empty result, rather than the expected string (“Always wear a helmet”) when DynamoDB interprets a dot in an expression as a document path separator. In this case, you must define an expression attribute names (such as #sw) as a substitute for Safety.Warning. Use the following projection-expression:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "#sw" 
  --expression-attribute-names '{"#sw":"Safety.Warning"}'

DynamoDB would then return the correct result.

Nested attributes Suppose that you want to access the nested attribute ProductReviews.OneStar, using the following projection-expression:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "ProductReviews.OneStar"

The result contains all of the one-star product reviews, which is expected.

But what if you want to use a projection-expression attribute instead? For example, you want to define #pr1star as a substitute for ProductReviews.OneStar:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "#pr1star" 
  --expression-attribute-names '{"#pr1star":"ProductReviews.OneStar"}'

DynamoDB returns an empty result instead of the expected map of one-star reviews when DynamoDB interprets a dot in an expression attribute value as a character within an attribute’s name. When DynamoDB evaluates the expression attribute name #pr1star, it determines that ProductReviews.OneStar refers to a scalar attribute, which is not what was intended.

The correct approach is to define an expression-attribute-names attribute for each element in the document path:

• #pr: ProductReviews
• #1star: OneStar

You then use #pr.#1star for the projection expression:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "#pr.#1star" 
  --expression-attribute-names '{"#pr":"ProductReviews", "#1star":"OneStar"}'

DynamoDB returns the correct result.

Repeat attribute names Expression attribute names are helpful when you must refer to the same attribute name repeatedly. For example, consider the following expression for retrieving reviews from a ProductCatalog item:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "ProductReviews.FiveStar, ProductReviews.ThreeStar, ProductReviews.OneStar"

To make this more concise, replace ProductReviews with an expression attribute name, such as #pr. The revised expression looks like the following:

aws dynamodb get-item 
  --table-name ProductCatalog 
  --key '{"Id":{"N":"123"}}' 
  --projection-expression "#pr.FiveStar, #pr.ThreeStar, #pr.OneStar" 
  --expression-attribute-names '{"#pr":"ProductReviews"}'

If you define an expression attribute name, you must use it consistently throughout the entire expression. Also, you cannot omit the # symbol.

Expression Attribute Values

If you must compare an attribute with a value, define an expression attribute value as a placeholder. Expression attribute values are substitutes for the actual values that you want to compare. These are values that you might not know until runtime. Use expression attribute values with condition expressions, update expressions, and filter expressions. An expression attribute value must begin with a colon (:) followed by one or more alphanumeric characters.

For example, you want to return all of the ProductCatalog items that are available in black and cost $500 or less. You could use a Scan operation with a filter-expression, as in this AWS CLI example:

aws dynamodb scan 
  --table-name ProductCatalog 
  --filter-expression "contains(Color, :c) and Price <= :p" 
  --expression-attribute-values file://values.json

The arguments for --expression-attribute-values are stored in the file values.json:

{
  ":c": { "S": "Black" },
  ":p": { "N": "500" }
}

If you define an expression-attribute-values attribute, you must use it consistently throughout the entire expression. Also, you cannot omit the colon (:) symbol.

Condition Expressions

To manipulate data in a DynamoDB table, use the PutItem, UpdateItem, and DeleteItem operations. You can also use BatchWriteItem to perform multiple PutItem or DeleteItem operations in a single call.

For these data manipulation operations, configure a condition expression to determine which items to modify. If the condition expression evaluates to true, the operation succeeds; otherwise, the operation fails.

The following AWS CLI examples include condition expressions that use the ProductCatalog table. The partition key for this table is Id; there is no sort key. The PutItem operation creates a sample ProductCatalog item in the examples:

aws dynamodb put-item 
  --table-name ProductCatalog 
  --item file://item.json

The arguments for --item are stored in the file item.json.

{
  "Id": {"N": "456" },
  "ProductCategory": {"S": "Sporting Goods" },
  "Price": {"N": "650" }
}

Update Expressions

To update an existing item in a table, use the UpdateItem operation, provide the key of the item that you want to update, and use an update expression, indicating the attributes that you want to modify and the values that you want to assign to them.

An update expression specifies how UpdateItem modifies the attributes of an item, such as setting a scalar value or removing elements from a list or a map. An update expression consists of one or more clauses. Each clause begins with a SET, REMOVE, ADD, or DELETE keyword. You can include any of these clauses in an update expression, in any order. However, each action keyword can appear only once. Each clause contains one or more actions, separated by commas.

Each of the following actions represents a data modification:

SET Updates the expression to add one or more attributes to an item. If any of these attributes already exists, it is overwritten by the new value.

REMOVE Updates the expression to remove one or more attributes from an item. To perform multiple Remove actions, separate the attributes by commas and use Remove to delete individual elements from a list.

ADD Updates the expression to add a new attribute and its values or values to an item. If the attribute already exists, then the behavior of ADD depends on the attribute’s data type:

• If the attribute is a number and the value you are adding is also a number, then the value is mathematically added to the existing attribute. If the value is a negative number, then it is subtracted from the existing attribute.
• If the attribute is a set, and the value you are adding is also a set, then the value is appended to the existing set.

DELETE Deletes the expression.

update-expression ::=
  [ SET action [, action] ... ]
  [ REMOVE action [, action] ...]
  [ ADD action [, action] ... ]
  [ DELETE action [, action] ...]

Key Condition Expression

To specify the search criteria, use a key condition expression. A key condition expression is a string that determines the items to be read from the table or index. You must configure the partition key name and value as an equality condition.

You can use any attribute name in a key condition expression as long as the first character is a–z or A–Z and the second character (if present) is a–z, A–Z, or 0–9.

In addition, the attribute name must not be a DynamoDB reserved word. If an attribute name does not meet these requirements, then define an expression attribute name as a placeholder.

For items with a given partition key value, DynamoDB stores these items close together, sorted by the sort key value. In an aws dynamodb Query operation, DynamoDB retrieves the items in sorted order and then processes the items using KeyConditionExpression and any FilterExpression that might be present. At that point, the aws dynamodb query results are sent to the client.

An aws dynamodb query operation generally returns a result set. If no matching items are found, the result set is empty. Query results sort by the sort key value. If the data type of the sort key is Number, the results return in numeric order; otherwise, the results are returned in the order of UTF-8 bytes. The sort order is ascending by default. If you want to reverse the order, set the ScanIndexForward parameter to false. A single Query operation can retrieve a maximum of 1 MB of data. This limit applies before any FilterExpression is applied to the results. If LastEvaluatedKey is present in the response and it is non-null, then paginate the result set.

aws dynamodb query 
  --table-name Thread 
  --key-condition-expression "ForumName = :name" 
  --expression-attribute-values '{":name":{"S":"Amazon DynamoDB"}}'

Filter Expressions for Query

You can use a filter expression to refine the Query results further. A filter expression determines which items within the aws dynamodb query results return. All other results are discarded. A filter expression is applied after an aws dynamodb query finishes, but before the results are returned. Therefore, a query consumes the same amount of read capacity, regardless of whether a filter expression is used. An aws dynamodb query operation can retrieve a maximum of 1 MB of data. This limit applies before the filter expression is evaluated. A filter expression cannot contain partition key or sort key attributes. Configure those attributes in the key condition expression, not the filter expression.

aws dynamodb query 
  --table-name Thread 
  --key-condition-expression "ForumName = :fn" 
  --filter-expression "#v >= :num" 
  --expression-attribute-names '{"#v": "Views"}' 
  --expression-attribute-values file://values.json

Read Consistency for Query

A Query operation performs eventually consistent reads by default. This means that the Query results might not reflect changes as the result of recently completed PutItem or UpdateItem operations. If you require strongly consistent reads, set the ConsistentRead parameter to true in the Query request.

How Encryption Works

DynamoDB encryption at rest provides an additional layer of data protection by securing your data from unauthorized access to the underlying storage. Organizational and industry policies, or government regulations and compliance requirements, might require the use of encryption at rest to protect your data. You can use encryption to increase the data security of the applications that you deploy to the cloud.

With encryption at rest, you can enable encryption for all of your DynamoDB data at rest, including the data that is persisted in your DynamoDB tables, local secondary indexes, and global secondary indexes. Encryption at rest encrypts your data by using 256-bit AES encryption, also known as AES-256 encryption. It works at the table level and encrypts both the base table and its indexes.

Encryption at rest automatically integrates with AWS KMS for managing the service default key to encrypt your tables. If a service default key does not exist when you create your encrypted DynamoDB table, AWS KMS automatically creates a new key. Encrypted tables that you create in the future use this key. AWS KMS combines secure, highly available hardware and software to provide a key management system scaled for the cloud.

Using the same AWS KMS service default key that encrypts the table, the following elements are also encrypted:

• DynamoDB base tables
• Local secondary indexes
• Global secondary indexes

After you encrypt your data, DynamoDB handles decryption of your data transparently with minimal impact on performance. You do not need to modify your applications to use encryption.

Considerations for Encryption at Rest

Before you enable encryption at rest on a DynamoDB table, consider the following:

• When you enable encryption for a table, all the data stored in that table is encrypted. You cannot encrypt only a subset of items in a table.
• DynamoDB uses a service default key for encrypting all of your tables. If this key does not exist, it is created for you. Remember, you cannot disable service default keys.
• Encryption at rest encrypts data only while it is static (at rest) on a persistent storage media. If data security is a concern for data in transit or data in use, you must take the following additional measures:
  • Data in transit: Protect your data while it is actively moving over a public or private network by encrypting sensitive data on the client side or by using encrypted connections, such as HTTPS, Secure Socket Layer (SSL), Transport Layer Security (TLS), and File Transfer Protocol Secure (FTPS).
  • Data in use: Protect your data before sending it to DynamoDB by using client-side encryption.
  • On-demand backup and restore: You can use on-demand backup and restore with encrypted tables, and you can create a backup of an encrypted table. The table that is restored with this backup has encryption enabled.

IAM Policy Conditions for Fine-Grained Access Control

When you grant permissions in DynamoDB, you can configure conditions that determine how a permissions policy takes effect. In DynamoDB, you can specify conditions when granting permissions by using an IAM policy. For example, you can set the following configurations:

• Grant permissions to allow users read-only access to certain items and attributes in a table or a secondary index.
• Grant permissions to allow users write-only access to certain attributes in a table, based on the identity of that user.

In DynamoDB, you can specify conditions in an IAM policy by using condition keys.

Examples of Permissions

In addition to controlling access to DynamoDB API actions, you can also control access to individual data items and attributes. For example, you can do the following:

• Grant permissions on a table but restrict access to specific items in that table based on certain primary key values. An example might be a social networking application for games, where all users’ game data is stored in a single table, but no users can access data items that they do not own, as shown in Figure 14.12.

  

• Hide information so that only a subset of attributes is visible to the user. An example might be an application that displays flight data for nearby airports based on the user’s location. Airline names, arrival and departure times, and flight numbers are displayed. However, attributes such as pilot names or number of passengers are hidden, as shown in Figure 14.13.

To implement this kind of fine-grained access control, write an IAM permissions policy that specifies conditions for accessing security credentials and the associated permissions and then apply the policy to IAM users, groups, or roles. Your IAM policy can restrict access to individual items in a table, access to the attributes in those items, or both at the same time.

Use the IAM condition element to implement a fine-grained access control policy. By adding a condition element to a permissions policy, you can allow or deny access to items and attributes in DynamoDB tables and indexes, based on your particular business requirements.

For example, in Figure 14.1, the game lets players select from and play a variety of games. The application uses a DynamoDB table named GameScores to track high scores and other user data. Each item in the table is uniquely identified by a user ID and the name of the game that the user played. The GameScores table has a primary key consisting of a partition key (UserId) and sort key (GameTitle). Users have access only to game data associated with their user ID. A user who wants to play a game must belong to an IAM role named GameRole, which has a security policy attached to it.

To manage user permissions in this application, you could write a permissions policy such as the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAccessToOnlyItemsMatchingUserID",
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:BatchGetItem",
        "dynamodb:Query",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem",
        "dynamodb:BatchWriteItem"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-west-2:123456789012:table/GameScores"
      ],
      "Condition": {
        "ForAllValues:StringEquals": {
          "dynamodb:LeadingKeys": [
            "${www.amazon.com:user_ID}"
              
          ],
          "dynamodb:Attributes": [
            "UserId",
            "GameTitle",
            "Wins",
            "Losses",
            "TopScore",
            "TopScoreDateTime"
          ]
        },
        "StringEqualsIfExists": {
          "dynamodb:Select": "SPECIFIC_ATTRIBUTES"
        }
      }
    }
  ]
}

In addition to granting permissions for specific DynamoDB actions (Action element) on the GameScores table (Resource element), the Condition element uses the condition keys specific to DynamoDB that limit the permissions as follows:

dynamodb:LeadingKeys This condition key enables users to access only the items where the partition key value matches their user ID. This ID, ${www.amazon.com:user_ID}, is a substitution variable.

dynamodb:Attributes This condition key limits access to the specified attributes so that only the actions listed in the permissions policy can return values for these attributes. In addition, the StringEqualsIfExists clause ensures that the application provides a list of specific attributes to act upon, and that the application cannot request all attributes.

When an IAM policy is evaluated, the result is either true (access is allowed) or false (access is denied). If any part of the Condition element is false, the entire policy evaluates to false and access is denied.

Configure Conditions with Condition Keys

AWS provides a set of predefined condition keys for all AWS offerings and services that support IAM for access control. For example, use the aws:SourceIp condition key to check the requester’s IP address before allowing an action to be performed.

Table 14.3 displays the DynamoDB service-specific condition keys that apply to DynamoDB.

Backups

When you create an on-demand backup, a time marker of the request is cataloged. The backup is created asynchronously by applying all changes until the time of the request to the last full table snapshot. Backup requests process instantaneously and become available for restore within minutes. Each time you create an on-demand backup, the entire table data is backed up. You can make an unlimited number of on-demand backups.

All backups in DynamoDB work without consuming any provisioned throughput on the table. DynamoDB backups do not enable causal consistency across items; however, the skew between updates in a backup is usually much less than a second. While a backup is in progress, you cannot perform certain operations, such as pausing or canceling the backup action, deleting the source table of the backup, or disabling backups on a table. However, you can use AWS Lambda functions to schedule periodic or future backups.

Restores

A table restores without consuming any provisioned throughput on the table. The destination table is set with the same provisioned read capacity units and write capacity units as the source table, as recorded at the time that you request the backup. The restore process also restores the local secondary indexes and the global secondary indexes.

You can only restore the entire table data to a new table from a backup. Restore times vary based on the size of the DynamoDB table that is being restored. You can write to the restored table only after it becomes active, and you cannot overwrite an existing table during a restore operation. You can use IAM policies for access control.

Point-in-Time Recovery

You can enable point-in-time recovery (PITR) and create on-demand backups for your DynamoDB tables. Point-in-time recovery helps protect your DynamoDB tables from accidental write or delete operations. With point-in-time recovery, you do not have to worry about creating, maintaining, or scheduling on-demand backups. DynamoDB maintains incremental backups of your table. In addition, point-in-time operations do not affect performance or API latencies. You can enable point-in-time recovery using the AWS Management Console, AWS CLI, or the DynamoDB API.

How Point-in-Time Recovery Works

When it is enabled, point-in-time recovery provides continuous backups until you explicitly turn it off. After you enable point-in-time recovery, you can restore to any point in time within EarliestRestorableDateTime and LatestRestorableDateTime. LatestRestorableDateTime is typically 5 minutes before the current time. The point-in-time recovery process always restores to a new table. For EarliestRestorableDateTime, you can restore your table to any point in time during the last 35 days. The retention period is a fixed 35 days (five calendar weeks) and cannot be modified. Any number of users can execute up to four concurrent restores (any type of restore) in a given account.

When you restore using point-in-time recovery, DynamoDB restores your table data to a new table and to the state based on the selected date and time (day:hour:minute:second). In addition to the data, the following are also included on the newly restored table using point-in-time recovery:

• Global secondary indexes
• Local secondary indexes
• Provisioned read and write capacity
• Encryption settings

After restoring a table, you must manually set up the following on the restored table:

• Scaling policies
• IAM policies
• Amazon CloudWatch metrics and alarms
• Tags
• Stream settings
• TTL settings
• Point-in-time recovery settings

Considerations for Point-in-Time Recovery

Before you enable point-in-time recovery on a DynamoDB table, consider the following:

• If you disable point-in-time recovery and then later re-enable it on a table, reset the start time for which you can recover that table. As a result, you can only immediately restore that table using the LatestRestorableDateTime.
• If you must recover a deleted table that had point-in-time recovery enabled, you must contact AWS Support to restore that table within the 35-day recovery window.
• You can enable point-in-time recovery on each local replica of a global table. When you restore the table, the backup restores to an independent table that is not part of the global table.
• You can enable point-in-time recovery on an encrypted table.
• AWS CloudTrail logs all console and API actions for point-in-time recovery to enable logging, continuous monitoring, and auditing.

Nodes

A node is the smallest building block of an ElastiCache deployment. A node is a fixed-size chunk of secure, network-attached RAM. Each node runs an instance of Memcached or Redis, depending on which you select when you create the cluster.

Clusters

Each ElastiCache deployment consists of one or more nodes in a cluster. When you create a cluster, you may choose from many different nodes based on the requirements of both your solution case and your capacity. One Memcached cluster can be as large as 20 nodes. Redis clusters consist of a single node; however, you can group multiple clusters into a Redis replication group.

The individual node types are derived from a subset of the Amazon EC2 instance type families, such as t2, m3, and r3. The t2 cache node family is ideal for development and low-volume applications with occasional bursts, but certain features may not be available. The m3 family is a mix of memory and compute, whereas the r3 family is optimized for memory-intensive workloads.

Based on your requirements, you may decide to have a few large nodes or many smaller nodes in your cluster or replication group. As demand for your application fluctuates, you may add or remove nodes over time. Each node type has a preconfigured amount of memory, with a small portion of that memory reserved for both the caching engine and operating system.

Though it is unlikely, always plan for the possible failure of an individual cache node. For a Memcached cluster, decrease the impact of the failure of a cache node by using a larger number of nodes with a smaller capacity instead of a few large nodes.

If ElastiCache detects the failure of a node, it provisions a replacement and then adds it back to the cluster. During this time, your database experiences increased load because any requests that would have been cached now need to be read from the database. For Redis clusters, ElastiCache detects failures and replaces the primary node. If you enable a Multi-AZ replication group, a read replica automatically is promoted to primary automatically to primary.

Replication group A replication group is a collection of Redis clusters with one primary read/write cluster and up to five secondary, read-only clusters called read replicas. Each read replica maintains a copy of the data from the primary cluster. Asynchronous replication mechanisms keep the read replicas synchronized with the primary cluster. Applications can read from any cluster in the replication group. Applications can write only to the primary cluster. Read replicas enhance scalability and guard against data loss.

Endpoint An endpoint is the unique address your application uses to connect to an ElastiCache node or cluster. Memcached and Redis have the following characteristics with respect to endpoints:

• A Memcached cluster has its own endpoint and a configuration endpoint.
• A standalone Redis cluster has an endpoint to connect to the cluster for both reads and writes.
• A Redis replication group has two types of endpoints.
  • The primary endpoint connects to the primary cluster in the replication group.
  • The read endpoint points to a specific cluster in the replication group.

Strategies for Caching

The strategy or strategies that you want to implement for populating and maintaining your cache depend on what data you are caching and the access patterns to that data. For example, you would likely not want to use the same strategy for a top-10 leaderboard on a gaming site, Facebook posts, and trending news stories.

Lazy Loading

Lazy loading loads data into the cache only when necessary. Whenever your application requests data, it first makes the request to the ElastiCache cache. If the data exists in the cache and it is current, ElastiCache returns the data to your application. If the data does not exist in the cache or the data in the cache has expired, your application requests the data from your data store, which returns the data to your application. Your application then writes the data received from the store to the cache so that it can be retrieved more quickly the next time that it is requested.

Advantages of Lazy Loading

• Only requested data is cached.

• Because most data is never requested, lazy loading avoids filling up the cache with data that is not requested.

• Node failures are not fatal.

• When a new, empty node replaces a failed node, the application continues to function, though with increased latency. As requests are made to the new node, each missed cache results in a query of the database and adding the data copy to the cache so that subsequent requests are retrieved from the cache.

Disadvantages of Lazy Loading

• There is a cache miss penalty.
  • Each cache miss results in three trips:
  1. Initial request for data from the cache
  2. Querying of the database for the data
  3. Writing the data to the cache
  • This can cause a noticeable delay in data getting to the application.

• Stale data.

• The application may receive stale data because another application may have updated the data in the database behind the scenes.

Figure 14.14 summarizes the advantages and disadvantages of lazy loading.

Data Access Patterns

Retrieving a flat key from an in-memory cache is faster than the most performance-tuned database query. Analyze the access pattern of the data before you determine whether you should store it in an in-memory cache.

Replication and Multi-AZ

Replication is an effective method for providing speedy recovery if a node fails and for serving high quantities of read queries beyond the capacities of a single node. ElastiCache clusters running Redis support both. In contrast, cache clusters running Memcached are standalone in-memory services that do not provide any data redundancy-protection services. Cache clusters running Redis support the notion of replication groups. A replication group consists of up to six clusters, with five of them designated as read replicas. By using a replication group, you can scale horizontally by developing code in the application to offload reads to one of the five replicas.

Multi-AZ Replication Groups

With ElastiCache, you can provision a Multi-AZ replication group that allows your application to raise the availability and reduce the loss of data. Multi-AZ streamlines the procedure of dealing with a failure by automating the replacement and failover from the primary node.

If the primary node goes down or is otherwise unhealthy, Multi-AZ selects a replica and promotes it to become the new primary; then a new node is provisioned to replace the failed one. ElastiCache updates the DNS entry of the new primary node to enable your application to continue processing without any changes to the configuration of the application and with only minimal disruption. ElastiCache replication is handled asynchronously, meaning that there will be a small delay before the data is available on all cluster nodes.

Creating a Bucket

Amazon S3 provides APIs for creating and managing buckets. By default, you can create up to 100 buckets in each of your accounts. If you need more buckets, increase your bucket limit by submitting a service limit increase.

When you create a bucket, provide a name for the bucket, and then choose the AWS Region where you want to create the bucket. You can store any number of objects in a bucket.

Create a bucket by using any of the following methods:

• Amazon S3 console
• Programmatically, using the AWS SDKs

When using the AWS SDKs, first create a client and then use the client to deliver a request to create a bucket. When you create the client, you can configure an AWS Region. US East (N. Virginia) is the default region. You can also configure an AWS Region in your request to create the bucket.

If you create a client specific to the US East (N. Virginia) Region, the client uses this endpoint to communicate with: Amazon S3: s3.amazonaws.com. You can use this client to create a bucket in any AWS Region in your create bucket request. If you do not specify a region, Amazon S3 creates the bucket in the US East (N. Virginia) Region. If you select an AWS Region, Amazon S3 creates the bucket in the specified region. If you create a client specific to any other AWS Region, it maps to the region-specific endpoint: s3-.amazonaws.com. For example, if you create a client and specify the us-east-2 Region, it maps to the following region-specific endpoint: s3-us-east-2.amazonaws.com.

Objects Objects are the principal items stored in Amazon S3. Objects consist of object data and metadata. The data part is opaque to Amazon S3. The metadata is a set of name-value pairs that characterize the object. These include certain default metadata, such as the date last modified and standard HTTP metadata, such as Content-Type. It is also possible for you to configure custom metadata at the time of object creation.

Keys A key is the unique identifier for an object within a bucket. Every object in a bucket has exactly one key. Because the combination of a bucket, key, and version ID uniquely identifies each object, Amazon S3 is like a basic data map between bucket + key + version and the object itself. Every object in Amazon S3 can be uniquely addressed through the combination of the web service endpoint, bucket name, key, and, optionally, a version. Figure 14.15 displays one object with a key and ID.

Versioning Versioning is a way to keep multiple variations of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. With versioning, you can recover from both unintended user actions and application failures.

In Figure 14.16, you can have two objects with the same key, but different version IDs, such as photo.gif (version 111111) and photo.gif (version 121212).

Versioning-enabled buckets allow you to recover objects from accidental deletion or overwrite. For instance:

• If you delete an object, instead of removing it permanently, Amazon S3 inserts a delete marker, which becomes the current object version. You can restore the previous version.
• You can delete object versions whenever you want.

In addition, you can also define lifecycle configuration rules for objects that have a well-defined lifecycle to request Amazon S3 to expire current object versions or permanently remove noncurrent object versions. When your bucket is version-enabled or versioning is suspended, the lifecycle configuration actions work as follows:

• The expiration action applies to the current object version. Instead of deleting the current object version, Amazon S3 retains the current version as a noncurrent version by adding a delete marker, which then becomes the current version.
• The NoncurrentVersionExpiration action applies to noncurrent object versions, and Amazon S3 permanently removes these object versions. You cannot recover permanently removed objects.

A Delete request has the following use cases:

• When versioning is enabled, a simple Delete request cannot permanently delete an object. Instead, Amazon S3 inserts a delete marker in the bucket, and that marker becomes the current version of the object with a new ID.
• When you send a Get request for an object whose current version is a delete marker, Amazon S3 treats it as though the object has been deleted (even though it has not been erased) and returns a 404 error. Figure 14.17 shows that a simple Delete request does not actually remove the specified object. Instead, Amazon S3 inserts a delete marker.
• To permanently delete versioned objects, you must use DELETE Object versionId. Figure 14.18 shows that deleting a specified object version permanently removes that object.
• If you overwrite an object, it results in a new object version in the bucket. You can restore the previous version.

Accessing a Bucket

Use the Amazon S3 console to access your bucket. You can perform almost all bucket operations without having to write any code. If you access a bucket programmatically, Amazon S3 supports the RESTful architecture wherein your buckets and objects are resources, each with a resource Uniform Resource Identifier (URI) that uniquely identifies the resource.

Amazon S3 supports both path-style URLs and virtual hosted-style URLs to access a bucket:

• In a virtual hosted-style URL, the bucket name is part of the domain name in the URL. Here’s an example:

• http://bucket.s3.amazonaws.com
• http://bucket.s3-aws-region.amazonaws.com

• In a path-style URL, the bucket name is not part of the domain (unless you use a region-specific endpoint). Here’s an example:

• US East (N. Virginia) Region endpoint: http://s3.amazonaws.com/bucket
• Region-specific endpoint: http://s3-aws-region.amazonaws.com/bucket

In a path-style URL, the endpoint you use must match the AWS Region where the bucket resides. For example, if your bucket is in the South America (São Paulo) Region, you must use the http://s3-saeast-1.amazonaws.com/bucket endpoint.

Bucket Restrictions and Limitations

The account that created the bucket owns it. By default, you can create up to 100 buckets in each of your accounts. If you need additional buckets, increase your bucket limit by submitting a service limit increase.

• Bucket ownership is not transferable; however, if a bucket is empty, you can delete it.

• After a bucket is deleted, the name becomes available for reuse, but the name may not be available for you to reuse for various reasons. For instance, another account could provision a bucket with the same name. Also, it might take some time before the name can be reused. Therefore, if you want to use the same bucket name after emptying the bucket, do not delete the bucket.

• There is no limit to the number of objects that you can store in a bucket and no difference in performance whether you use many buckets or only a few.

• Moreover, you can store all of your objects in one bucket, or you can arrange all of your objects across multiple buckets.

• You cannot create a bucket within another bucket.
• The high-availability engineering of Amazon S3 is focused on GET, PUT, LIST, and DELETE operations.

• Because bucket operations work against a centralized, global resource space, it is not appropriate to create or delete buckets on the high-availability code path of your application. It is better to create or delete buckets in a separate initialization or setup routine that you run less often.

Rules for Naming Buckets

After you create an S3 bucket, you cannot change the bucket name, so choose the name wisely.

The following are the rules for naming S3 buckets in all AWS Regions:

• Bucket names must be unique across all existing bucket names in Amazon S3.
• Bucket names must comply with DNS naming conventions.
• Bucket names must be between 3 and 63 characters long.
• Bucket names must not contain uppercase characters or underscores.
• Bucket names must start with a lowercase letter or number.
• Bucket names must be a series of one or more labels. Use a single period (.) to separate adjacent labels. Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.
• Bucket names must not be formatted as an IP address (for example, 192.168.4.5).
• When you use virtual hosted-style buckets with SSL, the SSL wildcard certificate only matches buckets that do not contain periods. To work around this, use HTTP, or write your own certificate verification logic. AWS recommends that you do not use periods (“.”) in bucket names when using virtual hosted-style buckets.

Working with Amazon S3 Buckets

Amazon S3 is cloud storage for the internet. To upload your data, such as images, videos, and documents, you must first create a bucket in one of the AWS Regions. You can then upload any number of objects to the bucket.

Amazon S3 is set up with buckets and objects as resources, and it includes APIs to interact with its resources. For instance, you can use the Amazon S3 API to create a bucket and then put objects in the bucket. Additionally, you can use the Amazon S3 web console to execute these actions. The console uses the Amazon S3 APIs to deliver requests to Amazon S3.

An Amazon S3 bucket name is globally unique regardless of the AWS Region in which you create the bucket. Configure the name at the time that you create the bucket. Amazon S3 creates buckets in a region that you choose. To minimize latency, reduce costs, or address regulatory requirements, choose any AWS Region that is geographically close to you.

The following are the most common operations executed through the API:

• Create a bucket Create and name the bucket where you will store your objects.
• Write an object Store data by creating or writing over an object. When you write an object, configure a unique key in the namespace of your bucket. Configure any access control that you want on the object.
• Read an object Retrieve data. You can download data through HTTP or BitTorrent.
• Delete an object Delete data.
• List keys List the keys contained in one of your buckets. You can filter the key list based on a prefix.
• Make requests Amazon S3 is a REST service. You can send requests to Amazon S3 using the REST API or the AWS SDK wrapper libraries. These libraries wrap the underlying Amazon S3 REST API, simplifying your programming tasks.
• Every interaction with Amazon S3 is either authenticated or anonymous. Authentication is a process of verifying the identity of the requester trying to access an AWS offering. Authenticated requests must include a signature value that authenticates the request sender. The signature value is, in part, generated from the requester’s AWS access keys (access key ID and secret access key).
• If you are using the AWS SDK, the libraries compute the signature from the keys you provide. However, if you make direct REST API calls in your application, you must write the code to compute the signature and then add it to the request.

Deleting or Emptying a Bucket

It is easy to delete an empty bucket. However, in certain situations, you may need to delete or empty a bucket that contains objects. In other situations, you may choose to empty a bucket instead of deleting it. This section explains various options that you can use to delete or empty a bucket that contains objects.

You can delete a bucket and its content programmatically using the AWS SDKs. You can also use lifecycle configuration on a bucket to empty its content and then delete the bucket. There are additional options, such as using Amazon S3 console and AWS CLI, but there are limitations on these methods based on the number of objects in your bucket and the bucket’s versioning status.

Bucket Configuration Options

Amazon S3 supports various options to configure your bucket. For example, you can configure your bucket for website hosting, managing the lifecycle of objects in the bucket, and enabling all access to the bucket. Amazon S3 supports subresources for you to store, and it manages the bucket configuration information. Using the Amazon S3 API, you can create and manage these subresources. You can also use the Amazon S3 console or the AWS SDKs.

Amazon S3 Consistency Model

Amazon S3 provides read-after-write consistency for PUT requests of new objects in your S3 bucket in all regions. However, if you make a HEAD or GET request to the key name (to determine whether the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.

Amazon S3 offers eventual consistency for overwriting PUT and DELETE requests in all regions.

Updates to a single key are atomic. For example, if you PUT to an existing key, a subsequent read might return the old data or the updated data, but it will not write corrupted or partial data.

Amazon S3 achieves high availability by replicating data across multiple servers within Amazon’s data centers. If a PUT request is successful, your data is safely stored. However, information about the changes must replicate across Amazon S3, which can take time, so you might observe the following behaviors:

• A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list.
• A process replaces an existing object and immediately attempts to read it. Until the change is fully propagated, Amazon S3 might return the prior data.
• A process deletes an existing object and immediately attempts to read it. Until the deletion is fully propagated, Amazon S3 might return the deleted data.
• A process deletes an existing object and immediately lists keys within its bucket. Until the deletion is fully propagated, Amazon S3 might list the deleted object.

Amazon S3 does not currently support object locking. If two PUT requests are simultaneously made to the same key, the request with the latest timestamp takes effect. If this is an issue, build an object-locking mechanism into your application. Updates are key-based; there is no way to make atomic updates across keys. For example, you cannot make the update of one key dependent on the update of another key unless you design this functionality into your application.

Table 14.5 describes the characteristics of eventually consistent reads and consistent reads.

Storage Classes for Frequently Accessed Objects

Amazon S3 provides storage classes for performance-sensitive use cases (millisecond access time) and frequently accessed data. Amazon S3 provides the following storage classes:

STANDARD Standard is the default storage class. If you do not specify the storage class when uploading an object, Amazon S3 assigns the STANDARD storage class.

REDUCED_REDUNDANCY The Reduced Redundancy Storage (RRS) storage class is designed for noncritical, reproducible data that you can store with less redundancy than the STANDARD storage class.

Regarding durability, RRS objects have an average annual expected loss of 0.01 percent of objects. If an RRS object is lost, when requests are made to that object, Amazon S3 returns a 405 error.

Storage Classes for Infrequently Accessed Objects

The STANDARD_IA and ONEZONE_IA storage classes are designed for data that is long-lived and infrequently accessed. STANDARD_IA and ONEZONE_IA objects are available for millisecond access (similar to the STANDARD storage class). Amazon S3 charges a fee for retrieving these objects; thus, they are most appropriate for infrequently accessed data.

Possible use cases for STANDARD_IA and ONEZONE_IA are as follows:

• For storing backups
• For older data that is accessed infrequently but that still requires millisecond access

These storage classes differ as follows:

STANDARD_IA Objects stored using this storage class are stored redundantly across multiple, geographically distinct Availability Zones (similar to the STANDARD storage class). STANDARD_IA objects are resilient to data loss of an Availability Zone. This storage class provides more availability, durability, and resiliency than the ONEZONE_IA class.

ONEZONE_IA Amazon S3 stores the object data in only one Availability Zone, which makes it less expensive than STANDARD_IA. However, the data is not resilient to the physical loss of the Availability Zone resulting from disasters, such as earthquakes and floods. The ONEZONE_IA storage class is as durable as STANDARD_IA, but it is less available and less resilient.

To determine when to use a particular storage class, follow these recommendations:

STANDARD_IA Use for your primary copy (or only copy) of data that cannot be regenerated.

ONEZONE_IA Use if you can regenerate the data if the Availability Zone fails.

GLACIER Storage Class

You use the GLACIER storage class to archive data where access is infrequent. Objects that you archive are not available for real-time access. The GLACIER storage class offers the same durability and resiliency as the STANDARD storage class.

When you store objects in Amazon S3 with the GLACIER storage class, Amazon S3 uses the low-cost Amazon Simple Storage Service Glacier (Amazon S3 Glacier) service to store these objects. Though the objects are stored in Amazon S3 Glacier, these remain Amazon S3 objects that are managed in Amazon S3, and they cannot be accessed directly through Amazon S3 Glacier.

At the time that you create an object, it is not possible to specify GLACIER as the storage class. The way that GLACIER objects are created is by uploading objects first using STANDARD as the storage class. You can transition these objects to the GLACIER storage class by using lifecycle management.

Setting the Storage Class of an Object

Amazon S3 APIs offer support for setting or updating the storage class of objects. When you create a new object, configure its storage class. For example, when you create objects with the PUT Object, POST Object, and Initiate Multipart Upload APIs, add the x-amz-storageclass request header to configure a storage class. If you do not add this header, Amazon S3 uses STANDARD, the default storage class.

You can also change the storage class of an object that is already stored in Amazon S3 by making a copy of the object using the PUT Object - Copy API. Copy the object in the same bucket with the same key name, and configure request headers as follows:

• Set the x-amz-metadata-directive header to COPY.
• Set the x-amz-storage-class to the storage class that you want to use.

In a versioning-enabled bucket, you cannot change the storage class of a specific version of an object. When you copy it, Amazon S3 gives it a new version ID. You can direct Amazon S3 to change the storage class of objects by adding a lifecycle configuration to a bucket.

Protecting Data Using Encryption

Data protection refers to protecting data while in transit (as it travels to and from Amazon S3), and at rest (while it is stored on disks in Amazon S3 data centers). You can protect data in transit by using SSL or by using client-side encryption with the following options of protecting data at rest in Amazon S3:

Use server-side encryption You request Amazon S3 to encrypt your object before saving it on disks in its data centers and then decrypt the object when you download it.

Use client-side encryption You can encrypt data on the client side and upload the encrypted data to Amazon S3 and then manage the encryption process, the encryption keys, and related tools.

Protecting Data Using Server-Side Encryption

Server-side encryption is about data encryption at rest; that is, Amazon S3 encrypts your data at the object level as it writes it to disks in its data centers and decrypts it for you when you access it. As long as you authenticate your request and you have access permissions, there is no difference in the way that you access encrypted or unencrypted objects. For example, if you share your objects using a presigned URL, that URL works the same way for both encrypted and unencrypted objects.

Consider the following mutually exclusive options, depending on how you choose to manage the encryption keys:

Use server-side encryption with Amazon S3 Managed Keys (SSE-S3) Each object is encrypted with a unique key employing strong multifactor encryption. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 server-side encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.

Use server-side encryption with AWS KMS Managed Keys (SSE-KMS) SSE-KMS is similar to SSE-S3, but with additional benefits and charges for using this service. There are separate permissions for the use of an envelope key (that is, a key that protects your data’s encryption key) that provides added protection against unauthorized access of your objects in Amazon S3.

SSE-KMS also provides you with an audit trail of when your key was used and by whom. Additionally, you can create and manage encryption keys yourself or use a default key that is unique to you, the service you are using, and the region in which you are working.

Use server-side encryption with customer-provided keys (SSE-C) You manage the encryption keys, and Amazon S3 manages the encryption as it writes to disks. You also manage decryption when you access your objects.

Object Keys and Metadata

Each Amazon S3 object is composed of several parts. These parts include the data, a key, and metadata. An object key (or key name) uniquely identifies the object in a bucket. Object metadata is a set of name-value pairs. You can set the object metadata at the time that you upload an object. However, after you upload the object, you cannot modify object metadata. The only way to modify object metadata after it has been uploaded is to create a copy of the object.

When you upload an object, set the key name, which uniquely identifies the object in that bucket. As an example, in the Amazon S3 console, when you select a bucket, a list of objects that reside in your bucket is displayed. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is limited to 1,024 bytes.

Object Key Naming Guidelines

Though you can use any UTF-8 characters in an object key name, the following best practices for key naming help ensure maximum compatibility with other applications. Each application might parse special characters differently. The following guidelines help you maximize compliance with DNS, web-safe characters, XML parsers, and other APIs.

The following character sets are generally safe for use in key names:

• a–z
• A–Z
• 0–9
• _ (underscore)
• – (dash)
• . (period)
• ! (exclamation point)
• * (asterisk)
• ’ (apostrophe)
• , (comma)

  The following are examples of valid object key names:

  • 2your-customer
  • your.great_photos-2018/jane/yourvacation.jpg
  • videos/2018/graduation/video1.wmv

The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of sub-buckets or subfolders; however, you can infer logical hierarchy by using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports the concept of folders.

Object Metadata

There are two kinds of object metadata: system metadata and user-defined metadata.

Object Tagging

You can use object tagging as a way to categorize your objects. Each tag is a key-value pair. The following are tagging examples:

• Suppose that an object contains protected health information (PHI) data. You might tag the object by using the following key-value pair:

  PHI=True

• Suppose that you store project files in your S3 bucket. You might tag these objects with a key called Project, and the following value:

  Project=Aqua

• You can set up multiple tags to an object, as shown in the following:

  Project=Y

  Classification=sensitive

  You can add tags to new objects and existing objects. Note the following:

• You can associate up to 10 tags with an object. Tags associated with an object must have unique tag keys.
• A tag key can be up to 128 Unicode characters in length, and tag values can be up to 256 Unicode characters in length.
• Key and values are case-sensitive.

  Object key name prefixes also enable you to categorize storage. However, prefix-based categorization is one-dimensional. Consider the following object key names:

• images/photo1.jpg
• projects/accountingproject/document.pdf
• project/financeproject/document2.pdf

These key names have the prefixes images/, projects/accountingproject/, and projects/financeproject/. These prefixes enable one-dimensional categorization; that is, everything under a prefix is one category. For example, the prefix projects/accountingproject identifies all documents related to the project accountingproject.

Tagging creates another dimension. If you want photo1 in the project financeproject category, tag the object accordingly. In addition to data classification, tagging offers the following additional benefits:

• Object tags enable fine-grained access control of permissions. For instance, you could grant an IAM user permissions to read-only objects with certain tags.
• Object tags enable fine-grained object lifecycle policies where you can configure tag-based filters, in addition to key name prefixes, in a lifecycle policy.
• When using Amazon S3 analytics, you can configure filters to group objects together for analysis by object tags, by key name prefix, or by both prefix and tags.
• You can customize Amazon CloudWatch metrics to display information by specific tag filters.

Operations on Objects

Amazon S3 enables you to store (POST and PUT), retrieve (GET), and delete (DELETE) objects. You can retrieve an entire object or a portion of an object. If you have enabled versioning on the bucket, you can retrieve a specific version of the object. You can also retrieve a subresource associated with your object and update it where applicable. You can make a copy of your existing object.

Depending on the size of the object, consider the following when uploading or copying a file:

Uploading objects You can upload objects of up to 5 GB in size in a single operation. If your object is larger than 5 GB, use the multipart upload API. By using the multipart upload API, you can upload objects up to 5 TB each.

Copying objects The Copy operation creates a copy of an object that is already stored in Amazon S3. You can create a copy of your object up to 5 GB in size in a single atomic operation. However, for copying an object greater than 5 GB, use the multipart upload API.

Request Rate and Performance Considerations

Amazon S3 scales to support high request rates. If your request rate grows steadily, Amazon S3 automatically partitions your buckets as needed to support higher request rates. However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, AWS recommends that you open a support case to prepare for the workload and avoid any temporary limits on your request rate.

If your requests are typically a mix of GET, PUT, DELETE, or GET Bucket (List Objects), choose the appropriate key names for your objects to ensure better performance by providing low-latency access to the Amazon S3 index. It also ensures scalability regardless of the number of requests that you send per second. If the bulk of your workload consists of GET requests, AWS recommends using the Amazon CloudFront content delivery service.

Workloads with Different Request Types

When you are uploading a large number of objects, you might use sequential numbers or date-and-time values as part of the key names. For instance, you might decide to use key names that include a combination of the date and time, as shown in the following example, where the prefix includes a timestamp:

demobucket/2018-31-05-16-00-00/order1234234/receipt1.jpg
demobucket/2018-31-05-16-00-00/order3857422/receipt2.jpg
demobucket/2018-31-05-16-00-00/order1248473/receipt2.jpg
demobucket/2018-31-05-16-00-00/order8474937/receipt2.jpg
demobucket/2018-31-05-16-00-00/order1248473/receipt3.jpg
…
demobucket/2018-31-05-16-00-00/order1248473/receipt4.jpg
demobucket/2018-31-05-16-00-00/order1248473/receipt5.jpg
demobucket/2018-31-05-16-00-00/order1248473/receipt6.jpg
demobucket/2018-31-05-16-00-00/order1248473/receipt7.jpg

The way these keys are named presents a performance problem. To get a better understanding of this issue, consider the way that Amazon S3 stores key names. Amazon S3 maintains an index of object key names in each AWS Region. These keys are stored in UTF-8 binary ordering across multiple partitions in the index. The key name dictates where (or which partition) the key is stored in. In this case, where you use a sequential prefix such as a timestamp, or an alphabetical sequence, would increase the chances that Amazon S3 targets a specific partition for a large number of your keys, overwhelming the I/O capacity of the partition. However, if you introduce randomness in your key name prefixes, the key names, and therefore the I/O load, distribute across more than one partition.

If you anticipate that your workload will consistently exceed 100 requests per second, avoid sequential key names. If you must use sequential numbers or date-and-time patterns in key names, add a random prefix to the key name. The randomness of the prefix more evenly distributes key names across multiple index partitions.

The guidelines for the key name prefixes also apply to the bucket names. When Amazon S3 stores a key name in the index, it stores the bucket names as part of the key name (demobucket/object.jpg).

One way to introduce randomness to key names is to add a hash string as a prefix to the key name. For instance, you can compute an MD5 hash of the character sequence that you plan to assign as the key name. From the hash, select a specific number of characters and add them as the prefix to the key name.

GET-Intensive Workloads

If your workload consists mostly of sending GET requests, then in addition to the preceding guidelines, consider using Amazon CloudFront for performance optimization.

Integrating CloudFront with Amazon S3 enables you to deliver content with low latency and a high data transfer rate. You will also send fewer direct requests to Amazon S3, which helps to lower your costs.

Suppose that you have a few objects that are popular. CloudFront fetches those objects from Amazon S3 and caches them. CloudFront can then serve future requests for the objects from its cache, reducing the number of GET requests it sends to Amazon S3.

How Amazon EFS Works with AWS Direct Connect

Using an Amazon EFS file system mounted on an on-premises server, you can migrate on-premises data into the AWS Cloud hosted in an Amazon EFS file system. You can also take advantage of bursting. This means that you can move data from your on-premises servers into Amazon EFS, analyze it on a fleet of Amazon EC2 instances in your Amazon VPC, and then store the results permanently in your file system or move the results back to your on-premises server.

Consider the following when using Amazon EFS with DX:

• Your on-premises server must have a Linux-based operating system. AWS recommends Linux kernel version 4.0 or later.
• For the sake of simplicity, AWS recommends mounting an Amazon EFS file system on an on-premises server using a mount target IP address instead of a DNS name.
• AWS Virtual Private Network (AWS VPN) is not supported for accessing an Amazon EFS file system from an on-premises server.

You can use any one of the mount targets in your Amazon VPC as long as the subnet of the mount target is reachable by using the DX connection between your on-premises server and Amazon VPC. To access Amazon EFS from an on-premises server, you must add a rule to your mount target security group to allow inbound traffic to the NFS port (2049) from your on-premises server.

In Amazon EFS, a file system is the primary resource. Each file system has properties such as ID, creation token, creation time, file system size in bytes, number of mount targets created for the file system, and the file system state.

Amazon EFS also supports other resources to configure the primary resource. These include mount targets and tags:

Mount Target

• To access your file system, create mount targets in your Amazon VPC. Each mount target has the following properties:
  • Mount target ID
  • Subnet ID where it is created
  • File system ID for which it is created
  • IP address at which the file system may be mounted
  • Mount target state

    You can use the IP address or the DNS name in your mount command.

Tags

• To help organize your file systems, assign your own metadata to each of the file systems that you create. Each tag is a key-value pair.

Each file system has a DNS name of the following form:

file-system-ID.efs.aws-region.amazonaws.com

You can configure this DNS name in your mount command to mount the Amazon EFS file system.

Suppose that you create an efs-mount-point subdirectory in your home directory on your EC2 instance or on-premises server. Use the mount command to mount the file system. For example, on an Amazon Linux AMI, you can use following mount command:

$ sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600, retrans=2,noresvport file-system-DNS-name:/ ~/efs-mount-point

You can think of mount targets and tags as subresources that do not exist without being associated with a file system.

Amazon EFS provides API operations for you to create and manage these resources. In addition to the create and delete operations for each resource, Amazon EFS also supports a describe operation that enables you to retrieve resource information. The following options are available for creating and managing these resources:

• Use the Amazon EFS console.
• Use the Amazon EFS command line interface (CLI).

You can also manage these resources programmatically as follows:

Use the AWS SDKs The AWS SDKs simplify your programming tasks by wrapping the underlying Amazon EFS API. The SDK clients also authenticate your requests by using access keys that you provide.

Call the Amazon EFS API directly from your application If you cannot use the SDKs, you can make the Amazon EFS API calls directly from your application. However, if you use this option, you must write the necessary code to authenticate your requests.

Authentication and Access Control

You must have valid credentials to make Amazon EFS API requests, such as creating a file system. In addition, you must also have permissions to create or access resources. By default, when you use the account root user credentials, you can create and access resources owned by that account. However, AWS does not recommend using account root user credentials. In addition to creating or accessing resources, you must grant permissions to any AWS IAM users and roles you create in your account.

Data Consistency in Amazon EFS

Amazon EFS provides the open-after-close consistency semantics that applications expect from NFS. In Amazon EFS, write operations are durably stored across Availability Zones when an application performs a synchronous write operation (for example, using the open Linux command with the O_DIRECT flag, or the fsync Linux command) and when an application closes a file.

Amazon EFS provides stronger consistency than open-after-close semantics, depending on the access pattern. Applications that perform synchronous data access and perform nonappending writes have read-after-write consistency for data access.

NFS-Level Users, Groups, and Permissions

Amazon EFS file system objects have a Unix-style mode associated with them. This value defines the permissions for performing actions on that object, and users familiar with Unix-style systems can understand how Amazon EFS manages these permissions.

Further, on Unix-style systems, users and groups are mapped to numeric identifiers, which Amazon EFS uses to represent file ownership. A single owner and a single group own file system objects, such as files or directories on Amazon EFS. Amazon EFS uses these numeric IDs to check permissions when a user attempts to access a file system object.

User and Group ID Permissions on Files and Directories within a File System

Files and directories in an Amazon EFS file system support standard Unix-style read/write/execute permissions based on the user ID and group ID asserted by the mounting NFSv4.1 client. When a user tries to access files and directories, Amazon EFS checks their user ID and group IDs to determine whether the user has permission to access the objects. Amazon EFS also uses these IDs as the owner and group owner for new files and directories that the user creates. Amazon EFS does not inspect user or group names—it uses only the numeric identifiers.

When you create a user on an EC2 instance, you can assign any numeric UID and GID to the user. The numeric user IDs are set in the /etc/passwd file on Linux systems. The numeric group IDs are in the /etc/group file. These files define the mappings between names and IDs. Outside of the EC2 instance, Amazon EFS does not perform any authentication of these IDs, including the root ID of 0.

If a user accesses an Amazon EFS file system from two different EC2 instances, depending on whether the UID for the user is the same or different on those instances, you may observe different behavior. If the user IDs are the same on both EC2 instances, Amazon EFS considers them the same user, regardless of the EC2 instance they use. The user experience when accessing the file system is the same from both EC2 instances. If the user IDs are not the same on both EC2 instances, Amazon EFS considers them to be different users, and the user experience will not be the same when accessing the Amazon EFS file system from the two different EC2 instances. If two different users on different EC2 instances share an ID, Amazon EFS considers them the same user.

Performance Modes

To support a wide variety of cloud storage workloads, Amazon EFS offers two performance modes: General Purpose and Max I/O. At the time that you create your file system, you select a file system’s performance mode. There are no additional charges associated with the two performance modes. Your Amazon EFS file system is billed and metered the same, irrespective of the performance mode chosen. You cannot change an Amazon EFS file system’s performance mode after you have created the file system.

General Purpose performance mode AWS recommends the General Purpose performance mode for the majority of your Amazon EFS file systems. General Purpose is ideal for latency-sensitive use cases, such as web serving environments, content management systems, home directories, and general file serving. If you do not choose a performance mode when you create your file system, Amazon EFS selects the General Purpose mode for you by default.

Max I/O performance mode File systems in the Max I/O mode can scale to higher levels of aggregate throughput and operations per second with a trade-off of slightly higher latencies for file operations. Highly parallelized applications and workloads, such as big data analysis, media processing, and genomics analysis can benefit from this mode.

Throughput Scaling in Amazon EFS

Throughput on Amazon EFS scales as a file system grows. Because file-based workloads are typically spiky, driving high levels of throughput for short periods of time and low levels of throughput the rest of the time, Amazon EFS is designed to burst to high throughput levels for periods of time.

All file systems, regardless of size, can burst to 100 MB/s of throughput, and those larger than 1 TB can burst to 100 MB/s per TB of data stored in the file system. For example, a 10-TB file system can burst to 1,000 MB/s of throughput (10 TB × 100 MB/s/TB). The portion of time a file system can burst is determined by its size, and the bursting model is designed so that typical file system workloads will be able to burst virtually any time they need to.

Amazon EFS uses a credit system to determine when file systems can burst. Each file system earns credits over time at a baseline rate that is determined by the size of the file system, and it uses credits whenever it reads or writes data. The baseline rate is 50 MB/s per TB of storage (equivalently, 50 KB/s per GB of storage).

Accumulated burst credits give the file system permission to drive throughput above its baseline rate. A file system can drive throughput continuously at its baseline rate. Whenever the file system is inactive or when it is driving throughput below its baseline rate, the file system accumulates burst credits.