Day 1: CRUD and Datatypes

Because the command-line interface (CLI) is of such primary importance to the Redis development team—and loved by users everywhere—we’re going to spend Day 1 looking at many of the 160+ commands available. Of primary importance are Redis’s sophisticated datatypes and how they can query in ways that go far beyond simply “retrieve the value of this key.”

Getting Started

Redis is available through a few package builders such as Homebrew for Mac but is also rather painless to build from source.^[57] We’ll be working off version 3.2.8. Once you have it installed, you can start up the server by calling this:

​	​$ ​​redis-server​

It won’t run in the background by default, but you can make that happen by appending &, or you can just open another terminal. Next, run the command-line tool, which should connect to the default port 6379 automatically.

​	​$ ​​redis-cli​

After you connect, let’s try to ping the server.

​	redis 127.0.0.1:6379> PING
​	PONG

If you cannot connect, you’ll receive an error message. Typing help will display a list of help options. Type help followed by a space and then start typing any command. If you don’t know any Redis commands, just start pressing Tab to cycle through your options.

​	redis 127.0.0.1:6379> help
​	redis-cli 3.2.8
​	To get help about Redis commands type:
​	"help @<group>" to get a list of commands in <group>
​	"help <command>" for help on <command>
​	"help <tab>" to get a list of possible help topics
​	"quit" to exit

Today we’re going to use Redis to build the back end for a URL shortener, such as tinyurl.com or bit.ly. A URL shortener is a service that takes a really long URL and maps it to a shorter version on their own domain—like mapping http://supercalifragilisticexpialidocious.com to http://bit.ly/VLD. Visiting that short URL redirects users to the longer, mapped URL, which saves the visitors from text messaging long strings and also provides the short URL creator with some statistics, such as a count of visits.

In Redis, we can use SET to key a shortcode like 7wks to a value like http://www.sevenweeks.org. SET always requires two parameters: a key and a value. Retrieving the value just needs GET and the key name.

​	redis 127.0.0.1:6379> SET 7wks http://www.sevenweeks.org/
​	OK
​	redis 127.0.0.1:6379> GET 7wks
​	"http://www.sevenweeks.org/"

To reduce traffic, we can also set multiple values with MSET, like any number of key-value pairs. Here we map Google.com to gog and Yahoo.com to yah.

​	redis 127.0.0.1:6379> MSET gog http://www.google.com yah http://www.yahoo.com
​	OK

Correlatively, MGET grabs multiple keys and returns values as an ordered list.

​	redis 127.0.0.1:6379> MGET gog yah
​	1) "http://www.google.com/"
​	2) "http://www.yahoo.com/"

Although Redis stores strings, it recognizes integers and provides some simple operations for them. If we want to keep a running total of how many short keys are in our dataset, we can create a count and then increment it with the INCR command.

​	redis 127.0.0.1:6379> SET count 2
​	OK
​	redis 127.0.0.1:6379> INCR count
​	(integer) 3
​	redis 127.0.0.1:6379> GET count
​	"3"

Although GET returns count as a string, INCR recognized it as an integer and added one to it. Any attempt to increment a non-integer ends poorly.

​	redis 127.0.0.1:6379> SET bad_count "a"
​	OK
​	redis 127.0.0.1:6379> INCR bad_count
​	(error) ERR value is not an integer or out of range

If the value can’t be resolved to an integer, Redis rightly complains. You can also increment by any integer (INCRBY) or decrement (DECR, DECRBY).

Transactions

You’ve seen transactions in previous databases (Postgres and Neo4j), and Redis’s MULTI block atomic commands are a similar concept. Wrapping two operations like SET and INCR in a single block will complete either successfully or not at all. But you will never end up with a partial operation.

Let’s key another shortcode to a URL and also increment the count all in one transaction. We begin the transaction with the MULTI command and execute it with EXEC.

​	redis 127.0.0.1:6379> MULTI
​	OK
​	redis 127.0.0.1:6379> SET prag http://pragprog.com
​	QUEUED
​	redis 127.0.0.1:6379> INCR count
​	QUEUED
​	redis 127.0.0.1:6379> EXEC
​	1) OK
​	2) (integer) 4

When using MULTI, the commands aren’t actually executed when we define them (similar to Postgres transactions). Instead, they are queued and then executed in sequence.

Similar to ROLLBACK in SQL, you can stop a transaction with the DISCARD command, which will clear the transaction queue. Unlike ROLLBACK, it won’t revert the database; it will simply not run the transaction at all. The effect is identical, although the underlying concept is a different mechanism (transaction rollback vs. operation cancellation).

Complex Datatypes

So far, we haven’t seen much complex behavior. Storing string and integer values under keys—even as transactions—is all fine and good, but most programming and data storage problems deal with many types of data. Storing lists, hashes, sets, and sorted sets natively helps explain Redis’s popularity, and after exploring the complex operations you can enact on them, you may find you agree.

These collection datatypes can contain a huge number of values (up to 2^32 elements or more than 4 billion) per key. That’s more than enough for all Facebook accounts to live as a list under a single key (though maybe not for much longer!).

While some Redis commands may appear cryptic, they generally follow a repeated pattern. SET commands begin with S, hashes with H, and sorted sets with Z. List commands generally start with either an L (for left) or an R (for right), depending on the direction of the operation (such as LPUSH).

Hashes

Hashes are like nested Redis objects that can take any number of key-value pairs. Let’s use a hash to keep track of users who sign up for our URL-shortening service.

Hashes are nice because they help you avoid storing data with artificial key prefixes. (Note that we will use colons [:] within our keys. This is a valid character that often logically separates a key into segments. It’s merely a matter of convention, with no deeper meaning in Redis.)

​	redis 127.0.0.1:6379> MSET user:luc:name "Luc" user:luc:password s3cret
​	OK
​	redis 127.0.0.1:6379> MGET user:luc:name user:luc:password
​	1) "Luc"
​	2) "s3cret"

Instead of separate keys, we can create a hash that contains its own key-value pairs.

​	redis 127.0.0.1:6379> HMSET user:luc name "Luc" password s3cret
​	OK

We only need to keep track of the single Redis key to retrieve all values of the hash.

​	redis 127.0.0.1:6379> HVALS user:luc
​	1) "Luc"
​	2) "s3cret"

Or we can retrieve all hash keys.

​	redis 127.0.0.1:6379> HVALS user:luc
​	1) "name"
​	2) "password"

Or we can get a single value by passing in the Redis key followed by the hash key. Here we get just the password.

​	redis 127.0.0.1:6379> HGET user:luc password
​	"s3cret"

Unlike the document databases MongoDB and CouchDB, hashes in Redis cannot nest (nor can any other complex datatype such as lists). In other words, hashes can store only string values and not, say, sets or nested hashes.

More hash-specific commands exist to delete hash fields (HDEL), increment an integer field value by some count (HINCRBY), retrieve the number of fields in a hash (HLEN), get all keys and values (HGETALL), set a value only if the key doesn’t yet exist (HSETNX), and more.

Lists

Lists contain multiple ordered values that can act both as queues (first value in, first value out) and as stacks (last value in, first value out). They also have more sophisticated actions for inserting somewhere in the middle of a list, constraining list size, and moving values between lists.

Because our URL-shortening service can now track users, we want to allow them to keep a wishlist of URLs they’d like to visit. To create a list of short-coded websites we’d like to visit, we set the key to USERNAME:wishlist and push any number of values to the right (end) of the list.

​	redis 127.0.0.1:6379> RPUSH eric:wishlist 7wks gog prag
​	(integer) 3

Like most collection value insertions, the Redis command returns the number of values pushed. In other words, we pushed three values into the list so it returns 3. You can get the list length at any time with LLEN.

Using the list range command LRANGE, we can retrieve any part of the list by specifying the first and last positions. All list operations in Redis use a zero-based index. A negative position means the number of steps from the end.

​	redis 127.0.0.1:6379> LRANGE eric:wishlist 0 -1
​	1) "7wks"
​	2) "gog"
​	3) "prag"

LREM removes from the given key some matching values. It also requires a number to know how many matches to remove. Setting the count to 0 as we do here just removes them all:

​	redis 127.0.0.1:6379> LREM eric:wishlist 0 gog

Setting the count greater than 0 will remove only that number of matches, and setting the count to a negative number will remove that number of matches but scan the list from the end (right side).

To remove and retrieve each value in the order we added them (like a queue), we can pop them off from the left (head) of the list.

​	redis 127.0.0.1:6379> LPOP eric:wishlist
​	"7wks"

To act as a stack, after you RPUSH the values, you would RPOP from the end of the list. All of these operations are performed in constant time (meaning that the size of the list shouldn’t impact performance).

On the previous combination of commands, you can use LPUSH and RPOP to similar effect (a queue) or LPUSH and LPOP to be a stack.

Suppose we wanted to remove values from our wishlist and move them to another list of visited sites. To execute this move atomically, we might try wrapping pop and push actions within a multiblock. In Ruby, these steps might look something like this (you can’t use the CLI here because you must save the popped value, so we’ll use the redis-rb gem):

​	redis.multi ​do​
​	site = redis.rpop(​'eric:wishlist'​)
​	redis.lpush(​'eric:visited'​, site)
​	​end​

Because the multi block queues requests, the above won’t work. This is really a feature of lists, not a bug because it prevents concurrent access to list members. Fortunately, Redis provides a single command for popping values from the tail of one list and pushing to the head of another. It’s called RPOPLPUSH (right pop, left push).

​	redis 127.0.0.1:6379> RPOPLPUSH eric:wishlist eric:visited
​	"prag"

If you find the range of the wishlist, prag will be gone; it now lives under visited. This is a useful mechanism for queuing commands.

If you looked through the Redis docs to find RPOPRPUSH, LPOPLPUSH, and LPOPRPUSH commands, you may be dismayed to learn they don’t exist. RPOPLPUSH is your only option, and you must build your list accordingly.

Blocking Lists

Now that our URL shortener is taking off, let’s add some social activities—like a real-time commenting system—where people can post about the websites they have visited.

Let’s write a simple messaging system where multiple clients can push comments and one client (the digester) pops messages from the queue. We’d like the digester to just listen for new comments and pop them as they arrive. Redis provides a few blocking commands for this sort of purpose.

First, open another terminal and start another redis-cli client. This will be our digester. The command to block until a value exists to pop is BRPOP. It requires the key to pop a value from and a timeout in seconds, which we’ll set to five minutes.

​	redis 127.0.0.1:6379> BRPOP comments 300

Then switch back to the first console and push a message to comments.

​	redis 127.0.0.1:6379> LPUSH comments "Prag is a great publisher!"

If you switch back to the digester console, two lines will be returned: the key and the popped value. The console will also output the length of time it spent blocking.

​	1) "comments"
​	2) "Prag is a great publisher!"
​	(7.88s)

There’s also a blocking version of left pop (BLPOP) and left push (BRPOPLPUSH). BLPOP will remove the first element in a list or block (just like BRPOP removed the last element) while BRPOPLPUSH is a blocking version of RPOPLPUSH.

Sets

Our URL shortener is shaping up nicely, but it would be nice to be able to group common URLs in some way.

Sets are unordered collections with no duplicate values and are an excellent choice for performing complex operations between two or more key values, such as unions or intersections.

If we wanted to categorize sets of URLs with a common key, we could add multiple values with SADD.

​	redis 127.0.0.1:6379> SADD news nytimes.com pragprog.com
​	(integer) 2

Redis added two values. We can retrieve the full set, in no particular order, using the SMEMBERS command.

​	redis 127.0.0.1:6379> SMEMBERS news
​	1) "pragprog.com"
​	2) "nytimes.com"

Let’s add another category called tech for technology-related sites.

​	redis 127.0.0.1:6379> SADD tech pragprog.com apple.com
​	(integer) 2

To find the intersection of websites that both provide news and are technology focused, we use the SINTER command.

​	redis 127.0.0.1:6379> SINTER news tech
​	1) "pragprog.com"

Just as easily, we can remove any matching values in one set from another. To find all news sites that are not tech sites, use SDIFF:

​	redis 127.0.0.1:6379> SDIFF news tech
​	1) "nytimes.com"

We can also build a union of websites that are either news or tech. Because it’s a set, any duplicates are dropped.

​	redis 127.0.0.1:6379> SUNION news tech
​	1) "apple.com"
​	2) "pragprog.com"
​	3) "nytimes.com"

That set of values can also be stored directly into a new set (SUNIONSTORE destination key [key ...]).

​	redis 127.0.0.1:6379> SUNIONSTORE websites news tech
​	redis 127.0.0.1:6379> SMEMBERS websites
​	1) "pragprog.com"
​	2) "nytimes.com"
​	3) "apple.com"

This also provides a useful trick for cloning a single key’s values to another key, such as SUNIONSTORE news_copy news. Similar commands exist for storing intersections (SINTERSTORE) and diffs (SDIFFSTORE).

Just as RPOPLPUSH moved values from one list to another, SMOVE does the same for sets; it’s just easier to remember. And like LLEN finds the length of a list, SCARD (set cardinality) counts the set; it’s just harder to remember.

Because sets are not ordered, there are no left, right, or other positional commands. Popping a random value from a set just requires SPOP key, and removing values is SREM key value [value ...].

Unlike lists, there are no blocking commands for sets. With no blocking or positional commands, sets are a subpar choice for some Redis use cases such as message queues, but they’re nonetheless great for all kinds of operations over collections.

Sorted Sets

Whereas other Redis datatypes we’ve looked at so far easily map to common programming language constructs, sorted sets take something from each of the previous datatypes. They are ordered like lists and are unique like sets. They have field-value pairs like hashes, but their fields are numeric scores that denote the order of the values rather than plain strings. You can think of sorted sets as similar to a random access priority queue. This power has a trade-off, however. Internally, sorted sets keep values in order, so inserts can take log(N) time to insert (where N is the size of the set), rather than the constant time complexity of hashes or lists.

Next, we want to keep track of the popularity of specific shortcodes. Every time someone visits a URL, the score gets increased. Like a hash, adding a value to a sorted set requires two values after the Redis key name: the score and the member.

​	redis 127.0.0.1:6379> ZADD visits 500 7wks 9 gog 9999 prag
​	(integer) 3

To increment a score, we can either re-add it with the new score, which just updates the score but does not add a new value, or increment by some number, which will return the new value.

​	redis 127.0.0.1:6379> ZINCRBY visits 1 prag
​	"10000"

You can decrement also by setting a negative number for ZINCRBY.

Ranges

To get values from our visits set, we can issue a range command, ZRANGE, which returns by position, just like the list datatype’s LRANGE command. Except in the case of a sorted set, the position is ordered by score from lowest to highest. So, to get the top two scoring visited sites (zero-based), use this:

​	redis 127.0.0.1:6379> ZRANGE visits 0 1
​	1) "gog"
​	2) "7wks"

To get the scores of each element as well, append WITHSCORES to the previous code. To get them in reverse, insert the word REV, as in ZREVRANGE.

​	redis 127.0.0.1:6379> ZREVRANGE visits 0 -1 WITHSCORES
​	1) "prag"
​	2) "10000"
​	3) "7wks"
​	4) "500"
​	5) "gog"
​	6) "9"

But if we’re using a sorted set, it’s more likely we want to range by score, rather than by position. ZRANGEBYSCORE has a slightly different syntax from ZRANGE. Because the low and high range numbers are inclusive by default, we can make a score number exclusive by prefixing it with an opening paren: (. So this will return all scores where 9 <= score <= 10,000:

​	redis 127.0.0.1:6379> ZRANGEBYSCORE visits 9 9999
​	1) "gog"
​	2) "7wks"

But the following will return 9 < score <= 10,000:

​	redis 127.0.0.1:6379> ZRANGEBYSCORE visits (9 9999
​	1) "7wks"

We can also range by both positive and negative values, including infinities. This returns the entire set.

​	redis 127.0.0.1:6379> ZRANGEBYSCORE visits -inf inf
​	1) "gog"
​	2) "7wks"
​	3) "prag"

You can list them in reverse, too, with ZREVRANGEBYSCORE.

​	redis 127.0.0.1:6379> ZREVRANGEBYSCORE visits inf -inf
​	1) "prag"
​	2) "7wks"
​	3) "gog"

Along with retrieving a range of values by rank (index) or score, ZREMRANGEBYRANK and ZREMRANGEBYSCORE, respectively, remove values by rank or score.

Unions

Just like the set datatype, we can create a destination key that contains the union or intersection of one or more keys. This is one of the more complex commands in Redis because it must not only join the keys—a relatively simple operation—but also merge (possibly) differing scores. The union operation looks like this:

​	ZUNIONSTORE destination numkeys key [key ...]
​	[WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX]

destination is the key to store into, and key is one or more keys to union. numkeys is simply the number of keys you’re about to join, while weight is the optional number to multiply each score of the relative key by (if you have two keys, you can have two weights, and so on). Finally, aggregate is the optional rule for resolving each weighted score and summing by default, but you can also choose the min or max between many scores.

Let’s use this command to measure the importance of a sorted set of shortcodes. First, we’ll create another key that scores our shortcodes by votes. Each visitor to a site can vote if they like the site or not, and each vote adds a point.

​	redis 127.0.0.1:6379> ZADD votes 2 7wks 0 gog 9001 prag
​	(integer) 3

We want to figure out the most important websites in our system as some combination of votes and visits. Votes are important, but to a lesser extent, website visits also carry some weight (perhaps people are so enchanted by the website, they simply forget to vote). We want to add the two types of scores together to compute a new importance score while giving votes a weight of double importance—multiplied by two.

​	redis 127.0.0.1:6379> ZUNIONSTORE imp 2 visits votes WEIGHTS 1 2 AGGREGATE SUM
​	(integer) 3
​	redis 127.0.0.1:6379> ZRANGEBYSCORE imp -inf inf WITHSCORES
​	1) "gog"
​	2) "9"
​	3) "7wks"
​	4) "504"
​	5) "prag"
​	6) "28002"

This command is powerful in other ways, too. For example, if you need to double all scores of a set, you can union a single key with a weight of 2 and store it back into itself.

​	redis 127.0.0.1:6379> ZUNIONSTORE votes 1 votes WEIGHTS 2
​	(integer) 2
​	redis 127.0.0.1:6379> ZRANGE votes 0 -1 WITHSCORES
​	1) "gog"
​	2) "0"
​	3) "7wks"
​	4) "4"
​	5) "prag"
​	6) "18002"

Sorted sets contain a similar command (ZINTERSTORE) to perform intersections.

Expiry

Another common use case for a key-value system like Redis is as a fast-access cache for data that’s more expensive to retrieve or compute. In just about any cache, ensuring that keys expire after a designated time period is essential to keeping the key set from growing unboundedly.

Marking a key for expiration requires the EXPIRE command, an existing key, and a time to live (in seconds). Here we set a key and set it to expire in ten seconds. We can check whether the key EXISTS within ten seconds and it returns a 1 (true). If we wait to execute, it will eventually return a 0 (false).

​	redis 127.0.0.1:6379> SET ice "I'm melting..."
​	OK
​	redis 127.0.0.1:6379> EXPIRE ice 10
​	(integer) 1
​	redis 127.0.0.1:6379> EXISTS ice
​	(integer) 1
​	redis 127.0.0.1:6379> EXISTS ice
​	(integer) 0

Setting and expiring keys is so common that Redis provides a shortcut command called SETEX.

​	redis 127.0.0.1:6379> SETEX ice 10 "I'm melting..."

You can query the time a key has to live with TTL. Setting ice to expire as shown earlier and checking its TTL will return the number of seconds left (or -2 if the key has already expired or doesn’t exist, which is the same thing).

​	redis 127.0.0.1:6379> TTL ice
​	(integer) 4

At any moment before the key expires, you can remove the timeout by running PERSIST key.

​	redis 127.0.0.1:6379> PERSIST ice

For marking a countdown to a specific time, EXPIREAT accepts a Unix timestamp (as seconds since January 1, 1970) rather than a number of seconds to count up to. In other words, EXPIREAT is for absolute timeouts, and EXPIRE is for relative timeouts.

A common trick for keeping only recently used keys is to update the expire time whenever you retrieve a value. This is the most recently used (MRU) caching algorithm to ensure that your most recently used keys will remain in Redis, while the unused keys will just expire as normal.

Database Namespaces

So far, we’ve interacted only with a single namespace. Sometimes we need to separate keys into multiple namespaces. For example, if you wrote an internationalized key-value store, you could store different translated responses in different namespaces. The key greeting could be set to “guten Tag” in a German namespace and “bonjour” in French. When a user selects their language, the application just pulls all values from the namespace assigned.

In Redis nomenclature, a namespace is called a database and is keyed by number. So far, we’ve always interacted with the default namespace 0 (also known as database 0). Here we set greeting to the English hello.

​	redis 127.0.0.1:6379> SET greeting hello
​	OK
​	redis 127.0.0.1:6379> GET greeting
​	"hello"

But if we switch to another database via the SELECT command, that key is unavailable.

​	redis 127.0.0.1:6379> SELECT 1
​	OK
​	redis 127.0.0.1:6379[1]> GET greeting
​	(nil)

And setting a value to this database’s namespace will not affect the value of the original.

​	redis 127.0.0.1:6379[1]> SET greeting "guten Tag"
​	OK
​	redis 127.0.0.1:6379[1]> SELECT 0
​	OK
​	redis 127.0.0.1:6379> GET greeting
​	"hello"

Because all databases are running in the same server instance, Redis lets us shuffle keys around with the MOVE command. Here we move greeting to database 2:

​	redis 127.0.0.1:6379> MOVE greeting 2
​	(integer) 2
​	redis 127.0.0.1:6379> SELECT 2
​	OK
​	redis 127.0.0.1:6379[2]> GET greeting
​	"hello"

This can be useful for running different applications against a single Redis server while still allowing these multiple applications to trade data between each other.

And There’s More

Redis has plenty of other commands for actions such as renaming keys (RENAME), determining the type of a key’s value (TYPE), and deleting a key-value (DEL). There’s also the painfully dangerous FLUSHDB, which removes all keys from this Redis database, and its apocalyptic cousin, FLUSHALL, which removes all keys from all Redis databases. We won’t cover all commands here, but we recommend checking out the online documentation for the full list of Redis commands (and we must say, the Redis documentation is truly a joy to read and interact with).

Day 1 Wrap-Up

The datatypes of Redis and the complex queries it can perform make it much more than a standard key-value store. It can act as a stack, queue, or priority queue; you can interact with it as you would an object store (via hashes); and Redis even can perform complex set operations such as unions, intersections, and subtractions (diff). It provides many atomic commands as well as a transaction mechanism for multistep commands. Plus, it has a built-in ability to expire keys, which is useful as a cache.

Day 1 Homework

Find

Find the complete Redis commands documentation, as well as the Big-O notated (O(x)) time complexity under the command details.

Do

1. Install your favorite programming language driver and connect to the Redis server. Insert and increment a value within a transaction.

2. Using your driver of choice, create a program that reads a blocking list and outputs somewhere (console, file, Socket.io, and so on) and another that writes to the same list.