When you have a lot of data and a lot of computation to be performed, you might start to crave more computing power. Amazon (http://aws.amazon.com) allows you to rent computing power by the hour. Thus, you can access a large amount of computing power without having to commit to purchasing a large number of machines (including the cost of managing the infrastructure). There are other competitors in this market, but Amazon is the largest player, so we briefly cover it here.
Amazon Web Services (AWS) is a large set of services. We will focus only on the Elastic Compute Cloud (EC2) service. This service offers you virtual machines and disk space, which can be allocated and deallocated quickly.
There are three modes of use. The first is a reserved mode, whereby you prepay to have cheaper per-hour access, a fixed per-hour rate, and a variable rate, which depends on the overall compute market (when there is less demand, the costs are lower; when there is more demand, the prices go up).
On top of this general system, there are several types of machines available with varying costs, from a single core to a multicore system with a lot of RAM or even graphical processing units (GPUs). We will later see that you can also get several of the cheaper machines and build yourself a virtual cluster. You can also choose to get a Linux or Windows server (with Linux being slightly cheaper). In this chapter, we will work on our examples on Linux, but most of this information will be valid for Windows machines as well.
For testing, you can use a single machine in the free tier. This allows you to play around with the system, get used to the interface, and so on. Note, though, that this machine contains a slow CPU.
The resources can be managed through a web interface. However, it's also possible to do so programmatically and to write scripts that allocate virtual machines, format hard disks, and perform all operations that are possible through the web interface. In fact, while the web interface changes very frequently (and some of the screenshots we show in the book may be out of date by the time it goes to press), the programmatic interface is more stable, and the general architecture has remained stable since the service was introduced.
Access to AWS services is performed through a traditional username/password system, although Amazon calls the username an access key and the password a secret key. They probably do so to keep it separate from the username/password you use to access the web interface. In fact, you can create as many access/secret key pairs as you wish and give them different permissions. This is helpful for a larger team, where a senior user with access to the full web panel can create other keys for developers with fewer privileges.
AWS is a very large topic and there are various books exclusively available covering it. The purpose of this chapter is to give you an overall impression of what is available and what is possible with AWS. In the practical spirit of this book, we do this by working through examples, but we will not exhaust all possibilities.