Amazon Elastic MapReduce (EMR) provides on-demand managed Hadoop clusters in the Amazon Web Services (AWS) cloud to perform your Hadoop MapReduce computations. EMR uses Amazon Elastic Compute Cloud (EC2) instances as the compute resources. EMR supports reading input data from Amazon Simple Storage Service (S3) and storing of the output data in Amazon S3 as well. EMR makes our life easier by taking care of the provisioning of cloud instances, configuring the Hadoop cluster and the execution of our MapReduce computational flows.
In this recipe, we are going to run the WordCount MapReduce sample (Refer to the Writing the WordCount MapReduce sample, bundling it and running it using standalone Hadoop recipe from Chapter 1, Getting Hadoop up and running in a Cluster) in the Amazon EC2 cloud using Amazon Elastic MapReduce.
Build the required c10-samples.jar by running the Ant build in the code samples for this chapter.
The steps for executing WordCount MapReduce application on Amazon Elastic MapReduce are as follows:
- Sign up for an AWS account by visiting http://aws.amazon.com.
- Open the Amazon S3 monitoring console at https://console.aws.amazon.com/s3 and sign in.
- Create a S3 bucket to upload the input data by clicking on Create Bucket. Provide a unique name for your bucket. Let's assume the name of the bucket as
wc-input-data. You can find more information on creating a S3 bucket in http://docs.amazonwebservices.com/AmazonS3/latest/gsg/CreatingABucket.html. There exist several third-party desktop clients for the Amazon S3. You can use one of those clients to manage your data in S3 as well. - Upload your input data to the above-created bucket by selecting the bucket and clicking on Upload. The input data for the WordCount sample should be one or more text files.
- Create a S3 bucket to upload the JAR file needed for our MapReduce computation. Let's assume the name of the bucket as
sample-jars. Upload theC10Samples.jarfile to the newly created bucket. - Create a S3 bucket to store the output data of the computation. Let's assume the name of this bucket as
ws-output-data. Create another S3 bucket to store the logs of the computation. Let's assume the name of this bucket asc10-logs. - Open the Amazon EMR console at https://console.aws.amazon.com/elasticmapreduce. Click on the Create New Job Flow button to create a new EMR MapReduce job flow. Provide a name for your job flow. Select Run your own application option under Create a Job Flow. Select the Custom Jar option from the drop-down menu below that. Click on Continue.
- Specify the S3 location of the
c10-samples.jarin the Jar Location textbox of the next tab (the Specify Parameters tab). You should specify the location of the JAR in the formatbucket_name/jar_name. In the JAR Arguments textbox, enterchapter1.WordCountfollowed by the bucket location where you uploaded the input data and the output path. The output path should not exist and we use a directory (wc-output-data/out1) inside the output bucket you created in Step 6 as the output path. You should specify the locations using the format,s3n://bucket_name/path. Click on Continue.
- Leave the default options and click on Continue in the next tab, Configure EC2 Instances. The default options use two EC2
m1.smallinstances for the Hadoop slave nodes and one EC2m1.smallinstance for the Hadoop master node. - In the Advanced Options tab, enter the path of S3 bucket you created above for the logs in the Amazon S3 Log Path textbox. Select Yes for the Enable Debugging. Click on Continue.
- Click on Continue in the Bootstrap Options. Review your job flow in the Review tab and click on Create Job Flow to launch instances and to run the MapReduce computation.
- Click on Refresh in the EMR console to monitor the progress of your MapReduce job. Select your job flow entry and click on Debug to view the logs and to debug the computation. As EMR uploads the logfiles periodically, you might have to wait and refresh to access the logfiles. Check the output of the computation in the output data bucket using the AWS S3 console.
- Writing the WordCount MapReduce sample, bundling it and running it using standalone Hadoop and Running WordCount program in a distributed cluster environment recipes from Chapter 1, Getting Hadoop up and running in a Cluster.