- Hadoop MapReduce Cookbook
- Table of Contents
- Hadoop MapReduce Cookbook
- Credits
- About the Authors
- About the Reviewers
- www.PacktPub.com
- Preface
- 1. Getting Hadoop Up and Running in a Cluster
- Introduction
- Setting up Hadoop on your machine
- Writing a WordCount MapReduce sample, bundling it, and running it using standalone Hadoop
- Adding the combiner step to the WordCount MapReduce program
- Setting up HDFS
- Using HDFS monitoring UI
- HDFS basic command-line file operations
- Setting Hadoop in a distributed cluster environment
- Running the WordCount program in a distributed cluster environment
- Using MapReduce monitoring UI
- 2. Advanced HDFS
- 3. Advanced Hadoop MapReduce Administration
- Introduction
- Tuning Hadoop configurations for cluster deployments
- Running benchmarks to verify the Hadoop installation
- Reusing Java VMs to improve the performance
- Fault tolerance and speculative execution
- Debug scripts – analyzing task failures
- Setting failure percentages and skipping bad records
- Shared-user Hadoop clusters – using fair and other schedulers
- Hadoop security – integrating with Kerberos
- Using the Hadoop Tool interface
- 4. Developing Complex Hadoop MapReduce Applications
- Introduction
- Choosing appropriate Hadoop data types
- Implementing a custom Hadoop Writable data type
- Implementing a custom Hadoop key type
- Emitting data of different value types from a mapper
- Choosing a suitable Hadoop InputFormat for your input data format
- Adding support for new input data formats – implementing a custom InputFormat
- Formatting the results of MapReduce computations – using Hadoop OutputFormats
- Hadoop intermediate (map to reduce) data partitioning
- Broadcasting and distributing shared resources to tasks in a MapReduce job – Hadoop DistributedCache
- Using Hadoop with legacy applications – Hadoop Streaming
- Adding dependencies between MapReduce jobs
- Hadoop counters for reporting custom metrics
- 5. Hadoop Ecosystem
- Introduction
- Installing HBase
- Data random access using Java client APIs
- Running MapReduce jobs on HBase (table input/output)
- Installing Pig
- Running your first Pig command
- Set operations (join, union) and sorting with Pig
- Installing Hive
- Running a SQL-style query with Hive
- Performing a join with Hive
- Installing Mahout
- Running K-means with Mahout
- Visualizing K-means results
- 6. Analytics
- Introduction
- Simple analytics using MapReduce
- Performing Group-By using MapReduce
- Calculating frequency distributions and sorting using MapReduce
- Plotting the Hadoop results using GNU Plot
- Calculating histograms using MapReduce
- Calculating scatter plots using MapReduce
- Parsing a complex dataset with Hadoop
- Joining two datasets using MapReduce
- 7. Searching and Indexing
- Introduction
- Generating an inverted index using Hadoop MapReduce
- Intra-domain web crawling using Apache Nutch
- Indexing and searching web documents using Apache Solr
- Configuring Apache HBase as the backend data store for Apache Nutch
- Deploying Apache HBase on a Hadoop cluster
- Whole web crawling with Apache Nutch using a Hadoop/HBase cluster
- ElasticSearch for indexing and searching
- Generating the in-links graph for crawled web pages
- 8. Classifications, Recommendations, and Finding Relationships
- 9. Mass Text Data Processing
- Introduction
- Data preprocessing (extract, clean, and format conversion) using Hadoop Streaming and Python
- Data de-duplication using Hadoop Streaming
- Loading large datasets to an Apache HBase data store using importtsv and bulkload tools
- Creating TF and TF-IDF vectors for the text data
- Clustering the text data
- Topic discovery using Latent Dirichlet Allocation (LDA)
- Document classification using Mahout Naive Bayes classifier
- 10. Cloud Deployments: Using Hadoop on Clouds
- Introduction
- Running Hadoop MapReduce computations using Amazon Elastic MapReduce (EMR)
- Saving money by using Amazon EC2 Spot Instances to execute EMR job flows
- Executing a Pig script using EMR
- Executing a Hive script using EMR
- Creating an Amazon EMR job flow using the Command Line Interface
- Deploying an Apache HBase Cluster on Amazon EC2 cloud using EMR
- Using EMR Bootstrap actions to configure VMs for the Amazon EMR jobs
- Using Apache Whirr to deploy an Apache Hadoop cluster in a cloud environment
- Using Apache Whirr to deploy an Apache HBase cluster in a cloud environment
- Index
As we described in the earlier chapters, you can use Hadoop MapReduce interface to program most of the applications. However, if we are writing an application that includes many MapReduce steps, programming them with MapReduce is complicated.
There are several higher-level programming interfaces such as Pig and Hive to program parallel applications built on top of MapReduce. We will discuss these two interfaces in the following recipes.
This section demonstrates how to install Pig.
- Download Pig 0.10.0 from http://pig.apache.org/releases.html.
- Unzip Pig distribution by running the following command. We will call it
PIG_HOME.> tar xvf pig-0.10.0.tar.gz - To run Pig commands, change the directory to
PIG_HOMEand run thepigcommand. It starts the grunt shell.>cd PIG_HOME >bin/pig --help >bin/pig-x local grunt>
You can issue the Pig commands from the grunt shell.
The preceding instructions set up Pig in the local mode, and you can use the grunt> shell to execute the Pig commands.
The preceding commands explain how to run Pig in the local mode. The link http://pig.apache.org/docs/r0.10.0/start.html#Running+the+Pig+Scripts+in+Mapreduce+Mode explains how to run HBase in the distributed mode.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.