- Hadoop MapReduce Cookbook
- Table of Contents
- Hadoop MapReduce Cookbook
- Credits
- About the Authors
- About the Reviewers
- www.PacktPub.com
- Preface
- 1. Getting Hadoop Up and Running in a Cluster
- Introduction
- Setting up Hadoop on your machine
- Writing a WordCount MapReduce sample, bundling it, and running it using standalone Hadoop
- Adding the combiner step to the WordCount MapReduce program
- Setting up HDFS
- Using HDFS monitoring UI
- HDFS basic command-line file operations
- Setting Hadoop in a distributed cluster environment
- Running the WordCount program in a distributed cluster environment
- Using MapReduce monitoring UI
- 2. Advanced HDFS
- 3. Advanced Hadoop MapReduce Administration
- Introduction
- Tuning Hadoop configurations for cluster deployments
- Running benchmarks to verify the Hadoop installation
- Reusing Java VMs to improve the performance
- Fault tolerance and speculative execution
- Debug scripts – analyzing task failures
- Setting failure percentages and skipping bad records
- Shared-user Hadoop clusters – using fair and other schedulers
- Hadoop security – integrating with Kerberos
- Using the Hadoop Tool interface
- 4. Developing Complex Hadoop MapReduce Applications
- Introduction
- Choosing appropriate Hadoop data types
- Implementing a custom Hadoop Writable data type
- Implementing a custom Hadoop key type
- Emitting data of different value types from a mapper
- Choosing a suitable Hadoop InputFormat for your input data format
- Adding support for new input data formats – implementing a custom InputFormat
- Formatting the results of MapReduce computations – using Hadoop OutputFormats
- Hadoop intermediate (map to reduce) data partitioning
- Broadcasting and distributing shared resources to tasks in a MapReduce job – Hadoop DistributedCache
- Using Hadoop with legacy applications – Hadoop Streaming
- Adding dependencies between MapReduce jobs
- Hadoop counters for reporting custom metrics
- 5. Hadoop Ecosystem
- Introduction
- Installing HBase
- Data random access using Java client APIs
- Running MapReduce jobs on HBase (table input/output)
- Installing Pig
- Running your first Pig command
- Set operations (join, union) and sorting with Pig
- Installing Hive
- Running a SQL-style query with Hive
- Performing a join with Hive
- Installing Mahout
- Running K-means with Mahout
- Visualizing K-means results
- 6. Analytics
- Introduction
- Simple analytics using MapReduce
- Performing Group-By using MapReduce
- Calculating frequency distributions and sorting using MapReduce
- Plotting the Hadoop results using GNU Plot
- Calculating histograms using MapReduce
- Calculating scatter plots using MapReduce
- Parsing a complex dataset with Hadoop
- Joining two datasets using MapReduce
- 7. Searching and Indexing
- Introduction
- Generating an inverted index using Hadoop MapReduce
- Intra-domain web crawling using Apache Nutch
- Indexing and searching web documents using Apache Solr
- Configuring Apache HBase as the backend data store for Apache Nutch
- Deploying Apache HBase on a Hadoop cluster
- Whole web crawling with Apache Nutch using a Hadoop/HBase cluster
- ElasticSearch for indexing and searching
- Generating the in-links graph for crawled web pages
- 8. Classifications, Recommendations, and Finding Relationships
- 9. Mass Text Data Processing
- Introduction
- Data preprocessing (extract, clean, and format conversion) using Hadoop Streaming and Python
- Data de-duplication using Hadoop Streaming
- Loading large datasets to an Apache HBase data store using importtsv and bulkload tools
- Creating TF and TF-IDF vectors for the text data
- Clustering the text data
- Topic discovery using Latent Dirichlet Allocation (LDA)
- Document classification using Mahout Naive Bayes classifier
- 10. Cloud Deployments: Using Hadoop on Clouds
- Introduction
- Running Hadoop MapReduce computations using Amazon Elastic MapReduce (EMR)
- Saving money by using Amazon EC2 Spot Instances to execute EMR job flows
- Executing a Pig script using EMR
- Executing a Hive script using EMR
- Creating an Amazon EMR job flow using the Command Line Interface
- Deploying an Apache HBase Cluster on Amazon EC2 cloud using EMR
- Using EMR Bootstrap actions to configure VMs for the Amazon EMR jobs
- Using Apache Whirr to deploy an Apache Hadoop cluster in a cloud environment
- Using Apache Whirr to deploy an Apache HBase cluster in a cloud environment
- Index
This recipe shows how to merge files in HDFS to create a single file. This is useful when retrieving the output of a MapReduce computation with multiple reducers where each reducer produces a part of the output.
The getmerge command has the following syntax:
hadoopfs -getmerge <src> <localdst> [addnl]
The getmerge command has three parameters. The first parameter, <src files> is the HDFS path to the directory that contains the files to be concatenated. <dist file> is the local filename of the merged file. addnl
is an optional parameter that adds a new line in the result file, after the data from each merged file.
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.