In this chapter, we will cover:
- Choosing appropriate Hadoop data types
- Implementing a custom Hadoop
Writabledata type - Implementing a custom Hadoop
keytype - Emitting data of different
valuetypes from a mapper - Choosing a suitable Hadoop
InputFormatfor your input data format - Adding support for new input data formats – implementing a custom
InputFormat - Formatting the results of MapReduce computations – using Hadoop
OutputFormats - Hadoop intermediate (map to reduce) data partitioning
- Broadcasting and distributing shared resources to tasks in a MapReduce job : Hadoop
DistributedCache - Using Hadoop with legacy applications – Hadoop Streaming
- Adding dependencies between MapReduce jobs
- Hadoop counters for reporting custom metrics
This chapter introduces you to several advanced Hadoop MapReduce features that will help you to develop highly customized, efficient MapReduce applications.
In this chapter, we will explore the different data types provided by Hadoop and the steps to implement custom data types for Hadoop MapReduce computations. We will also explore the different data input and output formats provided by Hadoop. This chapter will provide you with the basic understanding of how to add support for new data formats in Hadoop. We will also be discussing other advanced Hadoop features such as using DistributedCache for distribute data, using Hadoop Streaming for quick prototyping of Hadoop computations, and using Hadoop counters to report custom metrics for your computation as well as adding job dependencies to manage simple DAG-based workflows of Hadoop MapReduce computations.