Install and deploy Hadoop MapReduce and HDFS. Export the HADOOP_HOME environment variable to point to your Hadoop installation root folder.

Install Python on your Hadoop compute nodes, if Python is not already installed.

The following steps show how to remove duplicate mails, due to cross-posting across the lists, from the 20news dataset:

Mapper Python script outputs the message ID as the key. We use the message ID to identify the duplicated messages that are a result of cross-posting across different newsgroups.

Hadoop Streaming provides the Reducer input records of the each key group line by line to the Streaming reducer application through the standard input. However, Hadoop Streaming does not have a mechanism to distinguish when it starts to feed records of a new key to the process. The Streaming reducer applications need to keep track of the input key to identify new groups. Since we output the mapper results using the MessageID header as the key, the Reducer input gets grouped by MessageID. Any group with more than one value (that is, message) per MessageID contains duplicates.

#!/usr/bin/env python
import sys;

currentKey = ""

for line in sys.stdin:
  line = line.strip()
  key, value = line.split('	',1)
  if currentKey == key :
    continue
  print '%s	%s' % (key, value)

In the previous script, we use only the first value (message) of the record group and discard the others, which are the duplicate