Hadoop counters for reporting custom metrics

Hadoop uses a set of counters to aggregate the metrics for MapReduce computations. Hadoop counters are helpful to understand the behavior of our MapReduce programs and to track the progress of the MapReduce computations. We can define custom counters to track the application specific metrics in MapReduce computations.

How to do it...

The following steps show you how to define a custom counter to count the number of bad or corrupted records in our log processing application.

  1. Define the list of custom counters using an enum.
      public static num LOG_PROCESSOR_COUNTER {
     BAD_RECORDS
    };
  2. Increment the counter in your mapper or reducer:
    context.getCounter(LOG_PROCESSOR_COUNTER.BAD_RECORDS).increment(1);
  3. Add the following to your driver program to access the counters:
    Job job = new Job(getConf(), "log-analysis");
……    
Counters counters = job.getCounters();
Counter badRecordsCounter = counters.findCounter(LOG_PROCESSOR_COUNTER.BAD_RECORDS);
System.out.println("# of Bad Records:"+ badRecordsCounter.getValue());
  4. Execute your Hadoop MapReduce computation. You can also view the counter values in the admin console or in the command line.
    > bin/hadoop jar C4LogProcessor.jar demo.LogProcessor in out 1
………
12/07/29 23:59:01 INFO mapred.JobClient: Job complete: job_201207271742_0020
12/07/29 23:59:01 INFO mapred.JobClient: Counters: 30
12/07/29 23:59:01 INFO mapred.JobClient:   demo.LogProcessorMap$LOG_PROCESSOR_COUNTER
12/07/29 23:59:01 INFO mapred.JobClient:   BAD_RECORDS=1406
12/07/29 23:59:01 INFO mapred.JobClient:   Job Counters 
………
12/07/29 23:59:01 INFO mapred.JobClient:     Map output records=112349
# of Bad Records :1406

How it works...

You have to define your custom counters using enums. The set of counters in an enum will form a group of counters. The JobTracker aggregates the counter values reported by the mappers and the reducers.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset