When a user submits a job to Hadoop, this job needs to be assigned a resource (a computer/host) before execution. This process is called scheduling , and a scheduler decides when resources are assigned to a given job.
Hadoop is by default configured with a First in First out (FIFO) scheduler, which executes jobs in the same order as they arrive. However, for a deployment that is running many MapReduce jobs and shared by many users, more complex scheduling policies are needed.
The good news is that Hadoop scheduler is pluggable, and it comes with two other schedulers. Therefore, if required, it is possible to write your own scheduler as well.
- Fair scheduler: This defines pools and over time; each pool gets around the same amount of resources.
- Capacity scheduler: This defines queues, and each queue has a guaranteed capacity. The capacity scheduler shares computer resources allocated to a queue with other queues if those resources are not in use.
This recipe describes how to change the scheduler in Hadoop.
For this recipe, you need a working Hadoop deployment. Set up Hadoop using the Setting Hadoop in a distributed cluster environment recipe from Chapter 1, Getting Hadoop Up and Running in a Cluster.
- Shut down the Hadoop cluster.
- You need
hadoop-fairscheduler-1.0.0.jarin theHADOOP_HOME/lib. However, from Hadoop 1.0.0 and higher releases, this JAR file is in the right place in the Hadoop distribution. - Add the following code to the
HADOOP_HOME/conf/mapred-site.xml:<property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.FairScheduler</value> </property>
- Restart Hadoop.
- Verify that the new scheduler has been applied by going to
http://<job-tracker-host>:50030/schedulerin your installation. If the scheduler has been properly applied, the page will have the heading "Fair Scheduler Administration".
When you follow the preceding steps, Hadoop will load the new scheduler settings when it is started. The fair scheduler shares equal amount of resources between users unless it has been configured otherwise.
The fair scheduler supports users to configure it through two ways. There are several parameters of the mapred.fairscheduler.* form, and we can configure these parameters via HADOOP_HOME/conf/mapred-site.xml. Also additional parameters can be configured via HADOOP_HOME/conf/fair-scheduler.xml. More details about fair scheduler can be found from HADOOP_HOME/docs/fair_scheduler.html.