Sunday, May 4, 2014

An Insight into Hadoop Yarn Resource Manager

The Resource Manager is the core component in the Hadoop 2.0 framework (YARN). In analogy, it occupies the place of JobTracker of MRV1. YARN is designed to provide a generic and flexible framework to administer the computing resources in the cluster.
In this direction, the YARN Resource Manager Service (RM) is the central controlling authority for resource management and makes allocation decisions
ResourceManager has two main components: Scheduler and ApplicationsManager.
The Scheduler API is specifically designed to negotiate resources and not schedule tasks. The scheduler does not perform monitoring or tracking of status for the Applications. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc. Resource Manager does not guarantee about restarting failed tasks either due to application failure or hardware failures. Applications can request resources at different layers of the cluster topology such as nodes, racks etc. The scheduler determines how much and where to allocate based on resource availability and the configured sharing policy.
The Scheduler has a pluggable policy plug-in, which is responsible for partitioning the cluster resources among the various queues, applications etc. The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in
ApplicationsManager is responsible for maintaining a collection of submitted applications. It accepts job from the client and negotiates for a container to execute the application specific ApplicationMaster and it provide the service for restarting the ApplicationMaster in the case of failure. It also keeps a cache of completed applications so as to serve users’ requests via web UI or command line long after the applications in question finished.

No comments:

Post a Comment