Yarn
----
An Yarn Architecture has 3 core components
i)ResourceManager
It handles both resource management and scheduling in Hadoop
It replace JobTracker in hadoop2.
It has 2 core components
i)Application Manager
It manages running Application Masters in the cluster, i.e., it is responsible for starting application masters and for monitoring and restarting them on different nodes in case of failures.
It also manages resources
ii)Scheduler
Scheduler is responsible for allocating the resources to the running application.
Scheduler is pure scheduler it means that it performs no monitoring no tracking for the application.
The Scheduler has a pluggable policy plug-in,
The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler are some examples
ii)NodeManager
It is the slave daemon of Yarn
NM Manages containers.
It also keeps track of health.
iii)Application Master
One application master runs per application. It negotiates resources from the resource manager and works with the node manager.
It Manages the application life cycle.
The AM acquires containers from the RM’s Scheduler before contacting the corresponding NMs to start the application’s individual tasks.
Yarn Architecture Flow
-----------------------
1)User will submit a job to Resource Manager.
2)Resource Manager will issue a request to Application Manager to communicte with NodeManager and start Application Master.
3)In a yarn Architecture No of Node Managers is one per machine or System.
4)Number of Application Master running is equivalent to no of Application's submitted by user i,e MapReduce,Apache Tez,Spark .
Here All MapReduce jobs are treated as MapReduce Application.
5)Each Node Manager is associated with 1 Application Master(s) i,e 1 per Application
MapReduce--1 App Master.
Spark--1 App Master.
Apache Tez--1 App Master.
6) An Application Master is by default in sleep mode(passive mode)
7)Whenever Resource Manager needs to Process a job then it instruct NodeManager to lauch Application Master which is in sleep mode.
8)Once Node Manager launch Application Master,AM will request for containers to the scheduler of Resource Manager.
9)Here Scheduler allocate Resources for containers.
10)A container is collection of (RAM,cpu,disk,Network) in which Application tasks are executed.
11)Once Resources are allocated by Scheduler,Application Master will request NodeManager to launch Containers where the tasks are executed.
12)Node Manager will track and monitor the life cycle of containers and update the status to ResourceManger's Application Manager through Application Master.
13)Application Master acts a communication channel bewteen Nodemanger and ResourceManager.
14)Till 2.0 version there is a SPOF[single point of failure] for Resource Manager but from then it has a Stand By(Passive RM),if the active RM is down then the passive will become active immediately,this makes RM higly fault tolerant.
15)If any node is down where Node Manager and App master is running,then if the App Master is destroyed then the other NodeManagers reporting to App master can directly communicate with RM.
16)To Clarify the above stament,An Application Master will requesr for containers where ever data is available,An Application Master running in one Node will communicate with other NodeManager's running in other machine's,All these NodeManager's will report the status of Node to App Master and App Master will update the status to RM.
16)Here Application Master can Allocate containers in different Node's and Track the staus of Application through these NM as discussed above.
RM <---> APP Master<---->NM1
Scheduler
App Manager <--->NM2
<--->NM3
..
..
17)Here Application Manager of RM will take care of Monitoring and Tracking the status of Node's through App Master as shown above.
18)NodeManagers will send heart beats consistently to App Master and App Master update the same to Application Manager of RM.
18)If Application Manager is not receiving heart beats for atleast 10 mins from a NodeManager through Application Master ,then the Resource Manager assumes the NodeManager is down and assigns the Job to some other NodeManager running in the cluster.
----
An Yarn Architecture has 3 core components
i)ResourceManager
It handles both resource management and scheduling in Hadoop
It replace JobTracker in hadoop2.
It has 2 core components
i)Application Manager
It manages running Application Masters in the cluster, i.e., it is responsible for starting application masters and for monitoring and restarting them on different nodes in case of failures.
It also manages resources
ii)Scheduler
Scheduler is responsible for allocating the resources to the running application.
Scheduler is pure scheduler it means that it performs no monitoring no tracking for the application.
The Scheduler has a pluggable policy plug-in,
The current Map-Reduce schedulers such as the CapacityScheduler and the FairScheduler are some examples
ii)NodeManager
It is the slave daemon of Yarn
NM Manages containers.
It also keeps track of health.
iii)Application Master
One application master runs per application. It negotiates resources from the resource manager and works with the node manager.
It Manages the application life cycle.
The AM acquires containers from the RM’s Scheduler before contacting the corresponding NMs to start the application’s individual tasks.
Yarn Architecture Flow
-----------------------
1)User will submit a job to Resource Manager.
2)Resource Manager will issue a request to Application Manager to communicte with NodeManager and start Application Master.
3)In a yarn Architecture No of Node Managers is one per machine or System.
4)Number of Application Master running is equivalent to no of Application's submitted by user i,e MapReduce,Apache Tez,Spark .
Here All MapReduce jobs are treated as MapReduce Application.
5)Each Node Manager is associated with 1 Application Master(s) i,e 1 per Application
MapReduce--1 App Master.
Spark--1 App Master.
Apache Tez--1 App Master.
6) An Application Master is by default in sleep mode(passive mode)
7)Whenever Resource Manager needs to Process a job then it instruct NodeManager to lauch Application Master which is in sleep mode.
8)Once Node Manager launch Application Master,AM will request for containers to the scheduler of Resource Manager.
9)Here Scheduler allocate Resources for containers.
10)A container is collection of (RAM,cpu,disk,Network) in which Application tasks are executed.
11)Once Resources are allocated by Scheduler,Application Master will request NodeManager to launch Containers where the tasks are executed.
12)Node Manager will track and monitor the life cycle of containers and update the status to ResourceManger's Application Manager through Application Master.
13)Application Master acts a communication channel bewteen Nodemanger and ResourceManager.
14)Till 2.0 version there is a SPOF[single point of failure] for Resource Manager but from then it has a Stand By(Passive RM),if the active RM is down then the passive will become active immediately,this makes RM higly fault tolerant.
15)If any node is down where Node Manager and App master is running,then if the App Master is destroyed then the other NodeManagers reporting to App master can directly communicate with RM.
16)To Clarify the above stament,An Application Master will requesr for containers where ever data is available,An Application Master running in one Node will communicate with other NodeManager's running in other machine's,All these NodeManager's will report the status of Node to App Master and App Master will update the status to RM.
16)Here Application Master can Allocate containers in different Node's and Track the staus of Application through these NM as discussed above.
RM <---> APP Master<---->NM1
Scheduler
App Manager <--->NM2
<--->NM3
..
..
17)Here Application Manager of RM will take care of Monitoring and Tracking the status of Node's through App Master as shown above.
18)NodeManagers will send heart beats consistently to App Master and App Master update the same to Application Manager of RM.
18)If Application Manager is not receiving heart beats for atleast 10 mins from a NodeManager through Application Master ,then the Resource Manager assumes the NodeManager is down and assigns the Job to some other NodeManager running in the cluster.
No comments:
Post a Comment