Configuring YARN in Informatica Big Data Management®

Configuring YARN in Informatica Big Data Management®

Example. Using a Fair Scheduler

Example. Using a Fair Scheduler

You are a Hadoop administrator for a healthcare company that owns a Hadoop cluster where the company runs different data processes. You want to allocate resources to each of the processes.
The healthcare company might have two data processes that run simultaneously. One process might handle inventories while another process might handle prescriptions. The following table describes how you might divide the data processes into queues:
Queue Name
Weight
Description
Inventory
0.33
Queue for jobs that process inventory data from hospital pharmacies.
Prescriptions
0.67
Queue for jobs that process data on the prescriptions that doctors write.
At a point in time, the queue
Inventory
might be running 2 jobs, while the queue
Prescriptions
might be running 3 jobs. The following image shows the proportion of cluster resources that each job can use:
This image shows a pie chart of the resource allocations to two queues, Inventory and Prescriptions, on a cluster that uses a fair scheduler. The queue Inventory has 2 jobs that equally share the resources assigned to the queue. The queue Prescriptions has 3 jobs that equally share the resources assigned to the queue.
Note that the resources allocated to the queue
Inventory
are divided in half. Each halve is allocated to one of the two jobs in the queue. The resources allocated to the queue
Prescriptions
are divided into three parts. Each part is allocated to one of the three jobs in the queue.
The first job in the queue
Inventory
completes. The following image shows the proportion of cluster resources that are available to the remaining jobs:
This image shows a pie chart of the remaining jobs and the resources allocated to each job. The remaining inventory job has access to a proportion of 0.33 of the cluster resources. The remaining prescriptions jobs each have access to a proportion of 0.22 of the cluster resources.

0 COMMENTS

We’d like to hear from you!