A Novel Priority Based Hadoop Energy Efficient Job Scheduling and Migration Technique with Multi Level Queue on YARN Scheduler

Author Name(s): G. Joel Sunny Deol, Dr.O.NagaRaju
Author Email: sunnydeolgosu@gmail.com

Abstract

The process or the framework for MapReduce works in two parts as Mapper and Reducer. The reducer algorithm analyzes the input from the tasks characteristics and generate recommendations for the applicable allocation of the work and the mapper algorithm analyses the perfect or the best fit for the task or the programming running on the Hadoop clusters. The primary challenge is to manage the migration of the virtual machines to make these arrangements suitable to the Hadoop scheduling capabilities. Hence the demand from the research is to justly the Hadoop scheduling capabilities and test the performances of the scheduler strategies for diversified workloads. Also, it is important to design a virtual machine migration algorithm to justify the demands of low power consumptions. Accordingly, this work also coined an energy efficient technique for Hadoop MapReduce jobs scheduling and migration technique. The work results into a novel algorithm and provide significant improvement of the energy consumption. The outcome of the work also analyzes the improvement of other performance parameters like identification of ill-scheduled job and total execution time. This work demonstrates a significant 30% reduction of energy with nearly 40% reduction in job identification and migration time.

Introduction

The Apache Hadoop framework provides the best mashup of data and processing capabilities. The highest rated demand of enterprise applications like robustness and scalabilities can be achieved through Hadoop [Fig – 1] provided highly reliable and low cost storage structures [1]. The storage provided is HDFS enabling to host a large amount of data and process the jobs using MapReduce framework enabling the distributed and parallel processing of the Job [2]. The MapReduce framework [Fig – 2] is first introduced by Google in the early 2000. The MapReduce framework operates on <key, value> pairs, that is, the framework views the input to the job as a set of <key, value> pairs and produces a set of <key, value> pairs as the output of the job, conceivably of different types [3]. MapReduce libraries have been written in many programming dialects, with various dimensions of advancement. A prominent open-source execution that has bolster for disseminated rearranges is a piece of Apache Hadoop. The name MapReduce initially alluded to the restrictive Google innovation, yet has since been genericized [4]. The decrease work takes the yield from a guide as info and joins those information tuples into a littler arrangement of tuples. As the arrangement of the name MapReduce infers, the diminish work is constantly performed after the guide work [5].

Conclusion

The work exhibits the current methods with comprehension of the importance in time and information multifaceted nature worldview like FIFO scheduler, Fair Scheduler, Capacity Scheduler, Dynamic Priority Based Hybrid Scheduler, LATE Scheduler, SAMR Scheduler, Delay Scheduler and Context Aware Scheduler. The not well apportioned Job Identification calculation is been tried and the work delivers considerable lot of test results to demonstrate the curiosity and enhancement. The outcome demonstrates almost a huge 30% enhancement of vitality utilization over the current procedures and 88.89% lesser than the current systems alongside a critical 70% enhancement over the current methods for Job Reallocation Time. Critical accomplishment of this work additionally incorporates the begetting the novel execution assessment lattice for near comprehension of the proposed and existing procedure. The ultimate result of the work exhibits the centrality of line booking to accomplish comparative execution

1,182 total views, no views today

Download PDF File

About the author: admin