DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce

Zhihong LIU; Aimal KHAN; Peixin CHEN; Yaping LIU; Zhenghu GONG

doi:10.1587/transinf.2015EDL8255

DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce

Zhihong LIU, Aimal KHAN, Peixin CHEN, Yaping LIU, Zhenghu GONG

Full Text Views

0

Share
Cite this

Summary :

MapReduce still suffers from a problem known as skew, where load is unevenly distributed among tasks. Existing solutions follow a similar pattern that estimates the load of each task and then rebalances the load among tasks. However, these solutions often incur heavy overhead due to the load estimation and rebalancing. In this paper, we present DynamicAdjust, a dynamic resource adjustment technique for mitigating skew in MapReduce. Instead of rebalancing the load among tasks, DynamicAdjust adjusts resources dynamically for the tasks that need more computation, thereby accelerating these tasks. Through experiments using real MapReduce workloads on a 21-node Hadoop cluster, we show that DynamicAdjust can effectively mitigate the skew and speed up the job completion time by up to 37.27% compared to the native Hadoop YARN.

Publication: IEICE TRANSACTIONS on Information Vol.E99-D No.6 pp.1686-1689

Publication Date: 2016/06/01

Publicized: 2016/03/07

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2015EDL8255

Type of Manuscript: LETTER

Category: Fundamentals of Information Systems

Authors

Zhihong LIU
  National University of Defense Technology,University of Waterloo
Aimal KHAN
  University of Waterloo
Peixin CHEN
  National University of Defense Technology
Yaping LIU
  National University of Defense Technology
Zhenghu GONG
  National University of Defense Technology

Keyword

MapReduce, resource-aware scheduling, skew mitigation

Cite this

Copy

Zhihong LIU, Aimal KHAN, Peixin CHEN, Yaping LIU, Zhenghu GONG, "DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce" in IEICE TRANSACTIONS on Information, vol. E99-D, no. 6, pp. 1686-1689, June 2016, doi: 10.1587/transinf.2015EDL8255.
Abstract: MapReduce still suffers from a problem known as skew, where load is unevenly distributed among tasks. Existing solutions follow a similar pattern that estimates the load of each task and then rebalances the load among tasks. However, these solutions often incur heavy overhead due to the load estimation and rebalancing. In this paper, we present DynamicAdjust, a dynamic resource adjustment technique for mitigating skew in MapReduce. Instead of rebalancing the load among tasks, DynamicAdjust adjusts resources dynamically for the tasks that need more computation, thereby accelerating these tasks. Through experiments using real MapReduce workloads on a 21-node Hadoop cluster, we show that DynamicAdjust can effectively mitigate the skew and speed up the job completion time by up to 37.27% compared to the native Hadoop YARN.
URL: https://globals.ieice.org/en_transactions/information/10.1587/transinf.2015EDL8255/_p

Copy

@ARTICLE{e99-d_6_1686,
author={Zhihong LIU, Aimal KHAN, Peixin CHEN, Yaping LIU, Zhenghu GONG, },
journal={IEICE TRANSACTIONS on Information},
title={DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce},
year={2016},
volume={E99-D},
number={6},
pages={1686-1689},
abstract={MapReduce still suffers from a problem known as skew, where load is unevenly distributed among tasks. Existing solutions follow a similar pattern that estimates the load of each task and then rebalances the load among tasks. However, these solutions often incur heavy overhead due to the load estimation and rebalancing. In this paper, we present DynamicAdjust, a dynamic resource adjustment technique for mitigating skew in MapReduce. Instead of rebalancing the load among tasks, DynamicAdjust adjusts resources dynamically for the tasks that need more computation, thereby accelerating these tasks. Through experiments using real MapReduce workloads on a 21-node Hadoop cluster, we show that DynamicAdjust can effectively mitigate the skew and speed up the job completion time by up to 37.27% compared to the native Hadoop YARN.},
keywords={},
doi={10.1587/transinf.2015EDL8255},
ISSN={1745-1361},
month={June},}

Copy

TY - JOUR
TI - DynamicAdjust: Dynamic Resource Adjustment for Mitigating Skew in MapReduce
T2 - IEICE TRANSACTIONS on Information
SP - 1686
EP - 1689
AU - Zhihong LIU
AU - Aimal KHAN
AU - Peixin CHEN
AU - Yaping LIU
AU - Zhenghu GONG
PY - 2016
DO - 10.1587/transinf.2015EDL8255
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E99-D
IS - 6
JA - IEICE TRANSACTIONS on Information
Y1 - June 2016
AB - MapReduce still suffers from a problem known as skew, where load is unevenly distributed among tasks. Existing solutions follow a similar pattern that estimates the load of each task and then rebalances the load among tasks. However, these solutions often incur heavy overhead due to the load estimation and rebalancing. In this paper, we present DynamicAdjust, a dynamic resource adjustment technique for mitigating skew in MapReduce. Instead of rebalancing the load among tasks, DynamicAdjust adjusts resources dynamically for the tasks that need more computation, thereby accelerating these tasks. Through experiments using real MapReduce workloads on a 21-node Hadoop cluster, we show that DynamicAdjust can effectively mitigate the skew and speed up the job completion time by up to 37.27% compared to the native Hadoop YARN.
ER -