异构Hadoop集群下的比例数据分配策略

被引:6
作者
魏文娟
王黎明
机构
[1] 郑州大学信息工程学院
关键词
异构Hadoop集群; MapReduce; 数据转移; 比例数据分配; 计算速率;
D O I
暂无
中图分类号
TP311.13 [];
学科分类号
1201 ;
摘要
针对异构Hadoop环境下仍采用均等的数据分配方法将严重降低MapReduce的性能,提出比例数据分配策略。通过计算异构集群中各节点的计算比率,将已经分割好的数据块重新进行组合,形成数个按比例划分的数据块。每个节点根据自身性能来选择所分配和存储的数据块,从而使异构Hadoop集群中各节点处理数据的时间大致相同,降低节点之间数据的移动量。实验验证了提出的比例数据分配方法可以有效地提高MapReduce的性能,并使数据负载均衡。
引用
收藏
页码:316 / 319
页数:4
相关论文
共 13 条
[1]  
A dynamic MapReduce scheduler forheterogeneous workloads. Chao T,Zhou H,He Y,et al. . 2009
[2]  
MapReduce Optimization using Regulated Dynamic Prioritization. T. Sandholm,K. Lai. ACM SIGMETRICS’’09: International Conference on Measurement and Modeling of Computer Systems . 2009
[3]  
Above the clouds: A Berkeley view of cloud computing. Armbrust,M.et al. UCB/EECS- 2009 -28 . 2009
[4]  
MapReduce[J] . Jeffrey Dean,Sanjay Ghemawat. &nbspCommunications of the ACM . 2008 (1)
[5]  
Leen:Locality/fairness-aware key partitioning for mapreduce in the cloud. S. Ibrahim,H. Jin,L. Lu,S. Wu,B. He,L. Qi. Cloud Computing Technology and Science, IEEE International Conference on . 2010
[6]  
Sampling-based Partitioning in MapReduce for SkewedData. Yujie Xu,Peng Zou, et al. Seventh ChinaGrid Annual Conference . 2011
[7]  
Improving MapReduce performance through data placement inheterogeneous Hadoop clusters. Xie J,Yin S,Ruan X,et al. Proc19th Intel Heterogeneity in ComputingWorkshop . 2010
[8]  
Improving MapReduce Performance via Heterogeneity-Load-Aware Partition Function. Huifeng Sun,Junliang Chen,ChuanChang Liu. IEEE International Conference on Cluster Computing . 2011
[9]  
Performance Management of Accelerated MapReduce Workloads in Heterogeneous Clusters. J. Polo,D. Carrera,Y. Becerra,J. Torres,E. Ayguadé,M. Steinder,I. Whalley. 2010 39th International Conference on Parallel Processing . 2010
[10]  
Improving MapReduce performance in heterogeneous environments. Zaharia M,Konwinski A,Joseph A,et al. 8th Symposium on Operating Systems Design and Implementation . 2008