实时流处理系统Storm的调度优化综述

被引:9
作者
蔡宇 [1 ]
赵国锋 [1 ,2 ]
郭航 [1 ]
机构
[1] 重庆邮电大学电子信息与网络研究院
[2] 重庆市光通信与网络高校重点实验室
关键词
流式数据处理; Apache Storm; 性能优化; 调度;
D O I
暂无
中图分类号
TP311.13 [];
学科分类号
1201 ;
摘要
随着大数据技术的发展,相对于Hadoop等传统的批处理系统,流式处理系统具有更好的实时性特点。在已有的流式处理系统中,Storm系统具有良好的稳定性、高可扩展性以及高容错性等优点,使它在流式数据处理系统中脱颖而出。但是在任务调度方面,Storm系统并没有做过多的考虑,默认采用相对简单的轮询调度法,导致系统在性能上存在瓶颈。近年来针对Storm系统的调度问题,研究提出了各种优化方案。从实时流处理系统Storm的调度优化出发,将这些优化方法分为四类,并详细阐述各类中具有一定代表性的方法,分析其优缺点以及适用的场景。最后,讨论了在日益发展的新环境下,Storm系统的调度优化相关研究未来可能存在的方向。
引用
收藏
页码:2567 / 2573
页数:7
相关论文
共 53 条
[41]  
A Qo S-aware controller for apache Storm. Farahabady M R H,Samani H R D,Wang Yidan,et al. Proc of the 15th IEEE International Symposium on Network Computing and Applications . 2016
[42]  
Apache Hadoop YARN: yet another resource nego-tiator. Vavilapalli V K,Murthy A C,Douglas C, et al. Symposium on Cloud Computing . 2013
[43]  
Fast heuristics for near-optimal task allocation in data stream processing over clusters. Chatzistergiou A,Viglas S D. Proc of the 23rd ACM International Conference on Conference on Information and Knowledge Management . 2014
[44]  
Priority-based resource scheduling in distributed stream processing systems for big data applications. Bellavista P,Corradi A,Reale A,et al. Proc of the 7th IEEE/ACM International Conference on Utility and Cloud Computing . 2015
[45]  
G-Storm:a GPU-aware Storm scheduler. Chen Yiren,Lee C R. Proc of the 14th International Conference on Dependable,Autonomic and Secure Computing,14th International Conference on Pervasive Intelligence and Computing,2nd International Conference on Big Data Intelligence and Computing and Cyber Science and Technolo . 2016
[46]  
GPU-accelerated highthroughput online stream data processing. Chen Zhenhua,Xu Jielong,Tang Jian,et al. IEEE Trans on Big Data . 2018
[47]  
DBalancer:a tool for dynamic changing of workers number in Storm. Zang Zhida,Rao R N. Proc of the 4th International Conference on Computer Science and Network Technology . 2015
[48]  
Enabling elastic stream processing in shared clusters. Li J,Pu C,Chen Yuan et al. Proc of the 9th IEEE International Conference on Cloud Computing . 2017
[49]  
Elastic stateful stream processing in Storm. Cardellini V,Nardelli M,Luzi D. Proc of International Conference on High Performance Computing&Simulation . 2016
[50]  
Auto-scaling techniques for elastic data stream processing. Heinze T,Pappalardo V,Jerzak Z,et al. Proc of the 8th ACM International Conference on Distributed Event-Based Systems,IEEE International Conference on Data Engineering Workshops . 2014