| 基于稳定匹配的实时ETL弹性调度机制 |
| |
| 引用本文： | 刘旋律,顾进广. 基于稳定匹配的实时ETL弹性调度机制[J]. 计算机应用与软件, 2022, 0(2): 266-273 |
| |
| 作者姓名： | 刘旋律 顾进广 |
| |
| 作者单位： | 武汉科技大学计算机科学与技术学院;智能信息处理与实时工业系统湖北省重点实验室（武汉科技大学）;武汉科技大学大数据科学与工程研究院;国家新闻出版署富媒体数字出版内容组织与知识服务重点实验室 |
| |
| 基金项目： | 国家自然科学基金项目(61673304)； |
| |
| 摘 要： | 在数据生产速度波动较大的场景,为了实时ETL资源利用更合理,提出基于稳定匹配的ETL弹性调度机制.预测数据源的数据生产速度,并计算满足预测值的消费数据速度;使用贪婪负载均衡算法,调整ETL服务个数使节点负载均衡;确定ETL操作匹配关系,使消费数据速度最大且代价最小.该调度机制将匹配问题转化为最小费用最大流问题,并提出基...
|
| 关 键 词： | 实时ETL 弹性调度 稳定匹配 最小费用最大流 |
AN ELASTIC SCHEDULING MECHANISM FOR REAL TIME ETL BASED ON STABLE MATCHING |
| |
| Liu Xuanlü,Gu Jinguang. AN ELASTIC SCHEDULING MECHANISM FOR REAL TIME ETL BASED ON STABLE MATCHING[J]. Computer Applications and Software, 2022, 0(2): 266-273 |
| |
| Authors: | Liu Xuanlü Gu Jinguang |
| |
| Affiliation: | (School of Computer and Technology,Wuhan University of Science and Technology,Wuhan 430065,Hubei,China;Key Laboratory of Intelligent Information Processing and Real-time Industrial System in Hubei Province(Wuhan University of Science and Technology),Wuhan 430065,Hubei,China;Institute of Big Data Science and Engineering,Wuhan University of Science and Technology,Wuhan 430065,Hubei,China;Key Laboratory of Rich-media Knowledge Organization and Service of Digital Publishing Content,National Press and Publication Administration,Beijing 100038,China) |
| |
| Abstract: | In the case of large fluctuation of data production speed,in order to make real time ETL process resource utilization more reasonable,this paper proposes an ETL elastic scheduling mechanism based on stable matching.The data production speed of ETL data source was predicted,and the consumption data speed which needs to meet the predicted speed was calculated;greedy load balancing algorithm was adopted to adjust the number of ETL services to balance the load of nodes;we matched ETL operation relationship to make the consumption data speed the fastest and the cost the least.The ETL operation matching problem was transformed into the minimum cost maximum flow problem,and the improved algorithm based on Dicnic algorithm was proposed.The experimental results show that the scheduling mechanism has advantages in resource utilization. |
| |
| Keywords: | Real time ETL Elastic scheduling Stable matching Minimum cost maximum flow |
| 本文献已被 维普 等数据库收录！ |
|
